mirror of
https://github.com/xinnan-tech/xiaozhi-esp32-server.git
synced 2026-07-22 07:03:53 +08:00
Merge branch 'manager-web-logo-i18n' of https://github.com/xinnan-tech/xiaozhi-esp32-server into new
This commit is contained in:
rainv123
@@ -183,8 +183,8 @@ Spearheaded by Professor Siyuan Liu's Team (South China University of Technology
|
||||
#### 🚀 部署方式选择
|
||||
| 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **最简化安装** | 智能对话、IOT、MCP、视觉感知 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - |
|
||||
| **全模块安装** | 智能对话、IOT、MCP接入点、声纹识别、视觉感知、OTA、智控台 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **最简化安装** | 智能对话、单智能体管理 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - |
|
||||
| **全模块安装** | 智能对话、多用户管理、多智能体管理、智控台界面操作 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
常见问题及相关教程，可参考[这个链接](./docs/FAQ.md)
|
||||
|
||||
@@ -211,10 +211,10 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
| 模块名称 | 入门全免费设置 | 流式配置 |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR(语音识别) | FunASR(本地) | 👍FunASR(本地GPU模式) |
|
||||
| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍AliLLM(qwen3-235b-a22b-instruct-2507) 或 👍DoubaoLLM(doubao-1-5-pro-32k-250115) |
|
||||
| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
|
||||
| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) 或 👍AliyunStreamTTS(阿里云流式语音合成) |
|
||||
| ASR(语音识别) | FunASR(本地) | 👍XunfeiStreamASR(讯飞流式) |
|
||||
| LLM(大模型) | glm-4-flash(智谱) | 👍qwen-flash(阿里百炼) |
|
||||
| VLLM(视觉大模型) | glm-4v-flash(智谱) | 👍qwen2.5-vl-3b-instructh(阿里百炼) |
|
||||
| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山流式) |
|
||||
| Intent(意图识别) | function_call(函数调用) | function_call(函数调用) |
|
||||
| Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） |
|
||||
|
||||
@@ -260,7 +260,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
---
|
||||
|
||||
## 产品生态 👬
|
||||
小智是一个生态，当你使用这个产品时，也可以看看其他在这个生态圈的[优秀项目](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE)
|
||||
小智是一个生态，当你使用这个产品时，也可以看看其他在这个生态圈的[优秀项目](https://github.com/78/xiaozhi-esp32/blob/main/README_zh.md#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE)
|
||||
|
||||
---
|
||||
|
||||
|
||||
+7
-7
@@ -181,8 +181,8 @@ Dieses Projekt bietet zwei Bereitstellungsmethoden. Bitte wählen Sie basierend
|
||||
#### 🚀 Auswahl der Bereitstellungsmethode
|
||||
| Bereitstellungsmethode | Funktionen | Anwendungsszenarien | Deployment-Dokumente | Konfigurationsanforderungen | Video-Tutorials |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Vereinfachte Installation** | Intelligenter Dialog, IOT, MCP, visuelle Wahrnehmung | Umgebungen mit geringer Konfiguration, Daten in Konfigurationsdateien gespeichert, keine Datenbank erforderlich | [①Docker-Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Quellcode-Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 Kerne 4GB bei Verwendung von `FunASR`, 2 Kerne 2GB bei allen APIs | - |
|
||||
| **Vollständige Modulinstallation** | Intelligenter Dialog, IOT, MCP-Endpunkte, Stimmabdruckerkennung, visuelle Wahrnehmung, OTA, intelligente Steuerkonsole | Vollständige Funktionserfahrung, Daten in Datenbank gespeichert |[①Docker-Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Quellcode-Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Quellcode-Deployment Auto-Update-Tutorial](./docs/dev-ops-integration.md) | 4 Kerne 8GB bei Verwendung von `FunASR`, 2 Kerne 4GB bei allen APIs| [Video-Tutorial für lokalen Quellcode-Start](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Vereinfachte Installation** | Intelligenter Dialog, Einzel-Agenten-Verwaltung | Umgebungen mit geringer Konfiguration, Daten in Konfigurationsdateien gespeichert, keine Datenbank erforderlich | [①Docker-Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Quellcode-Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 Kerne 4GB bei Verwendung von `FunASR`, 2 Kerne 2GB bei allen APIs | - |
|
||||
| **Vollständige Modulinstallation** | Intelligenter Dialog, Mehrbenutzerverwaltung, Mehr-Agenten-Verwaltung, Intelligente Steuerkonsole-Bedienung | Vollständige Funktionserfahrung, Daten in Datenbank gespeichert |[①Docker-Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Quellcode-Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Quellcode-Deployment Auto-Update-Tutorial](./docs/dev-ops-integration.md) | 4 Kerne 8GB bei Verwendung von `FunASR`, 2 Kerne 4GB bei allen APIs| [Video-Tutorial für lokalen Quellcode-Start](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
Häufige Fragen und entsprechende Tutorials finden Sie unter [diesem Link](./docs/FAQ.md)
|
||||
|
||||
@@ -209,10 +209,10 @@ Websocket-Schnittstellenadresse: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
| Modulname | Einstiegslevel Kostenlose Einstellungen | Streaming-Konfiguration |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR (Spracherkennung) | FunASR (Lokal) | 👍FunASR (Lokaler GPU-Modus) |
|
||||
| LLM (Großes Modell) | ChatGLMLLM (Zhipu glm-4-flash) | 👍AliLLM (qwen3-235b-a22b-instruct-2507) oder 👍DoubaoLLM (doubao-1-5-pro-32k-250115) |
|
||||
| VLLM (Vision Large Model) | ChatGLMVLLM (Zhipu glm-4v-flash) | 👍QwenVLVLLM (Qwen qwen2.5-vl-3b-instructh) |
|
||||
| TTS (Sprachsynthese) | ✅LinkeraiTTS (Lingxi-Streaming) | 👍HuoshanDoubleStreamTTS (Volcano Dual-Stream-Sprachsynthese) oder 👍AliyunStreamTTS (Alibaba Cloud Streaming-Sprachsynthese) |
|
||||
| ASR (Spracherkennung) | FunASR (Lokal) | 👍XunfeiStreamASR (Xunfei-Streaming) |
|
||||
| LLM (Großes Modell) | glm-4-flash (Zhipu) | 👍qwen-flash (Alibaba Bailian) |
|
||||
| VLLM (Vision Large Model) | glm-4v-flash (Zhipu) | 👍qwen2.5-vl-3b-instructh (Alibaba Bailian) |
|
||||
| TTS (Sprachsynthese) | ✅LinkeraiTTS (Lingxi-Streaming) | 👍HuoshanDoubleStreamTTS (Volcano-Streaming) |
|
||||
| Intent (Absichtserkennung) | function_call (Funktionsaufruf) | function_call (Funktionsaufruf) |
|
||||
| Memory (Gedächtnisfunktion) | mem_local_short (Lokales Kurzzeitgedächtnis) | mem_local_short (Lokales Kurzzeitgedächtnis) |
|
||||
|
||||
@@ -258,7 +258,7 @@ Wenn Sie ein Softwareentwickler sind, finden Sie hier einen [Offenen Brief an En
|
||||
---
|
||||
|
||||
## Produktökosystem 👬
|
||||
Xiaozhi ist ein Ökosystem. Wenn Sie dieses Produkt verwenden, können Sie sich auch andere [hervorragende Projekte](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) in diesem Ökosystem ansehen
|
||||
Xiaozhi ist ein Ökosystem. Wenn Sie dieses Produkt verwenden, können Sie sich auch andere [hervorragende Projekte](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#related-open-source-projects) in diesem Ökosystem ansehen
|
||||
|
||||
---
|
||||
|
||||
|
||||
+7
-7
@@ -181,8 +181,8 @@ This project provides two deployment methods. Please choose based on your specif
|
||||
#### 🚀 Deployment Method Selection
|
||||
| Deployment Method | Features | Applicable Scenarios | Deployment Docs | Configuration Requirements | Video Tutorials |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Simplified Installation** | Intelligent dialogue, IOT, MCP, visual perception | Low-configuration environments, data stored in config files, no database required | [①Docker Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - |
|
||||
| **Full Module Installation** | Intelligent dialogue, IOT, MCP endpoints, voiceprint recognition, visual perception, OTA, intelligent control console | Complete functionality experience, data stored in database |[①Docker Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](./docs/dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Simplified Installation** | Intelligent dialogue, single agent management | Low-configuration environments, data stored in config files, no database required | [①Docker Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - |
|
||||
| **Full Module Installation** | Intelligent dialogue, multi-user management, multi-agent management, intelligent console interface operation | Complete functionality experience, data stored in database |[①Docker Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](./docs/dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
|
||||
> 💡 Note: Below is a test platform deployed with the latest code. You can burn and test if needed. Concurrent users: 6, data will be cleared daily.
|
||||
@@ -208,10 +208,10 @@ Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
| Module Name | Entry Level Free Settings | Streaming Configuration |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR(Speech Recognition) | FunASR(Local) | 👍FunASRServer or 👍DoubaoStreamASR |
|
||||
| LLM(Large Model) | ChatGLMLLM(Zhipu glm-4-flash) | 👍DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
|
||||
| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | 👍QwenVLVLLM(Qwen qwen2.5-vl-3b-instructh) |
|
||||
| TTS(Speech Synthesis) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano dual-stream speech synthesis) |
|
||||
| ASR(Speech Recognition) | FunASR(Local) | 👍XunfeiStreamASR(Xunfei Streaming) |
|
||||
| LLM(Large Model) | glm-4-flash(Zhipu) | 👍qwen-flash(Alibaba Bailian) |
|
||||
| VLLM(Vision Large Model) | glm-4v-flash(Zhipu) | 👍qwen2.5-vl-3b-instructh(Alibaba Bailian) |
|
||||
| TTS(Speech Synthesis) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano Streaming) |
|
||||
| Intent(Intent Recognition) | function_call(Function calling) | function_call(Function calling) |
|
||||
| Memory(Memory function) | mem_local_short(Local short-term memory) | mem_local_short(Local short-term memory) |
|
||||
|
||||
@@ -256,7 +256,7 @@ If you are a software developer, here is an [Open Letter to Developers](docs/con
|
||||
---
|
||||
|
||||
## Product Ecosystem 👬
|
||||
Xiaozhi is an ecosystem. When using this product, you can also check out other [excellent projects](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) in this ecosystem
|
||||
Xiaozhi is an ecosystem. When using this product, you can also check out other [excellent projects](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#related-open-source-projects) in this ecosystem
|
||||
|
||||
| Project Name | Project Address | Project Description |
|
||||
|:---------------------|:--------|:--------|
|
||||
|
||||
+7
-7
@@ -182,8 +182,8 @@ Dự án này cung cấp hai phương pháp triển khai, vui lòng chọn theo
|
||||
#### 🚀 Lựa chọn phương pháp triển khai
|
||||
| Phương pháp triển khai | Đặc điểm | Tình huống áp dụng | Tài liệu triển khai | Yêu cầu cấu hình | Video hướng dẫn |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Cài đặt tối giản** | Đối thoại thông minh, IOT, MCP, cảm nhận thị giác | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 nhân 4GB nếu dùng `FunASR`, 2 nhân 2GB nếu toàn API | - |
|
||||
| **Cài đặt toàn bộ module** | Đối thoại thông minh, IOT, điểm truy cập MCP, nhận dạng giọng nói, cảm nhận thị giác, OTA, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](./docs/dev-ops-integration.md) | 4 nhân 8GB nếu dùng `FunASR`, 2 nhân 4GB nếu toàn API| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **Cài đặt tối giản** | Đối thoại thông minh, quản lý đơn tác nhân | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 nhân 4GB nếu dùng `FunASR`, 2 nhân 2GB nếu toàn API | - |
|
||||
| **Cài đặt toàn bộ module** | Đối thoại thông minh, quản lý đa người dùng, quản lý đa tác nhân, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](./docs/dev-ops-integration.md) | 4 nhân 8GB nếu dùng `FunASR`, 2 nhân 4GB nếu toàn API| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
Câu hỏi thường gặp và hướng dẫn liên quan, vui lòng tham khảo [liên kết này](./docs/FAQ.md)
|
||||
|
||||
@@ -210,10 +210,10 @@ Công cụ kiểm tra dịch vụ: https://2662r3426b.vicp.fun/test/
|
||||
|
||||
| Tên module | Cài đặt miễn phí cho người mới | Cấu hình streaming |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR(Nhận dạng giọng nói) | FunASR(Local) | 👍FunASR(Chế độ GPU cục bộ) |
|
||||
| LLM(Mô hình lớn) | ChatGLMLLM(Zhipu glm-4-flash) | 👍AliLLM(qwen3-235b-a22b-instruct-2507) hoặc 👍DoubaoLLM(doubao-1-5-pro-32k-250115) |
|
||||
| VLLM(Mô hình lớn thị giác) | ChatGLMVLLM(Zhipu glm-4v-flash) | 👍QwenVLVLLM(Qwen qwen2.5-vl-3b-instructh) |
|
||||
| TTS(Tổng hợp giọng nói) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Tổng hợp giọng nói streaming kép Volcano) hoặc 👍AliyunStreamTTS(Tổng hợp giọng nói streaming Alibaba Cloud) |
|
||||
| ASR(Nhận dạng giọng nói) | FunASR(Local) | 👍XunfeiStreamASR(Xunfei Streaming) |
|
||||
| LLM(Mô hình lớn) | glm-4-flash(Zhipu) | 👍qwen-flash(Alibaba Bailian) |
|
||||
| VLLM(Mô hình lớn thị giác) | glm-4v-flash(Zhipu) | 👍qwen2.5-vl-3b-instructh(Alibaba Bailian) |
|
||||
| TTS(Tổng hợp giọng nói) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano Streaming) |
|
||||
| Intent(Nhận dạng ý định) | function_call(Gọi hàm) | function_call(Gọi hàm) |
|
||||
| Memory(Chức năng bộ nhớ) | mem_local_short(Bộ nhớ ngắn hạn cục bộ) | mem_local_short(Bộ nhớ ngắn hạn cục bộ) |
|
||||
|
||||
@@ -259,7 +259,7 @@ Nếu bạn là một nhà phát triển phần mềm, đây có một [Lá thư
|
||||
---
|
||||
|
||||
## Hệ sinh thái sản phẩm 👬
|
||||
Xiaozhi là một hệ sinh thái, khi bạn sử dụng sản phẩm này, bạn cũng có thể xem các [dự án xuất sắc](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) khác trong hệ sinh thái này
|
||||
Xiaozhi là một hệ sinh thái, khi bạn sử dụng sản phẩm này, bạn cũng có thể xem các [dự án xuất sắc](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#related-open-source-projects) khác trong hệ sinh thái này
|
||||
|
||||
---
|
||||
|
||||
|
||||
+5
-4
@@ -38,10 +38,10 @@ conda install conda-forge::ffmpeg
|
||||
|
||||
| 模块名称 | 入门全免费设置 | 流式配置 |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR(语音识别) | FunASR(本地) | 👍FunASR(本地GPU模式) |
|
||||
| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍AliLLM(qwen3-235b-a22b-instruct-2507) 或 👍DoubaoLLM(doubao-1-5-pro-32k-250115) |
|
||||
| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
|
||||
| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) 或 👍AliyunStreamTTS(阿里云流式语音合成) |
|
||||
| ASR(语音识别) | FunASR(本地) | 👍XunfeiStreamASR(讯飞流式) |
|
||||
| LLM(大模型) | glm-4-flash(智谱) | 👍qwen-flash(阿里百炼) |
|
||||
| VLLM(视觉大模型) | glm-4v-flash(智谱) | 👍qwen2.5-vl-3b-instructh(阿里百炼) |
|
||||
| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山流式) |
|
||||
| Intent(意图识别) | function_call(函数调用) | function_call(函数调用) |
|
||||
| Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） |
|
||||
|
||||
@@ -69,6 +69,7 @@ VAD:
|
||||
### 9、编译固件相关教程
|
||||
1、[如何自己编译小智固件](./firmware-build.md)<br/>
|
||||
2、[如何基于虾哥编译好的固件修改OTA地址](./firmware-setting.md)<br/>
|
||||
3、[单模块部署如何配置固件OTA自动升级](./ota-upgrade-guide.md)<br/>
|
||||
|
||||
### 10、拓展相关教程
|
||||
1、[如何开启手机号码注册智控台](./ali-sms-integration.md)<br/>
|
||||
|
||||
@@ -23,6 +23,8 @@
|
||||
|
||||
### 2.将音色资源ID分配给系统账号
|
||||
|
||||
使用超级管理员账号登录智控台，点击顶部`参数字典`，在下拉菜单中，点击`系统功能配置`页面。在页面上勾选`音色克隆`，点击保存配置。即可在顶部菜单看到`音色克隆`按钮。
|
||||
|
||||
使用超级管理员账号登录智控台，点击顶部【音色克隆】、【音色资源】。
|
||||
|
||||
点击新增按钮，在【平台名称】选择“火山双流式语音合成”；
|
||||
|
||||
@@ -76,6 +76,7 @@ MQTT_PORT=1883 # MQTT服务器端口
|
||||
UDP_PORT=8884 # UDP服务器端口
|
||||
API_PORT=8007 # 管理API端口
|
||||
MQTT_SIGNATURE_KEY=test # MQTT签名密钥
|
||||
SERVER_SECRET=Te1st12134 # 服务器密钥，请保持和智控台（server.secret）一致或者和xiaozhi-server里（server.auth_key）保持一致
|
||||
```
|
||||
请注意`PUBLIC_IP`配置，确保其与实际公网IP一致，如果有域名就填域名。
|
||||
|
||||
@@ -85,6 +86,13 @@ MQTT_SIGNATURE_KEY=test # MQTT签名密钥
|
||||
- 注意不要用简单的密码，比如`123456`、`test`等。
|
||||
- 注意不要用简单的密码，比如`123456`、`test`等。
|
||||
|
||||
`SERVER_SECRET` 是用生成websocket连接的认证信息。
|
||||
|
||||
1、如果你是全模块部署，且你的智控台的参数管理里`server.auth.enabled`设置成了`true`，那么，`SERVER_SECRET`需要和智控台（`server.secret`）保持一致。
|
||||
|
||||
2、如果你是单模块部署，且你在配置文件里把`server.auth.enabled`设置成了`true`，那么，`SERVER_SECRET`需要和配置文件里（`server.auth_key`）保持一致。
|
||||
|
||||
|
||||
6. 启动MQTT网关
|
||||
```
|
||||
# 启动服务
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
# 单模块部署固件OTA自动升级配置指南
|
||||
|
||||
本教程将指导你如何在**单模块部署**场景下配置固件OTA自动升级功能，实现设备固件的自动更新。
|
||||
|
||||
如果你已经使用**全模块部署**，请忽略本教程。
|
||||
|
||||
## 功能介绍
|
||||
|
||||
在单模块部署中，xiaozhi-server内置了OTA固件管理功能，可以自动检测设备版本并下发升级固件。系统会根据设备型号和当前版本，自动匹配并推送最新的固件版本。
|
||||
|
||||
## 前提条件
|
||||
|
||||
- 你已经成功进行**单模块部署**并运行xiaozhi-server
|
||||
- 设备能够正常连接到服务器
|
||||
|
||||
## 第一步 准备固件文件
|
||||
|
||||
### 1. 创建固件存放目录
|
||||
|
||||
固件文件需要放在`data/bin/`目录下。如果该目录不存在，请手动创建：
|
||||
|
||||
```bash
|
||||
mkdir -p data/bin
|
||||
```
|
||||
|
||||
### 2. 固件文件命名规则
|
||||
|
||||
固件文件必须遵循以下命名格式：
|
||||
|
||||
```
|
||||
{设备型号}_{版本号}.bin
|
||||
```
|
||||
|
||||
**命名规则说明：**
|
||||
- `设备型号`：设备的型号名称，例如 `lichuang-dev`、`bread-compact-wifi` 等
|
||||
- `版本号`：固件版本号，必须以数字开头，支持数字、字母、点号、下划线和短横线，例如 `1.6.6`、`2.0.0` 等
|
||||
- 文件扩展名必须是 `.bin`
|
||||
|
||||
**命名示例：**
|
||||
```
|
||||
bread-compact-wifi_1.6.6.bin
|
||||
lichuang-dev_2.0.0.bin
|
||||
```
|
||||
|
||||
### 3. 放置固件文件
|
||||
|
||||
将准备好的固件文件（.bin文件）复制到`data/bin/`目录下：
|
||||
|
||||
重要的事情说三遍：升级的bin文件是`xiaozhi.bin`，不是全量固件文件`merged-binary.bin`!
|
||||
|
||||
重要的事情说三遍：升级的bin文件是`xiaozhi.bin`，不是全量固件文件`merged-binary.bin`!
|
||||
|
||||
重要的事情说三遍：升级的bin文件是`xiaozhi.bin`，不是全量固件文件`merged-binary.bin`!
|
||||
|
||||
```bash
|
||||
cp xiaozhi.bin data/bin/设备型号_版本号.bin
|
||||
```
|
||||
|
||||
例如：
|
||||
```bash
|
||||
cp xiaozhi.bin data/bin/bread-compact-wifi_1.6.6.bin
|
||||
```
|
||||
|
||||
## 第二步 配置公网访问地址（仅公网部署需要）
|
||||
|
||||
**注意：此步骤仅适用于单模块公网部署的场景。**
|
||||
|
||||
如果你的xiaozhi-server是公网部署（使用公网IP或域名），**必须**配置`server.vision_explain`参数，因为OTA固件下载地址会使用该配置的域名和端口。
|
||||
|
||||
如果你是局域网部署，可以跳过此步骤。
|
||||
|
||||
### 为什么要配置这个参数？
|
||||
|
||||
在单模块部署中，系统生成固件下载地址时，会使用`vision_explain`配置的域名和端口作为基础地址。如果不配置或配置错误，设备将无法访问固件下载地址。
|
||||
|
||||
### 配置方法
|
||||
|
||||
打开`data/.config.yaml`文件，找到`server`配置段，设置`vision_explain`参数：
|
||||
|
||||
```yaml
|
||||
server:
|
||||
vision_explain: http://你的域名或IP:端口号/mcp/vision/explain
|
||||
```
|
||||
|
||||
**配置示例：**
|
||||
|
||||
局域网部署（默认）：
|
||||
```yaml
|
||||
server:
|
||||
vision_explain: http://192.168.1.100:8003/mcp/vision/explain
|
||||
```
|
||||
|
||||
公网域名部署：
|
||||
```yaml
|
||||
server:
|
||||
vision_explain: http://yourdomain.com:8003/mcp/vision/explain
|
||||
```
|
||||
|
||||
### 注意事项
|
||||
|
||||
- 域名或IP必须是设备能够访问的地址
|
||||
- 如果使用Docker部署，不能使用Docker内部地址（如127.0.0.1或localhost）
|
||||
- 如果你使用了nginx反向代理，请填写对外的地址和端口号，不是本项目运行的端口号
|
||||
|
||||
|
||||
## 常见问题
|
||||
|
||||
### 1. 设备收不到固件更新
|
||||
|
||||
**可能原因和解决方法：**
|
||||
|
||||
- 检查固件文件命名是否符合规则：`{型号}_{版本号}.bin`
|
||||
- 检查固件文件是否正确放置在`data/bin/`目录
|
||||
- 检查设备型号是否与固件文件名中的型号匹配
|
||||
- 检查固件版本号是否高于设备当前版本
|
||||
- 查看服务器日志，确认OTA请求是否正常处理
|
||||
|
||||
### 2. 设备报告下载地址无法访问
|
||||
|
||||
**可能原因和解决方法：**
|
||||
|
||||
- 检查`server.vision_explain`配置的域名或IP是否正确
|
||||
- 确认端口号配置正确（默认8003）
|
||||
- 如果是公网部署，确保设备能够访问该公网地址
|
||||
- 如果是Docker部署，确保不是使用了内部地址（127.0.0.1）
|
||||
- 检查防火墙是否开放了对应端口
|
||||
- 如果你使用了nginx反向代理，请填写对外的地址和端口号，不是本项目运行的端口号
|
||||
|
||||
### 3. 如何确认设备当前版本
|
||||
|
||||
查看OTA请求日志，日志中会显示设备上报的版本号：
|
||||
|
||||
```
|
||||
[ota_handler] - 设备 AA:BB:CC:DD:EE:FF 固件已是最新: 1.6.6
|
||||
```
|
||||
|
||||
### 4. 固件文件放置后没有生效
|
||||
|
||||
系统有30秒的缓存时间（默认），可以：
|
||||
- 等待30秒后再让设备发起OTA请求
|
||||
- 重启xiaozhi-server服务
|
||||
- 调整`firmware_cache_ttl`配置为更短的时间
|
||||
@@ -141,6 +141,11 @@ public interface Constant {
|
||||
*/
|
||||
String SERVER_MQTT_SECRET = "server.mqtt_signature_key";
|
||||
|
||||
/**
|
||||
* WebSocket认证开关
|
||||
*/
|
||||
String SERVER_AUTH_ENABLED = "server.auth.enabled";
|
||||
|
||||
/**
|
||||
* 无记忆
|
||||
*/
|
||||
@@ -299,7 +304,7 @@ public interface Constant {
|
||||
/**
|
||||
* 版本号
|
||||
*/
|
||||
public static final String VERSION = "0.8.9";
|
||||
public static final String VERSION = "0.8.10";
|
||||
|
||||
/**
|
||||
* 无效固件URL
|
||||
|
||||
+54
-3
@@ -1,6 +1,8 @@
|
||||
package xiaozhi.modules.device.service.impl;
|
||||
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.InvalidKeyException;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.time.Instant;
|
||||
import java.util.Base64;
|
||||
import java.util.Date;
|
||||
@@ -169,7 +171,22 @@ public class DeviceServiceImpl extends BaseServiceImpl<DeviceDao, DeviceEntity>
|
||||
DeviceReportRespDTO.Websocket websocket = new DeviceReportRespDTO.Websocket();
|
||||
// 从系统参数获取WebSocket URL，如果未配置则使用默认值
|
||||
String wsUrl = sysParamsService.getValue(Constant.SERVER_WEBSOCKET, true);
|
||||
websocket.setToken("");
|
||||
|
||||
// 检查是否启用认证并生成token
|
||||
String authEnabled = sysParamsService.getValue(Constant.SERVER_AUTH_ENABLED, true);
|
||||
if ("true".equalsIgnoreCase(authEnabled)) {
|
||||
try {
|
||||
// 生成token
|
||||
String token = generateWebSocketToken(clientId, macAddress);
|
||||
websocket.setToken(token);
|
||||
} catch (Exception e) {
|
||||
log.error("生成WebSocket token失败: {}", e.getMessage());
|
||||
websocket.setToken("");
|
||||
}
|
||||
} else {
|
||||
websocket.setToken("");
|
||||
}
|
||||
|
||||
if (StringUtils.isBlank(wsUrl) || wsUrl.equals("null")) {
|
||||
log.error("WebSocket地址未配置，请登录智控台，在参数管理找到【server.websocket】配置");
|
||||
wsUrl = "ws://xiaozhi.server.com:8000/xiaozhi/v1/";
|
||||
@@ -189,7 +206,7 @@ public class DeviceServiceImpl extends BaseServiceImpl<DeviceDao, DeviceEntity>
|
||||
|
||||
// 添加MQTT UDP配置
|
||||
// 从系统参数获取MQTT Gateway地址，仅在配置有效时使用
|
||||
String mqttUdpConfig = sysParamsService.getValue(Constant.SERVER_MQTT_GATEWAY, false);
|
||||
String mqttUdpConfig = sysParamsService.getValue(Constant.SERVER_MQTT_GATEWAY, true);
|
||||
if (mqttUdpConfig != null && !mqttUdpConfig.equals("null") && !mqttUdpConfig.isEmpty()) {
|
||||
try {
|
||||
String groupId = deviceById != null && deviceById.getBoard() != null ? deviceById.getBoard()
|
||||
@@ -494,6 +511,40 @@ public class DeviceServiceImpl extends BaseServiceImpl<DeviceDao, DeviceEntity>
|
||||
return Base64.getEncoder().encodeToString(signature);
|
||||
}
|
||||
|
||||
/**
|
||||
* 生成WebSocket认证token 遵循Python端AuthManager的实现逻辑：token = signature.timestamp
|
||||
*
|
||||
* @param clientId 客户端ID
|
||||
* @param username 用户名 (通常为deviceId/macAddress)
|
||||
* @return 认证token字符串
|
||||
*/
|
||||
private String generateWebSocketToken(String clientId, String username)
|
||||
throws NoSuchAlgorithmException, InvalidKeyException {
|
||||
// 从系统参数获取密钥
|
||||
String secretKey = sysParamsService.getValue(Constant.SERVER_SECRET, false);
|
||||
if (StringUtils.isBlank(secretKey)) {
|
||||
throw new IllegalStateException("WebSocket认证密钥未配置(server.secret)");
|
||||
}
|
||||
|
||||
// 获取当前时间戳(秒)
|
||||
long timestamp = System.currentTimeMillis() / 1000;
|
||||
|
||||
// 构建签名内容: clientId|username|timestamp
|
||||
String content = String.format("%s|%s|%d", clientId, username, timestamp);
|
||||
|
||||
// 生成HMAC-SHA256签名
|
||||
Mac hmac = Mac.getInstance("HmacSHA256");
|
||||
SecretKeySpec keySpec = new SecretKeySpec(secretKey.getBytes(StandardCharsets.UTF_8), "HmacSHA256");
|
||||
hmac.init(keySpec);
|
||||
byte[] signature = hmac.doFinal(content.getBytes(StandardCharsets.UTF_8));
|
||||
|
||||
// Base64 URL-safe编码签名(去除填充符=)
|
||||
String signatureBase64 = Base64.getUrlEncoder().withoutPadding().encodeToString(signature);
|
||||
|
||||
// 返回格式: signature.timestamp
|
||||
return String.format("%s.%d", signatureBase64, timestamp);
|
||||
}
|
||||
|
||||
/**
|
||||
* 构建MQTT配置信息
|
||||
*
|
||||
@@ -504,7 +555,7 @@ public class DeviceServiceImpl extends BaseServiceImpl<DeviceDao, DeviceEntity>
|
||||
private DeviceReportRespDTO.MQTT buildMqttConfig(String macAddress, String groupId)
|
||||
throws Exception {
|
||||
// 从环境变量或系统参数获取签名密钥
|
||||
String signatureKey = sysParamsService.getValue("server.mqtt_signature_key", false);
|
||||
String signatureKey = sysParamsService.getValue("server.mqtt_signature_key", true);
|
||||
if (StringUtils.isBlank(signatureKey)) {
|
||||
log.warn("缺少MQTT_SIGNATURE_KEY，跳过MQTT配置生成");
|
||||
return null;
|
||||
|
||||
+4
-4
@@ -6,6 +6,7 @@ import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.PutMapping;
|
||||
@@ -23,7 +24,9 @@ import xiaozhi.common.exception.ErrorCode;
|
||||
import xiaozhi.common.exception.RenException;
|
||||
import xiaozhi.common.page.TokenDTO;
|
||||
import xiaozhi.common.user.UserDetail;
|
||||
import xiaozhi.common.utils.JsonUtils;
|
||||
import xiaozhi.common.utils.Result;
|
||||
import xiaozhi.common.utils.Sm2DecryptUtil;
|
||||
import xiaozhi.common.validator.AssertUtils;
|
||||
import xiaozhi.common.validator.ValidatorUtils;
|
||||
import xiaozhi.modules.security.dto.LoginDTO;
|
||||
@@ -32,8 +35,6 @@ import xiaozhi.modules.security.password.PasswordUtils;
|
||||
import xiaozhi.modules.security.service.CaptchaService;
|
||||
import xiaozhi.modules.security.service.SysUserTokenService;
|
||||
import xiaozhi.modules.security.user.SecurityUser;
|
||||
import xiaozhi.common.utils.Sm2DecryptUtil;
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import xiaozhi.modules.sys.dto.PasswordDTO;
|
||||
import xiaozhi.modules.sys.dto.RetrievePasswordDTO;
|
||||
import xiaozhi.modules.sys.dto.SysUserDTO;
|
||||
@@ -41,7 +42,6 @@ import xiaozhi.modules.sys.service.SysDictDataService;
|
||||
import xiaozhi.modules.sys.service.SysParamsService;
|
||||
import xiaozhi.modules.sys.service.SysUserService;
|
||||
import xiaozhi.modules.sys.vo.SysDictDataItem;
|
||||
import xiaozhi.common.utils.JsonUtils;
|
||||
|
||||
/**
|
||||
* 登录控制层
|
||||
@@ -237,7 +237,7 @@ public class LoginController {
|
||||
config.put("sm2PublicKey", publicKey);
|
||||
|
||||
// 获取system-web.menu参数配置
|
||||
String menuConfig = sysParamsService.getValue("system-web.menu", false);
|
||||
String menuConfig = sysParamsService.getValue("system-web.menu", true);
|
||||
if (StringUtils.isNotBlank(menuConfig)) {
|
||||
config.put("systemWebMenu", JsonUtils.parseObject(menuConfig, Object.class));
|
||||
}
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
-- 删除server模块是否开启token认证参数
|
||||
delete from `sys_params` where param_code = 'server.auth.enabled';
|
||||
|
||||
-- 添加server模块是否开启token认证参数
|
||||
INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES
|
||||
(122, 'server.auth.enabled', 'true', 'boolean', 1, 'server模块是否开启token认证');
|
||||
@@ -0,0 +1 @@
|
||||
INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (311, 'enable_websocket_ping', 'false', 'boolean', 1, '是否启用WebSocket心跳保活机制');
|
||||
@@ -438,3 +438,17 @@ databaseChangeLog:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202512041515.sql
|
||||
- changeSet:
|
||||
id: 202512131453
|
||||
author: hrz
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202512131453.sql
|
||||
- changeSet:
|
||||
id: 202512161529
|
||||
author: RanChen
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202512161529.sql
|
||||
|
||||
@@ -235,7 +235,7 @@ function showAbout() {
|
||||
title: t('settings.aboutApp', { appName: import.meta.env.VITE_APP_TITLE }),
|
||||
content: t('settings.aboutContent', {
|
||||
appName: import.meta.env.VITE_APP_TITLE,
|
||||
version: '0.8.9'
|
||||
version: '0.8.10'
|
||||
}),
|
||||
showCancel: false,
|
||||
confirmText: t('common.confirm'),
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 1.9 KiB After Width: | Height: | Size: 1.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.8 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 7.0 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.8 KiB |
@@ -713,7 +713,7 @@ export default {
|
||||
'paramManagement.deleteFailed': 'Löschen fehlgeschlagen, bitte versuchen Sie es erneut',
|
||||
'paramManagement.operationCancelled': 'Löschen abgebrochen',
|
||||
'paramManagement.operationClosed': 'Operation geschlossen',
|
||||
'paramManagement.updateSuccess': 'Aktualisierung erfolgreich',
|
||||
'paramManagement.updateSuccess': 'Aktualisierung erfolgreich. Einige Konfigurationen werden erst nach Neustart des xiaozhi-server-Moduls wirksam.',
|
||||
'paramManagement.addSuccess': 'Hinzufügen erfolgreich',
|
||||
'paramManagement.updateFailed': 'Aktualisierung fehlgeschlagen',
|
||||
'paramManagement.addFailed': 'Hinzufügen fehlgeschlagen',
|
||||
@@ -821,7 +821,7 @@ export default {
|
||||
'modelConfig.rag': 'RAG',
|
||||
'modelConfig.modelId': 'Modell-ID',
|
||||
'modelConfig.modelName': 'Modellname',
|
||||
'modelConfig.provider': 'Anbieter',
|
||||
'modelConfig.provider': 'Schnittstellentyp',
|
||||
'modelConfig.unknown': 'Unbekannt',
|
||||
'modelConfig.isEnabled': 'Aktiviert',
|
||||
'modelConfig.isDefault': 'Standard',
|
||||
@@ -852,7 +852,7 @@ export default {
|
||||
'modelConfig.enableSuccess': 'Aktivieren erfolgreich',
|
||||
'modelConfig.disableSuccess': 'Deaktivieren erfolgreich',
|
||||
'modelConfig.operationFailed': 'Operation fehlgeschlagen',
|
||||
'modelConfig.setDefaultSuccess': 'Standardmodell erfolgreich gesetzt',
|
||||
'modelConfig.setDefaultSuccess': 'Standardmodell erfolgreich gesetzt, bitte starten Sie das xiaozhi-server-Modul zeitnah manuell neu',
|
||||
'modelConfig.itemsPerPage': '{items} Einträge/Seite',
|
||||
'modelConfig.firstPage': 'Erste Seite',
|
||||
'modelConfig.prevPage': 'Vorherige Seite',
|
||||
|
||||
@@ -713,7 +713,7 @@ export default {
|
||||
'paramManagement.deleteFailed': 'Deletion failed, please try again',
|
||||
'paramManagement.operationCancelled': 'Deletion cancelled',
|
||||
'paramManagement.operationClosed': 'Operation closed',
|
||||
'paramManagement.updateSuccess': 'Update successful',
|
||||
'paramManagement.updateSuccess': 'Update successful. Some configurations will take effect only after restarting the xiaozhi-server module.',
|
||||
'paramManagement.addSuccess': 'Add successful',
|
||||
'paramManagement.updateFailed': 'Update failed',
|
||||
'paramManagement.addFailed': 'Add failed',
|
||||
@@ -821,7 +821,7 @@ export default {
|
||||
'modelConfig.rag': 'RAG',
|
||||
'modelConfig.modelId': 'Model ID',
|
||||
'modelConfig.modelName': 'Model Name',
|
||||
'modelConfig.provider': 'Provider',
|
||||
'modelConfig.provider': 'Interface Type',
|
||||
'modelConfig.unknown': 'Unknown',
|
||||
'modelConfig.isEnabled': 'Enabled',
|
||||
'modelConfig.isDefault': 'Default',
|
||||
@@ -852,7 +852,7 @@ export default {
|
||||
'modelConfig.enableSuccess': 'Enable successful',
|
||||
'modelConfig.disableSuccess': 'Disable successful',
|
||||
'modelConfig.operationFailed': 'Operation failed',
|
||||
'modelConfig.setDefaultSuccess': 'Set default model successful',
|
||||
'modelConfig.setDefaultSuccess': 'Set default model successful, please restart the xiaozhi-server module manually in time',
|
||||
'modelConfig.itemsPerPage': '{items} items/page',
|
||||
'modelConfig.firstPage': 'First Page',
|
||||
'modelConfig.prevPage': 'Previous Page',
|
||||
|
||||
@@ -713,7 +713,7 @@ export default {
|
||||
'paramManagement.deleteFailed': 'Xóa thất bại, vui lòng thử lại',
|
||||
'paramManagement.operationCancelled': 'Đã hủy xóa',
|
||||
'paramManagement.operationClosed': 'Đã đóng thao tác',
|
||||
'paramManagement.updateSuccess': 'Cập nhật thành công',
|
||||
'paramManagement.updateSuccess': 'Cập nhật thành công. Một số cấu hình chỉ có hiệu lực sau khi khởi động lại mô-đun xiaozhi-server.',
|
||||
'paramManagement.addSuccess': 'Thêm thành công',
|
||||
'paramManagement.updateFailed': 'Cập nhật thất bại',
|
||||
'paramManagement.addFailed': 'Thêm thất bại',
|
||||
@@ -821,7 +821,7 @@ export default {
|
||||
'modelConfig.rag': 'RAG',
|
||||
'modelConfig.modelId': 'ID mô hình',
|
||||
'modelConfig.modelName': 'Tên mô hình',
|
||||
'modelConfig.provider': 'Nhà cung cấp',
|
||||
'modelConfig.provider': 'Loại giao diện',
|
||||
'modelConfig.unknown': 'Không xác định',
|
||||
'modelConfig.isEnabled': 'Đã bật',
|
||||
'modelConfig.isDefault': 'Mặc định',
|
||||
@@ -852,7 +852,7 @@ export default {
|
||||
'modelConfig.enableSuccess': 'Bật thành công',
|
||||
'modelConfig.disableSuccess': 'Tắt thành công',
|
||||
'modelConfig.operationFailed': 'Thao tác thất bại',
|
||||
'modelConfig.setDefaultSuccess': 'Đặt mô hình mặc định thành công',
|
||||
'modelConfig.setDefaultSuccess': 'Đặt mô hình mặc định thành công, vui lòng khởi động lại module xiaozhi-server thủ công kịp thời',
|
||||
'modelConfig.itemsPerPage': '{items} mục/trang',
|
||||
'modelConfig.firstPage': 'Trang đầu',
|
||||
'modelConfig.prevPage': 'Trang trước',
|
||||
|
||||
@@ -713,7 +713,7 @@ export default {
|
||||
'paramManagement.deleteFailed': '删除失败，请重试',
|
||||
'paramManagement.operationCancelled': '已取消删除操作',
|
||||
'paramManagement.operationClosed': '操作已关闭',
|
||||
'paramManagement.updateSuccess': '修改成功',
|
||||
'paramManagement.updateSuccess': '修改成功，部分配置需重启xiaozhi-server模块才生效',
|
||||
'paramManagement.addSuccess': '新增成功',
|
||||
'paramManagement.updateFailed': '更新失败',
|
||||
'paramManagement.addFailed': '新增失败',
|
||||
@@ -821,7 +821,7 @@ export default {
|
||||
'modelConfig.rag': '知识库',
|
||||
'modelConfig.modelId': '模型ID',
|
||||
'modelConfig.modelName': '模型名称',
|
||||
'modelConfig.provider': '提供商',
|
||||
'modelConfig.provider': '接口类型',
|
||||
'modelConfig.unknown': '未知',
|
||||
'modelConfig.isEnabled': '是否启用',
|
||||
'modelConfig.isDefault': '是否默认',
|
||||
@@ -852,7 +852,7 @@ export default {
|
||||
'modelConfig.enableSuccess': '启用成功',
|
||||
'modelConfig.disableSuccess': '禁用成功',
|
||||
'modelConfig.operationFailed': '操作失败',
|
||||
'modelConfig.setDefaultSuccess': '设置默认模型成功',
|
||||
'modelConfig.setDefaultSuccess': '设置默认模型成功，请及时手动重启xiaozhi-server模块',
|
||||
'modelConfig.itemsPerPage': '{items}条/页',
|
||||
'modelConfig.firstPage': '首页',
|
||||
'modelConfig.prevPage': '上一页',
|
||||
|
||||
@@ -713,7 +713,7 @@ export default {
|
||||
'paramManagement.deleteFailed': '刪除失敗，請重試',
|
||||
'paramManagement.operationCancelled': '已取消刪除操作',
|
||||
'paramManagement.operationClosed': '操作已關閉',
|
||||
'paramManagement.updateSuccess': '修改成功',
|
||||
'paramManagement.updateSuccess': '修改成功，部分配置需重啟xiaozhi-server模組才生效',
|
||||
'paramManagement.addSuccess': '新增成功',
|
||||
'paramManagement.updateFailed': '更新失敗',
|
||||
'paramManagement.addFailed': '新增失敗',
|
||||
@@ -821,7 +821,7 @@ export default {
|
||||
'modelConfig.rag': '知識庫',
|
||||
'modelConfig.modelId': '模型ID',
|
||||
'modelConfig.modelName': '模型名稱',
|
||||
'modelConfig.provider': '提供商',
|
||||
'modelConfig.provider': '接口類型',
|
||||
'modelConfig.unknown': '未知',
|
||||
'modelConfig.isEnabled': '是否啟用',
|
||||
'modelConfig.isDefault': '是否默認',
|
||||
@@ -852,7 +852,7 @@ export default {
|
||||
'modelConfig.enableSuccess': '啟用成功',
|
||||
'modelConfig.disableSuccess': '禁用成功',
|
||||
'modelConfig.operationFailed': '操作失敗',
|
||||
'modelConfig.setDefaultSuccess': '設置默認模型成功',
|
||||
'modelConfig.setDefaultSuccess': '設置默認模型成功，請及時手動重啟xiaozhi-server模組',
|
||||
'modelConfig.itemsPerPage': '{items}條/頁',
|
||||
'modelConfig.firstPage': '首頁',
|
||||
'modelConfig.prevPage': '上一頁',
|
||||
|
||||
@@ -69,6 +69,9 @@ enable_greeting: true
|
||||
enable_stop_tts_notify: false
|
||||
# 说完话是否开启提示音，音效地址
|
||||
stop_tts_notify_voice: "config/assets/tts_notify.mp3"
|
||||
# 是否启用WebSocket心跳保活机制
|
||||
enable_websocket_ping: false
|
||||
|
||||
|
||||
# TTS音频发送延迟配置
|
||||
# tts_audio_send_delay: 控制音频包发送间隔
|
||||
@@ -471,7 +474,6 @@ ASR:
|
||||
domain: slm # 识别领域，iat:日常用语，medical:医疗，finance:金融等
|
||||
language: zh_cn # 语言，zh_cn:中文，en_us:英文
|
||||
accent: mandarin # 方言，mandarin:普通话
|
||||
dwa: wpgs # 动态修正，wpgs:实时返回中间结果
|
||||
# 调整音频处理参数以提高长语音识别质量
|
||||
output_dir: tmp/
|
||||
|
||||
|
||||
@@ -68,6 +68,7 @@ async def get_config_from_api_async(config):
|
||||
"url": config["manager-api"].get("url", ""),
|
||||
"secret": config["manager-api"].get("secret", ""),
|
||||
}
|
||||
auth_enabled = config_data.get("server", {}).get("auth", {}).get("enabled", False)
|
||||
# server的配置以本地为准
|
||||
if config.get("server"):
|
||||
config_data["server"] = {
|
||||
@@ -77,6 +78,7 @@ async def get_config_from_api_async(config):
|
||||
"vision_explain": config["server"].get("vision_explain", ""),
|
||||
"auth_key": config["server"].get("auth_key", ""),
|
||||
}
|
||||
config_data["server"]["auth"] = {"enabled": auth_enabled}
|
||||
# 如果服务器没有prompt_template，则从本地配置读取
|
||||
if not config_data.get("prompt_template"):
|
||||
config_data["prompt_template"] = config.get("prompt_template")
|
||||
|
||||
@@ -5,7 +5,7 @@ from config.config_loader import load_config
|
||||
from config.settings import check_config_file
|
||||
from datetime import datetime
|
||||
|
||||
SERVER_VERSION = "0.8.9"
|
||||
SERVER_VERSION = "0.8.10"
|
||||
_logger_initialized = False
|
||||
|
||||
|
||||
|
||||
@@ -10,7 +10,15 @@ class BaseHandler:
|
||||
def _add_cors_headers(self, response):
|
||||
"""添加CORS头信息"""
|
||||
response.headers["Access-Control-Allow-Headers"] = (
|
||||
"client-id, content-type, device-id"
|
||||
"client-id, content-type, device-id, authorization"
|
||||
)
|
||||
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
|
||||
async def handle_options(self, request):
|
||||
"""处理OPTIONS请求，添加CORS头信息"""
|
||||
response = web.Response(body=b"", content_type="text/plain")
|
||||
self._add_cors_headers(response)
|
||||
# 添加允许的方法
|
||||
response.headers["Access-Control-Allow-Methods"] = "GET, POST, OPTIONS"
|
||||
return response
|
||||
|
||||
@@ -3,15 +3,46 @@ import time
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import os
|
||||
import re
|
||||
import glob
|
||||
from typing import Dict, List, Tuple
|
||||
from aiohttp import web
|
||||
|
||||
from core.auth import AuthManager
|
||||
from core.utils.util import get_local_ip
|
||||
from core.utils.util import get_local_ip, get_vision_url
|
||||
from core.api.base_handler import BaseHandler
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
def _safe_basename(filename: str) -> str:
|
||||
# Prevent directory traversal
|
||||
return os.path.basename(filename)
|
||||
|
||||
|
||||
def _parse_version(ver: str) -> Tuple[int, ...]:
|
||||
# conservative parser: split by non-digit, keep numeric parts
|
||||
parts = re.findall(r"\d+", ver)
|
||||
return tuple(int(p) for p in parts) if parts else (0,)
|
||||
|
||||
|
||||
def _is_higher_version(a: str, b: str) -> bool:
|
||||
"""Return True if version string a > b (semver-like numeric compare)."""
|
||||
ta = _parse_version(a)
|
||||
tb = _parse_version(b)
|
||||
# compare tuple lexicographically, but allow different lengths
|
||||
maxlen = max(len(ta), len(tb))
|
||||
for i in range(maxlen):
|
||||
ai = ta[i] if i < len(ta) else 0
|
||||
bi = tb[i] if i < len(tb) else 0
|
||||
if ai > bi:
|
||||
return True
|
||||
if ai < bi:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
class OTAHandler(BaseHandler):
|
||||
def __init__(self, config: dict):
|
||||
super().__init__(config)
|
||||
@@ -23,6 +54,54 @@ class OTAHandler(BaseHandler):
|
||||
expire_seconds = auth_config.get("expire_seconds")
|
||||
self.auth = AuthManager(secret_key=secret_key, expire_seconds=expire_seconds)
|
||||
|
||||
# firmware storage
|
||||
self.bin_dir = os.path.join(os.getcwd(), "data", "bin")
|
||||
# cache structure: { 'updated_at': timestamp, 'ttl': seconds, 'files_by_model': { model: [(version, filename), ...] } }
|
||||
self._bin_cache: Dict = {
|
||||
"updated_at": 0,
|
||||
"ttl": config.get("firmware_cache_ttl", 30),
|
||||
"files_by_model": {},
|
||||
}
|
||||
|
||||
def _refresh_bin_cache_if_needed(self):
|
||||
now = int(time.time())
|
||||
ttl = int(self._bin_cache.get("ttl", 30))
|
||||
if now - int(
|
||||
self._bin_cache.get("updated_at", 0)
|
||||
) < ttl and self._bin_cache.get("files_by_model"):
|
||||
return
|
||||
|
||||
files_by_model: Dict[str, List[Tuple[str, str]]] = {}
|
||||
try:
|
||||
if not os.path.isdir(self.bin_dir):
|
||||
os.makedirs(self.bin_dir, exist_ok=True)
|
||||
|
||||
# match files like model_1.2.3.bin (allow dots, dashes, underscores in model and version)
|
||||
pattern = os.path.join(self.bin_dir, "*.bin")
|
||||
for path in glob.glob(pattern):
|
||||
fname = os.path.basename(path)
|
||||
# filename format: {model}_{version}.bin
|
||||
m = re.match(r"^(.+?)_([0-9][A-Za-z0-9\.\-_]*)\.bin$", fname)
|
||||
if not m:
|
||||
# skip files not conforming to naming rule
|
||||
continue
|
||||
model = m.group(1)
|
||||
version = m.group(2)
|
||||
files_by_model.setdefault(model, []).append((version, fname))
|
||||
|
||||
# sort versions for each model descending
|
||||
for model, items in files_by_model.items():
|
||||
items.sort(key=lambda it: _parse_version(it[0]), reverse=True)
|
||||
|
||||
self._bin_cache["files_by_model"] = files_by_model
|
||||
self._bin_cache["updated_at"] = now
|
||||
self.logger.bind(tag=TAG).info(
|
||||
f"Firmware cache refreshed: {len(files_by_model)} models"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"刷新固件缓存失败: {e}")
|
||||
# keep previous cache if any
|
||||
|
||||
def generate_password_signature(self, content: str, secret_key: str) -> str:
|
||||
"""生成MQTT密码签名
|
||||
|
||||
@@ -62,7 +141,14 @@ class OTAHandler(BaseHandler):
|
||||
return f"ws://{local_ip}:{port}/xiaozhi/v1/"
|
||||
|
||||
async def handle_post(self, request):
|
||||
"""处理 OTA POST 请求"""
|
||||
"""处理 OTA POST 请求
|
||||
|
||||
This handler will:
|
||||
- read device id/client id (as before)
|
||||
- attempt to determine device model and current firmware version (prefer headers, fallback to body)
|
||||
- check data/bin for newer firmware for that model
|
||||
- if found a newer firmware, set firmware.url to the download endpoint
|
||||
"""
|
||||
try:
|
||||
data = await request.text()
|
||||
self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}")
|
||||
@@ -81,33 +167,76 @@ class OTAHandler(BaseHandler):
|
||||
else:
|
||||
raise Exception("OTA请求ClientID为空")
|
||||
|
||||
data_json = json.loads(data)
|
||||
data_json = {}
|
||||
try:
|
||||
data_json = json.loads(data) if data else {}
|
||||
except Exception:
|
||||
data_json = {}
|
||||
|
||||
server_config = self.config["server"]
|
||||
port = int(server_config.get("port", 8000))
|
||||
# Distinguish ports:
|
||||
# - websocket_port is used to construct websocket URL (server["port"])
|
||||
# - http_port is used to construct OTA download URLs (server["http_port"])
|
||||
websocket_port = int(server_config.get("port", 8000))
|
||||
http_port = int(server_config.get("http_port", 8003))
|
||||
local_ip = get_local_ip()
|
||||
|
||||
# Determine device model (prefer headers)
|
||||
device_model = ""
|
||||
# header candidates
|
||||
for h in ("device-model", "device_model", "model"):
|
||||
if h in request.headers:
|
||||
device_model = request.headers.get(h, "").strip()
|
||||
break
|
||||
# body fallback
|
||||
if not device_model:
|
||||
try:
|
||||
if "board" in data_json and isinstance(data_json["board"], dict):
|
||||
device_model = data_json["board"].get("type", "")
|
||||
elif "model" in data_json:
|
||||
device_model = data_json.get("model", "")
|
||||
except Exception:
|
||||
device_model = ""
|
||||
if not device_model:
|
||||
device_model = "default"
|
||||
|
||||
# Determine device current version (prefer headers)
|
||||
device_version = ""
|
||||
for h in (
|
||||
"device-version",
|
||||
"device_version",
|
||||
"firmware-version",
|
||||
"app-version",
|
||||
"application-version",
|
||||
):
|
||||
if h in request.headers:
|
||||
device_version = request.headers.get(h, "").strip()
|
||||
break
|
||||
if not device_version:
|
||||
try:
|
||||
device_version = data_json.get("application", {}).get("version", "")
|
||||
except Exception:
|
||||
device_version = ""
|
||||
if not device_version:
|
||||
device_version = "0.0.0"
|
||||
|
||||
return_json = {
|
||||
"server_time": {
|
||||
"timestamp": int(round(time.time() * 1000)),
|
||||
"timezone_offset": server_config.get("timezone_offset", 8) * 60,
|
||||
},
|
||||
"firmware": {
|
||||
"version": data_json["application"].get("version", "1.0.0"),
|
||||
"version": device_version,
|
||||
"url": "",
|
||||
},
|
||||
}
|
||||
|
||||
# existing mqtt/websocket logic (unchanged)
|
||||
mqtt_gateway_endpoint = server_config.get("mqtt_gateway")
|
||||
|
||||
if mqtt_gateway_endpoint: # 如果配置了非空字符串
|
||||
# 尝试从请求数据中获取设备型号
|
||||
device_model = "default"
|
||||
# 尝试从请求数据中获取设备型号（已解析 above）
|
||||
try:
|
||||
if "device" in data_json and isinstance(data_json["device"], dict):
|
||||
device_model = data_json["device"].get("model", "default")
|
||||
elif "model" in data_json:
|
||||
device_model = data_json["model"]
|
||||
group_id = f"GID_{device_model}".replace(":", "_").replace(" ", "_")
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"获取设备型号失败: {e}")
|
||||
@@ -159,20 +288,61 @@ class OTAHandler(BaseHandler):
|
||||
token = self.auth.generate_token(client_id, device_id)
|
||||
else:
|
||||
token = self.auth.generate_token(client_id, device_id)
|
||||
# NOTE: use websocket_port here
|
||||
return_json["websocket"] = {
|
||||
"url": self._get_websocket_url(local_ip, port),
|
||||
"url": self._get_websocket_url(local_ip, websocket_port),
|
||||
"token": token,
|
||||
}
|
||||
self.logger.bind(tag=TAG).info(
|
||||
f"未配置MQTT网关，为设备 {device_id} 下发WebSocket配置"
|
||||
)
|
||||
self.logger.bind(tag=TAG).info(f"{return_json}")
|
||||
|
||||
# Now check firmware files for updates
|
||||
try:
|
||||
self._refresh_bin_cache_if_needed()
|
||||
files_by_model = self._bin_cache.get("files_by_model", {})
|
||||
candidates = files_by_model.get(device_model, [])
|
||||
|
||||
self.logger.bind(tag=TAG).info(
|
||||
f"查找型号 {device_model} 的固件，找到 {len(candidates)} 个候选"
|
||||
)
|
||||
|
||||
chosen_url = ""
|
||||
chosen_version = device_version
|
||||
|
||||
# candidates are sorted descending by version
|
||||
for ver, fname in candidates:
|
||||
if _is_higher_version(ver, device_version):
|
||||
# build download url (only allow download via our download endpoint)
|
||||
chosen_version = ver
|
||||
# Use get_vision_url to get the base URL and replace the path
|
||||
vision_url = get_vision_url(self.config)
|
||||
# Replace the path from "/mcp/vision/explain" to "/xiaozhi/ota/download/{fname}"
|
||||
chosen_url = vision_url.replace(
|
||||
"/mcp/vision/explain", f"/xiaozhi/ota/download/{fname}"
|
||||
)
|
||||
break
|
||||
|
||||
if chosen_url:
|
||||
return_json["firmware"]["version"] = chosen_version
|
||||
return_json["firmware"]["url"] = chosen_url
|
||||
self.logger.bind(tag=TAG).info(
|
||||
f"为设备 {device_id} 下发固件 {chosen_version} [如果地址前缀有误，请检查配置文件中的server.vision_explain]-> {chosen_url} "
|
||||
)
|
||||
else:
|
||||
self.logger.bind(tag=TAG).info(
|
||||
f"设备 {device_id} 固件已是最新: {device_version}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"检查固件版本时出错: {e}")
|
||||
|
||||
response = web.Response(
|
||||
text=json.dumps(return_json, separators=(",", ":")),
|
||||
content_type="application/json",
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"OTA POST处理异常: {e}")
|
||||
return_json = {"success": False, "message": "request error."}
|
||||
response = web.Response(
|
||||
text=json.dumps(return_json, separators=(",", ":")),
|
||||
@@ -187,8 +357,9 @@ class OTAHandler(BaseHandler):
|
||||
try:
|
||||
server_config = self.config["server"]
|
||||
local_ip = get_local_ip()
|
||||
port = int(server_config.get("port", 8000))
|
||||
websocket_url = self._get_websocket_url(local_ip, port)
|
||||
# use websocket port for websocket URL
|
||||
websocket_port = int(server_config.get("port", 8000))
|
||||
websocket_url = self._get_websocket_url(local_ip, websocket_port)
|
||||
message = f"OTA接口运行正常，向设备发送的websocket地址是：{websocket_url}"
|
||||
response = web.Response(text=message, content_type="text/plain")
|
||||
except Exception as e:
|
||||
@@ -197,3 +368,48 @@ class OTAHandler(BaseHandler):
|
||||
finally:
|
||||
self._add_cors_headers(response)
|
||||
return response
|
||||
|
||||
async def handle_download(self, request):
|
||||
"""
|
||||
下载固件接口
|
||||
URL: /xiaozhi/ota/download/{filename}
|
||||
- 只允许下载 data/bin 目录下的 .bin 文件
|
||||
- filename 必须是 basename 且匹配安全的模式
|
||||
"""
|
||||
try:
|
||||
fname = request.match_info.get("filename", "")
|
||||
if not fname:
|
||||
raise web.HTTPBadRequest(text="filename required")
|
||||
|
||||
# sanitize
|
||||
fname = _safe_basename(fname)
|
||||
# pattern: allow letters, numbers, dot, underscore, dash
|
||||
if not re.match(r"^[A-Za-z0-9\.\-_]+\.bin$", fname):
|
||||
raise web.HTTPBadRequest(text="invalid filename")
|
||||
|
||||
file_path = os.path.join(self.bin_dir, fname)
|
||||
# ensure realpath is under bin_dir
|
||||
file_real = os.path.realpath(file_path)
|
||||
bin_dir_real = os.path.realpath(self.bin_dir)
|
||||
if (
|
||||
not file_real.startswith(bin_dir_real + os.sep)
|
||||
and file_real != bin_dir_real
|
||||
):
|
||||
raise web.HTTPForbidden(text="forbidden")
|
||||
|
||||
if not os.path.isfile(file_real):
|
||||
raise web.HTTPNotFound(text="file not found")
|
||||
|
||||
# use FileResponse to stream file
|
||||
resp = web.FileResponse(path=file_real)
|
||||
except web.HTTPError as e:
|
||||
resp = e
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"固件下载异常: {e}")
|
||||
resp = web.Response(text="download error", status=500)
|
||||
finally:
|
||||
try:
|
||||
self._add_cors_headers(resp)
|
||||
except Exception:
|
||||
pass
|
||||
return resp
|
||||
|
||||
@@ -2,6 +2,7 @@ import json
|
||||
import copy
|
||||
from aiohttp import web
|
||||
from config.logger import setup_logging
|
||||
from core.api.base_handler import BaseHandler
|
||||
from core.utils.util import get_vision_url, is_valid_image_file
|
||||
from core.utils.vllm import create_instance
|
||||
from config.config_loader import get_private_config_from_api
|
||||
@@ -16,10 +17,9 @@ TAG = __name__
|
||||
MAX_FILE_SIZE = 5 * 1024 * 1024
|
||||
|
||||
|
||||
class VisionHandler:
|
||||
class VisionHandler(BaseHandler):
|
||||
def __init__(self, config: dict):
|
||||
self.config = config
|
||||
self.logger = setup_logging()
|
||||
super().__init__(config)
|
||||
# 初始化认证工具
|
||||
self.auth = AuthToken(config["server"]["auth_key"])
|
||||
|
||||
@@ -172,11 +172,3 @@ class VisionHandler:
|
||||
finally:
|
||||
self._add_cors_headers(response)
|
||||
return response
|
||||
|
||||
def _add_cors_headers(self, response):
|
||||
"""添加CORS头信息"""
|
||||
response.headers["Access-Control-Allow-Headers"] = (
|
||||
"client-id, content-type, device-id"
|
||||
)
|
||||
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
|
||||
@@ -69,6 +69,7 @@ class ConnectionHandler:
|
||||
self.server = server # 保存server实例的引用
|
||||
|
||||
self.need_bind = False # 是否需要绑定设备
|
||||
self.bind_completed_event = asyncio.Event()
|
||||
self.bind_code = None # 绑定设备的验证码
|
||||
self.last_bind_prompt_time = 0 # 上次播放绑定提示的时间戳(秒)
|
||||
self.bind_prompt_interval = 60 # 绑定提示播放间隔(秒)
|
||||
@@ -268,30 +269,43 @@ class ConnectionHandler:
|
||||
f"保存记忆后关闭连接失败: {close_error}"
|
||||
)
|
||||
|
||||
async def _discard_message_with_bind_prompt(self):
|
||||
"""丢弃消息并检查是否需要播放绑定提示"""
|
||||
current_time = time.time()
|
||||
# 检查是否需要播放绑定提示
|
||||
if current_time - self.last_bind_prompt_time >= self.bind_prompt_interval:
|
||||
self.last_bind_prompt_time = current_time
|
||||
# 复用现有的绑定提示逻辑
|
||||
from core.handle.receiveAudioHandle import check_bind_device
|
||||
|
||||
asyncio.create_task(check_bind_device(self))
|
||||
|
||||
async def _route_message(self, message):
|
||||
"""消息路由"""
|
||||
# 检查是否已经获取到真实的绑定状态
|
||||
if not self.bind_completed_event.is_set():
|
||||
# 还没有获取到真实状态，等待直到获取到真实状态或超时
|
||||
try:
|
||||
await asyncio.wait_for(self.bind_completed_event.wait(), timeout=1)
|
||||
except asyncio.TimeoutError:
|
||||
# 超时仍未获取到真实状态，丢弃消息
|
||||
await self._discard_message_with_bind_prompt()
|
||||
return
|
||||
|
||||
# 已经获取到真实状态，检查是否需要绑定
|
||||
if self.need_bind:
|
||||
# 需要绑定，丢弃消息
|
||||
await self._discard_message_with_bind_prompt()
|
||||
return
|
||||
|
||||
# 不需要绑定，继续处理消息
|
||||
|
||||
if isinstance(message, str):
|
||||
await handleTextMessage(self, message)
|
||||
elif isinstance(message, bytes):
|
||||
if self.vad is None or self.asr is None:
|
||||
return
|
||||
|
||||
# 未绑定设备直接丢弃所有音频，不进行ASR处理
|
||||
if self.need_bind:
|
||||
current_time = time.time()
|
||||
# 检查是否需要播放绑定提示
|
||||
if (
|
||||
current_time - self.last_bind_prompt_time
|
||||
>= self.bind_prompt_interval
|
||||
):
|
||||
self.last_bind_prompt_time = current_time
|
||||
# 复用现有的绑定提示逻辑
|
||||
from core.handle.receiveAudioHandle import check_bind_device
|
||||
|
||||
asyncio.create_task(check_bind_device(self))
|
||||
# 直接丢弃音频，不进行ASR处理
|
||||
return
|
||||
|
||||
# 处理来自MQTT网关的音频包
|
||||
if self.conn_from_mqtt_gateway and len(message) >= 16:
|
||||
handled = await self._process_mqtt_audio_message(message)
|
||||
@@ -415,6 +429,15 @@ class ConnectionHandler:
|
||||
|
||||
def _initialize_components(self):
|
||||
try:
|
||||
if self.tts is None:
|
||||
self.tts = self._initialize_tts()
|
||||
# 打开语音合成通道
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self.tts.open_audio_channels(self), self.loop
|
||||
)
|
||||
if self.need_bind:
|
||||
self.bind_completed_event.set()
|
||||
return
|
||||
self.selected_module_str = build_module_string(
|
||||
self.config.get("selected_module", {})
|
||||
)
|
||||
@@ -438,17 +461,10 @@ class ConnectionHandler:
|
||||
|
||||
# 初始化声纹识别
|
||||
self._initialize_voiceprint()
|
||||
|
||||
# 打开语音识别通道
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self.asr.open_audio_channels(self), self.loop
|
||||
)
|
||||
if self.tts is None:
|
||||
self.tts = self._initialize_tts()
|
||||
# 打开语音合成通道
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self.tts.open_audio_channels(self), self.loop
|
||||
)
|
||||
|
||||
"""加载记忆"""
|
||||
self._initialize_memory()
|
||||
@@ -463,6 +479,7 @@ class ConnectionHandler:
|
||||
self.logger.bind(tag=TAG).error(f"实例化组件失败: {e}")
|
||||
|
||||
def _init_prompt_enhancement(self):
|
||||
|
||||
# 更新上下文信息
|
||||
self.prompt_manager.update_context_info(self, self.client_ip)
|
||||
enhanced_prompt = self.prompt_manager.build_enhanced_prompt(
|
||||
@@ -498,7 +515,11 @@ class ConnectionHandler:
|
||||
|
||||
def _initialize_asr(self):
|
||||
"""初始化ASR"""
|
||||
if self._asr.interface_type == InterfaceType.LOCAL:
|
||||
if (
|
||||
self._asr is not None
|
||||
and hasattr(self._asr, "interface_type")
|
||||
and self._asr.interface_type == InterfaceType.LOCAL
|
||||
):
|
||||
# 如果公共ASR是本地服务，则直接返回
|
||||
# 因为本地一个实例ASR，可以被多个连接共享
|
||||
asr = self._asr
|
||||
@@ -538,6 +559,8 @@ class ConnectionHandler:
|
||||
async def _initialize_private_config_async(self):
|
||||
"""从接口异步获取差异化配置（异步版本，不阻塞主循环）"""
|
||||
if not self.read_config_from_api:
|
||||
self.need_bind = False
|
||||
self.bind_completed_event.set()
|
||||
return
|
||||
try:
|
||||
begin_time = time.time()
|
||||
@@ -550,6 +573,8 @@ class ConnectionHandler:
|
||||
self.logger.bind(tag=TAG).info(
|
||||
f"{time.time() - begin_time} 秒，异步获取差异化配置成功: {json.dumps(filter_sensitive_info(private_config), ensure_ascii=False)}"
|
||||
)
|
||||
self.need_bind = False
|
||||
self.bind_completed_event.set()
|
||||
except DeviceNotFoundException as e:
|
||||
self.need_bind = True
|
||||
private_config = {}
|
||||
@@ -1173,6 +1198,11 @@ class ConnectionHandler:
|
||||
except queue.Empty:
|
||||
break
|
||||
|
||||
# 重置音频流控器（取消后台任务并清空队列）
|
||||
if hasattr(self, "audio_rate_controller") and self.audio_rate_controller:
|
||||
self.audio_rate_controller.reset()
|
||||
self.logger.bind(tag=TAG).debug("已重置音频流控器")
|
||||
|
||||
self.logger.bind(tag=TAG).debug(
|
||||
f"清理结束: TTS队列大小={self.tts.tts_text_queue.qsize()}, 音频队列大小={self.tts.tts_audio_queue.qsize()}"
|
||||
)
|
||||
|
||||
@@ -101,7 +101,7 @@ async def checkWakeupWords(conn, text):
|
||||
}
|
||||
|
||||
# 获取音频数据
|
||||
opus_packets = audio_to_data(response.get("file_path"))
|
||||
opus_packets = await audio_to_data(response.get("file_path"), use_cache=False)
|
||||
# 播放唤醒词回复
|
||||
conn.client_abort = False
|
||||
|
||||
|
||||
@@ -123,7 +123,7 @@ async def max_out_size(conn):
|
||||
text = "不好意思，我现在有点事情要忙，明天这个时候我们再聊，约好了哦！明天不见不散，拜拜！"
|
||||
await send_stt_message(conn, text)
|
||||
file_path = "config/assets/max_output_size.wav"
|
||||
opus_packets = audio_to_data(file_path)
|
||||
opus_packets = await audio_to_data(file_path)
|
||||
conn.tts.tts_audio_queue.put((SentenceType.LAST, opus_packets, text))
|
||||
conn.close_after_chat = True
|
||||
|
||||
@@ -142,7 +142,7 @@ async def check_bind_device(conn):
|
||||
|
||||
# 播放提示音
|
||||
music_path = "config/assets/bind_code.wav"
|
||||
opus_packets = audio_to_data(music_path)
|
||||
opus_packets = await audio_to_data(music_path)
|
||||
conn.tts.tts_audio_queue.put((SentenceType.FIRST, opus_packets, text))
|
||||
|
||||
# 逐个播放数字
|
||||
@@ -150,7 +150,7 @@ async def check_bind_device(conn):
|
||||
try:
|
||||
digit = conn.bind_code[i]
|
||||
num_path = f"config/assets/bind_code/{digit}.wav"
|
||||
num_packets = audio_to_data(num_path)
|
||||
num_packets = await audio_to_data(num_path)
|
||||
conn.tts.tts_audio_queue.put((SentenceType.MIDDLE, num_packets, None))
|
||||
except Exception as e:
|
||||
conn.logger.bind(tag=TAG).error(f"播放数字音频失败: {e}")
|
||||
@@ -162,5 +162,5 @@ async def check_bind_device(conn):
|
||||
text = f"没有找到该设备的版本信息，请正确配置 OTA地址，然后重新编译固件。"
|
||||
await send_stt_message(conn, text)
|
||||
music_path = "config/assets/bind_not_found.wav"
|
||||
opus_packets = audio_to_data(music_path)
|
||||
opus_packets = await audio_to_data(music_path)
|
||||
conn.tts.tts_audio_queue.put((SentenceType.LAST, opus_packets, text))
|
||||
|
||||
@@ -10,7 +10,6 @@ TTS上报功能已集成到ConnectionHandler类中。
|
||||
"""
|
||||
|
||||
import time
|
||||
import gc
|
||||
import opuslib_next
|
||||
|
||||
from config.manage_api_client import report as manage_report
|
||||
|
||||
@@ -7,6 +7,10 @@ from core.providers.tts.dto.dto import SentenceType
|
||||
from core.utils.audioRateController import AudioRateController
|
||||
|
||||
TAG = __name__
|
||||
# 音频帧时长（毫秒）
|
||||
AUDIO_FRAME_DURATION = 60
|
||||
# 预缓冲包数量，直接发送以减少延迟
|
||||
PRE_BUFFER_COUNT = 5
|
||||
|
||||
|
||||
async def sendAudioMessage(conn, sentenceType, audios, text):
|
||||
@@ -16,7 +20,19 @@ async def sendAudioMessage(conn, sentenceType, audios, text):
|
||||
await send_tts_message(conn, "start", None)
|
||||
|
||||
if sentenceType == SentenceType.FIRST:
|
||||
await send_tts_message(conn, "sentence_start", text)
|
||||
# 同一句子的后续消息加入流控队列，其他情况立即发送
|
||||
if (
|
||||
hasattr(conn, "audio_rate_controller")
|
||||
and conn.audio_rate_controller
|
||||
and getattr(conn, "audio_flow_control", {}).get("sentence_id")
|
||||
== conn.sentence_id
|
||||
):
|
||||
conn.audio_rate_controller.add_message(
|
||||
lambda: send_tts_message(conn, "sentence_start", text)
|
||||
)
|
||||
else:
|
||||
# 新句子或流控器未初始化，立即发送
|
||||
await send_tts_message(conn, "sentence_start", text)
|
||||
|
||||
await sendAudio(conn, audios)
|
||||
# 发送句子开始消息
|
||||
@@ -31,6 +47,29 @@ async def sendAudioMessage(conn, sentenceType, audios, text):
|
||||
await conn.close()
|
||||
|
||||
|
||||
async def _wait_for_audio_completion(conn):
|
||||
"""
|
||||
等待音频队列清空并等待预缓冲包播放完成
|
||||
|
||||
Args:
|
||||
conn: 连接对象
|
||||
"""
|
||||
if hasattr(conn, "audio_rate_controller") and conn.audio_rate_controller:
|
||||
rate_controller = conn.audio_rate_controller
|
||||
conn.logger.bind(tag=TAG).debug(
|
||||
f"等待音频发送完成，队列中还有 {len(rate_controller.queue)} 个包"
|
||||
)
|
||||
await rate_controller.queue_empty_event.wait()
|
||||
|
||||
# 等待预缓冲包播放完成
|
||||
# 前N个包直接发送，增加2个网络抖动包，需要额外等待它们在客户端播放完成
|
||||
frame_duration_ms = rate_controller.frame_duration
|
||||
pre_buffer_playback_time = (PRE_BUFFER_COUNT + 2) * frame_duration_ms / 1000.0
|
||||
await asyncio.sleep(pre_buffer_playback_time)
|
||||
|
||||
conn.logger.bind(tag=TAG).debug("音频发送完成")
|
||||
|
||||
|
||||
async def _send_to_mqtt_gateway(conn, opus_packet, timestamp, sequence):
|
||||
"""
|
||||
发送带16字节头部的opus数据包给mqtt_gateway
|
||||
@@ -53,139 +92,133 @@ async def _send_to_mqtt_gateway(conn, opus_packet, timestamp, sequence):
|
||||
await conn.websocket.send(complete_packet)
|
||||
|
||||
|
||||
# 播放音频 - 使用 AudioRateController 进行精确流控
|
||||
async def sendAudio(conn, audios, frame_duration=60):
|
||||
async def sendAudio(conn, audios, frame_duration=AUDIO_FRAME_DURATION):
|
||||
"""
|
||||
发送音频包，使用 AudioRateController 进行精确的流量控制
|
||||
|
||||
Args:
|
||||
conn: 连接对象
|
||||
audios: 单个opus包(bytes) 或 opus包列表
|
||||
frame_duration: 帧时长（毫秒），默认60ms
|
||||
|
||||
改进点：
|
||||
1. 使用单一时间基准，避免累积误差
|
||||
2. 每次检查队列时重新计算 elapsed_ms，更精准
|
||||
3. 支持高并发而不产生时间偏差
|
||||
frame_duration: 帧时长（毫秒），默认使用全局常量AUDIO_FRAME_DURATION
|
||||
"""
|
||||
if audios is None or len(audios) == 0:
|
||||
return
|
||||
|
||||
# 获取发送延迟配置
|
||||
send_delay = conn.config.get("tts_audio_send_delay", -1) / 1000.0
|
||||
is_single_packet = isinstance(audios, bytes)
|
||||
|
||||
if isinstance(audios, bytes):
|
||||
# 单个 opus 包处理
|
||||
await _sendAudio_single(conn, audios, send_delay, frame_duration)
|
||||
else:
|
||||
# 音频列表处理（如文件型音频）
|
||||
await _sendAudio_list(conn, audios, send_delay, frame_duration)
|
||||
# 初始化或获取 RateController
|
||||
rate_controller, flow_control = _get_or_create_rate_controller(
|
||||
conn, frame_duration, is_single_packet
|
||||
)
|
||||
|
||||
# 统一转换为列表处理
|
||||
audio_list = [audios] if is_single_packet else audios
|
||||
|
||||
# 发送音频包
|
||||
await _send_audio_with_rate_control(
|
||||
conn, audio_list, rate_controller, flow_control, send_delay
|
||||
)
|
||||
|
||||
|
||||
async def _sendAudio_single(conn, opus_packet, send_delay, frame_duration=60):
|
||||
def _get_or_create_rate_controller(conn, frame_duration, is_single_packet):
|
||||
"""
|
||||
发送单个 opus 包
|
||||
使用 AudioRateController 进行流控
|
||||
获取或创建 RateController 和 flow_control
|
||||
|
||||
Args:
|
||||
conn: 连接对象
|
||||
frame_duration: 帧时长
|
||||
is_single_packet: 是否单包模式（True: TTS流式单包, False: 批量包）
|
||||
|
||||
Returns:
|
||||
(rate_controller, flow_control)
|
||||
"""
|
||||
# 重置流控状态，第一次读取和会话发生转变时
|
||||
if not hasattr(conn, "audio_rate_controller") or conn.audio_flow_control.get("sentence_id") != conn.sentence_id:
|
||||
if hasattr(conn, "audio_rate_controller"):
|
||||
conn.audio_rate_controller.reset()
|
||||
else:
|
||||
# 判断是否需要重置：单包模式且 sentence_id 变化，或者控制器不存在
|
||||
need_reset = (
|
||||
is_single_packet
|
||||
and getattr(conn, "audio_flow_control", {}).get("sentence_id")
|
||||
!= conn.sentence_id
|
||||
) or not hasattr(conn, "audio_rate_controller")
|
||||
|
||||
if need_reset:
|
||||
# 创建或获取 rate_controller
|
||||
if not hasattr(conn, "audio_rate_controller"):
|
||||
conn.audio_rate_controller = AudioRateController(frame_duration)
|
||||
else:
|
||||
conn.audio_rate_controller.reset()
|
||||
|
||||
# 初始化 flow_control
|
||||
conn.audio_flow_control = {
|
||||
"packet_count": 0,
|
||||
"sequence": 0,
|
||||
"sentence_id": conn.sentence_id,
|
||||
}
|
||||
|
||||
if conn.client_abort:
|
||||
return
|
||||
# 启动后台发送循环
|
||||
_start_background_sender(
|
||||
conn, conn.audio_rate_controller, conn.audio_flow_control
|
||||
)
|
||||
|
||||
conn.last_activity_time = time.time() * 1000
|
||||
return conn.audio_rate_controller, conn.audio_flow_control
|
||||
|
||||
rate_controller = conn.audio_rate_controller
|
||||
flow_control = conn.audio_flow_control
|
||||
packet_count = flow_control["packet_count"]
|
||||
|
||||
# 预缓冲：前5个包直接发送，不做延迟
|
||||
pre_buffer_count = 5
|
||||
def _start_background_sender(conn, rate_controller, flow_control):
|
||||
"""
|
||||
启动后台发送循环任务
|
||||
|
||||
if packet_count < pre_buffer_count or send_delay > 0:
|
||||
# 预缓冲阶段或固定延迟模式，直接发送
|
||||
await _do_send_audio(conn, opus_packet, flow_control, frame_duration)
|
||||
Args:
|
||||
conn: 连接对象
|
||||
rate_controller: 速率控制器
|
||||
flow_control: 流控状态
|
||||
"""
|
||||
|
||||
async def send_callback(packet):
|
||||
# 检查是否应该中止
|
||||
if conn.client_abort:
|
||||
raise asyncio.CancelledError("客户端已中止")
|
||||
|
||||
conn.last_activity_time = time.time() * 1000
|
||||
await _do_send_audio(conn, packet, flow_control)
|
||||
conn.client_is_speaking = True
|
||||
|
||||
if send_delay > 0 and packet_count >= pre_buffer_count:
|
||||
await asyncio.sleep(send_delay)
|
||||
else:
|
||||
# 使用流控器进行精确的速率控制
|
||||
rate_controller.add_audio(opus_packet)
|
||||
|
||||
async def send_callback(packet):
|
||||
await _do_send_audio(conn, packet, flow_control, frame_duration)
|
||||
|
||||
await rate_controller.check_queue(send_callback)
|
||||
conn.client_is_speaking = True
|
||||
|
||||
# 更新流控状态
|
||||
flow_control["packet_count"] += 1
|
||||
flow_control["sequence"] += 1
|
||||
# 使用 start_sending 启动后台循环
|
||||
rate_controller.start_sending(send_callback)
|
||||
|
||||
|
||||
async def _sendAudio_list(conn, audios, send_delay, frame_duration=60):
|
||||
async def _send_audio_with_rate_control(
|
||||
conn, audio_list, rate_controller, flow_control, send_delay
|
||||
):
|
||||
"""
|
||||
发送音频列表（如文件型音频）
|
||||
使用 rate_controller 发送音频包
|
||||
|
||||
Args:
|
||||
conn: 连接对象
|
||||
audio_list: 音频包列表
|
||||
rate_controller: 速率控制器
|
||||
flow_control: 流控状态
|
||||
send_delay: 固定延迟（秒），-1表示使用动态流控
|
||||
"""
|
||||
if not audios:
|
||||
return
|
||||
|
||||
rate_controller = AudioRateController(frame_duration)
|
||||
rate_controller.reset()
|
||||
|
||||
flow_control = {
|
||||
"packet_count": 0,
|
||||
"sequence": 0,
|
||||
}
|
||||
|
||||
# 预缓冲：前5个包直接发送
|
||||
pre_buffer_frames = min(5, len(audios))
|
||||
for i in range(pre_buffer_frames):
|
||||
for packet in audio_list:
|
||||
if conn.client_abort:
|
||||
return
|
||||
await _do_send_audio(conn, audios[i], flow_control, frame_duration)
|
||||
conn.client_is_speaking = True
|
||||
|
||||
remaining_audios = audios[pre_buffer_frames:]
|
||||
|
||||
# 处理剩余音频帧
|
||||
for i, opus_packet in enumerate(remaining_audios):
|
||||
if conn.client_abort:
|
||||
break
|
||||
|
||||
conn.last_activity_time = time.time() * 1000
|
||||
|
||||
if send_delay > 0:
|
||||
# 预缓冲：前N个包直接发送
|
||||
if flow_control["packet_count"] < PRE_BUFFER_COUNT:
|
||||
await _do_send_audio(conn, packet, flow_control)
|
||||
conn.client_is_speaking = True
|
||||
elif send_delay > 0:
|
||||
# 固定延迟模式
|
||||
await asyncio.sleep(send_delay)
|
||||
else:
|
||||
# 使用流控器进行精确延迟
|
||||
rate_controller.add_audio(opus_packet)
|
||||
|
||||
async def send_callback(packet):
|
||||
await _do_send_audio(conn, packet, flow_control, frame_duration)
|
||||
|
||||
await rate_controller.check_queue(send_callback)
|
||||
await _do_send_audio(conn, packet, flow_control)
|
||||
conn.client_is_speaking = True
|
||||
continue
|
||||
|
||||
await _do_send_audio(conn, opus_packet, flow_control, frame_duration)
|
||||
conn.client_is_speaking = True
|
||||
else:
|
||||
# 动态流控模式：仅添加到队列，由后台循环负责发送
|
||||
rate_controller.add_audio(packet)
|
||||
|
||||
|
||||
async def _do_send_audio(conn, opus_packet, flow_control, frame_duration=60):
|
||||
async def _do_send_audio(conn, opus_packet, flow_control):
|
||||
"""
|
||||
执行实际的音频发送
|
||||
"""
|
||||
@@ -222,8 +255,10 @@ async def send_tts_message(conn, state, text=None):
|
||||
stop_tts_notify_voice = conn.config.get(
|
||||
"stop_tts_notify_voice", "config/assets/tts_notify.mp3"
|
||||
)
|
||||
audios = audio_to_data(stop_tts_notify_voice, is_opus=True)
|
||||
audios = await audio_to_data(stop_tts_notify_voice, is_opus=True)
|
||||
await sendAudio(conn, audios)
|
||||
# 等待所有音频包发送完成
|
||||
await _wait_for_audio_completion(conn)
|
||||
# 清除服务端讲话状态
|
||||
conn.clearSpeakStatus()
|
||||
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
import time
|
||||
import asyncio
|
||||
from typing import Dict, Any
|
||||
|
||||
from core.handle.receiveAudioHandle import handleAudioMessage, startToChat
|
||||
from core.handle.receiveAudioHandle import startToChat
|
||||
from core.handle.reportHandle import enqueue_asr_report
|
||||
from core.handle.sendAudioHandle import send_stt_message, send_tts_message
|
||||
from core.handle.textMessageHandler import TextMessageHandler
|
||||
from core.handle.textMessageType import TextMessageType
|
||||
from core.utils.util import remove_punctuation_and_length
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
|
||||
@@ -29,8 +31,18 @@ class ListenTextMessageHandler(TextMessageHandler):
|
||||
elif msg_json["state"] == "stop":
|
||||
conn.client_have_voice = True
|
||||
conn.client_voice_stop = True
|
||||
if len(conn.asr_audio) > 0:
|
||||
await handleAudioMessage(conn, b"")
|
||||
if conn.asr.interface_type == InterfaceType.STREAM:
|
||||
# 流式模式下，发送结束请求
|
||||
asyncio.create_task(conn.asr._send_stop_request())
|
||||
else:
|
||||
# 非流式模式：直接触发ASR识别
|
||||
if len(conn.asr_audio) > 0:
|
||||
asr_audio_task = conn.asr_audio.copy()
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
|
||||
if len(asr_audio_task) > 0:
|
||||
await conn.asr.handle_voice_stop(conn, asr_audio_task)
|
||||
elif msg_json["state"] == "detect":
|
||||
conn.client_have_voice = False
|
||||
conn.asr_audio.clear()
|
||||
@@ -57,6 +69,7 @@ class ListenTextMessageHandler(TextMessageHandler):
|
||||
enqueue_asr_report(conn, "嘿，你好呀", [])
|
||||
await startToChat(conn, "嘿，你好呀")
|
||||
else:
|
||||
conn.just_woken_up = True
|
||||
# 上报纯文字数据（复用ASR上报功能，但不提供音频数据）
|
||||
enqueue_asr_report(conn, original_text, [])
|
||||
# 否则需要LLM对文字内容进行答复
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
import json
|
||||
import time
|
||||
from typing import Dict, Any
|
||||
|
||||
from core.handle.textMessageHandler import TextMessageHandler
|
||||
from core.handle.textMessageType import TextMessageType
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
class PingMessageHandler(TextMessageHandler):
|
||||
"""Ping消息处理器，用于保持WebSocket连接"""
|
||||
|
||||
@property
|
||||
def message_type(self) -> TextMessageType:
|
||||
return TextMessageType.PING
|
||||
|
||||
async def handle(self, conn, msg_json: Dict[str, Any]) -> None:
|
||||
"""
|
||||
处理PING消息，发送PONG响应
|
||||
消息格式：{"type": "ping"}
|
||||
Args:
|
||||
conn: WebSocket连接对象
|
||||
msg_json: PING消息的JSON数据
|
||||
"""
|
||||
# 检查是否启用了WebSocket心跳功能
|
||||
enable_websocket_ping = conn.config.get("enable_websocket_ping", False)
|
||||
if not enable_websocket_ping:
|
||||
conn.logger.debug(f"WebSocket心跳功能未启用，忽略PING消息")
|
||||
return
|
||||
|
||||
try:
|
||||
conn.logger.debug(f"收到PING消息，发送PONG响应")
|
||||
conn.last_activity_time = time.time() * 1000
|
||||
# 构造PONG响应消息
|
||||
pong_message = {
|
||||
"type": "pong",
|
||||
"timestamp": time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()),
|
||||
}
|
||||
|
||||
# 发送PONG响应
|
||||
await conn.websocket.send(json.dumps(pong_message))
|
||||
|
||||
except Exception as e:
|
||||
conn.logger.error(f"处理PING消息时发生错误: {e}")
|
||||
@@ -7,6 +7,7 @@ from core.handle.textHandler.listenMessageHandler import ListenTextMessageHandle
|
||||
from core.handle.textHandler.mcpMessageHandler import McpTextMessageHandler
|
||||
from core.handle.textMessageHandler import TextMessageHandler
|
||||
from core.handle.textHandler.serverMessageHandler import ServerTextMessageHandler
|
||||
from core.handle.textHandler.pingMessageHandler import PingMessageHandler
|
||||
|
||||
TAG = __name__
|
||||
|
||||
@@ -27,6 +28,7 @@ class TextMessageHandlerRegistry:
|
||||
IotTextMessageHandler(),
|
||||
McpTextMessageHandler(),
|
||||
ServerTextMessageHandler(),
|
||||
PingMessageHandler(),
|
||||
]
|
||||
|
||||
for handler in handlers:
|
||||
|
||||
@@ -9,3 +9,4 @@ class TextMessageType(Enum):
|
||||
IOT = "iot"
|
||||
MCP = "mcp"
|
||||
SERVER = "server"
|
||||
PING = "ping"
|
||||
|
||||
@@ -33,38 +33,60 @@ class SimpleHttpServer:
|
||||
return f"ws://{local_ip}:{port}/xiaozhi/v1/"
|
||||
|
||||
async def start(self):
|
||||
server_config = self.config["server"]
|
||||
read_config_from_api = self.config.get("read_config_from_api", False)
|
||||
host = server_config.get("ip", "0.0.0.0")
|
||||
port = int(server_config.get("http_port", 8003))
|
||||
try:
|
||||
server_config = self.config["server"]
|
||||
read_config_from_api = self.config.get("read_config_from_api", False)
|
||||
host = server_config.get("ip", "0.0.0.0")
|
||||
port = int(server_config.get("http_port", 8003))
|
||||
|
||||
if port:
|
||||
app = web.Application()
|
||||
if port:
|
||||
app = web.Application()
|
||||
|
||||
if not read_config_from_api:
|
||||
# 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
|
||||
if not read_config_from_api:
|
||||
# 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
|
||||
app.add_routes(
|
||||
[
|
||||
web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
|
||||
web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
|
||||
web.options(
|
||||
"/xiaozhi/ota/", self.ota_handler.handle_options
|
||||
),
|
||||
# 下载接口，仅提供 data/bin/*.bin 下载
|
||||
web.get(
|
||||
"/xiaozhi/ota/download/{filename}",
|
||||
self.ota_handler.handle_download,
|
||||
),
|
||||
web.options(
|
||||
"/xiaozhi/ota/download/{filename}",
|
||||
self.ota_handler.handle_options,
|
||||
),
|
||||
]
|
||||
)
|
||||
# 添加路由
|
||||
app.add_routes(
|
||||
[
|
||||
web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
|
||||
web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
|
||||
web.options("/xiaozhi/ota/", self.ota_handler.handle_post),
|
||||
web.get("/mcp/vision/explain", self.vision_handler.handle_get),
|
||||
web.post(
|
||||
"/mcp/vision/explain", self.vision_handler.handle_post
|
||||
),
|
||||
web.options(
|
||||
"/mcp/vision/explain", self.vision_handler.handle_options
|
||||
),
|
||||
]
|
||||
)
|
||||
# 添加路由
|
||||
app.add_routes(
|
||||
[
|
||||
web.get("/mcp/vision/explain", self.vision_handler.handle_get),
|
||||
web.post("/mcp/vision/explain", self.vision_handler.handle_post),
|
||||
web.options("/mcp/vision/explain", self.vision_handler.handle_post),
|
||||
]
|
||||
)
|
||||
|
||||
# 运行服务
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, host, port)
|
||||
await site.start()
|
||||
# 运行服务
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, host, port)
|
||||
await site.start()
|
||||
|
||||
# 保持服务运行
|
||||
while True:
|
||||
await asyncio.sleep(3600) # 每隔 1 小时检查一次
|
||||
# 保持服务运行
|
||||
while True:
|
||||
await asyncio.sleep(3600) # 每隔 1 小时检查一次
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"HTTP服务器启动失败: {e}")
|
||||
import traceback
|
||||
|
||||
self.logger.bind(tag=TAG).error(f"错误堆栈: {traceback.format_exc()}")
|
||||
raise
|
||||
|
||||
@@ -5,12 +5,9 @@ import hmac
|
||||
import base64
|
||||
import hashlib
|
||||
import asyncio
|
||||
import gc
|
||||
import requests
|
||||
import websockets
|
||||
import opuslib_next
|
||||
import random
|
||||
from typing import Optional, Tuple, List
|
||||
from urllib import parse
|
||||
from datetime import datetime
|
||||
from config.logger import setup_logging
|
||||
@@ -140,13 +137,13 @@ class ASRProvider(ASRProviderBase):
|
||||
conn.asr_audio.append(audio)
|
||||
conn.asr_audio = conn.asr_audio[-10:]
|
||||
|
||||
# 只在有声音且没有连接时建立连接
|
||||
if audio_have_voice and not self.is_processing:
|
||||
# 只在有声音且没有连接时建立连接（排除正在停止的情况）
|
||||
if audio_have_voice and not self.is_processing and not self.asr_ws:
|
||||
try:
|
||||
await self._start_recognition(conn)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"开始识别失败: {str(e)}")
|
||||
await self._cleanup(conn)
|
||||
await self._cleanup()
|
||||
return
|
||||
|
||||
if self.asr_ws and self.is_processing and self.server_ready:
|
||||
@@ -186,10 +183,8 @@ class ASRProvider(ASRProviderBase):
|
||||
"header": {
|
||||
"namespace": "SpeechTranscriber",
|
||||
"name": "StartTranscription",
|
||||
"status": 20000000,
|
||||
"message_id": uuid.uuid4().hex,
|
||||
"task_id": self.task_id,
|
||||
"status_text": "Gateway:SUCCESS:Success.",
|
||||
"appkey": self.appkey
|
||||
},
|
||||
"payload": {
|
||||
@@ -208,18 +203,21 @@ class ASRProvider(ASRProviderBase):
|
||||
async def _forward_results(self, conn):
|
||||
"""转发识别结果"""
|
||||
try:
|
||||
while self.asr_ws and not conn.stop_event.is_set():
|
||||
while not conn.stop_event.is_set():
|
||||
try:
|
||||
response = await asyncio.wait_for(self.asr_ws.recv(), timeout=1.0)
|
||||
result = json.loads(response)
|
||||
|
||||
|
||||
header = result.get("header", {})
|
||||
payload = result.get("payload", {})
|
||||
message_name = header.get("name", "")
|
||||
status = header.get("status", 0)
|
||||
|
||||
|
||||
if status != 20000000:
|
||||
if status in [40000004, 40010004]: # 连接超时或客户端断开
|
||||
if status == 40010004:
|
||||
logger.bind(tag=TAG).warning(f"请在服务端响应完成后再关闭链接，状态码: {status}")
|
||||
break
|
||||
if status in [40000004, 40010003]: # 连接超时或客户端断开
|
||||
logger.bind(tag=TAG).warning(f"连接问题，状态码: {status}")
|
||||
break
|
||||
elif status in [40270002, 40270003]: # 音频问题
|
||||
@@ -228,12 +226,12 @@ class ASRProvider(ASRProviderBase):
|
||||
else:
|
||||
logger.bind(tag=TAG).error(f"识别错误，状态码: {status}, 消息: {header.get('status_text', '')}")
|
||||
continue
|
||||
|
||||
|
||||
# 收到TranscriptionStarted表示服务器准备好接收音频数据
|
||||
if message_name == "TranscriptionStarted":
|
||||
self.server_ready = True
|
||||
logger.bind(tag=TAG).debug("服务器已准备，开始发送缓存音频...")
|
||||
|
||||
|
||||
# 发送缓存音频
|
||||
if conn.asr_audio:
|
||||
for cached_audio in conn.asr_audio[-10:]:
|
||||
@@ -244,89 +242,89 @@ class ASRProvider(ASRProviderBase):
|
||||
logger.bind(tag=TAG).warning(f"发送缓存音频失败: {e}")
|
||||
break
|
||||
continue
|
||||
|
||||
if message_name == "TranscriptionResultChanged":
|
||||
# 中间结果
|
||||
text = payload.get("result", "")
|
||||
if text:
|
||||
self.text = text
|
||||
elif message_name == "SentenceEnd":
|
||||
# 最终结果
|
||||
# 句子结束（每个句子都会触发）
|
||||
text = payload.get("result", "")
|
||||
if text:
|
||||
self.text = text
|
||||
conn.reset_vad_states()
|
||||
# 传递缓存的音频数据
|
||||
audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
# 清空缓存
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
break
|
||||
elif message_name == "TranscriptionCompleted":
|
||||
# 识别完成
|
||||
self.is_processing = False
|
||||
break
|
||||
|
||||
logger.bind(tag=TAG).info(f"识别到文本: {text}")
|
||||
|
||||
# 手动模式下累积识别结果
|
||||
if conn.client_listen_mode == "manual":
|
||||
if self.text:
|
||||
self.text += text
|
||||
else:
|
||||
self.text = text
|
||||
|
||||
# 手动模式下，只有在收到stop信号后才触发处理（仅处理一次）
|
||||
if conn.client_voice_stop:
|
||||
audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
|
||||
if len(audio_data) > 0:
|
||||
logger.bind(tag=TAG).debug("收到最终识别结果，触发处理")
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
# 清理音频缓存
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
break
|
||||
else:
|
||||
# 自动模式下直接覆盖
|
||||
self.text = text
|
||||
conn.reset_vad_states()
|
||||
audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
break
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
continue
|
||||
except websockets.exceptions.ConnectionClosed:
|
||||
logger.bind(tag=TAG).error("接收结果超时")
|
||||
break
|
||||
except websockets.ConnectionClosed:
|
||||
logger.bind(tag=TAG).info("ASR服务连接已关闭")
|
||||
self.is_processing = False
|
||||
break
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"处理结果失败: {str(e)}")
|
||||
break
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"结果转发失败: {str(e)}")
|
||||
finally:
|
||||
await self._cleanup(conn)
|
||||
# 清理连接的音频缓存
|
||||
await self._cleanup()
|
||||
if conn:
|
||||
if hasattr(conn, 'asr_audio_for_voiceprint'):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
if hasattr(conn, 'asr_audio'):
|
||||
conn.asr_audio = []
|
||||
|
||||
async def _cleanup(self, conn):
|
||||
"""清理资源"""
|
||||
logger.bind(tag=TAG).debug(f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}")
|
||||
|
||||
# 清理连接的音频缓存
|
||||
if conn and hasattr(conn, 'asr_audio_for_voiceprint'):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
|
||||
# 判断是否需要发送终止请求
|
||||
should_stop = self.is_processing or self.server_ready
|
||||
|
||||
# 发送停止识别请求
|
||||
if self.asr_ws and should_stop:
|
||||
async def _send_stop_request(self):
|
||||
"""发送停止识别请求（不关闭连接）"""
|
||||
if self.asr_ws:
|
||||
try:
|
||||
# 先停止音频发送
|
||||
self.is_processing = False
|
||||
|
||||
stop_msg = {
|
||||
"header": {
|
||||
"namespace": "SpeechTranscriber",
|
||||
"name": "StopTranscription",
|
||||
"status": 20000000,
|
||||
"message_id": uuid.uuid4().hex,
|
||||
"task_id": self.task_id,
|
||||
"status_text": "Client:Stop",
|
||||
"appkey": self.appkey
|
||||
}
|
||||
}
|
||||
logger.bind(tag=TAG).debug("正在发送ASR终止请求")
|
||||
logger.bind(tag=TAG).debug("停止识别请求已发送")
|
||||
await self.asr_ws.send(json.dumps(stop_msg, ensure_ascii=False))
|
||||
await asyncio.sleep(0.1)
|
||||
logger.bind(tag=TAG).debug("ASR终止请求已发送")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"ASR终止请求发送失败: {e}")
|
||||
|
||||
# 状态重置（在终止请求发送后）
|
||||
logger.bind(tag=TAG).error(f"发送停止识别请求失败: {e}")
|
||||
|
||||
async def _cleanup(self):
|
||||
"""清理资源（关闭连接）"""
|
||||
logger.bind(tag=TAG).debug(f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}")
|
||||
|
||||
# 状态重置
|
||||
self.is_processing = False
|
||||
self.server_ready = False
|
||||
logger.bind(tag=TAG).debug("ASR状态已重置")
|
||||
|
||||
# 清理任务
|
||||
if self.forward_task and not self.forward_task.done():
|
||||
self.forward_task.cancel()
|
||||
try:
|
||||
await asyncio.wait_for(self.forward_task, timeout=1.0)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).debug(f"forward_task取消异常: {e}")
|
||||
finally:
|
||||
self.forward_task = None
|
||||
|
||||
# 关闭连接
|
||||
if self.asr_ws:
|
||||
try:
|
||||
@@ -337,7 +335,10 @@ class ASRProvider(ASRProviderBase):
|
||||
logger.bind(tag=TAG).error(f"关闭WebSocket连接失败: {e}")
|
||||
finally:
|
||||
self.asr_ws = None
|
||||
|
||||
|
||||
# 清理任务引用
|
||||
self.forward_task = None
|
||||
|
||||
logger.bind(tag=TAG).debug("ASR会话清理完成")
|
||||
|
||||
async def speech_to_text(self, opus_data, session_id, audio_format):
|
||||
|
||||
@@ -9,8 +9,6 @@ import asyncio
|
||||
import traceback
|
||||
import threading
|
||||
import opuslib_next
|
||||
import concurrent.futures
|
||||
import gc
|
||||
from abc import ABC, abstractmethod
|
||||
from config.logger import setup_logging
|
||||
from typing import Optional, Tuple, List
|
||||
@@ -54,121 +52,89 @@ class ASRProviderBase(ABC):
|
||||
|
||||
# 接收音频
|
||||
async def receive_audio(self, conn, audio, audio_have_voice):
|
||||
if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
|
||||
have_voice = audio_have_voice
|
||||
if conn.client_listen_mode == "manual":
|
||||
# 手动模式：缓存音频用于ASR识别
|
||||
conn.asr_audio.append(audio)
|
||||
else:
|
||||
have_voice = conn.client_have_voice
|
||||
|
||||
conn.asr_audio.append(audio)
|
||||
if not have_voice and not conn.client_have_voice:
|
||||
conn.asr_audio = conn.asr_audio[-10:]
|
||||
return
|
||||
# 自动/实时模式：使用VAD检测
|
||||
have_voice = audio_have_voice
|
||||
|
||||
if conn.client_voice_stop:
|
||||
asr_audio_task = conn.asr_audio.copy()
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
conn.asr_audio.append(audio)
|
||||
if not have_voice and not conn.client_have_voice:
|
||||
conn.asr_audio = conn.asr_audio[-10:]
|
||||
return
|
||||
|
||||
if len(asr_audio_task) > 15:
|
||||
await self.handle_voice_stop(conn, asr_audio_task)
|
||||
# 自动模式下通过VAD检测到语音停止时触发识别
|
||||
if conn.client_voice_stop:
|
||||
asr_audio_task = conn.asr_audio.copy()
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
|
||||
if len(asr_audio_task) > 15:
|
||||
await self.handle_voice_stop(conn, asr_audio_task)
|
||||
|
||||
# 处理语音停止
|
||||
async def handle_voice_stop(self, conn, asr_audio_task: List[bytes]):
|
||||
"""并行处理ASR和声纹识别"""
|
||||
try:
|
||||
total_start_time = time.monotonic()
|
||||
|
||||
|
||||
# 准备音频数据
|
||||
if conn.audio_format == "pcm":
|
||||
pcm_data = asr_audio_task
|
||||
else:
|
||||
pcm_data = self.decode_opus(asr_audio_task)
|
||||
|
||||
|
||||
combined_pcm_data = b"".join(pcm_data)
|
||||
|
||||
|
||||
# 预先准备WAV数据
|
||||
wav_data = None
|
||||
if conn.voiceprint_provider and combined_pcm_data:
|
||||
wav_data = self._pcm_to_wav(combined_pcm_data)
|
||||
|
||||
|
||||
# 定义ASR任务
|
||||
def run_asr():
|
||||
start_time = time.monotonic()
|
||||
try:
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
result = loop.run_until_complete(
|
||||
self.speech_to_text(asr_audio_task, conn.session_id, conn.audio_format)
|
||||
)
|
||||
end_time = time.monotonic()
|
||||
logger.bind(tag=TAG).debug(f"ASR耗时: {end_time - start_time:.3f}s")
|
||||
return result
|
||||
finally:
|
||||
loop.close()
|
||||
except Exception as e:
|
||||
end_time = time.monotonic()
|
||||
logger.bind(tag=TAG).error(f"ASR失败: {e}")
|
||||
return ("", None)
|
||||
|
||||
# 定义声纹识别任务
|
||||
def run_voiceprint():
|
||||
if not wav_data:
|
||||
return None
|
||||
try:
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# 使用连接的声纹识别提供者
|
||||
result = loop.run_until_complete(
|
||||
conn.voiceprint_provider.identify_speaker(wav_data, conn.session_id)
|
||||
)
|
||||
return result
|
||||
finally:
|
||||
loop.close()
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"声纹识别失败: {e}")
|
||||
return None
|
||||
|
||||
# 使用线程池执行器并行运行
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as thread_executor:
|
||||
asr_future = thread_executor.submit(run_asr)
|
||||
|
||||
if conn.voiceprint_provider and wav_data:
|
||||
voiceprint_future = thread_executor.submit(run_voiceprint)
|
||||
|
||||
# 等待两个线程都完成
|
||||
asr_result = asr_future.result(timeout=15)
|
||||
voiceprint_result = voiceprint_future.result(timeout=15)
|
||||
|
||||
results = {"asr": asr_result, "voiceprint": voiceprint_result}
|
||||
else:
|
||||
asr_result = asr_future.result(timeout=15)
|
||||
results = {"asr": asr_result, "voiceprint": None}
|
||||
|
||||
|
||||
# 处理结果
|
||||
raw_text, _ = results.get("asr", ("", None))
|
||||
speaker_name = results.get("voiceprint", None)
|
||||
|
||||
# 记录识别结果
|
||||
asr_task = self.speech_to_text(asr_audio_task, conn.session_id, conn.audio_format)
|
||||
|
||||
if conn.voiceprint_provider and wav_data:
|
||||
voiceprint_task = conn.voiceprint_provider.identify_speaker(wav_data, conn.session_id)
|
||||
# 并发等待两个结果
|
||||
asr_result, voiceprint_result = await asyncio.gather(
|
||||
asr_task, voiceprint_task, return_exceptions=True
|
||||
)
|
||||
else:
|
||||
asr_result = await asr_task
|
||||
voiceprint_result = None
|
||||
|
||||
# 记录识别结果 - 检查是否为异常
|
||||
if isinstance(asr_result, Exception):
|
||||
logger.bind(tag=TAG).error(f"ASR识别失败: {asr_result}")
|
||||
raw_text = ""
|
||||
else:
|
||||
raw_text, _ = asr_result
|
||||
|
||||
if isinstance(voiceprint_result, Exception):
|
||||
logger.bind(tag=TAG).error(f"声纹识别失败: {voiceprint_result}")
|
||||
speaker_name = ""
|
||||
else:
|
||||
speaker_name = voiceprint_result
|
||||
|
||||
if raw_text:
|
||||
logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
|
||||
if speaker_name:
|
||||
logger.bind(tag=TAG).info(f"识别说话人: {speaker_name}")
|
||||
|
||||
|
||||
# 性能监控
|
||||
total_time = time.monotonic() - total_start_time
|
||||
logger.bind(tag=TAG).debug(f"总处理耗时: {total_time:.3f}s")
|
||||
|
||||
|
||||
# 检查文本长度
|
||||
text_len, _ = remove_punctuation_and_length(raw_text)
|
||||
self.stop_ws_connection()
|
||||
|
||||
|
||||
if text_len > 0:
|
||||
# 构建包含说话人信息的JSON字符串
|
||||
enhanced_text = self._build_enhanced_text(raw_text, speaker_name)
|
||||
|
||||
|
||||
# 使用自定义模块进行上报
|
||||
await startToChat(conn, enhanced_text)
|
||||
enqueue_asr_report(conn, enhanced_text, asr_audio_task)
|
||||
|
||||
@@ -4,7 +4,6 @@ import uuid
|
||||
import asyncio
|
||||
import websockets
|
||||
import opuslib_next
|
||||
import gc
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from config.logger import setup_logging
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
@@ -19,8 +18,6 @@ class ASRProvider(ASRProviderBase):
|
||||
self.interface_type = InterfaceType.STREAM
|
||||
self.config = config
|
||||
self.text = ""
|
||||
self.max_retries = 3
|
||||
self.retry_delay = 2
|
||||
self.decoder = opuslib_next.Decoder(16000, 1)
|
||||
self.asr_ws = None
|
||||
self.forward_task = None
|
||||
@@ -57,14 +54,13 @@ class ASRProvider(ASRProviderBase):
|
||||
async def receive_audio(self, conn, audio, audio_have_voice):
|
||||
conn.asr_audio.append(audio)
|
||||
conn.asr_audio = conn.asr_audio[-10:]
|
||||
|
||||
# 存储音频数据
|
||||
if not hasattr(conn, 'asr_audio_for_voiceprint'):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
conn.asr_audio_for_voiceprint.append(audio)
|
||||
|
||||
|
||||
# 当没有音频数据时处理完整语音片段
|
||||
if not audio and len(conn.asr_audio_for_voiceprint) > 0:
|
||||
if conn.client_listen_mode != "manual" and not audio and len(conn.asr_audio_for_voiceprint) > 0:
|
||||
await self.handle_voice_stop(conn, conn.asr_audio_for_voiceprint)
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
|
||||
@@ -180,6 +176,7 @@ class ASRProvider(ASRProviderBase):
|
||||
payload.get("audio_info", {}).get("duration", 0) > 2000
|
||||
and not utterances
|
||||
and not payload["result"].get("text")
|
||||
and conn.client_listen_mode != "manual"
|
||||
):
|
||||
logger.bind(tag=TAG).error(f"识别文本：空")
|
||||
self.text = ""
|
||||
@@ -188,15 +185,44 @@ class ASRProvider(ASRProviderBase):
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
break
|
||||
|
||||
# 专门处理没有文本的识别结果（手动模式下可能已经识别完成但是没松按键）
|
||||
elif not payload["result"].get("text") and not utterances:
|
||||
if conn.client_listen_mode == "manual" and conn.client_voice_stop and len(audio_data) > 0:
|
||||
logger.bind(tag=TAG).debug("消息结束收到停止信号，触发处理")
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
# 清理音频缓存
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
break
|
||||
|
||||
for utterance in utterances:
|
||||
if utterance.get("definite", False):
|
||||
self.text = utterance["text"]
|
||||
current_text = utterance["text"]
|
||||
logger.bind(tag=TAG).info(
|
||||
f"识别到文本: {self.text}"
|
||||
f"识别到文本: {current_text}"
|
||||
)
|
||||
conn.reset_vad_states()
|
||||
if len(audio_data) > 15: # 确保有足够音频数据
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
|
||||
# 手动模式下累积识别结果
|
||||
if conn.client_listen_mode == "manual":
|
||||
if self.text:
|
||||
self.text += current_text
|
||||
else:
|
||||
self.text = current_text
|
||||
|
||||
# 在接收消息中途时收到停止信号
|
||||
if conn.client_voice_stop and len(audio_data) > 0:
|
||||
logger.bind(tag=TAG).debug("消息中途收到停止信号，触发处理")
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
# 清理音频缓存
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
break
|
||||
else:
|
||||
# 自动模式下直接覆盖
|
||||
self.text = current_text
|
||||
conn.reset_vad_states()
|
||||
if len(audio_data) > 15: # 确保有足够音频数据
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
break
|
||||
elif "error" in payload:
|
||||
error_msg = payload.get("error", "未知错误")
|
||||
@@ -228,8 +254,6 @@ class ASRProvider(ASRProviderBase):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
if hasattr(conn, 'asr_audio'):
|
||||
conn.asr_audio = []
|
||||
if hasattr(conn, 'has_valid_voice'):
|
||||
conn.has_valid_voice = False
|
||||
|
||||
def stop_ws_connection(self):
|
||||
if self.asr_ws:
|
||||
@@ -237,6 +261,20 @@ class ASRProvider(ASRProviderBase):
|
||||
self.asr_ws = None
|
||||
self.is_processing = False
|
||||
|
||||
async def _send_stop_request(self):
|
||||
"""发送最后一个音频帧以通知服务器结束"""
|
||||
if self.asr_ws:
|
||||
try:
|
||||
# 发送结束标记的音频帧（gzip压缩的空数据）
|
||||
empty_payload = gzip.compress(b"")
|
||||
last_audio_request = bytearray(self.generate_last_audio_default_header())
|
||||
last_audio_request.extend(len(empty_payload).to_bytes(4, "big"))
|
||||
last_audio_request.extend(empty_payload)
|
||||
await self.asr_ws.send(last_audio_request)
|
||||
logger.bind(tag=TAG).debug("已发送结束音频帧")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).debug(f"发送结束音频帧时出错: {e}")
|
||||
|
||||
def construct_request(self, reqid):
|
||||
req = {
|
||||
"app": {
|
||||
@@ -388,5 +426,3 @@ class ASRProvider(ASRProviderBase):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
if hasattr(conn, 'asr_audio'):
|
||||
conn.asr_audio = []
|
||||
if hasattr(conn, 'has_valid_voice'):
|
||||
conn.has_valid_voice = False
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
import time
|
||||
import os
|
||||
import sys
|
||||
import io
|
||||
import sys
|
||||
import time
|
||||
import shutil
|
||||
import psutil
|
||||
import asyncio
|
||||
|
||||
from config.logger import setup_logging
|
||||
from typing import Optional, Tuple, List
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from funasr import AutoModel
|
||||
from funasr.utils.postprocess_utils import rich_transcription_postprocess
|
||||
import shutil
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
@@ -90,16 +92,17 @@ class ASRProvider(ASRProviderBase):
|
||||
else:
|
||||
file_path = self.save_audio_to_file(pcm_data, session_id)
|
||||
|
||||
# 语音识别
|
||||
# 语音识别 - 使用线程池避免阻塞事件循环
|
||||
start_time = time.time()
|
||||
result = self.model.generate(
|
||||
result = await asyncio.to_thread(
|
||||
self.model.generate,
|
||||
input=combined_pcm_data,
|
||||
cache={},
|
||||
language="auto",
|
||||
use_itn=True,
|
||||
batch_size_s=60,
|
||||
)
|
||||
text = rich_transcription_postprocess(result[0]["text"])
|
||||
text = await asyncio.to_thread(rich_transcription_postprocess, result[0]["text"])
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {text}"
|
||||
)
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
import os
|
||||
import json
|
||||
import asyncio
|
||||
import tempfile
|
||||
import difflib
|
||||
from typing import Optional, Tuple, List
|
||||
import dashscope
|
||||
from config.logger import setup_logging
|
||||
@@ -16,7 +13,8 @@ logger = setup_logging()
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config: dict, delete_audio_file: bool):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.STREAM
|
||||
# 音频文件上传类型，流式文本识别输出
|
||||
self.interface_type = InterfaceType.NON_STREAM
|
||||
"""Qwen3-ASR-Flash ASR初始化"""
|
||||
|
||||
# 配置参数
|
||||
@@ -130,27 +128,11 @@ class ASRProvider(ASRProviderBase):
|
||||
|
||||
# 处理流式响应
|
||||
full_text = ""
|
||||
last_text = "" # 用于存储上一个文本片段
|
||||
for chunk in response:
|
||||
try:
|
||||
text = chunk["output"]["choices"][0]["message"].content[0]["text"]
|
||||
# 标准化文本片段（去除首尾空格）
|
||||
normalized_text = text.strip()
|
||||
# 只有当新文本片段与上一个不同时才处理
|
||||
if normalized_text != last_text:
|
||||
# 提取新增的文本部分
|
||||
# 通过比较当前文本和上一个文本，找到新增的部分
|
||||
if normalized_text.startswith(last_text):
|
||||
# 如果当前文本以最后一个文本开头，则新增部分是两者的差集
|
||||
new_part = normalized_text[len(last_text):]
|
||||
else:
|
||||
# 如果不以最后一个文本开头，说明识别结果发生了较大变化，直接使用当前文本
|
||||
new_part = normalized_text
|
||||
|
||||
# 将新增部分添加到完整文本中
|
||||
full_text += new_part
|
||||
last_text = normalized_text
|
||||
# 这里可以实时处理文本片段，例如通过回调函数
|
||||
# 更新为最新的完整文本
|
||||
full_text = text.strip()
|
||||
except:
|
||||
pass
|
||||
|
||||
|
||||
@@ -35,9 +35,6 @@ class ASRProvider(ASRProviderBase):
|
||||
self.forward_task = None
|
||||
self.is_processing = False
|
||||
self.server_ready = False
|
||||
self.last_frame_sent = False # 标记是否已发送最终帧
|
||||
self.best_text = "" # 保存最佳识别结果
|
||||
self.has_final_result = False # 标记是否收到最终识别结果
|
||||
|
||||
# 讯飞配置
|
||||
self.app_id = config.get("app_id")
|
||||
@@ -52,7 +49,6 @@ class ASRProvider(ASRProviderBase):
|
||||
"domain": config.get("domain", "slm"),
|
||||
"language": config.get("language", "zh_cn"),
|
||||
"accent": config.get("accent", "mandarin"),
|
||||
"dwa": config.get("dwa", "wpgs"),
|
||||
"result": {"encoding": "utf8", "compress": "raw", "format": "plain"},
|
||||
}
|
||||
|
||||
@@ -116,7 +112,7 @@ class ASRProvider(ASRProviderBase):
|
||||
await self._start_recognition(conn)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"建立ASR连接失败: {str(e)}")
|
||||
await self._cleanup(conn)
|
||||
await self._cleanup()
|
||||
return
|
||||
|
||||
# 发送当前音频数据
|
||||
@@ -126,7 +122,7 @@ class ASRProvider(ASRProviderBase):
|
||||
await self._send_audio_frame(pcm_frame, STATUS_CONTINUE_FRAME)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).warning(f"发送音频数据时发生错误: {e}")
|
||||
await self._cleanup(conn)
|
||||
await self._cleanup()
|
||||
|
||||
async def _start_recognition(self, conn):
|
||||
"""开始识别会话"""
|
||||
@@ -136,6 +132,10 @@ class ASRProvider(ASRProviderBase):
|
||||
ws_url = self.create_url()
|
||||
logger.bind(tag=TAG).info(f"正在连接ASR服务: {ws_url[:50]}...")
|
||||
|
||||
# 如果为手动模式,设置超时时长为一分钟
|
||||
if conn.client_listen_mode == "manual":
|
||||
self.iat_params["eos"] = 60000
|
||||
|
||||
self.asr_ws = await websockets.connect(
|
||||
ws_url,
|
||||
max_size=1000000000,
|
||||
@@ -146,8 +146,6 @@ class ASRProvider(ASRProviderBase):
|
||||
|
||||
logger.bind(tag=TAG).info("ASR WebSocket连接已建立")
|
||||
self.server_ready = False
|
||||
self.last_frame_sent = False
|
||||
self.best_text = ""
|
||||
self.forward_task = asyncio.create_task(self._forward_results(conn))
|
||||
|
||||
# 发送首帧音频
|
||||
@@ -196,23 +194,12 @@ class ASRProvider(ASRProviderBase):
|
||||
|
||||
await self.asr_ws.send(json.dumps(frame_data, ensure_ascii=False))
|
||||
|
||||
# 标记是否发送了最终帧
|
||||
if status == STATUS_LAST_FRAME:
|
||||
self.last_frame_sent = True
|
||||
logger.bind(tag=TAG).info("标记最终帧已发送")
|
||||
|
||||
async def _forward_results(self, conn):
|
||||
"""转发识别结果"""
|
||||
try:
|
||||
while self.asr_ws and not conn.stop_event.is_set():
|
||||
# 获取当前连接的音频数据
|
||||
audio_data = getattr(conn, "asr_audio_for_voiceprint", [])
|
||||
while not conn.stop_event.is_set():
|
||||
try:
|
||||
# 如果已发送最终帧，增加超时时间等待完整结果
|
||||
timeout = 3.0 if self.last_frame_sent else 30.0
|
||||
response = await asyncio.wait_for(
|
||||
self.asr_ws.recv(), timeout=timeout
|
||||
)
|
||||
response = await asyncio.wait_for(self.asr_ws.recv(), timeout=60)
|
||||
result = json.loads(response)
|
||||
logger.bind(tag=TAG).debug(f"收到ASR结果: {result}")
|
||||
|
||||
@@ -236,144 +223,27 @@ class ASRProvider(ASRProviderBase):
|
||||
# 解码base64文本
|
||||
decoded_text = base64.b64decode(text_data).decode("utf-8")
|
||||
text_json = json.loads(decoded_text)
|
||||
|
||||
# 提取文本内容
|
||||
text_ws = text_json.get("ws", [])
|
||||
result_text = ""
|
||||
for i in text_ws:
|
||||
for j in i.get("cw", []):
|
||||
w = j.get("w", "")
|
||||
result_text += w
|
||||
self.text += w
|
||||
|
||||
# 更新识别文本 - 实时更新策略
|
||||
# 只检查是否为空字符串，不再过滤任何标点符号
|
||||
# 这样可以确保所有识别到的内容，包括标点符号都能被实时更新
|
||||
if result_text and result_text.strip():
|
||||
# 实时更新：正常情况下都更新，提高响应速度
|
||||
should_update = True
|
||||
|
||||
# 保存最佳文本
|
||||
# 1. 如果是识别完成状态或最终帧后收到的结果，优先保存
|
||||
# 2. 否则保存最长的有意义文本
|
||||
# 取消对标点符号的过滤，只检查是否为空
|
||||
# 这样可以保留所有识别到的内容，包括各种标点符号
|
||||
is_valid_text = len(result_text.strip()) > 0
|
||||
|
||||
if (
|
||||
self.last_frame_sent or status == 2
|
||||
) and is_valid_text:
|
||||
self.best_text = result_text
|
||||
self.has_final_result = True # 标记已收到最终结果
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"保存最终识别结果: {self.best_text}"
|
||||
)
|
||||
elif (
|
||||
len(result_text) > len(self.best_text)
|
||||
and is_valid_text
|
||||
and not self.has_final_result
|
||||
):
|
||||
self.best_text = result_text
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"保存中间最佳文本: {self.best_text}"
|
||||
)
|
||||
|
||||
# 如果已发送最终帧，只过滤空文本
|
||||
if self.last_frame_sent:
|
||||
# 只拒绝完全空的结果
|
||||
if not result_text.strip():
|
||||
should_update = False
|
||||
logger.bind(tag=TAG).warning(
|
||||
f"最终帧后拒绝空文本"
|
||||
)
|
||||
|
||||
if should_update:
|
||||
# 处理流式识别结果，避免简单替换导致内容丢失
|
||||
# 1. 如果是中间状态（非最终帧后），可能需要替换为更完整的识别
|
||||
# 2. 如果是最终帧后收到的结果，可能是对前面文本的补充
|
||||
if self.last_frame_sent:
|
||||
# 最终帧后收到的结果可能是标点符号等补充内容
|
||||
# 检查是否需要合并文本而不是替换
|
||||
# 如果当前文本是纯标点而前面已有内容，应该追加而不是替换
|
||||
if len(
|
||||
self.text
|
||||
) > 0 and result_text.strip() in [
|
||||
"。",
|
||||
".",
|
||||
"?",
|
||||
"？",
|
||||
"!",
|
||||
"！",
|
||||
",",
|
||||
"，",
|
||||
";",
|
||||
"；",
|
||||
]:
|
||||
# 对于标点符号，追加到现有文本后
|
||||
self.text = (
|
||||
self.text.rstrip().rstrip("。.")
|
||||
+ result_text
|
||||
)
|
||||
else:
|
||||
# 其他情况保持替换逻辑
|
||||
self.text = result_text
|
||||
else:
|
||||
# 中间状态替换为新的识别结果
|
||||
self.text = result_text
|
||||
|
||||
logger.bind(tag=TAG).info(
|
||||
f"实时更新识别文本: {self.text} (最终帧已发送: {self.last_frame_sent})"
|
||||
)
|
||||
|
||||
# 识别完成，但如果还没发送最终帧，继续等待
|
||||
if status == 2:
|
||||
logger.bind(tag=TAG).info(
|
||||
f"识别完成状态已到达，当前识别文本: {self.text}"
|
||||
)
|
||||
|
||||
# 如果还没发送最终帧，继续等待
|
||||
if not self.last_frame_sent:
|
||||
logger.bind(tag=TAG).info(
|
||||
"识别完成但最终帧未发送，继续等待..."
|
||||
)
|
||||
continue
|
||||
|
||||
# 已发送最终帧且收到完成状态，使用最佳策略选择最终结果
|
||||
# 优先使用识别完成状态下的最新结果，而不是仅仅基于长度
|
||||
if self.best_text:
|
||||
# 如果当前文本是在最终帧发送后或识别完成状态下收到的，优先使用
|
||||
if (
|
||||
self.last_frame_sent or status == 2
|
||||
) and self.text.strip():
|
||||
logger.bind(tag=TAG).info(
|
||||
f"使用完成状态下的最新识别结果: {self.text}"
|
||||
)
|
||||
elif len(self.best_text) > len(self.text):
|
||||
logger.bind(tag=TAG).info(
|
||||
f"使用更长的最佳文本作为最终结果: {self.text} -> {self.best_text}"
|
||||
)
|
||||
self.text = self.best_text
|
||||
|
||||
logger.bind(tag=TAG).info(f"获取到最终完整文本: {self.text}")
|
||||
if conn.client_listen_mode == "manual":
|
||||
audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
|
||||
if len(audio_data) > 0:
|
||||
logger.bind(tag=TAG).debug("收到最终识别结果，触发处理")
|
||||
await self.handle_voice_stop(conn, audio_data)
|
||||
# 清理音频缓存
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
if len(audio_data) > 15: # 确保有足够音频数据
|
||||
# 准备处理结果
|
||||
pass
|
||||
break
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
if self.last_frame_sent:
|
||||
# 超时时也使用最佳文本
|
||||
if self.best_text and len(self.best_text) > len(self.text):
|
||||
logger.bind(tag=TAG).info(
|
||||
f"超时，使用最佳文本: {self.text} -> {self.best_text}"
|
||||
)
|
||||
self.text = self.best_text
|
||||
logger.bind(tag=TAG).info(
|
||||
f"最终帧后超时，使用结果: {self.text}"
|
||||
)
|
||||
break
|
||||
# 如果还没发送最终帧，继续等待
|
||||
continue
|
||||
logger.bind(tag=TAG).error("接收结果超时")
|
||||
break
|
||||
except websockets.ConnectionClosed:
|
||||
logger.bind(tag=TAG).info("ASR服务连接已关闭")
|
||||
self.is_processing = False
|
||||
@@ -390,17 +260,15 @@ class ASRProvider(ASRProviderBase):
|
||||
if hasattr(e, "__cause__") and e.__cause__:
|
||||
logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
|
||||
finally:
|
||||
if self.asr_ws:
|
||||
await self.asr_ws.close()
|
||||
self.asr_ws = None
|
||||
self.is_processing = False
|
||||
# 清理连接资源
|
||||
await self._cleanup()
|
||||
|
||||
# 清理连接的音频缓存
|
||||
if conn:
|
||||
if hasattr(conn, "asr_audio_for_voiceprint"):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
if hasattr(conn, "asr_audio"):
|
||||
conn.asr_audio = []
|
||||
if hasattr(conn, "has_valid_voice"):
|
||||
conn.has_valid_voice = False
|
||||
|
||||
async def handle_voice_stop(self, conn, asr_audio_task: List[bytes]):
|
||||
"""处理语音停止，发送最后一帧并处理识别结果"""
|
||||
@@ -408,22 +276,13 @@ class ASRProvider(ASRProviderBase):
|
||||
# 先发送最后一帧表示音频结束
|
||||
if self.asr_ws and self.is_processing:
|
||||
try:
|
||||
# 取最后一个有效的音频帧作为最后一帧数据
|
||||
last_frame = b""
|
||||
if asr_audio_task:
|
||||
last_audio = asr_audio_task[-1]
|
||||
last_frame = self.decoder.decode(last_audio, 960)
|
||||
await self._send_audio_frame(last_frame, STATUS_LAST_FRAME)
|
||||
logger.bind(tag=TAG).info("已发送最后一帧")
|
||||
await self._send_audio_frame(b"", STATUS_LAST_FRAME)
|
||||
logger.bind(tag=TAG).debug(f"已发送停止请求")
|
||||
|
||||
# 发送最终帧后，给_forward_results适当时间处理最终结果
|
||||
await asyncio.sleep(0.25)
|
||||
|
||||
logger.bind(tag=TAG).info(f"准备处理最终识别结果: {self.text}")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"发送最后一帧失败: {e}")
|
||||
logger.bind(tag=TAG).error(f"发送停止请求失败: {e}")
|
||||
|
||||
# 调用父类的handle_voice_stop方法处理识别结果
|
||||
await super().handle_voice_stop(conn, asr_audio_task)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"处理语音停止失败: {e}")
|
||||
@@ -437,40 +296,27 @@ class ASRProvider(ASRProviderBase):
|
||||
self.asr_ws = None
|
||||
self.is_processing = False
|
||||
|
||||
async def _cleanup(self, conn):
|
||||
"""清理资源"""
|
||||
logger.bind(tag=TAG).info(
|
||||
async def _send_stop_request(self):
|
||||
"""发送停止识别请求（不关闭连接）"""
|
||||
if self.asr_ws:
|
||||
try:
|
||||
# 先停止音频发送
|
||||
self.is_processing = False
|
||||
await self._send_audio_frame(b"", STATUS_LAST_FRAME)
|
||||
logger.bind(tag=TAG).debug("已发送停止请求")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"发送停止请求失败: {e}")
|
||||
|
||||
async def _cleanup(self):
|
||||
"""清理资源（关闭连接）"""
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}"
|
||||
)
|
||||
|
||||
# 发送最后一帧
|
||||
if self.asr_ws and self.is_processing:
|
||||
try:
|
||||
await self._send_audio_frame(b"", STATUS_LAST_FRAME)
|
||||
await asyncio.sleep(0.1)
|
||||
logger.bind(tag=TAG).info("已发送最后一帧")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"发送最后一帧失败: {e}")
|
||||
|
||||
# 状态重置
|
||||
self.is_processing = False
|
||||
self.server_ready = False
|
||||
self.last_frame_sent = False
|
||||
self.best_text = ""
|
||||
self.has_final_result = False
|
||||
logger.bind(tag=TAG).info("ASR状态已重置")
|
||||
|
||||
# 清理任务
|
||||
if self.forward_task and not self.forward_task.done():
|
||||
self.forward_task.cancel()
|
||||
try:
|
||||
await asyncio.wait_for(self.forward_task, timeout=1.0)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).debug(f"forward_task取消异常: {e}")
|
||||
finally:
|
||||
self.forward_task = None
|
||||
logger.bind(tag=TAG).debug("ASR状态已重置")
|
||||
|
||||
# 关闭连接
|
||||
if self.asr_ws:
|
||||
@@ -483,16 +329,10 @@ class ASRProvider(ASRProviderBase):
|
||||
finally:
|
||||
self.asr_ws = None
|
||||
|
||||
# 清理连接的音频缓存
|
||||
if conn:
|
||||
if hasattr(conn, "asr_audio_for_voiceprint"):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
if hasattr(conn, "asr_audio"):
|
||||
conn.asr_audio = []
|
||||
if hasattr(conn, "has_valid_voice"):
|
||||
conn.has_valid_voice = False
|
||||
# 清理任务引用
|
||||
self.forward_task = None
|
||||
|
||||
logger.bind(tag=TAG).info("ASR会话清理完成")
|
||||
logger.bind(tag=TAG).debug("ASR会话清理完成")
|
||||
|
||||
async def speech_to_text(self, opus_data, session_id, audio_format):
|
||||
"""获取识别结果"""
|
||||
@@ -513,7 +353,7 @@ class ASRProvider(ASRProviderBase):
|
||||
pass
|
||||
self.forward_task = None
|
||||
self.is_processing = False
|
||||
|
||||
|
||||
# 显式释放decoder资源
|
||||
if hasattr(self, 'decoder') and self.decoder is not None:
|
||||
try:
|
||||
@@ -530,5 +370,3 @@ class ASRProvider(ASRProviderBase):
|
||||
conn.asr_audio_for_voiceprint = []
|
||||
if hasattr(conn, "asr_audio"):
|
||||
conn.asr_audio = []
|
||||
if hasattr(conn, "has_valid_voice"):
|
||||
conn.has_valid_voice = False
|
||||
|
||||
@@ -2,7 +2,6 @@ import time
|
||||
import numpy as np
|
||||
import torch
|
||||
import opuslib_next
|
||||
import gc
|
||||
from config.logger import setup_logging
|
||||
from core.providers.vad.base import VADProviderBase
|
||||
|
||||
@@ -45,6 +44,10 @@ class VADProvider(VADProviderBase):
|
||||
pass
|
||||
|
||||
def is_vad(self, conn, opus_packet):
|
||||
# 手动模式：直接返回True，不进行实时VAD检测，所有音频都缓存
|
||||
if conn.client_listen_mode == "manual":
|
||||
return True
|
||||
|
||||
try:
|
||||
pcm_frame = self.decoder.decode(opus_packet, 960)
|
||||
conn.client_audio_buffer.extend(pcm_frame) # 将新数据加入缓冲区
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import time
|
||||
import asyncio
|
||||
from collections import deque
|
||||
from config.logger import setup_logging
|
||||
|
||||
TAG = __name__
|
||||
@@ -10,12 +11,6 @@ class AudioRateController:
|
||||
"""
|
||||
音频速率控制器 - 按照60ms帧时长精确控制音频发送
|
||||
解决高并发下的时间累积误差问题
|
||||
|
||||
关键改进：
|
||||
1. 单一时间基准（start_timestamp 只初始化一次）
|
||||
2. 每次检查队列时重新计算 elapsed_ms，避免累积误差
|
||||
3. 分离"检查时间"和"发送"两个操作
|
||||
4. 支持高并发而不产生延迟
|
||||
"""
|
||||
|
||||
def __init__(self, frame_duration=60):
|
||||
@@ -24,35 +19,52 @@ class AudioRateController:
|
||||
frame_duration: 单个音频帧时长（毫秒），默认60ms
|
||||
"""
|
||||
self.frame_duration = frame_duration
|
||||
self.queue = []
|
||||
self.queue = deque()
|
||||
self.play_position = 0 # 虚拟播放位置（毫秒）
|
||||
self.start_timestamp = None # 开始时间戳（只读，不修改）
|
||||
self.pending_send_task = None
|
||||
self.logger = logger
|
||||
self.queue_empty_event = asyncio.Event() # 队列清空事件
|
||||
self.queue_empty_event.set() # 初始为空状态
|
||||
self.queue_has_data_event = asyncio.Event() # 队列数据事件
|
||||
|
||||
def reset(self):
|
||||
"""重置控制器状态"""
|
||||
if self.pending_send_task and not self.pending_send_task.done():
|
||||
self.pending_send_task.cancel()
|
||||
try:
|
||||
# 等待任务取消完成
|
||||
asyncio.get_event_loop().run_until_complete(self.pending_send_task)
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
# 取消任务后，任务会在下次事件循环时清理，无需阻塞等待
|
||||
|
||||
self.queue.clear()
|
||||
self.play_position = 0
|
||||
self.start_timestamp = time.time()
|
||||
self.start_timestamp = None # 由首个音频包设置
|
||||
# 相关事件处理
|
||||
self.queue_empty_event.set()
|
||||
self.queue_has_data_event.clear()
|
||||
|
||||
def add_audio(self, opus_packet):
|
||||
"""添加音频包到队列"""
|
||||
self.queue.append(("audio", opus_packet))
|
||||
# 相关事件处理
|
||||
self.queue_empty_event.clear()
|
||||
self.queue_has_data_event.set()
|
||||
|
||||
def add_message(self, message_callback):
|
||||
"""
|
||||
添加消息到队列（立即发送，不占用播放时间）
|
||||
|
||||
Args:
|
||||
message_callback: 消息发送回调函数 async def()
|
||||
"""
|
||||
self.queue.append(("message", message_callback))
|
||||
# 相关事件处理
|
||||
self.queue_empty_event.clear()
|
||||
self.queue_has_data_event.set()
|
||||
|
||||
def _get_elapsed_ms(self):
|
||||
"""获取已经过的时间（毫秒）"""
|
||||
if self.start_timestamp is None:
|
||||
return 0
|
||||
return (time.time() - self.start_timestamp) * 1000
|
||||
return (time.monotonic() - self.start_timestamp) * 1000
|
||||
|
||||
async def check_queue(self, send_audio_callback):
|
||||
"""
|
||||
@@ -61,64 +73,80 @@ class AudioRateController:
|
||||
Args:
|
||||
send_audio_callback: 发送音频的回调函数 async def(opus_packet)
|
||||
"""
|
||||
if self.start_timestamp is None:
|
||||
self.reset()
|
||||
|
||||
while self.queue:
|
||||
item = self.queue[0]
|
||||
item_type = item[0]
|
||||
|
||||
if item_type == "audio":
|
||||
if item_type == "message":
|
||||
# 消息类型：立即发送，不占用播放时间
|
||||
_, message_callback = item
|
||||
self.queue.popleft()
|
||||
try:
|
||||
await message_callback()
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"发送消息失败: {e}")
|
||||
raise
|
||||
|
||||
elif item_type == "audio":
|
||||
if self.start_timestamp is None:
|
||||
self.start_timestamp = time.monotonic()
|
||||
|
||||
_, opus_packet = item
|
||||
|
||||
# 计算时间差
|
||||
elapsed_ms = self._get_elapsed_ms()
|
||||
output_ms = self.play_position
|
||||
# 循环等待直到时间到达
|
||||
while True:
|
||||
# 计算时间差
|
||||
elapsed_ms = self._get_elapsed_ms()
|
||||
output_ms = self.play_position
|
||||
|
||||
if elapsed_ms < output_ms:
|
||||
# 还不到发送时间，计算等待时长
|
||||
wait_ms = output_ms - elapsed_ms
|
||||
if elapsed_ms < output_ms:
|
||||
# 还不到发送时间，计算等待时长
|
||||
wait_ms = output_ms - elapsed_ms
|
||||
|
||||
# 等待后继续检查（允许被中断）
|
||||
try:
|
||||
await asyncio.sleep(wait_ms / 1000)
|
||||
except asyncio.CancelledError:
|
||||
self.logger.bind(tag=TAG).debug("音频发送任务被取消")
|
||||
raise
|
||||
# 等待后继续检查（允许被中断）
|
||||
try:
|
||||
await asyncio.sleep(wait_ms / 1000)
|
||||
except asyncio.CancelledError:
|
||||
self.logger.bind(tag=TAG).debug("音频发送任务被取消")
|
||||
raise
|
||||
# 等待结束后重新检查时间（循环回到 while True）
|
||||
else:
|
||||
# 时间已到，跳出等待循环
|
||||
break
|
||||
|
||||
# 继续循环检查（时间可能已到）
|
||||
continue
|
||||
|
||||
# 时间已到，发送音频
|
||||
self.queue.pop(0)
|
||||
# 时间已到，从队列移除并发送
|
||||
self.queue.popleft()
|
||||
self.play_position += self.frame_duration
|
||||
|
||||
try:
|
||||
await send_audio_callback(opus_packet)
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"发送音频失败: {e}")
|
||||
raise
|
||||
|
||||
# 队列处理完后清除事件
|
||||
self.queue_empty_event.set()
|
||||
self.queue_has_data_event.clear()
|
||||
|
||||
async def start_sending(self, send_audio_callback, send_message_callback=None):
|
||||
def start_sending(self, send_audio_callback):
|
||||
"""
|
||||
启动异步发送任务
|
||||
|
||||
Args:
|
||||
send_audio_callback: 发送音频的回调函数
|
||||
send_message_callback: 发送消息的回调函数
|
||||
|
||||
Returns:
|
||||
asyncio.Task: 发送任务
|
||||
"""
|
||||
|
||||
async def _send_loop():
|
||||
try:
|
||||
while True:
|
||||
await self.check_queue(send_audio_callback, send_message_callback)
|
||||
# 如果队列空了，短暂等待后再检查（避免 busy loop）
|
||||
await asyncio.sleep(0.01)
|
||||
# 等待队列数据事件，不轮询等待占用CPU
|
||||
await self.queue_has_data_event.wait()
|
||||
|
||||
await self.check_queue(send_audio_callback)
|
||||
except asyncio.CancelledError:
|
||||
self.logger.bind(tag=TAG).info("音频发送循环已停止")
|
||||
self.logger.bind(tag=TAG).debug("音频发送循环已停止")
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"音频发送循环异常: {e}")
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ class CacheType(Enum):
|
||||
CONFIG = "config"
|
||||
DEVICE_PROMPT = "device_prompt"
|
||||
VOICEPRINT_HEALTH = "voiceprint_health" # 声纹识别健康检查
|
||||
AUDIO_DATA = "audio_data" # 音频数据缓存
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -58,5 +59,8 @@ class CacheConfig:
|
||||
CacheType.VOICEPRINT_HEALTH: cls(
|
||||
strategy=CacheStrategy.TTL, ttl=600, max_size=100 # 10分钟过期
|
||||
),
|
||||
CacheType.AUDIO_DATA: cls(
|
||||
strategy=CacheStrategy.TTL, ttl=600, max_size=100 # 10分钟过期
|
||||
),
|
||||
}
|
||||
return configs.get(cache_type, cls())
|
||||
|
||||
@@ -6,7 +6,6 @@ Opus编码工具类
|
||||
import logging
|
||||
import traceback
|
||||
import numpy as np
|
||||
import gc
|
||||
from opuslib_next import Encoder
|
||||
from opuslib_next import constants
|
||||
from typing import Optional, Callable, Any
|
||||
@@ -103,6 +102,9 @@ class OpusEncoderUtils:
|
||||
def _encode(self, frame: np.ndarray) -> Optional[bytes]:
|
||||
"""编码一帧音频数据"""
|
||||
try:
|
||||
# 编码器已释放，跳过编码
|
||||
if not hasattr(self, 'encoder') or self.encoder is None:
|
||||
return None
|
||||
# 将numpy数组转换为bytes
|
||||
frame_bytes = frame.tobytes()
|
||||
# opuslib要求输入字节数必须是channels*2的倍数
|
||||
|
||||
@@ -184,10 +184,25 @@ class PromptManager:
|
||||
def update_context_info(self, conn, client_ip: str):
|
||||
"""同步更新上下文信息"""
|
||||
try:
|
||||
# 获取位置信息（使用全局缓存）
|
||||
local_address = self._get_location_info(client_ip)
|
||||
# 获取天气信息（使用全局缓存）
|
||||
self._get_weather_info(conn, local_address)
|
||||
local_address = ""
|
||||
if (
|
||||
client_ip
|
||||
and self.base_prompt_template
|
||||
and (
|
||||
"local_address" in self.base_prompt_template
|
||||
or "weather_info" in self.base_prompt_template
|
||||
)
|
||||
):
|
||||
# 获取位置信息（使用全局缓存）
|
||||
local_address = self._get_location_info(client_ip)
|
||||
|
||||
if (
|
||||
self.base_prompt_template
|
||||
and "weather_info" in self.base_prompt_template
|
||||
and local_address
|
||||
):
|
||||
# 获取天气信息（使用全局缓存）
|
||||
self._get_weather_info(conn, local_address)
|
||||
|
||||
# 获取配置的上下文数据
|
||||
if hasattr(conn, "device_id") and conn.device_id:
|
||||
|
||||
@@ -4,11 +4,11 @@ import json
|
||||
import copy
|
||||
import wave
|
||||
import socket
|
||||
import asyncio
|
||||
import requests
|
||||
import subprocess
|
||||
import numpy as np
|
||||
import opuslib_next
|
||||
import gc
|
||||
from io import BytesIO
|
||||
from core.utils import p3
|
||||
from pydub import AudioSegment
|
||||
@@ -269,56 +269,82 @@ def audio_to_data_stream(
|
||||
pcm_to_data_stream(raw_data, is_opus, callback)
|
||||
|
||||
|
||||
def audio_to_data(audio_file_path: str, is_opus: bool = True) -> list[bytes]:
|
||||
async def audio_to_data(
|
||||
audio_file_path: str, is_opus: bool = True, use_cache: bool = True
|
||||
) -> list[bytes]:
|
||||
"""
|
||||
将音频文件转换为Opus/PCM编码的帧列表
|
||||
Args:
|
||||
audio_file_path: 音频文件路径
|
||||
is_opus: 是否进行Opus编码
|
||||
use_cache: 是否使用缓存
|
||||
"""
|
||||
# 获取文件后缀名
|
||||
file_type = os.path.splitext(audio_file_path)[1]
|
||||
if file_type:
|
||||
file_type = file_type.lstrip(".")
|
||||
# 读取音频文件，-nostdin 参数：不要从标准输入读取数据，否则FFmpeg会阻塞
|
||||
audio = AudioSegment.from_file(
|
||||
audio_file_path, format=file_type, parameters=["-nostdin"]
|
||||
)
|
||||
from core.utils.cache.manager import cache_manager
|
||||
from core.utils.cache.config import CacheType
|
||||
|
||||
# 转换为单声道/16kHz采样率/16位小端编码（确保与编码器匹配）
|
||||
audio = audio.set_channels(1).set_frame_rate(16000).set_sample_width(2)
|
||||
# 生成缓存键，包含文件路径和编码类型
|
||||
cache_key = f"{audio_file_path}:{is_opus}"
|
||||
|
||||
# 获取原始PCM数据（16位小端）
|
||||
raw_data = audio.raw_data
|
||||
# 尝试从缓存获取结果
|
||||
if use_cache:
|
||||
cached_result = cache_manager.get(CacheType.AUDIO_DATA, cache_key)
|
||||
if cached_result is not None:
|
||||
return cached_result
|
||||
|
||||
# 初始化Opus编码器
|
||||
encoder = opuslib_next.Encoder(16000, 1, opuslib_next.APPLICATION_AUDIO)
|
||||
def _sync_audio_to_data():
|
||||
# 获取文件后缀名
|
||||
file_type = os.path.splitext(audio_file_path)[1]
|
||||
if file_type:
|
||||
file_type = file_type.lstrip(".")
|
||||
# 读取音频文件，-nostdin 参数：不要从标准输入读取数据，否则FFmpeg会阻塞
|
||||
audio = AudioSegment.from_file(
|
||||
audio_file_path, format=file_type, parameters=["-nostdin"]
|
||||
)
|
||||
|
||||
# 编码参数
|
||||
frame_duration = 60 # 60ms per frame
|
||||
frame_size = int(16000 * frame_duration / 1000) # 960 samples/frame
|
||||
# 转换为单声道/16kHz采样率/16位小端编码（确保与编码器匹配）
|
||||
audio = audio.set_channels(1).set_frame_rate(16000).set_sample_width(2)
|
||||
|
||||
datas = []
|
||||
# 按帧处理所有音频数据（包括最后一帧可能补零）
|
||||
for i in range(0, len(raw_data), frame_size * 2): # 16bit=2bytes/sample
|
||||
# 获取当前帧的二进制数据
|
||||
chunk = raw_data[i : i + frame_size * 2]
|
||||
# 获取原始PCM数据（16位小端）
|
||||
raw_data = audio.raw_data
|
||||
|
||||
# 如果最后一帧不足，补零
|
||||
if len(chunk) < frame_size * 2:
|
||||
chunk += b"\x00" * (frame_size * 2 - len(chunk))
|
||||
# 初始化Opus编码器
|
||||
encoder = opuslib_next.Encoder(16000, 1, opuslib_next.APPLICATION_AUDIO)
|
||||
|
||||
if is_opus:
|
||||
# 转换为numpy数组处理
|
||||
np_frame = np.frombuffer(chunk, dtype=np.int16)
|
||||
# 编码Opus数据
|
||||
frame_data = encoder.encode(np_frame.tobytes(), frame_size)
|
||||
else:
|
||||
frame_data = chunk if isinstance(chunk, bytes) else bytes(chunk)
|
||||
# 编码参数
|
||||
frame_duration = 60 # 60ms per frame
|
||||
frame_size = int(16000 * frame_duration / 1000) # 960 samples/frame
|
||||
|
||||
datas.append(frame_data)
|
||||
datas = []
|
||||
# 按帧处理所有音频数据（包括最后一帧可能补零）
|
||||
for i in range(0, len(raw_data), frame_size * 2): # 16bit=2bytes/sample
|
||||
# 获取当前帧的二进制数据
|
||||
chunk = raw_data[i : i + frame_size * 2]
|
||||
|
||||
return datas
|
||||
# 如果最后一帧不足，补零
|
||||
if len(chunk) < frame_size * 2:
|
||||
chunk += b"\x00" * (frame_size * 2 - len(chunk))
|
||||
|
||||
if is_opus:
|
||||
# 转换为numpy数组处理
|
||||
np_frame = np.frombuffer(chunk, dtype=np.int16)
|
||||
# 编码Opus数据
|
||||
frame_data = encoder.encode(np_frame.tobytes(), frame_size)
|
||||
else:
|
||||
frame_data = chunk if isinstance(chunk, bytes) else bytes(chunk)
|
||||
|
||||
datas.append(frame_data)
|
||||
|
||||
return datas
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
# 在单独的线程中执行同步的音频处理操作
|
||||
result = await loop.run_in_executor(None, _sync_audio_to_data)
|
||||
|
||||
# 将结果存入缓存，使用配置中定义的TTL（10分钟）
|
||||
if use_cache:
|
||||
cache_manager.set(CacheType.AUDIO_DATA, cache_key, result)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def audio_bytes_to_data_stream(
|
||||
|
||||
@@ -195,7 +195,7 @@ def get_news_from_chinanews(
|
||||
|
||||
# 否则，获取新闻列表并随机选择一条
|
||||
# 从配置中获取RSS URL
|
||||
rss_config = conn.config["plugins"]["get_news_from_chinanews"]
|
||||
rss_config = conn.config.get("plugins", {}).get("get_news_from_chinanews", {})
|
||||
default_rss_url = rss_config.get(
|
||||
"default_rss_url", "https://www.chinanews.com.cn/rss/society.xml"
|
||||
)
|
||||
|
||||
@@ -120,10 +120,10 @@ def fetch_news_from_api(conn, source="thepaper"):
|
||||
"""从API获取新闻列表"""
|
||||
try:
|
||||
api_url = f"https://newsnow.busiyi.world/api/s?id={source}"
|
||||
if conn.config["plugins"].get("get_news_from_newsnow") and conn.config[
|
||||
"plugins"
|
||||
]["get_news_from_newsnow"].get("url"):
|
||||
api_url = conn.config["plugins"]["get_news_from_newsnow"]["url"] + source
|
||||
|
||||
news_config = conn.config.get("plugins", {}).get("get_news_from_newsnow", {})
|
||||
if news_config.get("url"):
|
||||
api_url = news_config["url"] + source
|
||||
|
||||
headers = {"User-Agent": "Mozilla/5.0"}
|
||||
response = requests.get(api_url, headers=headers, timeout=10)
|
||||
|
||||
@@ -158,13 +158,10 @@ def parse_weather_info(soup):
|
||||
def get_weather(conn, location: str = None, lang: str = "zh_CN"):
|
||||
from core.utils.cache.manager import cache_manager, CacheType
|
||||
|
||||
api_host = conn.config["plugins"]["get_weather"].get(
|
||||
"api_host", "mj7p3y7naa.re.qweatherapi.com"
|
||||
)
|
||||
api_key = conn.config["plugins"]["get_weather"].get(
|
||||
"api_key", "a861d0d5e7bf4ee1a83d9a9e4f96d4da"
|
||||
)
|
||||
default_location = conn.config["plugins"]["get_weather"]["default_location"]
|
||||
weather_config = conn.config.get("plugins", {}).get("get_weather", {})
|
||||
api_host = weather_config.get("api_host", "mj7p3y7naa.re.qweatherapi.com")
|
||||
api_key = weather_config.get("api_key", "a861d0d5e7bf4ee1a83d9a9e4f96d4da")
|
||||
default_location = weather_config.get("default_location", "广州")
|
||||
client_ip = conn.client_ip
|
||||
|
||||
# 优先使用用户提供的location参数
|
||||
|
||||
@@ -11,15 +11,17 @@ def append_devices_to_prompt(conn):
|
||||
"functions", []
|
||||
)
|
||||
|
||||
# 安全地获取插件配置
|
||||
plugins_config = conn.config.get("plugins", {})
|
||||
config_source = (
|
||||
"home_assistant"
|
||||
if conn.config["plugins"].get("home_assistant")
|
||||
if plugins_config.get("home_assistant")
|
||||
else "hass_get_state"
|
||||
)
|
||||
|
||||
if "hass_get_state" in funcs or "hass_set_state" in funcs:
|
||||
prompt = "\n下面是我家智能设备列表（位置，设备名，entity_id），可以通过homeassistant控制\n"
|
||||
deviceStr = conn.config["plugins"].get(config_source, {}).get("devices", "")
|
||||
deviceStr = plugins_config.get(config_source, {}).get("devices", "")
|
||||
conn.prompt += prompt + deviceStr + "\n"
|
||||
# 更新提示词
|
||||
conn.dialogue.update_system_message(conn.prompt)
|
||||
@@ -30,17 +32,17 @@ def initialize_hass_handler(conn):
|
||||
if not conn.load_function_plugin:
|
||||
return ha_config
|
||||
|
||||
# 安全地获取插件配置
|
||||
plugins_config = conn.config.get("plugins", {})
|
||||
# 确定配置来源
|
||||
config_source = (
|
||||
"home_assistant"
|
||||
if conn.config["plugins"].get("home_assistant")
|
||||
else "hass_get_state"
|
||||
"home_assistant" if plugins_config.get("home_assistant") else "hass_get_state"
|
||||
)
|
||||
if not conn.config["plugins"].get(config_source):
|
||||
if not plugins_config.get(config_source):
|
||||
return ha_config
|
||||
|
||||
# 统一获取配置
|
||||
plugin_config = conn.config["plugins"][config_source]
|
||||
plugin_config = plugins_config[config_source]
|
||||
ha_config["base_url"] = plugin_config.get("base_url")
|
||||
ha_config["api_key"] = plugin_config.get("api_key")
|
||||
|
||||
|
||||
@@ -118,8 +118,9 @@ def get_music_files(music_dir, music_ext):
|
||||
def initialize_music_handler(conn):
|
||||
global MUSIC_CACHE
|
||||
if MUSIC_CACHE == {}:
|
||||
if "play_music" in conn.config["plugins"]:
|
||||
MUSIC_CACHE["music_config"] = conn.config["plugins"]["play_music"]
|
||||
plugins_config = conn.config.get("plugins", {})
|
||||
if "play_music" in plugins_config:
|
||||
MUSIC_CACHE["music_config"] = plugins_config["play_music"]
|
||||
MUSIC_CACHE["music_dir"] = os.path.abspath(
|
||||
MUSIC_CACHE["music_config"].get("music_dir", "./music") # 默认路径修改
|
||||
)
|
||||
|
||||
@@ -32,9 +32,10 @@ def search_from_ragflow(conn, question=None):
|
||||
else:
|
||||
question = str(question) if question is not None else ""
|
||||
|
||||
base_url = conn.config["plugins"]["search_from_ragflow"].get("base_url", "")
|
||||
api_key = conn.config["plugins"]["search_from_ragflow"].get("api_key", "")
|
||||
dataset_ids = conn.config["plugins"]["search_from_ragflow"].get("dataset_ids", [])
|
||||
ragflow_config = conn.config.get("plugins", {}).get("search_from_ragflow", {})
|
||||
base_url = ragflow_config.get("base_url", "")
|
||||
api_key = ragflow_config.get("api_key", "")
|
||||
dataset_ids = ragflow_config.get("dataset_ids", [])
|
||||
|
||||
url = base_url + "/api/v1/retrieval"
|
||||
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
||||
|
||||
@@ -13,7 +13,7 @@ pydub==0.25.1
|
||||
funasr==1.2.7
|
||||
openai==2.8.1
|
||||
google-generativeai==0.8.5
|
||||
edge_tts==7.2.3
|
||||
edge_tts==7.2.6
|
||||
httpx==0.28.1
|
||||
aiohttp==3.13.2
|
||||
aiohttp_cors==0.8.1
|
||||
|
||||
@@ -267,6 +267,7 @@ span.connection-status.llm-emoji {
|
||||
.llm-emoji .status {
|
||||
font-size: 14px !important;
|
||||
padding: 8px 20px !important;
|
||||
line-height: 1.2 !important;
|
||||
}
|
||||
|
||||
.emoji-large {
|
||||
@@ -393,7 +394,7 @@ span.connection-status.llm-emoji {
|
||||
/* ==================== 会话记录和日志 ==================== */
|
||||
.flex-container {
|
||||
display: flex;
|
||||
margin-top: 10px;
|
||||
margin-top: 20px;
|
||||
background-color: #f9fafb;
|
||||
}
|
||||
|
||||
|
||||
@@ -249,6 +249,41 @@ export class AudioPlayer {
|
||||
this.playBufferedAudio();
|
||||
this.startAudioBuffering();
|
||||
}
|
||||
|
||||
// 获取音频包统计信息
|
||||
getAudioStats() {
|
||||
if (!this.streamingContext) {
|
||||
return {
|
||||
pendingDecode: 0,
|
||||
pendingPlay: 0,
|
||||
totalPending: 0
|
||||
};
|
||||
}
|
||||
|
||||
const pendingDecode = this.streamingContext.getPendingDecodeCount();
|
||||
const pendingPlay = this.streamingContext.getPendingPlayCount();
|
||||
|
||||
return {
|
||||
pendingDecode, // 待解码包数
|
||||
pendingPlay, // 待播放包数
|
||||
totalPending: pendingDecode + pendingPlay // 总待处理包数
|
||||
};
|
||||
}
|
||||
|
||||
// 清空所有音频缓冲并停止播放
|
||||
clearAllAudio() {
|
||||
log('AudioPlayer: 清空所有音频', 'info');
|
||||
|
||||
// 清空接收队列（使用clear方法保持对象引用）
|
||||
this.queue.clear();
|
||||
|
||||
// 清空流上下文的所有缓冲
|
||||
if (this.streamingContext) {
|
||||
this.streamingContext.clearAllBuffers();
|
||||
}
|
||||
|
||||
log('AudioPlayer: 音频已清空', 'success');
|
||||
}
|
||||
}
|
||||
|
||||
// 创建单例
|
||||
|
||||
@@ -240,6 +240,24 @@ export class AudioRecorder {
|
||||
if (this.isRecording) return false;
|
||||
|
||||
try {
|
||||
// 检查是否有WebSocketHandler实例
|
||||
const { getWebSocketHandler } = await import('../network/websocket.js');
|
||||
const wsHandler = getWebSocketHandler();
|
||||
|
||||
// 如果机器正在说话，发送打断消息
|
||||
if (wsHandler && wsHandler.isRemoteSpeaking && wsHandler.currentSessionId) {
|
||||
const abortMessage = {
|
||||
session_id: wsHandler.currentSessionId,
|
||||
type: 'abort',
|
||||
reason: 'wake_word_detected'
|
||||
};
|
||||
|
||||
if (this.websocket && this.websocket.readyState === WebSocket.OPEN) {
|
||||
this.websocket.send(JSON.stringify(abortMessage));
|
||||
log('发送打断消息', 'info');
|
||||
}
|
||||
}
|
||||
|
||||
if (!this.initEncoder()) {
|
||||
log('无法启动录音: Opus编码器初始化失败', 'error');
|
||||
return false;
|
||||
|
||||
@@ -22,6 +22,7 @@ export class StreamingContext {
|
||||
this.source = null; // 当前音频源
|
||||
this.totalSamples = 0; // 累积的总样本数
|
||||
this.lastPlayTime = 0; // 上次播放的时间戳
|
||||
this.scheduledEndTime = 0; // 已调度音频的结束时间
|
||||
}
|
||||
|
||||
// 缓存音频数组
|
||||
@@ -31,17 +32,19 @@ export class StreamingContext {
|
||||
|
||||
// 获取需要处理缓存队列，单线程：在audioBufferQueue一直更新的状态下不会出现安全问题
|
||||
async getPendingAudioBufferQueue() {
|
||||
// 原子交换 + 清空
|
||||
[this.pendingAudioBufferQueue, this.audioBufferQueue] = [await this.audioBufferQueue.dequeue(), new BlockingQueue()];
|
||||
// 等待数据到达并获取
|
||||
const data = await this.audioBufferQueue.dequeue();
|
||||
// 赋值给待处理队列
|
||||
this.pendingAudioBufferQueue = data;
|
||||
}
|
||||
|
||||
// 获取正在播放已解码的PCM队列，单线程：在activeQueue一直更新的状态下不会出现安全问题
|
||||
async getQueue(minSamples) {
|
||||
let TepArray = [];
|
||||
const num = minSamples - this.queue.length > 0 ? minSamples - this.queue.length : 1;
|
||||
// 原子交换 + 清空
|
||||
[TepArray, this.activeQueue] = [await this.activeQueue.dequeue(num), new BlockingQueue()];
|
||||
this.queue.push(...TepArray);
|
||||
|
||||
// 等待数据并获取
|
||||
const tempArray = await this.activeQueue.dequeue(num);
|
||||
this.queue.push(...tempArray);
|
||||
}
|
||||
|
||||
// 将Int16音频数据转换为Float32音频数据
|
||||
@@ -54,6 +57,57 @@ export class StreamingContext {
|
||||
return float32Data;
|
||||
}
|
||||
|
||||
// 获取待解码包数
|
||||
getPendingDecodeCount() {
|
||||
return this.audioBufferQueue.length + this.pendingAudioBufferQueue.length;
|
||||
}
|
||||
|
||||
// 获取待播放样本数（转换为包数，每包960样本）
|
||||
getPendingPlayCount() {
|
||||
// 计算已在队列中的样本
|
||||
const queuedSamples = this.activeQueue.length + this.queue.length;
|
||||
|
||||
// 计算已调度但未播放的样本（在Web Audio缓冲区中）
|
||||
let scheduledSamples = 0;
|
||||
if (this.playing && this.scheduledEndTime) {
|
||||
const currentTime = this.audioContext.currentTime;
|
||||
const remainingTime = Math.max(0, this.scheduledEndTime - currentTime);
|
||||
scheduledSamples = Math.floor(remainingTime * this.sampleRate);
|
||||
}
|
||||
|
||||
const totalSamples = queuedSamples + scheduledSamples;
|
||||
return Math.ceil(totalSamples / 960);
|
||||
}
|
||||
|
||||
// 清空所有音频缓冲
|
||||
clearAllBuffers() {
|
||||
log('清空所有音频缓冲', 'info');
|
||||
|
||||
// 清空所有队列（使用clear方法保持对象引用）
|
||||
this.audioBufferQueue.clear();
|
||||
this.pendingAudioBufferQueue = [];
|
||||
this.activeQueue.clear();
|
||||
this.queue = [];
|
||||
|
||||
// 停止当前播放的音频源
|
||||
if (this.source) {
|
||||
try {
|
||||
this.source.stop();
|
||||
this.source.disconnect();
|
||||
} catch (e) {
|
||||
// 忽略已经停止的错误
|
||||
}
|
||||
this.source = null;
|
||||
}
|
||||
|
||||
// 重置状态
|
||||
this.playing = false;
|
||||
this.scheduledEndTime = this.audioContext.currentTime;
|
||||
this.totalSamples = 0;
|
||||
|
||||
log('音频缓冲已清空', 'success');
|
||||
}
|
||||
|
||||
// 将Opus数据解码为PCM
|
||||
async decodeOpusFrames() {
|
||||
if (!this.opusDecoder) {
|
||||
@@ -97,7 +151,7 @@ export class StreamingContext {
|
||||
|
||||
// 开始播放音频
|
||||
async startPlaying() {
|
||||
let scheduledEndTime = this.audioContext.currentTime; // 跟踪已调度音频的结束时间
|
||||
this.scheduledEndTime = this.audioContext.currentTime; // 跟踪已调度音频的结束时间
|
||||
|
||||
while (true) {
|
||||
// 初始缓冲：等待足够的样本再开始播放
|
||||
@@ -126,7 +180,7 @@ export class StreamingContext {
|
||||
|
||||
// 精确调度播放时间
|
||||
const currentTime = this.audioContext.currentTime;
|
||||
const startTime = Math.max(scheduledEndTime, currentTime);
|
||||
const startTime = Math.max(this.scheduledEndTime, currentTime);
|
||||
|
||||
// 直接连接到输出
|
||||
this.source.connect(this.audioContext.destination);
|
||||
@@ -136,7 +190,7 @@ export class StreamingContext {
|
||||
|
||||
// 更新下一个音频块的调度时间
|
||||
const duration = audioBuffer.duration;
|
||||
scheduledEndTime = startTime + duration;
|
||||
this.scheduledEndTime = startTime + duration;
|
||||
this.lastPlayTime = startTime;
|
||||
|
||||
// 如果队列中数据不足，等待新数据
|
||||
|
||||
@@ -123,7 +123,12 @@ export class WebSocketHandler {
|
||||
} else if (message.state === 'sentence_end') {
|
||||
log(`语音段结束: ${message.text}`, 'info');
|
||||
} else if (message.state === 'stop') {
|
||||
log('服务器语音传输结束', 'info');
|
||||
log('服务器语音传输结束，清空所有音频缓冲', 'info');
|
||||
|
||||
// 清空所有音频缓冲并停止播放
|
||||
const audioPlayer = getAudioPlayer();
|
||||
audioPlayer.clearAllAudio();
|
||||
|
||||
this.isRemoteSpeaking = false;
|
||||
if (this.onRecordButtonStateChange) {
|
||||
this.onRecordButtonStateChange(false);
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
import { loadConfig, saveConfig } from '../config/manager.js';
|
||||
import { getAudioRecorder } from '../core/audio/recorder.js';
|
||||
import { getWebSocketHandler } from '../core/network/websocket.js';
|
||||
import { getAudioPlayer } from '../core/audio/player.js';
|
||||
|
||||
// UI控制器类
|
||||
export class UIController {
|
||||
@@ -9,6 +10,7 @@ export class UIController {
|
||||
this.isEditing = false;
|
||||
this.visualizerCanvas = null;
|
||||
this.visualizerContext = null;
|
||||
this.audioStatsTimer = null;
|
||||
}
|
||||
|
||||
// 初始化
|
||||
@@ -18,6 +20,7 @@ export class UIController {
|
||||
|
||||
this.initVisualizer();
|
||||
this.initEventListeners();
|
||||
this.startAudioStatsMonitor();
|
||||
loadConfig();
|
||||
}
|
||||
|
||||
@@ -86,17 +89,20 @@ export class UIController {
|
||||
const sessionStatus = document.getElementById('sessionStatus');
|
||||
if (!sessionStatus) return;
|
||||
|
||||
// 保留背景元素
|
||||
const bgHtml = '<span id="sessionStatusBg" style="position: absolute; left: 0; top: 0; bottom: 0; width: 0%; background: linear-gradient(90deg, rgba(76, 175, 80, 0.2), rgba(33, 150, 243, 0.2)); transition: width 0.15s ease-out, background 0.3s ease; z-index: 0; border-radius: 20px;"></span>';
|
||||
|
||||
if (isSpeaking === null) {
|
||||
// 离线状态
|
||||
sessionStatus.innerHTML = '<span class="emoji-large">😶</span> 小智离线中';
|
||||
sessionStatus.innerHTML = bgHtml + '<span style="position: relative; z-index: 1;"><span class="emoji-large">😶</span> 小智离线中</span>';
|
||||
sessionStatus.className = 'status offline';
|
||||
} else if (isSpeaking) {
|
||||
// 说话中
|
||||
sessionStatus.innerHTML = '<span class="emoji-large">😶</span> 小智说话中';
|
||||
sessionStatus.innerHTML = bgHtml + '<span style="position: relative; z-index: 1;"><span class="emoji-large">😶</span> 小智说话中</span>';
|
||||
sessionStatus.className = 'status speaking';
|
||||
} else {
|
||||
// 聆听中
|
||||
sessionStatus.innerHTML = '<span class="emoji-large">😶</span> 小智聆听中';
|
||||
sessionStatus.innerHTML = bgHtml + '<span style="position: relative; z-index: 1;"><span class="emoji-large">😶</span> 小智聆听中</span>';
|
||||
sessionStatus.className = 'status listening';
|
||||
}
|
||||
}
|
||||
@@ -110,8 +116,72 @@ export class UIController {
|
||||
let currentText = sessionStatus.textContent;
|
||||
// 移除现有的表情符号
|
||||
currentText = currentText.replace(/[\u{1F300}-\u{1F9FF}]|[\u{2600}-\u{26FF}]|[\u{2700}-\u{27BF}]/gu, '').trim();
|
||||
|
||||
// 保留背景元素
|
||||
const bgHtml = '<span id="sessionStatusBg" style="position: absolute; left: 0; top: 0; bottom: 0; width: 0%; background: linear-gradient(90deg, rgba(76, 175, 80, 0.2), rgba(33, 150, 243, 0.2)); transition: width 0.15s ease-out, background 0.3s ease; z-index: 0; border-radius: 20px;"></span>';
|
||||
|
||||
// 使用 innerHTML 添加带样式的表情
|
||||
sessionStatus.innerHTML = `<span class="emoji-large">${emoji}</span> ${currentText}`;
|
||||
sessionStatus.innerHTML = bgHtml + `<span style="position: relative; z-index: 1;"><span class="emoji-large">${emoji}</span> ${currentText}</span>`;
|
||||
}
|
||||
|
||||
// 更新音频统计信息
|
||||
updateAudioStats() {
|
||||
const audioPlayer = getAudioPlayer();
|
||||
const stats = audioPlayer.getAudioStats();
|
||||
|
||||
const sessionStatus = document.getElementById('sessionStatus');
|
||||
const sessionStatusBg = document.getElementById('sessionStatusBg');
|
||||
|
||||
// 只在说话状态下显示背景进度
|
||||
if (sessionStatus && sessionStatus.classList.contains('speaking') && sessionStatusBg) {
|
||||
if (stats.pendingPlay > 0) {
|
||||
// 计算进度：5包=50%，10包及以上=100%
|
||||
let percentage;
|
||||
if (stats.pendingPlay >= 10) {
|
||||
percentage = 100;
|
||||
} else {
|
||||
percentage = (stats.pendingPlay / 10) * 100;
|
||||
}
|
||||
|
||||
sessionStatusBg.style.width = `${percentage}%`;
|
||||
|
||||
// 根据缓冲量改变背景颜色
|
||||
if (stats.pendingPlay < 5) {
|
||||
// 缓冲不足：橙红色半透明
|
||||
sessionStatusBg.style.background = 'linear-gradient(90deg, rgba(255, 152, 0, 0.25), rgba(255, 87, 34, 0.25))';
|
||||
} else if (stats.pendingPlay < 10) {
|
||||
// 一般：黄绿色半透明
|
||||
sessionStatusBg.style.background = 'linear-gradient(90deg, rgba(205, 220, 57, 0.25), rgba(76, 175, 80, 0.25))';
|
||||
} else {
|
||||
// 充足：绿蓝色半透明
|
||||
sessionStatusBg.style.background = 'linear-gradient(90deg, rgba(76, 175, 80, 0.25), rgba(33, 150, 243, 0.25))';
|
||||
}
|
||||
} else {
|
||||
// 没有缓冲，隐藏背景
|
||||
sessionStatusBg.style.width = '0%';
|
||||
}
|
||||
} else {
|
||||
// 非说话状态，隐藏背景
|
||||
if (sessionStatusBg) {
|
||||
sessionStatusBg.style.width = '0%';
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 启动音频统计监控
|
||||
startAudioStatsMonitor() {
|
||||
// 每100ms更新一次音频统计
|
||||
this.audioStatsTimer = setInterval(() => {
|
||||
this.updateAudioStats();
|
||||
}, 100);
|
||||
}
|
||||
|
||||
// 停止音频统计监控
|
||||
stopAudioStatsMonitor() {
|
||||
if (this.audioStatsTimer) {
|
||||
clearInterval(this.audioStatsTimer);
|
||||
this.audioStatsTimer = null;
|
||||
}
|
||||
}
|
||||
|
||||
// 绘制音频可视化效果
|
||||
|
||||
@@ -95,4 +95,9 @@ export default class BlockingQueue {
|
||||
get length() {
|
||||
return this.#items.length;
|
||||
}
|
||||
|
||||
/* 清空队列（保持对象引用，不影响等待者） */
|
||||
clear() {
|
||||
this.#items.length = 0;
|
||||
}
|
||||
}
|
||||
@@ -91,6 +91,7 @@
|
||||
<input type="text" id="serverUrl" value="" readonly disabled placeholder="点击连接按钮后，自动从OTA接口获取" />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
@@ -113,9 +114,12 @@
|
||||
<canvas id="audioVisualizer" class="audio-visualizer"></canvas>
|
||||
</div>
|
||||
|
||||
<div style="margin: -10px 0px 5px 0px;">
|
||||
<div style="margin: -10px 0px 10px 0px;">
|
||||
<span class="connection-status llm-emoji">
|
||||
<span id="sessionStatus" class="status offline"><span class="emoji-large">😶</span> 小智离线中</span>
|
||||
<span id="sessionStatus" class="status offline" style="position: relative; display: inline-block;">
|
||||
<span id="sessionStatusBg" style="position: absolute; left: 0; top: 0; bottom: 0; width: 0%; background: linear-gradient(90deg, rgba(76, 175, 80, 0.2), rgba(33, 150, 243, 0.2)); transition: width 0.15s ease-out, background 0.3s ease; z-index: 0; border-radius: 20px;"></span>
|
||||
<span style="position: relative; z-index: 1;"><span class="emoji-large">😶</span> 小智离线中</span>
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="flex-container">
|
||||
|
||||
Reference in New Issue
Block a user