diff --git a/README.md b/README.md index 943501a7..eb718c24 100644 --- a/README.md +++ b/README.md @@ -183,8 +183,8 @@ Spearheaded by Professor Siyuan Liu's Team (South China University of Technology #### 🚀 部署方式选择 | 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | |---------|------|---------|---------|---------|---------| -| **最简化安装** | 智能对话、IOT、MCP、视觉感知 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | -| **全模块安装** | 智能对话、IOT、MCP接入点、声纹识别、视觉感知、OTA、智控台 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | +| **最简化安装** | 智能对话、单智能体管理 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | +| **全模块安装** | 智能对话、多用户管理、多智能体管理、智控台界面操作 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 常见问题及相关教程，可参考[这个链接](./docs/FAQ.md) @@ -211,10 +211,10 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/ | 模块名称 | 入门全免费设置 | 流式配置 | |:---:|:---:|:---:| -| ASR(语音识别) | FunASR(本地) | 👍FunASR(本地GPU模式) | -| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍AliLLM(qwen3-235b-a22b-instruct-2507) 或 👍DoubaoLLM(doubao-1-5-pro-32k-250115) | -| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) | -| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) 或 👍AliyunStreamTTS(阿里云流式语音合成) | +| ASR(语音识别) | FunASR(本地) | 👍XunfeiStreamASR(讯飞流式) | +| LLM(大模型) | glm-4-flash(智谱) | 👍qwen-flash(阿里百炼) | +| VLLM(视觉大模型) | glm-4v-flash(智谱) | 👍qwen2.5-vl-3b-instructh(阿里百炼) | +| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山流式) | | Intent(意图识别) | function_call(函数调用) | function_call(函数调用) | | Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） | @@ -260,7 +260,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/ --- ## 产品生态 👬 -小智是一个生态，当你使用这个产品时，也可以看看其他在这个生态圈的[优秀项目](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) +小智是一个生态，当你使用这个产品时，也可以看看其他在这个生态圈的[优秀项目](https://github.com/78/xiaozhi-esp32/blob/main/README_zh.md#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) --- diff --git a/README_de.md b/README_de.md index c3578fec..0e4c74d5 100644 --- a/README_de.md +++ b/README_de.md @@ -181,8 +181,8 @@ Dieses Projekt bietet zwei Bereitstellungsmethoden. Bitte wählen Sie basierend #### 🚀 Auswahl der Bereitstellungsmethode | Bereitstellungsmethode | Funktionen | Anwendungsszenarien | Deployment-Dokumente | Konfigurationsanforderungen | Video-Tutorials | |---------|------|---------|---------|---------|---------| -| **Vereinfachte Installation** | Intelligenter Dialog, IOT, MCP, visuelle Wahrnehmung | Umgebungen mit geringer Konfiguration, Daten in Konfigurationsdateien gespeichert, keine Datenbank erforderlich | [①Docker-Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Quellcode-Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 Kerne 4GB bei Verwendung von `FunASR`, 2 Kerne 2GB bei allen APIs | - | -| **Vollständige Modulinstallation** | Intelligenter Dialog, IOT, MCP-Endpunkte, Stimmabdruckerkennung, visuelle Wahrnehmung, OTA, intelligente Steuerkonsole | Vollständige Funktionserfahrung, Daten in Datenbank gespeichert |[①Docker-Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Quellcode-Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Quellcode-Deployment Auto-Update-Tutorial](./docs/dev-ops-integration.md) | 4 Kerne 8GB bei Verwendung von `FunASR`, 2 Kerne 4GB bei allen APIs| [Video-Tutorial für lokalen Quellcode-Start](https://www.bilibili.com/video/BV1wBJhz4Ewe) | +| **Vereinfachte Installation** | Intelligenter Dialog, Einzel-Agenten-Verwaltung | Umgebungen mit geringer Konfiguration, Daten in Konfigurationsdateien gespeichert, keine Datenbank erforderlich | [①Docker-Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Quellcode-Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 Kerne 4GB bei Verwendung von `FunASR`, 2 Kerne 2GB bei allen APIs | - | +| **Vollständige Modulinstallation** | Intelligenter Dialog, Mehrbenutzerverwaltung, Mehr-Agenten-Verwaltung, Intelligente Steuerkonsole-Bedienung | Vollständige Funktionserfahrung, Daten in Datenbank gespeichert |[①Docker-Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Quellcode-Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Quellcode-Deployment Auto-Update-Tutorial](./docs/dev-ops-integration.md) | 4 Kerne 8GB bei Verwendung von `FunASR`, 2 Kerne 4GB bei allen APIs| [Video-Tutorial für lokalen Quellcode-Start](https://www.bilibili.com/video/BV1wBJhz4Ewe) | Häufige Fragen und entsprechende Tutorials finden Sie unter [diesem Link](./docs/FAQ.md) @@ -209,10 +209,10 @@ Websocket-Schnittstellenadresse: wss://2662r3426b.vicp.fun/xiaozhi/v1/ | Modulname | Einstiegslevel Kostenlose Einstellungen | Streaming-Konfiguration | |:---:|:---:|:---:| -| ASR (Spracherkennung) | FunASR (Lokal) | 👍FunASR (Lokaler GPU-Modus) | -| LLM (Großes Modell) | ChatGLMLLM (Zhipu glm-4-flash) | 👍AliLLM (qwen3-235b-a22b-instruct-2507) oder 👍DoubaoLLM (doubao-1-5-pro-32k-250115) | -| VLLM (Vision Large Model) | ChatGLMVLLM (Zhipu glm-4v-flash) | 👍QwenVLVLLM (Qwen qwen2.5-vl-3b-instructh) | -| TTS (Sprachsynthese) | ✅LinkeraiTTS (Lingxi-Streaming) | 👍HuoshanDoubleStreamTTS (Volcano Dual-Stream-Sprachsynthese) oder 👍AliyunStreamTTS (Alibaba Cloud Streaming-Sprachsynthese) | +| ASR (Spracherkennung) | FunASR (Lokal) | 👍XunfeiStreamASR (Xunfei-Streaming) | +| LLM (Großes Modell) | glm-4-flash (Zhipu) | 👍qwen-flash (Alibaba Bailian) | +| VLLM (Vision Large Model) | glm-4v-flash (Zhipu) | 👍qwen2.5-vl-3b-instructh (Alibaba Bailian) | +| TTS (Sprachsynthese) | ✅LinkeraiTTS (Lingxi-Streaming) | 👍HuoshanDoubleStreamTTS (Volcano-Streaming) | | Intent (Absichtserkennung) | function_call (Funktionsaufruf) | function_call (Funktionsaufruf) | | Memory (Gedächtnisfunktion) | mem_local_short (Lokales Kurzzeitgedächtnis) | mem_local_short (Lokales Kurzzeitgedächtnis) | @@ -258,7 +258,7 @@ Wenn Sie ein Softwareentwickler sind, finden Sie hier einen [Offenen Brief an En --- ## Produktökosystem 👬 -Xiaozhi ist ein Ökosystem. Wenn Sie dieses Produkt verwenden, können Sie sich auch andere [hervorragende Projekte](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) in diesem Ökosystem ansehen +Xiaozhi ist ein Ökosystem. Wenn Sie dieses Produkt verwenden, können Sie sich auch andere [hervorragende Projekte](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#related-open-source-projects) in diesem Ökosystem ansehen --- diff --git a/README_en.md b/README_en.md index e8437832..196cf16f 100644 --- a/README_en.md +++ b/README_en.md @@ -181,8 +181,8 @@ This project provides two deployment methods. Please choose based on your specif #### 🚀 Deployment Method Selection | Deployment Method | Features | Applicable Scenarios | Deployment Docs | Configuration Requirements | Video Tutorials | |---------|------|---------|---------|---------|---------| -| **Simplified Installation** | Intelligent dialogue, IOT, MCP, visual perception | Low-configuration environments, data stored in config files, no database required | [①Docker Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - | -| **Full Module Installation** | Intelligent dialogue, IOT, MCP endpoints, voiceprint recognition, visual perception, OTA, intelligent control console | Complete functionality experience, data stored in database |[①Docker Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](./docs/dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) | +| **Simplified Installation** | Intelligent dialogue, single agent management | Low-configuration environments, data stored in config files, no database required | [①Docker Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - | +| **Full Module Installation** | Intelligent dialogue, multi-user management, multi-agent management, intelligent console interface operation | Complete functionality experience, data stored in database |[①Docker Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](./docs/dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) | > 💡 Note: Below is a test platform deployed with the latest code. You can burn and test if needed. Concurrent users: 6, data will be cleared daily. @@ -208,10 +208,10 @@ Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/ | Module Name | Entry Level Free Settings | Streaming Configuration | |:---:|:---:|:---:| -| ASR(Speech Recognition) | FunASR(Local) | 👍FunASRServer or 👍DoubaoStreamASR | -| LLM(Large Model) | ChatGLMLLM(Zhipu glm-4-flash) | 👍DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) | -| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | 👍QwenVLVLLM(Qwen qwen2.5-vl-3b-instructh) | -| TTS(Speech Synthesis) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano dual-stream speech synthesis) | +| ASR(Speech Recognition) | FunASR(Local) | 👍XunfeiStreamASR(Xunfei Streaming) | +| LLM(Large Model) | glm-4-flash(Zhipu) | 👍qwen-flash(Alibaba Bailian) | +| VLLM(Vision Large Model) | glm-4v-flash(Zhipu) | 👍qwen2.5-vl-3b-instructh(Alibaba Bailian) | +| TTS(Speech Synthesis) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano Streaming) | | Intent(Intent Recognition) | function_call(Function calling) | function_call(Function calling) | | Memory(Memory function) | mem_local_short(Local short-term memory) | mem_local_short(Local short-term memory) | @@ -256,7 +256,7 @@ If you are a software developer, here is an [Open Letter to Developers](docs/con --- ## Product Ecosystem 👬 -Xiaozhi is an ecosystem. When using this product, you can also check out other [excellent projects](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) in this ecosystem +Xiaozhi is an ecosystem. When using this product, you can also check out other [excellent projects](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#related-open-source-projects) in this ecosystem | Project Name | Project Address | Project Description | |:---------------------|:--------|:--------| diff --git a/README_vi.md b/README_vi.md index 92697551..e6d5f489 100644 --- a/README_vi.md +++ b/README_vi.md @@ -182,8 +182,8 @@ Dự án này cung cấp hai phương pháp triển khai, vui lòng chọn theo #### 🚀 Lựa chọn phương pháp triển khai | Phương pháp triển khai | Đặc điểm | Tình huống áp dụng | Tài liệu triển khai | Yêu cầu cấu hình | Video hướng dẫn | |---------|------|---------|---------|---------|---------| -| **Cài đặt tối giản** | Đối thoại thông minh, IOT, MCP, cảm nhận thị giác | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 nhân 4GB nếu dùng `FunASR`, 2 nhân 2GB nếu toàn API | - | -| **Cài đặt toàn bộ module** | Đối thoại thông minh, IOT, điểm truy cập MCP, nhận dạng giọng nói, cảm nhận thị giác, OTA, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](./docs/dev-ops-integration.md) | 4 nhân 8GB nếu dùng `FunASR`, 2 nhân 4GB nếu toàn API| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) | +| **Cài đặt tối giản** | Đối thoại thông minh, quản lý đơn tác nhân | Môi trường cấu hình thấp, dữ liệu lưu trong tệp cấu hình, không cần cơ sở dữ liệu | [①Phiên bản Docker](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Triển khai mã nguồn](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 nhân 4GB nếu dùng `FunASR`, 2 nhân 2GB nếu toàn API | - | +| **Cài đặt toàn bộ module** | Đối thoại thông minh, quản lý đa người dùng, quản lý đa tác nhân, bảng điều khiển thông minh | Trải nghiệm đầy đủ tính năng, dữ liệu lưu trong cơ sở dữ liệu |[①Phiên bản Docker](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Triển khai mã nguồn](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Hướng dẫn tự động cập nhật triển khai mã nguồn](./docs/dev-ops-integration.md) | 4 nhân 8GB nếu dùng `FunASR`, 2 nhân 4GB nếu toàn API| [Video hướng dẫn khởi động mã nguồn cục bộ](https://www.bilibili.com/video/BV1wBJhz4Ewe) | Câu hỏi thường gặp và hướng dẫn liên quan, vui lòng tham khảo [liên kết này](./docs/FAQ.md) @@ -210,10 +210,10 @@ Công cụ kiểm tra dịch vụ: https://2662r3426b.vicp.fun/test/ | Tên module | Cài đặt miễn phí cho người mới | Cấu hình streaming | |:---:|:---:|:---:| -| ASR(Nhận dạng giọng nói) | FunASR(Local) | 👍FunASR(Chế độ GPU cục bộ) | -| LLM(Mô hình lớn) | ChatGLMLLM(Zhipu glm-4-flash) | 👍AliLLM(qwen3-235b-a22b-instruct-2507) hoặc 👍DoubaoLLM(doubao-1-5-pro-32k-250115) | -| VLLM(Mô hình lớn thị giác) | ChatGLMVLLM(Zhipu glm-4v-flash) | 👍QwenVLVLLM(Qwen qwen2.5-vl-3b-instructh) | -| TTS(Tổng hợp giọng nói) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Tổng hợp giọng nói streaming kép Volcano) hoặc 👍AliyunStreamTTS(Tổng hợp giọng nói streaming Alibaba Cloud) | +| ASR(Nhận dạng giọng nói) | FunASR(Local) | 👍XunfeiStreamASR(Xunfei Streaming) | +| LLM(Mô hình lớn) | glm-4-flash(Zhipu) | 👍qwen-flash(Alibaba Bailian) | +| VLLM(Mô hình lớn thị giác) | glm-4v-flash(Zhipu) | 👍qwen2.5-vl-3b-instructh(Alibaba Bailian) | +| TTS(Tổng hợp giọng nói) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano Streaming) | | Intent(Nhận dạng ý định) | function_call(Gọi hàm) | function_call(Gọi hàm) | | Memory(Chức năng bộ nhớ) | mem_local_short(Bộ nhớ ngắn hạn cục bộ) | mem_local_short(Bộ nhớ ngắn hạn cục bộ) | @@ -259,7 +259,7 @@ Nếu bạn là một nhà phát triển phần mềm, đây có một [Lá thư --- ## Hệ sinh thái sản phẩm 👬 -Xiaozhi là một hệ sinh thái, khi bạn sử dụng sản phẩm này, bạn cũng có thể xem các [dự án xuất sắc](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#%E7%9B%B8%E5%85%B3%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE) khác trong hệ sinh thái này +Xiaozhi là một hệ sinh thái, khi bạn sử dụng sản phẩm này, bạn cũng có thể xem các [dự án xuất sắc](https://github.com/78/xiaozhi-esp32?tab=readme-ov-file#related-open-source-projects) khác trong hệ sinh thái này --- diff --git a/docs/FAQ.md b/docs/FAQ.md index c40e7784..69494c3d 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -38,10 +38,10 @@ conda install conda-forge::ffmpeg | 模块名称 | 入门全免费设置 | 流式配置 | |:---:|:---:|:---:| -| ASR(语音识别) | FunASR(本地) | 👍FunASR(本地GPU模式) | -| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍AliLLM(qwen3-235b-a22b-instruct-2507) 或 👍DoubaoLLM(doubao-1-5-pro-32k-250115) | -| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) | -| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) 或 👍AliyunStreamTTS(阿里云流式语音合成) | +| ASR(语音识别) | FunASR(本地) | 👍XunfeiStreamASR(讯飞流式) | +| LLM(大模型) | glm-4-flash(智谱) | 👍qwen-flash(阿里百炼) | +| VLLM(视觉大模型) | glm-4v-flash(智谱) | 👍qwen2.5-vl-3b-instructh(阿里百炼) | +| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山流式) | | Intent(意图识别) | function_call(函数调用) | function_call(函数调用) | | Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） | @@ -69,6 +69,7 @@ VAD: ### 9、编译固件相关教程 1、[如何自己编译小智固件](./firmware-build.md)
2、[如何基于虾哥编译好的固件修改OTA地址](./firmware-setting.md)
+3、[单模块部署如何配置固件OTA自动升级](./ota-upgrade-guide.md)
### 10、拓展相关教程 1、[如何开启手机号码注册智控台](./ali-sms-integration.md)
diff --git a/docs/huoshan-streamTTS-voice-cloning.md b/docs/huoshan-streamTTS-voice-cloning.md index c2d261d9..304a8c90 100644 --- a/docs/huoshan-streamTTS-voice-cloning.md +++ b/docs/huoshan-streamTTS-voice-cloning.md @@ -23,6 +23,8 @@ ### 2.将音色资源ID分配给系统账号 +使用超级管理员账号登录智控台，点击顶部`参数字典`，在下拉菜单中，点击`系统功能配置`页面。在页面上勾选`音色克隆`，点击保存配置。即可在顶部菜单看到`音色克隆`按钮。 + 使用超级管理员账号登录智控台，点击顶部【音色克隆】、【音色资源】。 点击新增按钮，在【平台名称】选择“火山双流式语音合成”； diff --git a/docs/mqtt-gateway-integration.md b/docs/mqtt-gateway-integration.md index b2f3068c..2b96255f 100644 --- a/docs/mqtt-gateway-integration.md +++ b/docs/mqtt-gateway-integration.md @@ -76,6 +76,7 @@ MQTT_PORT=1883 # MQTT服务器端口 UDP_PORT=8884 # UDP服务器端口 API_PORT=8007 # 管理API端口 MQTT_SIGNATURE_KEY=test # MQTT签名密钥 +SERVER_SECRET=Te1st12134 # 服务器密钥，请保持和智控台（server.secret）一致或者和xiaozhi-server里（server.auth_key）保持一致 ``` 请注意`PUBLIC_IP`配置，确保其与实际公网IP一致，如果有域名就填域名。 @@ -85,6 +86,13 @@ MQTT_SIGNATURE_KEY=test # MQTT签名密钥 - 注意不要用简单的密码，比如`123456`、`test`等。 - 注意不要用简单的密码，比如`123456`、`test`等。 +`SERVER_SECRET` 是用生成websocket连接的认证信息。 + +1、如果你是全模块部署，且你的智控台的参数管理里`server.auth.enabled`设置成了`true`，那么，`SERVER_SECRET`需要和智控台（`server.secret`）保持一致。 + +2、如果你是单模块部署，且你在配置文件里把`server.auth.enabled`设置成了`true`，那么，`SERVER_SECRET`需要和配置文件里（`server.auth_key`）保持一致。 + + 6. 启动MQTT网关 ``` # 启动服务 diff --git a/docs/ota-upgrade-guide.md b/docs/ota-upgrade-guide.md new file mode 100644 index 00000000..0df4a2ff --- /dev/null +++ b/docs/ota-upgrade-guide.md @@ -0,0 +1,142 @@ +# 单模块部署固件OTA自动升级配置指南 + +本教程将指导你如何在**单模块部署**场景下配置固件OTA自动升级功能，实现设备固件的自动更新。 + +如果你已经使用**全模块部署**，请忽略本教程。 + +## 功能介绍 + +在单模块部署中，xiaozhi-server内置了OTA固件管理功能，可以自动检测设备版本并下发升级固件。系统会根据设备型号和当前版本，自动匹配并推送最新的固件版本。 + +## 前提条件 + +- 你已经成功进行**单模块部署**并运行xiaozhi-server +- 设备能够正常连接到服务器 + +## 第一步 准备固件文件 + +### 1. 创建固件存放目录 + +固件文件需要放在`data/bin/`目录下。如果该目录不存在，请手动创建： + +```bash +mkdir -p data/bin +``` + +### 2. 固件文件命名规则 + +固件文件必须遵循以下命名格式： + +``` +{设备型号}_{版本号}.bin +``` + +**命名规则说明：** +- `设备型号`：设备的型号名称，例如 `lichuang-dev`、`bread-compact-wifi` 等 +- `版本号`：固件版本号，必须以数字开头，支持数字、字母、点号、下划线和短横线，例如 `1.6.6`、`2.0.0` 等 +- 文件扩展名必须是 `.bin` + +**命名示例：** +``` +bread-compact-wifi_1.6.6.bin +lichuang-dev_2.0.0.bin +``` + +### 3. 放置固件文件 + +将准备好的固件文件（.bin文件）复制到`data/bin/`目录下： + +重要的事情说三遍：升级的bin文件是`xiaozhi.bin`，不是全量固件文件`merged-binary.bin`! + +重要的事情说三遍：升级的bin文件是`xiaozhi.bin`，不是全量固件文件`merged-binary.bin`! + +重要的事情说三遍：升级的bin文件是`xiaozhi.bin`，不是全量固件文件`merged-binary.bin`! + +```bash +cp xiaozhi.bin data/bin/设备型号_版本号.bin +``` + +例如： +```bash +cp xiaozhi.bin data/bin/bread-compact-wifi_1.6.6.bin +``` + +## 第二步 配置公网访问地址（仅公网部署需要） + +**注意：此步骤仅适用于单模块公网部署的场景。** + +如果你的xiaozhi-server是公网部署（使用公网IP或域名），**必须**配置`server.vision_explain`参数，因为OTA固件下载地址会使用该配置的域名和端口。 + +如果你是局域网部署，可以跳过此步骤。 + +### 为什么要配置这个参数？ + +在单模块部署中，系统生成固件下载地址时，会使用`vision_explain`配置的域名和端口作为基础地址。如果不配置或配置错误，设备将无法访问固件下载地址。 + +### 配置方法 + +打开`data/.config.yaml`文件，找到`server`配置段，设置`vision_explain`参数： + +```yaml +server: + vision_explain: http://你的域名或IP:端口号/mcp/vision/explain +``` + +**配置示例：** + +局域网部署（默认）： +```yaml +server: + vision_explain: http://192.168.1.100:8003/mcp/vision/explain +``` + +公网域名部署： +```yaml +server: + vision_explain: http://yourdomain.com:8003/mcp/vision/explain +``` + +### 注意事项 + +- 域名或IP必须是设备能够访问的地址 +- 如果使用Docker部署，不能使用Docker内部地址（如127.0.0.1或localhost） +- 如果你使用了nginx反向代理，请填写对外的地址和端口号，不是本项目运行的端口号 + + +## 常见问题 + +### 1. 设备收不到固件更新 + +**可能原因和解决方法：** + +- 检查固件文件命名是否符合规则：`{型号}_{版本号}.bin` +- 检查固件文件是否正确放置在`data/bin/`目录 +- 检查设备型号是否与固件文件名中的型号匹配 +- 检查固件版本号是否高于设备当前版本 +- 查看服务器日志，确认OTA请求是否正常处理 + +### 2. 设备报告下载地址无法访问 + +**可能原因和解决方法：** + +- 检查`server.vision_explain`配置的域名或IP是否正确 +- 确认端口号配置正确（默认8003） +- 如果是公网部署，确保设备能够访问该公网地址 +- 如果是Docker部署，确保不是使用了内部地址（127.0.0.1） +- 检查防火墙是否开放了对应端口 +- 如果你使用了nginx反向代理，请填写对外的地址和端口号，不是本项目运行的端口号 + +### 3. 如何确认设备当前版本 + +查看OTA请求日志，日志中会显示设备上报的版本号： + +``` +[ota_handler] - 设备 AA:BB:CC:DD:EE:FF 固件已是最新: 1.6.6 +``` + +### 4. 固件文件放置后没有生效 + +系统有30秒的缓存时间（默认），可以： +- 等待30秒后再让设备发起OTA请求 +- 重启xiaozhi-server服务 +- 调整`firmware_cache_ttl`配置为更短的时间 diff --git a/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java b/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java index 1ea19355..c3f498d4 100644 --- a/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java +++ b/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java @@ -141,6 +141,11 @@ public interface Constant { */ String SERVER_MQTT_SECRET = "server.mqtt_signature_key"; + /** + * WebSocket认证开关 + */ + String SERVER_AUTH_ENABLED = "server.auth.enabled"; + /** * 无记忆 */ @@ -299,7 +304,7 @@ public interface Constant { /** * 版本号 */ - public static final String VERSION = "0.8.9"; + public static final String VERSION = "0.8.10"; /** * 无效固件URL diff --git a/main/manager-api/src/main/java/xiaozhi/modules/device/service/impl/DeviceServiceImpl.java b/main/manager-api/src/main/java/xiaozhi/modules/device/service/impl/DeviceServiceImpl.java index 5c3f854d..44977730 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/device/service/impl/DeviceServiceImpl.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/device/service/impl/DeviceServiceImpl.java @@ -1,6 +1,8 @@ package xiaozhi.modules.device.service.impl; import java.nio.charset.StandardCharsets; +import java.security.InvalidKeyException; +import java.security.NoSuchAlgorithmException; import java.time.Instant; import java.util.Base64; import java.util.Date; @@ -169,7 +171,22 @@ public class DeviceServiceImpl extends BaseServiceImpl DeviceReportRespDTO.Websocket websocket = new DeviceReportRespDTO.Websocket(); // 从系统参数获取WebSocket URL，如果未配置则使用默认值 String wsUrl = sysParamsService.getValue(Constant.SERVER_WEBSOCKET, true); - websocket.setToken(""); + + // 检查是否启用认证并生成token + String authEnabled = sysParamsService.getValue(Constant.SERVER_AUTH_ENABLED, true); + if ("true".equalsIgnoreCase(authEnabled)) { + try { + // 生成token + String token = generateWebSocketToken(clientId, macAddress); + websocket.setToken(token); + } catch (Exception e) { + log.error("生成WebSocket token失败: {}", e.getMessage()); + websocket.setToken(""); + } + } else { + websocket.setToken(""); + } + if (StringUtils.isBlank(wsUrl) || wsUrl.equals("null")) { log.error("WebSocket地址未配置，请登录智控台，在参数管理找到【server.websocket】配置"); wsUrl = "ws://xiaozhi.server.com:8000/xiaozhi/v1/"; @@ -189,7 +206,7 @@ public class DeviceServiceImpl extends BaseServiceImpl // 添加MQTT UDP配置 // 从系统参数获取MQTT Gateway地址，仅在配置有效时使用 - String mqttUdpConfig = sysParamsService.getValue(Constant.SERVER_MQTT_GATEWAY, false); + String mqttUdpConfig = sysParamsService.getValue(Constant.SERVER_MQTT_GATEWAY, true); if (mqttUdpConfig != null && !mqttUdpConfig.equals("null") && !mqttUdpConfig.isEmpty()) { try { String groupId = deviceById != null && deviceById.getBoard() != null ? deviceById.getBoard() @@ -494,6 +511,40 @@ public class DeviceServiceImpl extends BaseServiceImpl return Base64.getEncoder().encodeToString(signature); } + /** + * 生成WebSocket认证token 遵循Python端AuthManager的实现逻辑：token = signature.timestamp + * + * @param clientId 客户端ID + * @param username 用户名 (通常为deviceId/macAddress) + * @return 认证token字符串 + */ + private String generateWebSocketToken(String clientId, String username) + throws NoSuchAlgorithmException, InvalidKeyException { + // 从系统参数获取密钥 + String secretKey = sysParamsService.getValue(Constant.SERVER_SECRET, false); + if (StringUtils.isBlank(secretKey)) { + throw new IllegalStateException("WebSocket认证密钥未配置(server.secret)"); + } + + // 获取当前时间戳(秒) + long timestamp = System.currentTimeMillis() / 1000; + + // 构建签名内容: clientId|username|timestamp + String content = String.format("%s|%s|%d", clientId, username, timestamp); + + // 生成HMAC-SHA256签名 + Mac hmac = Mac.getInstance("HmacSHA256"); + SecretKeySpec keySpec = new SecretKeySpec(secretKey.getBytes(StandardCharsets.UTF_8), "HmacSHA256"); + hmac.init(keySpec); + byte[] signature = hmac.doFinal(content.getBytes(StandardCharsets.UTF_8)); + + // Base64 URL-safe编码签名(去除填充符=) + String signatureBase64 = Base64.getUrlEncoder().withoutPadding().encodeToString(signature); + + // 返回格式: signature.timestamp + return String.format("%s.%d", signatureBase64, timestamp); + } + /** * 构建MQTT配置信息 * @@ -504,7 +555,7 @@ public class DeviceServiceImpl extends BaseServiceImpl private DeviceReportRespDTO.MQTT buildMqttConfig(String macAddress, String groupId) throws Exception { // 从环境变量或系统参数获取签名密钥 - String signatureKey = sysParamsService.getValue("server.mqtt_signature_key", false); + String signatureKey = sysParamsService.getValue("server.mqtt_signature_key", true); if (StringUtils.isBlank(signatureKey)) { log.warn("缺少MQTT_SIGNATURE_KEY，跳过MQTT配置生成"); return null; diff --git a/main/manager-api/src/main/java/xiaozhi/modules/security/controller/LoginController.java b/main/manager-api/src/main/java/xiaozhi/modules/security/controller/LoginController.java index 2c7bab4a..1ab9809d 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/security/controller/LoginController.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/security/controller/LoginController.java @@ -6,6 +6,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import org.apache.commons.lang3.StringUtils; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.PutMapping; @@ -23,7 +24,9 @@ import xiaozhi.common.exception.ErrorCode; import xiaozhi.common.exception.RenException; import xiaozhi.common.page.TokenDTO; import xiaozhi.common.user.UserDetail; +import xiaozhi.common.utils.JsonUtils; import xiaozhi.common.utils.Result; +import xiaozhi.common.utils.Sm2DecryptUtil; import xiaozhi.common.validator.AssertUtils; import xiaozhi.common.validator.ValidatorUtils; import xiaozhi.modules.security.dto.LoginDTO; @@ -32,8 +35,6 @@ import xiaozhi.modules.security.password.PasswordUtils; import xiaozhi.modules.security.service.CaptchaService; import xiaozhi.modules.security.service.SysUserTokenService; import xiaozhi.modules.security.user.SecurityUser; -import xiaozhi.common.utils.Sm2DecryptUtil; -import org.apache.commons.lang3.StringUtils; import xiaozhi.modules.sys.dto.PasswordDTO; import xiaozhi.modules.sys.dto.RetrievePasswordDTO; import xiaozhi.modules.sys.dto.SysUserDTO; @@ -41,7 +42,6 @@ import xiaozhi.modules.sys.service.SysDictDataService; import xiaozhi.modules.sys.service.SysParamsService; import xiaozhi.modules.sys.service.SysUserService; import xiaozhi.modules.sys.vo.SysDictDataItem; -import xiaozhi.common.utils.JsonUtils; /** * 登录控制层 @@ -237,7 +237,7 @@ public class LoginController { config.put("sm2PublicKey", publicKey); // 获取system-web.menu参数配置 - String menuConfig = sysParamsService.getValue("system-web.menu", false); + String menuConfig = sysParamsService.getValue("system-web.menu", true); if (StringUtils.isNotBlank(menuConfig)) { config.put("systemWebMenu", JsonUtils.parseObject(menuConfig, Object.class)); } diff --git a/main/manager-api/src/main/resources/db/changelog/202512131453.sql b/main/manager-api/src/main/resources/db/changelog/202512131453.sql new file mode 100644 index 00000000..27ea4a1b --- /dev/null +++ b/main/manager-api/src/main/resources/db/changelog/202512131453.sql @@ -0,0 +1,6 @@ +-- 删除server模块是否开启token认证参数 +delete from `sys_params` where param_code = 'server.auth.enabled'; + +-- 添加server模块是否开启token认证参数 +INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES +(122, 'server.auth.enabled', 'true', 'boolean', 1, 'server模块是否开启token认证'); \ No newline at end of file diff --git a/main/manager-api/src/main/resources/db/changelog/202512161529.sql b/main/manager-api/src/main/resources/db/changelog/202512161529.sql new file mode 100644 index 00000000..40f154c7 --- /dev/null +++ b/main/manager-api/src/main/resources/db/changelog/202512161529.sql @@ -0,0 +1 @@ +INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (311, 'enable_websocket_ping', 'false', 'boolean', 1, '是否启用WebSocket心跳保活机制'); \ No newline at end of file diff --git a/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml b/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml index b0cdbc63..67d912a1 100755 --- a/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml +++ b/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml @@ -438,3 +438,17 @@ databaseChangeLog: - sqlFile: encoding: utf8 path: classpath:db/changelog/202512041515.sql + - changeSet: + id: 202512131453 + author: hrz + changes: + - sqlFile: + encoding: utf8 + path: classpath:db/changelog/202512131453.sql + - changeSet: + id: 202512161529 + author: RanChen + changes: + - sqlFile: + encoding: utf8 + path: classpath:db/changelog/202512161529.sql diff --git a/main/manager-mobile/src/pages/settings/index.vue b/main/manager-mobile/src/pages/settings/index.vue index 2f7be6b4..b1fa487b 100644 --- a/main/manager-mobile/src/pages/settings/index.vue +++ b/main/manager-mobile/src/pages/settings/index.vue @@ -235,7 +235,7 @@ function showAbout() { title: t('settings.aboutApp', { appName: import.meta.env.VITE_APP_TITLE }), content: t('settings.aboutContent', { appName: import.meta.env.VITE_APP_TITLE, - version: '0.8.9' + version: '0.8.10' }), showCancel: false, confirmText: t('common.confirm'), diff --git a/main/manager-web/src/assets/xiaozhi-ai.png b/main/manager-web/src/assets/xiaozhi-ai.png index ef3834af..8a595493 100644 Binary files a/main/manager-web/src/assets/xiaozhi-ai.png and b/main/manager-web/src/assets/xiaozhi-ai.png differ diff --git a/main/manager-web/src/assets/xiaozhi-ai_de.png b/main/manager-web/src/assets/xiaozhi-ai_de.png new file mode 100644 index 00000000..f2c54d80 Binary files /dev/null and b/main/manager-web/src/assets/xiaozhi-ai_de.png differ diff --git a/main/manager-web/src/assets/xiaozhi-ai_en.png b/main/manager-web/src/assets/xiaozhi-ai_en.png new file mode 100644 index 00000000..d35fd499 Binary files /dev/null and b/main/manager-web/src/assets/xiaozhi-ai_en.png differ diff --git a/main/manager-web/src/assets/xiaozhi-ai_vi.png b/main/manager-web/src/assets/xiaozhi-ai_vi.png new file mode 100644 index 00000000..a54913ad Binary files /dev/null and b/main/manager-web/src/assets/xiaozhi-ai_vi.png differ diff --git a/main/manager-web/src/assets/xiaozhi-ai_zh_CN.png b/main/manager-web/src/assets/xiaozhi-ai_zh_CN.png new file mode 100644 index 00000000..ef3834af Binary files /dev/null and b/main/manager-web/src/assets/xiaozhi-ai_zh_CN.png differ diff --git a/main/manager-web/src/assets/xiaozhi-ai_zh_TW.png b/main/manager-web/src/assets/xiaozhi-ai_zh_TW.png new file mode 100644 index 00000000..ac5f59aa Binary files /dev/null and b/main/manager-web/src/assets/xiaozhi-ai_zh_TW.png differ diff --git a/main/manager-web/src/i18n/de.js b/main/manager-web/src/i18n/de.js index 4a9ab40c..22512e8c 100644 --- a/main/manager-web/src/i18n/de.js +++ b/main/manager-web/src/i18n/de.js @@ -713,7 +713,7 @@ export default { 'paramManagement.deleteFailed': 'Löschen fehlgeschlagen, bitte versuchen Sie es erneut', 'paramManagement.operationCancelled': 'Löschen abgebrochen', 'paramManagement.operationClosed': 'Operation geschlossen', - 'paramManagement.updateSuccess': 'Aktualisierung erfolgreich', + 'paramManagement.updateSuccess': 'Aktualisierung erfolgreich. Einige Konfigurationen werden erst nach Neustart des xiaozhi-server-Moduls wirksam.', 'paramManagement.addSuccess': 'Hinzufügen erfolgreich', 'paramManagement.updateFailed': 'Aktualisierung fehlgeschlagen', 'paramManagement.addFailed': 'Hinzufügen fehlgeschlagen', @@ -821,7 +821,7 @@ export default { 'modelConfig.rag': 'RAG', 'modelConfig.modelId': 'Modell-ID', 'modelConfig.modelName': 'Modellname', - 'modelConfig.provider': 'Anbieter', + 'modelConfig.provider': 'Schnittstellentyp', 'modelConfig.unknown': 'Unbekannt', 'modelConfig.isEnabled': 'Aktiviert', 'modelConfig.isDefault': 'Standard', @@ -852,7 +852,7 @@ export default { 'modelConfig.enableSuccess': 'Aktivieren erfolgreich', 'modelConfig.disableSuccess': 'Deaktivieren erfolgreich', 'modelConfig.operationFailed': 'Operation fehlgeschlagen', - 'modelConfig.setDefaultSuccess': 'Standardmodell erfolgreich gesetzt', + 'modelConfig.setDefaultSuccess': 'Standardmodell erfolgreich gesetzt, bitte starten Sie das xiaozhi-server-Modul zeitnah manuell neu', 'modelConfig.itemsPerPage': '{items} Einträge/Seite', 'modelConfig.firstPage': 'Erste Seite', 'modelConfig.prevPage': 'Vorherige Seite', diff --git a/main/manager-web/src/i18n/en.js b/main/manager-web/src/i18n/en.js index 298d1e17..9bcc1f7d 100644 --- a/main/manager-web/src/i18n/en.js +++ b/main/manager-web/src/i18n/en.js @@ -713,7 +713,7 @@ export default { 'paramManagement.deleteFailed': 'Deletion failed, please try again', 'paramManagement.operationCancelled': 'Deletion cancelled', 'paramManagement.operationClosed': 'Operation closed', - 'paramManagement.updateSuccess': 'Update successful', + 'paramManagement.updateSuccess': 'Update successful. Some configurations will take effect only after restarting the xiaozhi-server module.', 'paramManagement.addSuccess': 'Add successful', 'paramManagement.updateFailed': 'Update failed', 'paramManagement.addFailed': 'Add failed', @@ -821,7 +821,7 @@ export default { 'modelConfig.rag': 'RAG', 'modelConfig.modelId': 'Model ID', 'modelConfig.modelName': 'Model Name', - 'modelConfig.provider': 'Provider', + 'modelConfig.provider': 'Interface Type', 'modelConfig.unknown': 'Unknown', 'modelConfig.isEnabled': 'Enabled', 'modelConfig.isDefault': 'Default', @@ -852,7 +852,7 @@ export default { 'modelConfig.enableSuccess': 'Enable successful', 'modelConfig.disableSuccess': 'Disable successful', 'modelConfig.operationFailed': 'Operation failed', - 'modelConfig.setDefaultSuccess': 'Set default model successful', + 'modelConfig.setDefaultSuccess': 'Set default model successful, please restart the xiaozhi-server module manually in time', 'modelConfig.itemsPerPage': '{items} items/page', 'modelConfig.firstPage': 'First Page', 'modelConfig.prevPage': 'Previous Page', diff --git a/main/manager-web/src/i18n/vi.js b/main/manager-web/src/i18n/vi.js index 9ca108ef..c9d9e1cf 100644 --- a/main/manager-web/src/i18n/vi.js +++ b/main/manager-web/src/i18n/vi.js @@ -713,7 +713,7 @@ export default { 'paramManagement.deleteFailed': 'Xóa thất bại, vui lòng thử lại', 'paramManagement.operationCancelled': 'Đã hủy xóa', 'paramManagement.operationClosed': 'Đã đóng thao tác', - 'paramManagement.updateSuccess': 'Cập nhật thành công', + 'paramManagement.updateSuccess': 'Cập nhật thành công. Một số cấu hình chỉ có hiệu lực sau khi khởi động lại mô-đun xiaozhi-server.', 'paramManagement.addSuccess': 'Thêm thành công', 'paramManagement.updateFailed': 'Cập nhật thất bại', 'paramManagement.addFailed': 'Thêm thất bại', @@ -821,7 +821,7 @@ export default { 'modelConfig.rag': 'RAG', 'modelConfig.modelId': 'ID mô hình', 'modelConfig.modelName': 'Tên mô hình', - 'modelConfig.provider': 'Nhà cung cấp', + 'modelConfig.provider': 'Loại giao diện', 'modelConfig.unknown': 'Không xác định', 'modelConfig.isEnabled': 'Đã bật', 'modelConfig.isDefault': 'Mặc định', @@ -852,7 +852,7 @@ export default { 'modelConfig.enableSuccess': 'Bật thành công', 'modelConfig.disableSuccess': 'Tắt thành công', 'modelConfig.operationFailed': 'Thao tác thất bại', - 'modelConfig.setDefaultSuccess': 'Đặt mô hình mặc định thành công', + 'modelConfig.setDefaultSuccess': 'Đặt mô hình mặc định thành công, vui lòng khởi động lại module xiaozhi-server thủ công kịp thời', 'modelConfig.itemsPerPage': '{items} mục/trang', 'modelConfig.firstPage': 'Trang đầu', 'modelConfig.prevPage': 'Trang trước', diff --git a/main/manager-web/src/i18n/zh_CN.js b/main/manager-web/src/i18n/zh_CN.js index 392b9a8b..44d8e400 100644 --- a/main/manager-web/src/i18n/zh_CN.js +++ b/main/manager-web/src/i18n/zh_CN.js @@ -713,7 +713,7 @@ export default { 'paramManagement.deleteFailed': '删除失败，请重试', 'paramManagement.operationCancelled': '已取消删除操作', 'paramManagement.operationClosed': '操作已关闭', - 'paramManagement.updateSuccess': '修改成功', + 'paramManagement.updateSuccess': '修改成功，部分配置需重启xiaozhi-server模块才生效', 'paramManagement.addSuccess': '新增成功', 'paramManagement.updateFailed': '更新失败', 'paramManagement.addFailed': '新增失败', @@ -821,7 +821,7 @@ export default { 'modelConfig.rag': '知识库', 'modelConfig.modelId': '模型ID', 'modelConfig.modelName': '模型名称', - 'modelConfig.provider': '提供商', + 'modelConfig.provider': '接口类型', 'modelConfig.unknown': '未知', 'modelConfig.isEnabled': '是否启用', 'modelConfig.isDefault': '是否默认', @@ -852,7 +852,7 @@ export default { 'modelConfig.enableSuccess': '启用成功', 'modelConfig.disableSuccess': '禁用成功', 'modelConfig.operationFailed': '操作失败', - 'modelConfig.setDefaultSuccess': '设置默认模型成功', + 'modelConfig.setDefaultSuccess': '设置默认模型成功，请及时手动重启xiaozhi-server模块', 'modelConfig.itemsPerPage': '{items}条/页', 'modelConfig.firstPage': '首页', 'modelConfig.prevPage': '上一页', diff --git a/main/manager-web/src/i18n/zh_TW.js b/main/manager-web/src/i18n/zh_TW.js index 69bacd67..ca5a5221 100644 --- a/main/manager-web/src/i18n/zh_TW.js +++ b/main/manager-web/src/i18n/zh_TW.js @@ -713,7 +713,7 @@ export default { 'paramManagement.deleteFailed': '刪除失敗，請重試', 'paramManagement.operationCancelled': '已取消刪除操作', 'paramManagement.operationClosed': '操作已關閉', - 'paramManagement.updateSuccess': '修改成功', + 'paramManagement.updateSuccess': '修改成功，部分配置需重啟xiaozhi-server模組才生效', 'paramManagement.addSuccess': '新增成功', 'paramManagement.updateFailed': '更新失敗', 'paramManagement.addFailed': '新增失敗', @@ -821,7 +821,7 @@ export default { 'modelConfig.rag': '知識庫', 'modelConfig.modelId': '模型ID', 'modelConfig.modelName': '模型名稱', - 'modelConfig.provider': '提供商', + 'modelConfig.provider': '接口類型', 'modelConfig.unknown': '未知', 'modelConfig.isEnabled': '是否啟用', 'modelConfig.isDefault': '是否默認', @@ -852,7 +852,7 @@ export default { 'modelConfig.enableSuccess': '啟用成功', 'modelConfig.disableSuccess': '禁用成功', 'modelConfig.operationFailed': '操作失敗', - 'modelConfig.setDefaultSuccess': '設置默認模型成功', + 'modelConfig.setDefaultSuccess': '設置默認模型成功，請及時手動重啟xiaozhi-server模組', 'modelConfig.itemsPerPage': '{items}條/頁', 'modelConfig.firstPage': '首頁', 'modelConfig.prevPage': '上一頁', diff --git a/main/xiaozhi-server/config.yaml b/main/xiaozhi-server/config.yaml index 2c005356..f2bbe9be 100644 --- a/main/xiaozhi-server/config.yaml +++ b/main/xiaozhi-server/config.yaml @@ -69,6 +69,9 @@ enable_greeting: true enable_stop_tts_notify: false # 说完话是否开启提示音，音效地址 stop_tts_notify_voice: "config/assets/tts_notify.mp3" +# 是否启用WebSocket心跳保活机制 +enable_websocket_ping: false + # TTS音频发送延迟配置 # tts_audio_send_delay: 控制音频包发送间隔 @@ -471,7 +474,6 @@ ASR: domain: slm # 识别领域，iat:日常用语，medical:医疗，finance:金融等 language: zh_cn # 语言，zh_cn:中文，en_us:英文 accent: mandarin # 方言，mandarin:普通话 - dwa: wpgs # 动态修正，wpgs:实时返回中间结果 # 调整音频处理参数以提高长语音识别质量 output_dir: tmp/ diff --git a/main/xiaozhi-server/config/config_loader.py b/main/xiaozhi-server/config/config_loader.py index c85510b8..e0220e33 100644 --- a/main/xiaozhi-server/config/config_loader.py +++ b/main/xiaozhi-server/config/config_loader.py @@ -68,6 +68,7 @@ async def get_config_from_api_async(config): "url": config["manager-api"].get("url", ""), "secret": config["manager-api"].get("secret", ""), } + auth_enabled = config_data.get("server", {}).get("auth", {}).get("enabled", False) # server的配置以本地为准 if config.get("server"): config_data["server"] = { @@ -77,6 +78,7 @@ async def get_config_from_api_async(config): "vision_explain": config["server"].get("vision_explain", ""), "auth_key": config["server"].get("auth_key", ""), } + config_data["server"]["auth"] = {"enabled": auth_enabled} # 如果服务器没有prompt_template，则从本地配置读取 if not config_data.get("prompt_template"): config_data["prompt_template"] = config.get("prompt_template") diff --git a/main/xiaozhi-server/config/logger.py b/main/xiaozhi-server/config/logger.py index a0b64dd7..bf64893c 100644 --- a/main/xiaozhi-server/config/logger.py +++ b/main/xiaozhi-server/config/logger.py @@ -5,7 +5,7 @@ from config.config_loader import load_config from config.settings import check_config_file from datetime import datetime -SERVER_VERSION = "0.8.9" +SERVER_VERSION = "0.8.10" _logger_initialized = False diff --git a/main/xiaozhi-server/core/api/base_handler.py b/main/xiaozhi-server/core/api/base_handler.py index 7330185e..db277543 100644 --- a/main/xiaozhi-server/core/api/base_handler.py +++ b/main/xiaozhi-server/core/api/base_handler.py @@ -10,7 +10,15 @@ class BaseHandler: def _add_cors_headers(self, response): """添加CORS头信息""" response.headers["Access-Control-Allow-Headers"] = ( - "client-id, content-type, device-id" + "client-id, content-type, device-id, authorization" ) response.headers["Access-Control-Allow-Credentials"] = "true" response.headers["Access-Control-Allow-Origin"] = "*" + + async def handle_options(self, request): + """处理OPTIONS请求，添加CORS头信息""" + response = web.Response(body=b"", content_type="text/plain") + self._add_cors_headers(response) + # 添加允许的方法 + response.headers["Access-Control-Allow-Methods"] = "GET, POST, OPTIONS" + return response diff --git a/main/xiaozhi-server/core/api/ota_handler.py b/main/xiaozhi-server/core/api/ota_handler.py index b6c88dff..1e3b3bd0 100644 --- a/main/xiaozhi-server/core/api/ota_handler.py +++ b/main/xiaozhi-server/core/api/ota_handler.py @@ -3,15 +3,46 @@ import time import base64 import hashlib import hmac +import os +import re +import glob +from typing import Dict, List, Tuple from aiohttp import web from core.auth import AuthManager -from core.utils.util import get_local_ip +from core.utils.util import get_local_ip, get_vision_url from core.api.base_handler import BaseHandler TAG = __name__ +def _safe_basename(filename: str) -> str: + # Prevent directory traversal + return os.path.basename(filename) + + +def _parse_version(ver: str) -> Tuple[int, ...]: + # conservative parser: split by non-digit, keep numeric parts + parts = re.findall(r"\d+", ver) + return tuple(int(p) for p in parts) if parts else (0,) + + +def _is_higher_version(a: str, b: str) -> bool: + """Return True if version string a > b (semver-like numeric compare).""" + ta = _parse_version(a) + tb = _parse_version(b) + # compare tuple lexicographically, but allow different lengths + maxlen = max(len(ta), len(tb)) + for i in range(maxlen): + ai = ta[i] if i < len(ta) else 0 + bi = tb[i] if i < len(tb) else 0 + if ai > bi: + return True + if ai < bi: + return False + return False + + class OTAHandler(BaseHandler): def __init__(self, config: dict): super().__init__(config) @@ -23,6 +54,54 @@ class OTAHandler(BaseHandler): expire_seconds = auth_config.get("expire_seconds") self.auth = AuthManager(secret_key=secret_key, expire_seconds=expire_seconds) + # firmware storage + self.bin_dir = os.path.join(os.getcwd(), "data", "bin") + # cache structure: { 'updated_at': timestamp, 'ttl': seconds, 'files_by_model': { model: [(version, filename), ...] } } + self._bin_cache: Dict = { + "updated_at": 0, + "ttl": config.get("firmware_cache_ttl", 30), + "files_by_model": {}, + } + + def _refresh_bin_cache_if_needed(self): + now = int(time.time()) + ttl = int(self._bin_cache.get("ttl", 30)) + if now - int( + self._bin_cache.get("updated_at", 0) + ) < ttl and self._bin_cache.get("files_by_model"): + return + + files_by_model: Dict[str, List[Tuple[str, str]]] = {} + try: + if not os.path.isdir(self.bin_dir): + os.makedirs(self.bin_dir, exist_ok=True) + + # match files like model_1.2.3.bin (allow dots, dashes, underscores in model and version) + pattern = os.path.join(self.bin_dir, "*.bin") + for path in glob.glob(pattern): + fname = os.path.basename(path) + # filename format: {model}_{version}.bin + m = re.match(r"^(.+?)_([0-9][A-Za-z0-9\.\-_]*)\.bin$", fname) + if not m: + # skip files not conforming to naming rule + continue + model = m.group(1) + version = m.group(2) + files_by_model.setdefault(model, []).append((version, fname)) + + # sort versions for each model descending + for model, items in files_by_model.items(): + items.sort(key=lambda it: _parse_version(it[0]), reverse=True) + + self._bin_cache["files_by_model"] = files_by_model + self._bin_cache["updated_at"] = now + self.logger.bind(tag=TAG).info( + f"Firmware cache refreshed: {len(files_by_model)} models" + ) + except Exception as e: + self.logger.bind(tag=TAG).error(f"刷新固件缓存失败: {e}") + # keep previous cache if any + def generate_password_signature(self, content: str, secret_key: str) -> str: """生成MQTT密码签名 @@ -62,7 +141,14 @@ class OTAHandler(BaseHandler): return f"ws://{local_ip}:{port}/xiaozhi/v1/" async def handle_post(self, request): - """处理 OTA POST 请求""" + """处理 OTA POST 请求 + + This handler will: + - read device id/client id (as before) + - attempt to determine device model and current firmware version (prefer headers, fallback to body) + - check data/bin for newer firmware for that model + - if found a newer firmware, set firmware.url to the download endpoint + """ try: data = await request.text() self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}") @@ -81,33 +167,76 @@ class OTAHandler(BaseHandler): else: raise Exception("OTA请求ClientID为空") - data_json = json.loads(data) + data_json = {} + try: + data_json = json.loads(data) if data else {} + except Exception: + data_json = {} server_config = self.config["server"] - port = int(server_config.get("port", 8000)) + # Distinguish ports: + # - websocket_port is used to construct websocket URL (server["port"]) + # - http_port is used to construct OTA download URLs (server["http_port"]) + websocket_port = int(server_config.get("port", 8000)) + http_port = int(server_config.get("http_port", 8003)) local_ip = get_local_ip() + # Determine device model (prefer headers) + device_model = "" + # header candidates + for h in ("device-model", "device_model", "model"): + if h in request.headers: + device_model = request.headers.get(h, "").strip() + break + # body fallback + if not device_model: + try: + if "board" in data_json and isinstance(data_json["board"], dict): + device_model = data_json["board"].get("type", "") + elif "model" in data_json: + device_model = data_json.get("model", "") + except Exception: + device_model = "" + if not device_model: + device_model = "default" + + # Determine device current version (prefer headers) + device_version = "" + for h in ( + "device-version", + "device_version", + "firmware-version", + "app-version", + "application-version", + ): + if h in request.headers: + device_version = request.headers.get(h, "").strip() + break + if not device_version: + try: + device_version = data_json.get("application", {}).get("version", "") + except Exception: + device_version = "" + if not device_version: + device_version = "0.0.0" + return_json = { "server_time": { "timestamp": int(round(time.time() * 1000)), "timezone_offset": server_config.get("timezone_offset", 8) * 60, }, "firmware": { - "version": data_json["application"].get("version", "1.0.0"), + "version": device_version, "url": "", }, } + # existing mqtt/websocket logic (unchanged) mqtt_gateway_endpoint = server_config.get("mqtt_gateway") if mqtt_gateway_endpoint: # 如果配置了非空字符串 - # 尝试从请求数据中获取设备型号 - device_model = "default" + # 尝试从请求数据中获取设备型号（已解析 above） try: - if "device" in data_json and isinstance(data_json["device"], dict): - device_model = data_json["device"].get("model", "default") - elif "model" in data_json: - device_model = data_json["model"] group_id = f"GID_{device_model}".replace(":", "_").replace(" ", "_") except Exception as e: self.logger.bind(tag=TAG).error(f"获取设备型号失败: {e}") @@ -159,20 +288,61 @@ class OTAHandler(BaseHandler): token = self.auth.generate_token(client_id, device_id) else: token = self.auth.generate_token(client_id, device_id) + # NOTE: use websocket_port here return_json["websocket"] = { - "url": self._get_websocket_url(local_ip, port), + "url": self._get_websocket_url(local_ip, websocket_port), "token": token, } self.logger.bind(tag=TAG).info( f"未配置MQTT网关，为设备 {device_id} 下发WebSocket配置" ) - self.logger.bind(tag=TAG).info(f"{return_json}") + + # Now check firmware files for updates + try: + self._refresh_bin_cache_if_needed() + files_by_model = self._bin_cache.get("files_by_model", {}) + candidates = files_by_model.get(device_model, []) + + self.logger.bind(tag=TAG).info( + f"查找型号 {device_model} 的固件，找到 {len(candidates)} 个候选" + ) + + chosen_url = "" + chosen_version = device_version + + # candidates are sorted descending by version + for ver, fname in candidates: + if _is_higher_version(ver, device_version): + # build download url (only allow download via our download endpoint) + chosen_version = ver + # Use get_vision_url to get the base URL and replace the path + vision_url = get_vision_url(self.config) + # Replace the path from "/mcp/vision/explain" to "/xiaozhi/ota/download/{fname}" + chosen_url = vision_url.replace( + "/mcp/vision/explain", f"/xiaozhi/ota/download/{fname}" + ) + break + + if chosen_url: + return_json["firmware"]["version"] = chosen_version + return_json["firmware"]["url"] = chosen_url + self.logger.bind(tag=TAG).info( + f"为设备 {device_id} 下发固件 {chosen_version} [如果地址前缀有误，请检查配置文件中的server.vision_explain]-> {chosen_url} " + ) + else: + self.logger.bind(tag=TAG).info( + f"设备 {device_id} 固件已是最新: {device_version}" + ) + + except Exception as e: + self.logger.bind(tag=TAG).error(f"检查固件版本时出错: {e}") response = web.Response( text=json.dumps(return_json, separators=(",", ":")), content_type="application/json", ) except Exception as e: + self.logger.bind(tag=TAG).error(f"OTA POST处理异常: {e}") return_json = {"success": False, "message": "request error."} response = web.Response( text=json.dumps(return_json, separators=(",", ":")), @@ -187,8 +357,9 @@ class OTAHandler(BaseHandler): try: server_config = self.config["server"] local_ip = get_local_ip() - port = int(server_config.get("port", 8000)) - websocket_url = self._get_websocket_url(local_ip, port) + # use websocket port for websocket URL + websocket_port = int(server_config.get("port", 8000)) + websocket_url = self._get_websocket_url(local_ip, websocket_port) message = f"OTA接口运行正常，向设备发送的websocket地址是：{websocket_url}" response = web.Response(text=message, content_type="text/plain") except Exception as e: @@ -197,3 +368,48 @@ class OTAHandler(BaseHandler): finally: self._add_cors_headers(response) return response + + async def handle_download(self, request): + """ + 下载固件接口 + URL: /xiaozhi/ota/download/{filename} + - 只允许下载 data/bin 目录下的 .bin 文件 + - filename 必须是 basename 且匹配安全的模式 + """ + try: + fname = request.match_info.get("filename", "") + if not fname: + raise web.HTTPBadRequest(text="filename required") + + # sanitize + fname = _safe_basename(fname) + # pattern: allow letters, numbers, dot, underscore, dash + if not re.match(r"^[A-Za-z0-9\.\-_]+\.bin$", fname): + raise web.HTTPBadRequest(text="invalid filename") + + file_path = os.path.join(self.bin_dir, fname) + # ensure realpath is under bin_dir + file_real = os.path.realpath(file_path) + bin_dir_real = os.path.realpath(self.bin_dir) + if ( + not file_real.startswith(bin_dir_real + os.sep) + and file_real != bin_dir_real + ): + raise web.HTTPForbidden(text="forbidden") + + if not os.path.isfile(file_real): + raise web.HTTPNotFound(text="file not found") + + # use FileResponse to stream file + resp = web.FileResponse(path=file_real) + except web.HTTPError as e: + resp = e + except Exception as e: + self.logger.bind(tag=TAG).error(f"固件下载异常: {e}") + resp = web.Response(text="download error", status=500) + finally: + try: + self._add_cors_headers(resp) + except Exception: + pass + return resp diff --git a/main/xiaozhi-server/core/api/vision_handler.py b/main/xiaozhi-server/core/api/vision_handler.py index a4817a84..28f48753 100644 --- a/main/xiaozhi-server/core/api/vision_handler.py +++ b/main/xiaozhi-server/core/api/vision_handler.py @@ -2,6 +2,7 @@ import json import copy from aiohttp import web from config.logger import setup_logging +from core.api.base_handler import BaseHandler from core.utils.util import get_vision_url, is_valid_image_file from core.utils.vllm import create_instance from config.config_loader import get_private_config_from_api @@ -16,10 +17,9 @@ TAG = __name__ MAX_FILE_SIZE = 5 * 1024 * 1024 -class VisionHandler: +class VisionHandler(BaseHandler): def __init__(self, config: dict): - self.config = config - self.logger = setup_logging() + super().__init__(config) # 初始化认证工具 self.auth = AuthToken(config["server"]["auth_key"]) @@ -172,11 +172,3 @@ class VisionHandler: finally: self._add_cors_headers(response) return response - - def _add_cors_headers(self, response): - """添加CORS头信息""" - response.headers["Access-Control-Allow-Headers"] = ( - "client-id, content-type, device-id" - ) - response.headers["Access-Control-Allow-Credentials"] = "true" - response.headers["Access-Control-Allow-Origin"] = "*" diff --git a/main/xiaozhi-server/core/connection.py b/main/xiaozhi-server/core/connection.py index 43c77f93..ad8b4385 100644 --- a/main/xiaozhi-server/core/connection.py +++ b/main/xiaozhi-server/core/connection.py @@ -69,6 +69,7 @@ class ConnectionHandler: self.server = server # 保存server实例的引用 self.need_bind = False # 是否需要绑定设备 + self.bind_completed_event = asyncio.Event() self.bind_code = None # 绑定设备的验证码 self.last_bind_prompt_time = 0 # 上次播放绑定提示的时间戳(秒) self.bind_prompt_interval = 60 # 绑定提示播放间隔(秒) @@ -268,30 +269,43 @@ class ConnectionHandler: f"保存记忆后关闭连接失败: {close_error}" ) + async def _discard_message_with_bind_prompt(self): + """丢弃消息并检查是否需要播放绑定提示""" + current_time = time.time() + # 检查是否需要播放绑定提示 + if current_time - self.last_bind_prompt_time >= self.bind_prompt_interval: + self.last_bind_prompt_time = current_time + # 复用现有的绑定提示逻辑 + from core.handle.receiveAudioHandle import check_bind_device + + asyncio.create_task(check_bind_device(self)) + async def _route_message(self, message): """消息路由""" + # 检查是否已经获取到真实的绑定状态 + if not self.bind_completed_event.is_set(): + # 还没有获取到真实状态，等待直到获取到真实状态或超时 + try: + await asyncio.wait_for(self.bind_completed_event.wait(), timeout=1) + except asyncio.TimeoutError: + # 超时仍未获取到真实状态，丢弃消息 + await self._discard_message_with_bind_prompt() + return + + # 已经获取到真实状态，检查是否需要绑定 + if self.need_bind: + # 需要绑定，丢弃消息 + await self._discard_message_with_bind_prompt() + return + + # 不需要绑定，继续处理消息 + if isinstance(message, str): await handleTextMessage(self, message) elif isinstance(message, bytes): if self.vad is None or self.asr is None: return - # 未绑定设备直接丢弃所有音频，不进行ASR处理 - if self.need_bind: - current_time = time.time() - # 检查是否需要播放绑定提示 - if ( - current_time - self.last_bind_prompt_time - >= self.bind_prompt_interval - ): - self.last_bind_prompt_time = current_time - # 复用现有的绑定提示逻辑 - from core.handle.receiveAudioHandle import check_bind_device - - asyncio.create_task(check_bind_device(self)) - # 直接丢弃音频，不进行ASR处理 - return - # 处理来自MQTT网关的音频包 if self.conn_from_mqtt_gateway and len(message) >= 16: handled = await self._process_mqtt_audio_message(message) @@ -415,6 +429,15 @@ class ConnectionHandler: def _initialize_components(self): try: + if self.tts is None: + self.tts = self._initialize_tts() + # 打开语音合成通道 + asyncio.run_coroutine_threadsafe( + self.tts.open_audio_channels(self), self.loop + ) + if self.need_bind: + self.bind_completed_event.set() + return self.selected_module_str = build_module_string( self.config.get("selected_module", {}) ) @@ -438,17 +461,10 @@ class ConnectionHandler: # 初始化声纹识别 self._initialize_voiceprint() - # 打开语音识别通道 asyncio.run_coroutine_threadsafe( self.asr.open_audio_channels(self), self.loop ) - if self.tts is None: - self.tts = self._initialize_tts() - # 打开语音合成通道 - asyncio.run_coroutine_threadsafe( - self.tts.open_audio_channels(self), self.loop - ) """加载记忆""" self._initialize_memory() @@ -463,6 +479,7 @@ class ConnectionHandler: self.logger.bind(tag=TAG).error(f"实例化组件失败: {e}") def _init_prompt_enhancement(self): + # 更新上下文信息 self.prompt_manager.update_context_info(self, self.client_ip) enhanced_prompt = self.prompt_manager.build_enhanced_prompt( @@ -498,7 +515,11 @@ class ConnectionHandler: def _initialize_asr(self): """初始化ASR""" - if self._asr.interface_type == InterfaceType.LOCAL: + if ( + self._asr is not None + and hasattr(self._asr, "interface_type") + and self._asr.interface_type == InterfaceType.LOCAL + ): # 如果公共ASR是本地服务，则直接返回 # 因为本地一个实例ASR，可以被多个连接共享 asr = self._asr @@ -538,6 +559,8 @@ class ConnectionHandler: async def _initialize_private_config_async(self): """从接口异步获取差异化配置（异步版本，不阻塞主循环）""" if not self.read_config_from_api: + self.need_bind = False + self.bind_completed_event.set() return try: begin_time = time.time() @@ -550,6 +573,8 @@ class ConnectionHandler: self.logger.bind(tag=TAG).info( f"{time.time() - begin_time} 秒，异步获取差异化配置成功: {json.dumps(filter_sensitive_info(private_config), ensure_ascii=False)}" ) + self.need_bind = False + self.bind_completed_event.set() except DeviceNotFoundException as e: self.need_bind = True private_config = {} @@ -1173,6 +1198,11 @@ class ConnectionHandler: except queue.Empty: break + # 重置音频流控器（取消后台任务并清空队列） + if hasattr(self, "audio_rate_controller") and self.audio_rate_controller: + self.audio_rate_controller.reset() + self.logger.bind(tag=TAG).debug("已重置音频流控器") + self.logger.bind(tag=TAG).debug( f"清理结束: TTS队列大小={self.tts.tts_text_queue.qsize()}, 音频队列大小={self.tts.tts_audio_queue.qsize()}" ) diff --git a/main/xiaozhi-server/core/handle/helloHandle.py b/main/xiaozhi-server/core/handle/helloHandle.py index 248c5f73..a4220d5a 100644 --- a/main/xiaozhi-server/core/handle/helloHandle.py +++ b/main/xiaozhi-server/core/handle/helloHandle.py @@ -101,7 +101,7 @@ async def checkWakeupWords(conn, text): } # 获取音频数据 - opus_packets = audio_to_data(response.get("file_path")) + opus_packets = await audio_to_data(response.get("file_path"), use_cache=False) # 播放唤醒词回复 conn.client_abort = False diff --git a/main/xiaozhi-server/core/handle/receiveAudioHandle.py b/main/xiaozhi-server/core/handle/receiveAudioHandle.py index 4eaf9ab1..8879564f 100644 --- a/main/xiaozhi-server/core/handle/receiveAudioHandle.py +++ b/main/xiaozhi-server/core/handle/receiveAudioHandle.py @@ -123,7 +123,7 @@ async def max_out_size(conn): text = "不好意思，我现在有点事情要忙，明天这个时候我们再聊，约好了哦！明天不见不散，拜拜！" await send_stt_message(conn, text) file_path = "config/assets/max_output_size.wav" - opus_packets = audio_to_data(file_path) + opus_packets = await audio_to_data(file_path) conn.tts.tts_audio_queue.put((SentenceType.LAST, opus_packets, text)) conn.close_after_chat = True @@ -142,7 +142,7 @@ async def check_bind_device(conn): # 播放提示音 music_path = "config/assets/bind_code.wav" - opus_packets = audio_to_data(music_path) + opus_packets = await audio_to_data(music_path) conn.tts.tts_audio_queue.put((SentenceType.FIRST, opus_packets, text)) # 逐个播放数字 @@ -150,7 +150,7 @@ async def check_bind_device(conn): try: digit = conn.bind_code[i] num_path = f"config/assets/bind_code/{digit}.wav" - num_packets = audio_to_data(num_path) + num_packets = await audio_to_data(num_path) conn.tts.tts_audio_queue.put((SentenceType.MIDDLE, num_packets, None)) except Exception as e: conn.logger.bind(tag=TAG).error(f"播放数字音频失败: {e}") @@ -162,5 +162,5 @@ async def check_bind_device(conn): text = f"没有找到该设备的版本信息，请正确配置 OTA地址，然后重新编译固件。" await send_stt_message(conn, text) music_path = "config/assets/bind_not_found.wav" - opus_packets = audio_to_data(music_path) + opus_packets = await audio_to_data(music_path) conn.tts.tts_audio_queue.put((SentenceType.LAST, opus_packets, text)) diff --git a/main/xiaozhi-server/core/handle/reportHandle.py b/main/xiaozhi-server/core/handle/reportHandle.py index 973ebb38..053e8f2e 100644 --- a/main/xiaozhi-server/core/handle/reportHandle.py +++ b/main/xiaozhi-server/core/handle/reportHandle.py @@ -10,7 +10,6 @@ TTS上报功能已集成到ConnectionHandler类中。 """ import time -import gc import opuslib_next from config.manage_api_client import report as manage_report diff --git a/main/xiaozhi-server/core/handle/sendAudioHandle.py b/main/xiaozhi-server/core/handle/sendAudioHandle.py index ea952fbe..8198dfb0 100644 --- a/main/xiaozhi-server/core/handle/sendAudioHandle.py +++ b/main/xiaozhi-server/core/handle/sendAudioHandle.py @@ -7,6 +7,10 @@ from core.providers.tts.dto.dto import SentenceType from core.utils.audioRateController import AudioRateController TAG = __name__ +# 音频帧时长（毫秒） +AUDIO_FRAME_DURATION = 60 +# 预缓冲包数量，直接发送以减少延迟 +PRE_BUFFER_COUNT = 5 async def sendAudioMessage(conn, sentenceType, audios, text): @@ -16,7 +20,19 @@ async def sendAudioMessage(conn, sentenceType, audios, text): await send_tts_message(conn, "start", None) if sentenceType == SentenceType.FIRST: - await send_tts_message(conn, "sentence_start", text) + # 同一句子的后续消息加入流控队列，其他情况立即发送 + if ( + hasattr(conn, "audio_rate_controller") + and conn.audio_rate_controller + and getattr(conn, "audio_flow_control", {}).get("sentence_id") + == conn.sentence_id + ): + conn.audio_rate_controller.add_message( + lambda: send_tts_message(conn, "sentence_start", text) + ) + else: + # 新句子或流控器未初始化，立即发送 + await send_tts_message(conn, "sentence_start", text) await sendAudio(conn, audios) # 发送句子开始消息 @@ -31,6 +47,29 @@ async def sendAudioMessage(conn, sentenceType, audios, text): await conn.close() +async def _wait_for_audio_completion(conn): + """ + 等待音频队列清空并等待预缓冲包播放完成 + + Args: + conn: 连接对象 + """ + if hasattr(conn, "audio_rate_controller") and conn.audio_rate_controller: + rate_controller = conn.audio_rate_controller + conn.logger.bind(tag=TAG).debug( + f"等待音频发送完成，队列中还有 {len(rate_controller.queue)} 个包" + ) + await rate_controller.queue_empty_event.wait() + + # 等待预缓冲包播放完成 + # 前N个包直接发送，增加2个网络抖动包，需要额外等待它们在客户端播放完成 + frame_duration_ms = rate_controller.frame_duration + pre_buffer_playback_time = (PRE_BUFFER_COUNT + 2) * frame_duration_ms / 1000.0 + await asyncio.sleep(pre_buffer_playback_time) + + conn.logger.bind(tag=TAG).debug("音频发送完成") + + async def _send_to_mqtt_gateway(conn, opus_packet, timestamp, sequence): """ 发送带16字节头部的opus数据包给mqtt_gateway @@ -53,139 +92,133 @@ async def _send_to_mqtt_gateway(conn, opus_packet, timestamp, sequence): await conn.websocket.send(complete_packet) -# 播放音频 - 使用 AudioRateController 进行精确流控 -async def sendAudio(conn, audios, frame_duration=60): +async def sendAudio(conn, audios, frame_duration=AUDIO_FRAME_DURATION): """ 发送音频包，使用 AudioRateController 进行精确的流量控制 Args: conn: 连接对象 audios: 单个opus包(bytes) 或 opus包列表 - frame_duration: 帧时长（毫秒），默认60ms - - 改进点： - 1. 使用单一时间基准，避免累积误差 - 2. 每次检查队列时重新计算 elapsed_ms，更精准 - 3. 支持高并发而不产生时间偏差 + frame_duration: 帧时长（毫秒），默认使用全局常量AUDIO_FRAME_DURATION """ if audios is None or len(audios) == 0: return - # 获取发送延迟配置 send_delay = conn.config.get("tts_audio_send_delay", -1) / 1000.0 + is_single_packet = isinstance(audios, bytes) - if isinstance(audios, bytes): - # 单个 opus 包处理 - await _sendAudio_single(conn, audios, send_delay, frame_duration) - else: - # 音频列表处理（如文件型音频） - await _sendAudio_list(conn, audios, send_delay, frame_duration) + # 初始化或获取 RateController + rate_controller, flow_control = _get_or_create_rate_controller( + conn, frame_duration, is_single_packet + ) + + # 统一转换为列表处理 + audio_list = [audios] if is_single_packet else audios + + # 发送音频包 + await _send_audio_with_rate_control( + conn, audio_list, rate_controller, flow_control, send_delay + ) -async def _sendAudio_single(conn, opus_packet, send_delay, frame_duration=60): +def _get_or_create_rate_controller(conn, frame_duration, is_single_packet): """ - 发送单个 opus 包 - 使用 AudioRateController 进行流控 + 获取或创建 RateController 和 flow_control + + Args: + conn: 连接对象 + frame_duration: 帧时长 + is_single_packet: 是否单包模式（True: TTS流式单包, False: 批量包） + + Returns: + (rate_controller, flow_control) """ - # 重置流控状态，第一次读取和会话发生转变时 - if not hasattr(conn, "audio_rate_controller") or conn.audio_flow_control.get("sentence_id") != conn.sentence_id: - if hasattr(conn, "audio_rate_controller"): - conn.audio_rate_controller.reset() - else: + # 判断是否需要重置：单包模式且 sentence_id 变化，或者控制器不存在 + need_reset = ( + is_single_packet + and getattr(conn, "audio_flow_control", {}).get("sentence_id") + != conn.sentence_id + ) or not hasattr(conn, "audio_rate_controller") + + if need_reset: + # 创建或获取 rate_controller + if not hasattr(conn, "audio_rate_controller"): conn.audio_rate_controller = AudioRateController(frame_duration) + else: conn.audio_rate_controller.reset() + # 初始化 flow_control conn.audio_flow_control = { "packet_count": 0, "sequence": 0, "sentence_id": conn.sentence_id, } - if conn.client_abort: - return + # 启动后台发送循环 + _start_background_sender( + conn, conn.audio_rate_controller, conn.audio_flow_control + ) - conn.last_activity_time = time.time() * 1000 + return conn.audio_rate_controller, conn.audio_flow_control - rate_controller = conn.audio_rate_controller - flow_control = conn.audio_flow_control - packet_count = flow_control["packet_count"] - # 预缓冲：前5个包直接发送，不做延迟 - pre_buffer_count = 5 +def _start_background_sender(conn, rate_controller, flow_control): + """ + 启动后台发送循环任务 - if packet_count < pre_buffer_count or send_delay > 0: - # 预缓冲阶段或固定延迟模式，直接发送 - await _do_send_audio(conn, opus_packet, flow_control, frame_duration) + Args: + conn: 连接对象 + rate_controller: 速率控制器 + flow_control: 流控状态 + """ + + async def send_callback(packet): + # 检查是否应该中止 + if conn.client_abort: + raise asyncio.CancelledError("客户端已中止") + + conn.last_activity_time = time.time() * 1000 + await _do_send_audio(conn, packet, flow_control) conn.client_is_speaking = True - if send_delay > 0 and packet_count >= pre_buffer_count: - await asyncio.sleep(send_delay) - else: - # 使用流控器进行精确的速率控制 - rate_controller.add_audio(opus_packet) - - async def send_callback(packet): - await _do_send_audio(conn, packet, flow_control, frame_duration) - - await rate_controller.check_queue(send_callback) - conn.client_is_speaking = True - - # 更新流控状态 - flow_control["packet_count"] += 1 - flow_control["sequence"] += 1 + # 使用 start_sending 启动后台循环 + rate_controller.start_sending(send_callback) -async def _sendAudio_list(conn, audios, send_delay, frame_duration=60): +async def _send_audio_with_rate_control( + conn, audio_list, rate_controller, flow_control, send_delay +): """ - 发送音频列表（如文件型音频） + 使用 rate_controller 发送音频包 + + Args: + conn: 连接对象 + audio_list: 音频包列表 + rate_controller: 速率控制器 + flow_control: 流控状态 + send_delay: 固定延迟（秒），-1表示使用动态流控 """ - if not audios: - return - - rate_controller = AudioRateController(frame_duration) - rate_controller.reset() - - flow_control = { - "packet_count": 0, - "sequence": 0, - } - - # 预缓冲：前5个包直接发送 - pre_buffer_frames = min(5, len(audios)) - for i in range(pre_buffer_frames): + for packet in audio_list: if conn.client_abort: return - await _do_send_audio(conn, audios[i], flow_control, frame_duration) - conn.client_is_speaking = True - - remaining_audios = audios[pre_buffer_frames:] - - # 处理剩余音频帧 - for i, opus_packet in enumerate(remaining_audios): - if conn.client_abort: - break conn.last_activity_time = time.time() * 1000 - if send_delay > 0: + # 预缓冲：前N个包直接发送 + if flow_control["packet_count"] < PRE_BUFFER_COUNT: + await _do_send_audio(conn, packet, flow_control) + conn.client_is_speaking = True + elif send_delay > 0: # 固定延迟模式 await asyncio.sleep(send_delay) - else: - # 使用流控器进行精确延迟 - rate_controller.add_audio(opus_packet) - - async def send_callback(packet): - await _do_send_audio(conn, packet, flow_control, frame_duration) - - await rate_controller.check_queue(send_callback) + await _do_send_audio(conn, packet, flow_control) conn.client_is_speaking = True - continue - - await _do_send_audio(conn, opus_packet, flow_control, frame_duration) - conn.client_is_speaking = True + else: + # 动态流控模式：仅添加到队列，由后台循环负责发送 + rate_controller.add_audio(packet) -async def _do_send_audio(conn, opus_packet, flow_control, frame_duration=60): +async def _do_send_audio(conn, opus_packet, flow_control): """ 执行实际的音频发送 """ @@ -222,8 +255,10 @@ async def send_tts_message(conn, state, text=None): stop_tts_notify_voice = conn.config.get( "stop_tts_notify_voice", "config/assets/tts_notify.mp3" ) - audios = audio_to_data(stop_tts_notify_voice, is_opus=True) + audios = await audio_to_data(stop_tts_notify_voice, is_opus=True) await sendAudio(conn, audios) + # 等待所有音频包发送完成 + await _wait_for_audio_completion(conn) # 清除服务端讲话状态 conn.clearSpeakStatus() diff --git a/main/xiaozhi-server/core/handle/textHandler/listenMessageHandler.py b/main/xiaozhi-server/core/handle/textHandler/listenMessageHandler.py index 97286dfe..c71649ed 100644 --- a/main/xiaozhi-server/core/handle/textHandler/listenMessageHandler.py +++ b/main/xiaozhi-server/core/handle/textHandler/listenMessageHandler.py @@ -1,12 +1,14 @@ import time +import asyncio from typing import Dict, Any -from core.handle.receiveAudioHandle import handleAudioMessage, startToChat +from core.handle.receiveAudioHandle import startToChat from core.handle.reportHandle import enqueue_asr_report from core.handle.sendAudioHandle import send_stt_message, send_tts_message from core.handle.textMessageHandler import TextMessageHandler from core.handle.textMessageType import TextMessageType from core.utils.util import remove_punctuation_and_length +from core.providers.asr.dto.dto import InterfaceType TAG = __name__ @@ -29,8 +31,18 @@ class ListenTextMessageHandler(TextMessageHandler): elif msg_json["state"] == "stop": conn.client_have_voice = True conn.client_voice_stop = True - if len(conn.asr_audio) > 0: - await handleAudioMessage(conn, b"") + if conn.asr.interface_type == InterfaceType.STREAM: + # 流式模式下，发送结束请求 + asyncio.create_task(conn.asr._send_stop_request()) + else: + # 非流式模式：直接触发ASR识别 + if len(conn.asr_audio) > 0: + asr_audio_task = conn.asr_audio.copy() + conn.asr_audio.clear() + conn.reset_vad_states() + + if len(asr_audio_task) > 0: + await conn.asr.handle_voice_stop(conn, asr_audio_task) elif msg_json["state"] == "detect": conn.client_have_voice = False conn.asr_audio.clear() @@ -57,6 +69,7 @@ class ListenTextMessageHandler(TextMessageHandler): enqueue_asr_report(conn, "嘿，你好呀", []) await startToChat(conn, "嘿，你好呀") else: + conn.just_woken_up = True # 上报纯文字数据（复用ASR上报功能，但不提供音频数据） enqueue_asr_report(conn, original_text, []) # 否则需要LLM对文字内容进行答复 diff --git a/main/xiaozhi-server/core/handle/textHandler/pingMessageHandler.py b/main/xiaozhi-server/core/handle/textHandler/pingMessageHandler.py new file mode 100644 index 00000000..f3af85eb --- /dev/null +++ b/main/xiaozhi-server/core/handle/textHandler/pingMessageHandler.py @@ -0,0 +1,45 @@ +import json +import time +from typing import Dict, Any + +from core.handle.textMessageHandler import TextMessageHandler +from core.handle.textMessageType import TextMessageType + +TAG = __name__ + + +class PingMessageHandler(TextMessageHandler): + """Ping消息处理器，用于保持WebSocket连接""" + + @property + def message_type(self) -> TextMessageType: + return TextMessageType.PING + + async def handle(self, conn, msg_json: Dict[str, Any]) -> None: + """ + 处理PING消息，发送PONG响应 + 消息格式：{"type": "ping"} + Args: + conn: WebSocket连接对象 + msg_json: PING消息的JSON数据 + """ + # 检查是否启用了WebSocket心跳功能 + enable_websocket_ping = conn.config.get("enable_websocket_ping", False) + if not enable_websocket_ping: + conn.logger.debug(f"WebSocket心跳功能未启用，忽略PING消息") + return + + try: + conn.logger.debug(f"收到PING消息，发送PONG响应") + conn.last_activity_time = time.time() * 1000 + # 构造PONG响应消息 + pong_message = { + "type": "pong", + "timestamp": time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()), + } + + # 发送PONG响应 + await conn.websocket.send(json.dumps(pong_message)) + + except Exception as e: + conn.logger.error(f"处理PING消息时发生错误: {e}") diff --git a/main/xiaozhi-server/core/handle/textMessageHandlerRegistry.py b/main/xiaozhi-server/core/handle/textMessageHandlerRegistry.py index e90d7231..65e9474f 100644 --- a/main/xiaozhi-server/core/handle/textMessageHandlerRegistry.py +++ b/main/xiaozhi-server/core/handle/textMessageHandlerRegistry.py @@ -7,6 +7,7 @@ from core.handle.textHandler.listenMessageHandler import ListenTextMessageHandle from core.handle.textHandler.mcpMessageHandler import McpTextMessageHandler from core.handle.textMessageHandler import TextMessageHandler from core.handle.textHandler.serverMessageHandler import ServerTextMessageHandler +from core.handle.textHandler.pingMessageHandler import PingMessageHandler TAG = __name__ @@ -27,6 +28,7 @@ class TextMessageHandlerRegistry: IotTextMessageHandler(), McpTextMessageHandler(), ServerTextMessageHandler(), + PingMessageHandler(), ] for handler in handlers: diff --git a/main/xiaozhi-server/core/handle/textMessageType.py b/main/xiaozhi-server/core/handle/textMessageType.py index 53e71b71..bd04d289 100644 --- a/main/xiaozhi-server/core/handle/textMessageType.py +++ b/main/xiaozhi-server/core/handle/textMessageType.py @@ -9,3 +9,4 @@ class TextMessageType(Enum): IOT = "iot" MCP = "mcp" SERVER = "server" + PING = "ping" diff --git a/main/xiaozhi-server/core/http_server.py b/main/xiaozhi-server/core/http_server.py index edbdf1fe..feb96f3b 100644 --- a/main/xiaozhi-server/core/http_server.py +++ b/main/xiaozhi-server/core/http_server.py @@ -33,38 +33,60 @@ class SimpleHttpServer: return f"ws://{local_ip}:{port}/xiaozhi/v1/" async def start(self): - server_config = self.config["server"] - read_config_from_api = self.config.get("read_config_from_api", False) - host = server_config.get("ip", "0.0.0.0") - port = int(server_config.get("http_port", 8003)) + try: + server_config = self.config["server"] + read_config_from_api = self.config.get("read_config_from_api", False) + host = server_config.get("ip", "0.0.0.0") + port = int(server_config.get("http_port", 8003)) - if port: - app = web.Application() + if port: + app = web.Application() - if not read_config_from_api: - # 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口 + if not read_config_from_api: + # 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口 + app.add_routes( + [ + web.get("/xiaozhi/ota/", self.ota_handler.handle_get), + web.post("/xiaozhi/ota/", self.ota_handler.handle_post), + web.options( + "/xiaozhi/ota/", self.ota_handler.handle_options + ), + # 下载接口，仅提供 data/bin/*.bin 下载 + web.get( + "/xiaozhi/ota/download/{filename}", + self.ota_handler.handle_download, + ), + web.options( + "/xiaozhi/ota/download/{filename}", + self.ota_handler.handle_options, + ), + ] + ) + # 添加路由 app.add_routes( [ - web.get("/xiaozhi/ota/", self.ota_handler.handle_get), - web.post("/xiaozhi/ota/", self.ota_handler.handle_post), - web.options("/xiaozhi/ota/", self.ota_handler.handle_post), + web.get("/mcp/vision/explain", self.vision_handler.handle_get), + web.post( + "/mcp/vision/explain", self.vision_handler.handle_post + ), + web.options( + "/mcp/vision/explain", self.vision_handler.handle_options + ), ] ) - # 添加路由 - app.add_routes( - [ - web.get("/mcp/vision/explain", self.vision_handler.handle_get), - web.post("/mcp/vision/explain", self.vision_handler.handle_post), - web.options("/mcp/vision/explain", self.vision_handler.handle_post), - ] - ) - # 运行服务 - runner = web.AppRunner(app) - await runner.setup() - site = web.TCPSite(runner, host, port) - await site.start() + # 运行服务 + runner = web.AppRunner(app) + await runner.setup() + site = web.TCPSite(runner, host, port) + await site.start() - # 保持服务运行 - while True: - await asyncio.sleep(3600) # 每隔 1 小时检查一次 + # 保持服务运行 + while True: + await asyncio.sleep(3600) # 每隔 1 小时检查一次 + except Exception as e: + self.logger.bind(tag=TAG).error(f"HTTP服务器启动失败: {e}") + import traceback + + self.logger.bind(tag=TAG).error(f"错误堆栈: {traceback.format_exc()}") + raise diff --git a/main/xiaozhi-server/core/providers/asr/aliyun_stream.py b/main/xiaozhi-server/core/providers/asr/aliyun_stream.py index 179685bd..4ce588a7 100644 --- a/main/xiaozhi-server/core/providers/asr/aliyun_stream.py +++ b/main/xiaozhi-server/core/providers/asr/aliyun_stream.py @@ -5,12 +5,9 @@ import hmac import base64 import hashlib import asyncio -import gc import requests import websockets import opuslib_next -import random -from typing import Optional, Tuple, List from urllib import parse from datetime import datetime from config.logger import setup_logging @@ -140,13 +137,13 @@ class ASRProvider(ASRProviderBase): conn.asr_audio.append(audio) conn.asr_audio = conn.asr_audio[-10:] - # 只在有声音且没有连接时建立连接 - if audio_have_voice and not self.is_processing: + # 只在有声音且没有连接时建立连接（排除正在停止的情况） + if audio_have_voice and not self.is_processing and not self.asr_ws: try: await self._start_recognition(conn) except Exception as e: logger.bind(tag=TAG).error(f"开始识别失败: {str(e)}") - await self._cleanup(conn) + await self._cleanup() return if self.asr_ws and self.is_processing and self.server_ready: @@ -186,10 +183,8 @@ class ASRProvider(ASRProviderBase): "header": { "namespace": "SpeechTranscriber", "name": "StartTranscription", - "status": 20000000, "message_id": uuid.uuid4().hex, "task_id": self.task_id, - "status_text": "Gateway:SUCCESS:Success.", "appkey": self.appkey }, "payload": { @@ -208,18 +203,21 @@ class ASRProvider(ASRProviderBase): async def _forward_results(self, conn): """转发识别结果""" try: - while self.asr_ws and not conn.stop_event.is_set(): + while not conn.stop_event.is_set(): try: response = await asyncio.wait_for(self.asr_ws.recv(), timeout=1.0) result = json.loads(response) - + header = result.get("header", {}) payload = result.get("payload", {}) message_name = header.get("name", "") status = header.get("status", 0) - + if status != 20000000: - if status in [40000004, 40010004]: # 连接超时或客户端断开 + if status == 40010004: + logger.bind(tag=TAG).warning(f"请在服务端响应完成后再关闭链接，状态码: {status}") + break + if status in [40000004, 40010003]: # 连接超时或客户端断开 logger.bind(tag=TAG).warning(f"连接问题，状态码: {status}") break elif status in [40270002, 40270003]: # 音频问题 @@ -228,12 +226,12 @@ class ASRProvider(ASRProviderBase): else: logger.bind(tag=TAG).error(f"识别错误，状态码: {status}, 消息: {header.get('status_text', '')}") continue - + # 收到TranscriptionStarted表示服务器准备好接收音频数据 if message_name == "TranscriptionStarted": self.server_ready = True logger.bind(tag=TAG).debug("服务器已准备，开始发送缓存音频...") - + # 发送缓存音频 if conn.asr_audio: for cached_audio in conn.asr_audio[-10:]: @@ -244,89 +242,89 @@ class ASRProvider(ASRProviderBase): logger.bind(tag=TAG).warning(f"发送缓存音频失败: {e}") break continue - - if message_name == "TranscriptionResultChanged": - # 中间结果 - text = payload.get("result", "") - if text: - self.text = text elif message_name == "SentenceEnd": - # 最终结果 + # 句子结束（每个句子都会触发） text = payload.get("result", "") if text: - self.text = text - conn.reset_vad_states() - # 传递缓存的音频数据 - audio_data = getattr(conn, 'asr_audio_for_voiceprint', []) - await self.handle_voice_stop(conn, audio_data) - # 清空缓存 - conn.asr_audio_for_voiceprint = [] - break - elif message_name == "TranscriptionCompleted": - # 识别完成 - self.is_processing = False - break - + logger.bind(tag=TAG).info(f"识别到文本: {text}") + + # 手动模式下累积识别结果 + if conn.client_listen_mode == "manual": + if self.text: + self.text += text + else: + self.text = text + + # 手动模式下，只有在收到stop信号后才触发处理（仅处理一次） + if conn.client_voice_stop: + audio_data = getattr(conn, 'asr_audio_for_voiceprint', []) + if len(audio_data) > 0: + logger.bind(tag=TAG).debug("收到最终识别结果，触发处理") + await self.handle_voice_stop(conn, audio_data) + # 清理音频缓存 + conn.asr_audio.clear() + conn.reset_vad_states() + break + else: + # 自动模式下直接覆盖 + self.text = text + conn.reset_vad_states() + audio_data = getattr(conn, 'asr_audio_for_voiceprint', []) + await self.handle_voice_stop(conn, audio_data) + break + except asyncio.TimeoutError: - continue - except websockets.exceptions.ConnectionClosed: + logger.bind(tag=TAG).error("接收结果超时") + break + except websockets.ConnectionClosed: + logger.bind(tag=TAG).info("ASR服务连接已关闭") + self.is_processing = False break except Exception as e: logger.bind(tag=TAG).error(f"处理结果失败: {str(e)}") break - + except Exception as e: logger.bind(tag=TAG).error(f"结果转发失败: {str(e)}") finally: - await self._cleanup(conn) + # 清理连接的音频缓存 + await self._cleanup() + if conn: + if hasattr(conn, 'asr_audio_for_voiceprint'): + conn.asr_audio_for_voiceprint = [] + if hasattr(conn, 'asr_audio'): + conn.asr_audio = [] - async def _cleanup(self, conn): - """清理资源""" - logger.bind(tag=TAG).debug(f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}") - - # 清理连接的音频缓存 - if conn and hasattr(conn, 'asr_audio_for_voiceprint'): - conn.asr_audio_for_voiceprint = [] - - # 判断是否需要发送终止请求 - should_stop = self.is_processing or self.server_ready - - # 发送停止识别请求 - if self.asr_ws and should_stop: + async def _send_stop_request(self): + """发送停止识别请求（不关闭连接）""" + if self.asr_ws: try: + # 先停止音频发送 + self.is_processing = False + stop_msg = { "header": { "namespace": "SpeechTranscriber", "name": "StopTranscription", - "status": 20000000, "message_id": uuid.uuid4().hex, "task_id": self.task_id, - "status_text": "Client:Stop", "appkey": self.appkey } } - logger.bind(tag=TAG).debug("正在发送ASR终止请求") + logger.bind(tag=TAG).debug("停止识别请求已发送") await self.asr_ws.send(json.dumps(stop_msg, ensure_ascii=False)) - await asyncio.sleep(0.1) - logger.bind(tag=TAG).debug("ASR终止请求已发送") except Exception as e: - logger.bind(tag=TAG).error(f"ASR终止请求发送失败: {e}") - - # 状态重置（在终止请求发送后） + logger.bind(tag=TAG).error(f"发送停止识别请求失败: {e}") + + async def _cleanup(self): + """清理资源（关闭连接）""" + logger.bind(tag=TAG).debug(f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}") + + # 状态重置 self.is_processing = False self.server_ready = False logger.bind(tag=TAG).debug("ASR状态已重置") - # 清理任务 - if self.forward_task and not self.forward_task.done(): - self.forward_task.cancel() - try: - await asyncio.wait_for(self.forward_task, timeout=1.0) - except Exception as e: - logger.bind(tag=TAG).debug(f"forward_task取消异常: {e}") - finally: - self.forward_task = None - # 关闭连接 if self.asr_ws: try: @@ -337,7 +335,10 @@ class ASRProvider(ASRProviderBase): logger.bind(tag=TAG).error(f"关闭WebSocket连接失败: {e}") finally: self.asr_ws = None - + + # 清理任务引用 + self.forward_task = None + logger.bind(tag=TAG).debug("ASR会话清理完成") async def speech_to_text(self, opus_data, session_id, audio_format): diff --git a/main/xiaozhi-server/core/providers/asr/base.py b/main/xiaozhi-server/core/providers/asr/base.py index 7e36de5c..6c6969fd 100644 --- a/main/xiaozhi-server/core/providers/asr/base.py +++ b/main/xiaozhi-server/core/providers/asr/base.py @@ -9,8 +9,6 @@ import asyncio import traceback import threading import opuslib_next -import concurrent.futures -import gc from abc import ABC, abstractmethod from config.logger import setup_logging from typing import Optional, Tuple, List @@ -54,121 +52,89 @@ class ASRProviderBase(ABC): # 接收音频 async def receive_audio(self, conn, audio, audio_have_voice): - if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime": - have_voice = audio_have_voice + if conn.client_listen_mode == "manual": + # 手动模式：缓存音频用于ASR识别 + conn.asr_audio.append(audio) else: - have_voice = conn.client_have_voice - - conn.asr_audio.append(audio) - if not have_voice and not conn.client_have_voice: - conn.asr_audio = conn.asr_audio[-10:] - return + # 自动/实时模式：使用VAD检测 + have_voice = audio_have_voice - if conn.client_voice_stop: - asr_audio_task = conn.asr_audio.copy() - conn.asr_audio.clear() - conn.reset_vad_states() + conn.asr_audio.append(audio) + if not have_voice and not conn.client_have_voice: + conn.asr_audio = conn.asr_audio[-10:] + return - if len(asr_audio_task) > 15: - await self.handle_voice_stop(conn, asr_audio_task) + # 自动模式下通过VAD检测到语音停止时触发识别 + if conn.client_voice_stop: + asr_audio_task = conn.asr_audio.copy() + conn.asr_audio.clear() + conn.reset_vad_states() + + if len(asr_audio_task) > 15: + await self.handle_voice_stop(conn, asr_audio_task) # 处理语音停止 async def handle_voice_stop(self, conn, asr_audio_task: List[bytes]): """并行处理ASR和声纹识别""" try: total_start_time = time.monotonic() - + # 准备音频数据 if conn.audio_format == "pcm": pcm_data = asr_audio_task else: pcm_data = self.decode_opus(asr_audio_task) - + combined_pcm_data = b"".join(pcm_data) - + # 预先准备WAV数据 wav_data = None if conn.voiceprint_provider and combined_pcm_data: wav_data = self._pcm_to_wav(combined_pcm_data) - + # 定义ASR任务 - def run_asr(): - start_time = time.monotonic() - try: - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) - try: - result = loop.run_until_complete( - self.speech_to_text(asr_audio_task, conn.session_id, conn.audio_format) - ) - end_time = time.monotonic() - logger.bind(tag=TAG).debug(f"ASR耗时: {end_time - start_time:.3f}s") - return result - finally: - loop.close() - except Exception as e: - end_time = time.monotonic() - logger.bind(tag=TAG).error(f"ASR失败: {e}") - return ("", None) - - # 定义声纹识别任务 - def run_voiceprint(): - if not wav_data: - return None - try: - loop = asyncio.new_event_loop() - asyncio.set_event_loop(loop) - try: - # 使用连接的声纹识别提供者 - result = loop.run_until_complete( - conn.voiceprint_provider.identify_speaker(wav_data, conn.session_id) - ) - return result - finally: - loop.close() - except Exception as e: - logger.bind(tag=TAG).error(f"声纹识别失败: {e}") - return None - - # 使用线程池执行器并行运行 - with concurrent.futures.ThreadPoolExecutor(max_workers=2) as thread_executor: - asr_future = thread_executor.submit(run_asr) - - if conn.voiceprint_provider and wav_data: - voiceprint_future = thread_executor.submit(run_voiceprint) - - # 等待两个线程都完成 - asr_result = asr_future.result(timeout=15) - voiceprint_result = voiceprint_future.result(timeout=15) - - results = {"asr": asr_result, "voiceprint": voiceprint_result} - else: - asr_result = asr_future.result(timeout=15) - results = {"asr": asr_result, "voiceprint": None} - - - # 处理结果 - raw_text, _ = results.get("asr", ("", None)) - speaker_name = results.get("voiceprint", None) - - # 记录识别结果 + asr_task = self.speech_to_text(asr_audio_task, conn.session_id, conn.audio_format) + + if conn.voiceprint_provider and wav_data: + voiceprint_task = conn.voiceprint_provider.identify_speaker(wav_data, conn.session_id) + # 并发等待两个结果 + asr_result, voiceprint_result = await asyncio.gather( + asr_task, voiceprint_task, return_exceptions=True + ) + else: + asr_result = await asr_task + voiceprint_result = None + + # 记录识别结果 - 检查是否为异常 + if isinstance(asr_result, Exception): + logger.bind(tag=TAG).error(f"ASR识别失败: {asr_result}") + raw_text = "" + else: + raw_text, _ = asr_result + + if isinstance(voiceprint_result, Exception): + logger.bind(tag=TAG).error(f"声纹识别失败: {voiceprint_result}") + speaker_name = "" + else: + speaker_name = voiceprint_result + if raw_text: logger.bind(tag=TAG).info(f"识别文本: {raw_text}") if speaker_name: logger.bind(tag=TAG).info(f"识别说话人: {speaker_name}") - + # 性能监控 total_time = time.monotonic() - total_start_time logger.bind(tag=TAG).debug(f"总处理耗时: {total_time:.3f}s") - + # 检查文本长度 text_len, _ = remove_punctuation_and_length(raw_text) self.stop_ws_connection() - + if text_len > 0: # 构建包含说话人信息的JSON字符串 enhanced_text = self._build_enhanced_text(raw_text, speaker_name) - + # 使用自定义模块进行上报 await startToChat(conn, enhanced_text) enqueue_asr_report(conn, enhanced_text, asr_audio_task) diff --git a/main/xiaozhi-server/core/providers/asr/doubao_stream.py b/main/xiaozhi-server/core/providers/asr/doubao_stream.py index 2d179b86..0c1cf727 100644 --- a/main/xiaozhi-server/core/providers/asr/doubao_stream.py +++ b/main/xiaozhi-server/core/providers/asr/doubao_stream.py @@ -4,7 +4,6 @@ import uuid import asyncio import websockets import opuslib_next -import gc from core.providers.asr.base import ASRProviderBase from config.logger import setup_logging from core.providers.asr.dto.dto import InterfaceType @@ -19,8 +18,6 @@ class ASRProvider(ASRProviderBase): self.interface_type = InterfaceType.STREAM self.config = config self.text = "" - self.max_retries = 3 - self.retry_delay = 2 self.decoder = opuslib_next.Decoder(16000, 1) self.asr_ws = None self.forward_task = None @@ -57,14 +54,13 @@ class ASRProvider(ASRProviderBase): async def receive_audio(self, conn, audio, audio_have_voice): conn.asr_audio.append(audio) conn.asr_audio = conn.asr_audio[-10:] - # 存储音频数据 if not hasattr(conn, 'asr_audio_for_voiceprint'): conn.asr_audio_for_voiceprint = [] conn.asr_audio_for_voiceprint.append(audio) - + # 当没有音频数据时处理完整语音片段 - if not audio and len(conn.asr_audio_for_voiceprint) > 0: + if conn.client_listen_mode != "manual" and not audio and len(conn.asr_audio_for_voiceprint) > 0: await self.handle_voice_stop(conn, conn.asr_audio_for_voiceprint) conn.asr_audio_for_voiceprint = [] @@ -180,6 +176,7 @@ class ASRProvider(ASRProviderBase): payload.get("audio_info", {}).get("duration", 0) > 2000 and not utterances and not payload["result"].get("text") + and conn.client_listen_mode != "manual" ): logger.bind(tag=TAG).error(f"识别文本：空") self.text = "" @@ -188,15 +185,44 @@ class ASRProvider(ASRProviderBase): await self.handle_voice_stop(conn, audio_data) break + # 专门处理没有文本的识别结果（手动模式下可能已经识别完成但是没松按键） + elif not payload["result"].get("text") and not utterances: + if conn.client_listen_mode == "manual" and conn.client_voice_stop and len(audio_data) > 0: + logger.bind(tag=TAG).debug("消息结束收到停止信号，触发处理") + await self.handle_voice_stop(conn, audio_data) + # 清理音频缓存 + conn.asr_audio.clear() + conn.reset_vad_states() + break + for utterance in utterances: if utterance.get("definite", False): - self.text = utterance["text"] + current_text = utterance["text"] logger.bind(tag=TAG).info( - f"识别到文本: {self.text}" + f"识别到文本: {current_text}" ) - conn.reset_vad_states() - if len(audio_data) > 15: # 确保有足够音频数据 - await self.handle_voice_stop(conn, audio_data) + + # 手动模式下累积识别结果 + if conn.client_listen_mode == "manual": + if self.text: + self.text += current_text + else: + self.text = current_text + + # 在接收消息中途时收到停止信号 + if conn.client_voice_stop and len(audio_data) > 0: + logger.bind(tag=TAG).debug("消息中途收到停止信号，触发处理") + await self.handle_voice_stop(conn, audio_data) + # 清理音频缓存 + conn.asr_audio.clear() + conn.reset_vad_states() + break + else: + # 自动模式下直接覆盖 + self.text = current_text + conn.reset_vad_states() + if len(audio_data) > 15: # 确保有足够音频数据 + await self.handle_voice_stop(conn, audio_data) break elif "error" in payload: error_msg = payload.get("error", "未知错误") @@ -228,8 +254,6 @@ class ASRProvider(ASRProviderBase): conn.asr_audio_for_voiceprint = [] if hasattr(conn, 'asr_audio'): conn.asr_audio = [] - if hasattr(conn, 'has_valid_voice'): - conn.has_valid_voice = False def stop_ws_connection(self): if self.asr_ws: @@ -237,6 +261,20 @@ class ASRProvider(ASRProviderBase): self.asr_ws = None self.is_processing = False + async def _send_stop_request(self): + """发送最后一个音频帧以通知服务器结束""" + if self.asr_ws: + try: + # 发送结束标记的音频帧（gzip压缩的空数据） + empty_payload = gzip.compress(b"") + last_audio_request = bytearray(self.generate_last_audio_default_header()) + last_audio_request.extend(len(empty_payload).to_bytes(4, "big")) + last_audio_request.extend(empty_payload) + await self.asr_ws.send(last_audio_request) + logger.bind(tag=TAG).debug("已发送结束音频帧") + except Exception as e: + logger.bind(tag=TAG).debug(f"发送结束音频帧时出错: {e}") + def construct_request(self, reqid): req = { "app": { @@ -388,5 +426,3 @@ class ASRProvider(ASRProviderBase): conn.asr_audio_for_voiceprint = [] if hasattr(conn, 'asr_audio'): conn.asr_audio = [] - if hasattr(conn, 'has_valid_voice'): - conn.has_valid_voice = False diff --git a/main/xiaozhi-server/core/providers/asr/fun_local.py b/main/xiaozhi-server/core/providers/asr/fun_local.py index 217f17ff..2fd49c36 100644 --- a/main/xiaozhi-server/core/providers/asr/fun_local.py +++ b/main/xiaozhi-server/core/providers/asr/fun_local.py @@ -1,14 +1,16 @@ -import time import os -import sys import io +import sys +import time +import shutil import psutil +import asyncio + from config.logger import setup_logging from typing import Optional, Tuple, List -from core.providers.asr.base import ASRProviderBase from funasr import AutoModel from funasr.utils.postprocess_utils import rich_transcription_postprocess -import shutil +from core.providers.asr.base import ASRProviderBase from core.providers.asr.dto.dto import InterfaceType TAG = __name__ @@ -90,16 +92,17 @@ class ASRProvider(ASRProviderBase): else: file_path = self.save_audio_to_file(pcm_data, session_id) - # 语音识别 + # 语音识别 - 使用线程池避免阻塞事件循环 start_time = time.time() - result = self.model.generate( + result = await asyncio.to_thread( + self.model.generate, input=combined_pcm_data, cache={}, language="auto", use_itn=True, batch_size_s=60, ) - text = rich_transcription_postprocess(result[0]["text"]) + text = await asyncio.to_thread(rich_transcription_postprocess, result[0]["text"]) logger.bind(tag=TAG).debug( f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {text}" ) diff --git a/main/xiaozhi-server/core/providers/asr/qwen3_asr_flash.py b/main/xiaozhi-server/core/providers/asr/qwen3_asr_flash.py index 84fe1979..51c84a37 100644 --- a/main/xiaozhi-server/core/providers/asr/qwen3_asr_flash.py +++ b/main/xiaozhi-server/core/providers/asr/qwen3_asr_flash.py @@ -1,8 +1,5 @@ import os -import json -import asyncio import tempfile -import difflib from typing import Optional, Tuple, List import dashscope from config.logger import setup_logging @@ -16,7 +13,8 @@ logger = setup_logging() class ASRProvider(ASRProviderBase): def __init__(self, config: dict, delete_audio_file: bool): super().__init__() - self.interface_type = InterfaceType.STREAM + # 音频文件上传类型，流式文本识别输出 + self.interface_type = InterfaceType.NON_STREAM """Qwen3-ASR-Flash ASR初始化""" # 配置参数 @@ -130,27 +128,11 @@ class ASRProvider(ASRProviderBase): # 处理流式响应 full_text = "" - last_text = "" # 用于存储上一个文本片段 for chunk in response: try: text = chunk["output"]["choices"][0]["message"].content[0]["text"] - # 标准化文本片段（去除首尾空格） - normalized_text = text.strip() - # 只有当新文本片段与上一个不同时才处理 - if normalized_text != last_text: - # 提取新增的文本部分 - # 通过比较当前文本和上一个文本，找到新增的部分 - if normalized_text.startswith(last_text): - # 如果当前文本以最后一个文本开头，则新增部分是两者的差集 - new_part = normalized_text[len(last_text):] - else: - # 如果不以最后一个文本开头，说明识别结果发生了较大变化，直接使用当前文本 - new_part = normalized_text - - # 将新增部分添加到完整文本中 - full_text += new_part - last_text = normalized_text - # 这里可以实时处理文本片段，例如通过回调函数 + # 更新为最新的完整文本 + full_text = text.strip() except: pass diff --git a/main/xiaozhi-server/core/providers/asr/xunfei_stream.py b/main/xiaozhi-server/core/providers/asr/xunfei_stream.py index 6a895ebc..b7e7886e 100644 --- a/main/xiaozhi-server/core/providers/asr/xunfei_stream.py +++ b/main/xiaozhi-server/core/providers/asr/xunfei_stream.py @@ -35,9 +35,6 @@ class ASRProvider(ASRProviderBase): self.forward_task = None self.is_processing = False self.server_ready = False - self.last_frame_sent = False # 标记是否已发送最终帧 - self.best_text = "" # 保存最佳识别结果 - self.has_final_result = False # 标记是否收到最终识别结果 # 讯飞配置 self.app_id = config.get("app_id") @@ -52,7 +49,6 @@ class ASRProvider(ASRProviderBase): "domain": config.get("domain", "slm"), "language": config.get("language", "zh_cn"), "accent": config.get("accent", "mandarin"), - "dwa": config.get("dwa", "wpgs"), "result": {"encoding": "utf8", "compress": "raw", "format": "plain"}, } @@ -116,7 +112,7 @@ class ASRProvider(ASRProviderBase): await self._start_recognition(conn) except Exception as e: logger.bind(tag=TAG).error(f"建立ASR连接失败: {str(e)}") - await self._cleanup(conn) + await self._cleanup() return # 发送当前音频数据 @@ -126,7 +122,7 @@ class ASRProvider(ASRProviderBase): await self._send_audio_frame(pcm_frame, STATUS_CONTINUE_FRAME) except Exception as e: logger.bind(tag=TAG).warning(f"发送音频数据时发生错误: {e}") - await self._cleanup(conn) + await self._cleanup() async def _start_recognition(self, conn): """开始识别会话""" @@ -136,6 +132,10 @@ class ASRProvider(ASRProviderBase): ws_url = self.create_url() logger.bind(tag=TAG).info(f"正在连接ASR服务: {ws_url[:50]}...") + # 如果为手动模式,设置超时时长为一分钟 + if conn.client_listen_mode == "manual": + self.iat_params["eos"] = 60000 + self.asr_ws = await websockets.connect( ws_url, max_size=1000000000, @@ -146,8 +146,6 @@ class ASRProvider(ASRProviderBase): logger.bind(tag=TAG).info("ASR WebSocket连接已建立") self.server_ready = False - self.last_frame_sent = False - self.best_text = "" self.forward_task = asyncio.create_task(self._forward_results(conn)) # 发送首帧音频 @@ -196,23 +194,12 @@ class ASRProvider(ASRProviderBase): await self.asr_ws.send(json.dumps(frame_data, ensure_ascii=False)) - # 标记是否发送了最终帧 - if status == STATUS_LAST_FRAME: - self.last_frame_sent = True - logger.bind(tag=TAG).info("标记最终帧已发送") - async def _forward_results(self, conn): """转发识别结果""" try: - while self.asr_ws and not conn.stop_event.is_set(): - # 获取当前连接的音频数据 - audio_data = getattr(conn, "asr_audio_for_voiceprint", []) + while not conn.stop_event.is_set(): try: - # 如果已发送最终帧，增加超时时间等待完整结果 - timeout = 3.0 if self.last_frame_sent else 30.0 - response = await asyncio.wait_for( - self.asr_ws.recv(), timeout=timeout - ) + response = await asyncio.wait_for(self.asr_ws.recv(), timeout=60) result = json.loads(response) logger.bind(tag=TAG).debug(f"收到ASR结果: {result}") @@ -236,144 +223,27 @@ class ASRProvider(ASRProviderBase): # 解码base64文本 decoded_text = base64.b64decode(text_data).decode("utf-8") text_json = json.loads(decoded_text) - # 提取文本内容 text_ws = text_json.get("ws", []) - result_text = "" for i in text_ws: for j in i.get("cw", []): w = j.get("w", "") - result_text += w + self.text += w - # 更新识别文本 - 实时更新策略 - # 只检查是否为空字符串，不再过滤任何标点符号 - # 这样可以确保所有识别到的内容，包括标点符号都能被实时更新 - if result_text and result_text.strip(): - # 实时更新：正常情况下都更新，提高响应速度 - should_update = True - - # 保存最佳文本 - # 1. 如果是识别完成状态或最终帧后收到的结果，优先保存 - # 2. 否则保存最长的有意义文本 - # 取消对标点符号的过滤，只检查是否为空 - # 这样可以保留所有识别到的内容，包括各种标点符号 - is_valid_text = len(result_text.strip()) > 0 - - if ( - self.last_frame_sent or status == 2 - ) and is_valid_text: - self.best_text = result_text - self.has_final_result = True # 标记已收到最终结果 - logger.bind(tag=TAG).debug( - f"保存最终识别结果: {self.best_text}" - ) - elif ( - len(result_text) > len(self.best_text) - and is_valid_text - and not self.has_final_result - ): - self.best_text = result_text - logger.bind(tag=TAG).debug( - f"保存中间最佳文本: {self.best_text}" - ) - - # 如果已发送最终帧，只过滤空文本 - if self.last_frame_sent: - # 只拒绝完全空的结果 - if not result_text.strip(): - should_update = False - logger.bind(tag=TAG).warning( - f"最终帧后拒绝空文本" - ) - - if should_update: - # 处理流式识别结果，避免简单替换导致内容丢失 - # 1. 如果是中间状态（非最终帧后），可能需要替换为更完整的识别 - # 2. 如果是最终帧后收到的结果，可能是对前面文本的补充 - if self.last_frame_sent: - # 最终帧后收到的结果可能是标点符号等补充内容 - # 检查是否需要合并文本而不是替换 - # 如果当前文本是纯标点而前面已有内容，应该追加而不是替换 - if len( - self.text - ) > 0 and result_text.strip() in [ - "。", - ".", - "?", - "？", - "!", - "！", - ",", - "，", - ";", - "；", - ]: - # 对于标点符号，追加到现有文本后 - self.text = ( - self.text.rstrip().rstrip("。.") - + result_text - ) - else: - # 其他情况保持替换逻辑 - self.text = result_text - else: - # 中间状态替换为新的识别结果 - self.text = result_text - - logger.bind(tag=TAG).info( - f"实时更新识别文本: {self.text} (最终帧已发送: {self.last_frame_sent})" - ) - - # 识别完成，但如果还没发送最终帧，继续等待 if status == 2: - logger.bind(tag=TAG).info( - f"识别完成状态已到达，当前识别文本: {self.text}" - ) - - # 如果还没发送最终帧，继续等待 - if not self.last_frame_sent: - logger.bind(tag=TAG).info( - "识别完成但最终帧未发送，继续等待..." - ) - continue - - # 已发送最终帧且收到完成状态，使用最佳策略选择最终结果 - # 优先使用识别完成状态下的最新结果，而不是仅仅基于长度 - if self.best_text: - # 如果当前文本是在最终帧发送后或识别完成状态下收到的，优先使用 - if ( - self.last_frame_sent or status == 2 - ) and self.text.strip(): - logger.bind(tag=TAG).info( - f"使用完成状态下的最新识别结果: {self.text}" - ) - elif len(self.best_text) > len(self.text): - logger.bind(tag=TAG).info( - f"使用更长的最佳文本作为最终结果: {self.text} -> {self.best_text}" - ) - self.text = self.best_text - - logger.bind(tag=TAG).info(f"获取到最终完整文本: {self.text}") + if conn.client_listen_mode == "manual": + audio_data = getattr(conn, 'asr_audio_for_voiceprint', []) + if len(audio_data) > 0: + logger.bind(tag=TAG).debug("收到最终识别结果，触发处理") + await self.handle_voice_stop(conn, audio_data) + # 清理音频缓存 + conn.asr_audio.clear() conn.reset_vad_states() - if len(audio_data) > 15: # 确保有足够音频数据 - # 准备处理结果 - pass break except asyncio.TimeoutError: - if self.last_frame_sent: - # 超时时也使用最佳文本 - if self.best_text and len(self.best_text) > len(self.text): - logger.bind(tag=TAG).info( - f"超时，使用最佳文本: {self.text} -> {self.best_text}" - ) - self.text = self.best_text - logger.bind(tag=TAG).info( - f"最终帧后超时，使用结果: {self.text}" - ) - break - # 如果还没发送最终帧，继续等待 - continue + logger.bind(tag=TAG).error("接收结果超时") + break except websockets.ConnectionClosed: logger.bind(tag=TAG).info("ASR服务连接已关闭") self.is_processing = False @@ -390,17 +260,15 @@ class ASRProvider(ASRProviderBase): if hasattr(e, "__cause__") and e.__cause__: logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}") finally: - if self.asr_ws: - await self.asr_ws.close() - self.asr_ws = None - self.is_processing = False + # 清理连接资源 + await self._cleanup() + + # 清理连接的音频缓存 if conn: if hasattr(conn, "asr_audio_for_voiceprint"): conn.asr_audio_for_voiceprint = [] if hasattr(conn, "asr_audio"): conn.asr_audio = [] - if hasattr(conn, "has_valid_voice"): - conn.has_valid_voice = False async def handle_voice_stop(self, conn, asr_audio_task: List[bytes]): """处理语音停止，发送最后一帧并处理识别结果""" @@ -408,22 +276,13 @@ class ASRProvider(ASRProviderBase): # 先发送最后一帧表示音频结束 if self.asr_ws and self.is_processing: try: - # 取最后一个有效的音频帧作为最后一帧数据 - last_frame = b"" - if asr_audio_task: - last_audio = asr_audio_task[-1] - last_frame = self.decoder.decode(last_audio, 960) - await self._send_audio_frame(last_frame, STATUS_LAST_FRAME) - logger.bind(tag=TAG).info("已发送最后一帧") + await self._send_audio_frame(b"", STATUS_LAST_FRAME) + logger.bind(tag=TAG).debug(f"已发送停止请求") - # 发送最终帧后，给_forward_results适当时间处理最终结果 await asyncio.sleep(0.25) - - logger.bind(tag=TAG).info(f"准备处理最终识别结果: {self.text}") except Exception as e: - logger.bind(tag=TAG).error(f"发送最后一帧失败: {e}") + logger.bind(tag=TAG).error(f"发送停止请求失败: {e}") - # 调用父类的handle_voice_stop方法处理识别结果 await super().handle_voice_stop(conn, asr_audio_task) except Exception as e: logger.bind(tag=TAG).error(f"处理语音停止失败: {e}") @@ -437,40 +296,27 @@ class ASRProvider(ASRProviderBase): self.asr_ws = None self.is_processing = False - async def _cleanup(self, conn): - """清理资源""" - logger.bind(tag=TAG).info( + async def _send_stop_request(self): + """发送停止识别请求（不关闭连接）""" + if self.asr_ws: + try: + # 先停止音频发送 + self.is_processing = False + await self._send_audio_frame(b"", STATUS_LAST_FRAME) + logger.bind(tag=TAG).debug("已发送停止请求") + except Exception as e: + logger.bind(tag=TAG).error(f"发送停止请求失败: {e}") + + async def _cleanup(self): + """清理资源（关闭连接）""" + logger.bind(tag=TAG).debug( f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}" ) - # 发送最后一帧 - if self.asr_ws and self.is_processing: - try: - await self._send_audio_frame(b"", STATUS_LAST_FRAME) - await asyncio.sleep(0.1) - logger.bind(tag=TAG).info("已发送最后一帧") - except Exception as e: - logger.bind(tag=TAG).error(f"发送最后一帧失败: {e}") - # 状态重置 self.is_processing = False self.server_ready = False - self.last_frame_sent = False - self.best_text = "" - self.has_final_result = False - logger.bind(tag=TAG).info("ASR状态已重置") - - # 清理任务 - if self.forward_task and not self.forward_task.done(): - self.forward_task.cancel() - try: - await asyncio.wait_for(self.forward_task, timeout=1.0) - except asyncio.CancelledError: - pass - except Exception as e: - logger.bind(tag=TAG).debug(f"forward_task取消异常: {e}") - finally: - self.forward_task = None + logger.bind(tag=TAG).debug("ASR状态已重置") # 关闭连接 if self.asr_ws: @@ -483,16 +329,10 @@ class ASRProvider(ASRProviderBase): finally: self.asr_ws = None - # 清理连接的音频缓存 - if conn: - if hasattr(conn, "asr_audio_for_voiceprint"): - conn.asr_audio_for_voiceprint = [] - if hasattr(conn, "asr_audio"): - conn.asr_audio = [] - if hasattr(conn, "has_valid_voice"): - conn.has_valid_voice = False + # 清理任务引用 + self.forward_task = None - logger.bind(tag=TAG).info("ASR会话清理完成") + logger.bind(tag=TAG).debug("ASR会话清理完成") async def speech_to_text(self, opus_data, session_id, audio_format): """获取识别结果""" @@ -513,7 +353,7 @@ class ASRProvider(ASRProviderBase): pass self.forward_task = None self.is_processing = False - + # 显式释放decoder资源 if hasattr(self, 'decoder') and self.decoder is not None: try: @@ -530,5 +370,3 @@ class ASRProvider(ASRProviderBase): conn.asr_audio_for_voiceprint = [] if hasattr(conn, "asr_audio"): conn.asr_audio = [] - if hasattr(conn, "has_valid_voice"): - conn.has_valid_voice = False diff --git a/main/xiaozhi-server/core/providers/vad/silero.py b/main/xiaozhi-server/core/providers/vad/silero.py index ae51b88f..81215681 100644 --- a/main/xiaozhi-server/core/providers/vad/silero.py +++ b/main/xiaozhi-server/core/providers/vad/silero.py @@ -2,7 +2,6 @@ import time import numpy as np import torch import opuslib_next -import gc from config.logger import setup_logging from core.providers.vad.base import VADProviderBase @@ -45,6 +44,10 @@ class VADProvider(VADProviderBase): pass def is_vad(self, conn, opus_packet): + # 手动模式：直接返回True，不进行实时VAD检测，所有音频都缓存 + if conn.client_listen_mode == "manual": + return True + try: pcm_frame = self.decoder.decode(opus_packet, 960) conn.client_audio_buffer.extend(pcm_frame) # 将新数据加入缓冲区 diff --git a/main/xiaozhi-server/core/utils/audioRateController.py b/main/xiaozhi-server/core/utils/audioRateController.py index b404e6f1..01d71b05 100644 --- a/main/xiaozhi-server/core/utils/audioRateController.py +++ b/main/xiaozhi-server/core/utils/audioRateController.py @@ -1,5 +1,6 @@ import time import asyncio +from collections import deque from config.logger import setup_logging TAG = __name__ @@ -10,12 +11,6 @@ class AudioRateController: """ 音频速率控制器 - 按照60ms帧时长精确控制音频发送 解决高并发下的时间累积误差问题 - - 关键改进： - 1. 单一时间基准（start_timestamp 只初始化一次） - 2. 每次检查队列时重新计算 elapsed_ms，避免累积误差 - 3. 分离"检查时间"和"发送"两个操作 - 4. 支持高并发而不产生延迟 """ def __init__(self, frame_duration=60): @@ -24,35 +19,52 @@ class AudioRateController: frame_duration: 单个音频帧时长（毫秒），默认60ms """ self.frame_duration = frame_duration - self.queue = [] + self.queue = deque() self.play_position = 0 # 虚拟播放位置（毫秒） self.start_timestamp = None # 开始时间戳（只读，不修改） self.pending_send_task = None self.logger = logger + self.queue_empty_event = asyncio.Event() # 队列清空事件 + self.queue_empty_event.set() # 初始为空状态 + self.queue_has_data_event = asyncio.Event() # 队列数据事件 def reset(self): """重置控制器状态""" if self.pending_send_task and not self.pending_send_task.done(): self.pending_send_task.cancel() - try: - # 等待任务取消完成 - asyncio.get_event_loop().run_until_complete(self.pending_send_task) - except asyncio.CancelledError: - pass + # 取消任务后，任务会在下次事件循环时清理，无需阻塞等待 self.queue.clear() self.play_position = 0 - self.start_timestamp = time.time() + self.start_timestamp = None # 由首个音频包设置 + # 相关事件处理 + self.queue_empty_event.set() + self.queue_has_data_event.clear() def add_audio(self, opus_packet): """添加音频包到队列""" self.queue.append(("audio", opus_packet)) + # 相关事件处理 + self.queue_empty_event.clear() + self.queue_has_data_event.set() + + def add_message(self, message_callback): + """ + 添加消息到队列（立即发送，不占用播放时间） + + Args: + message_callback: 消息发送回调函数 async def() + """ + self.queue.append(("message", message_callback)) + # 相关事件处理 + self.queue_empty_event.clear() + self.queue_has_data_event.set() def _get_elapsed_ms(self): """获取已经过的时间（毫秒）""" if self.start_timestamp is None: return 0 - return (time.time() - self.start_timestamp) * 1000 + return (time.monotonic() - self.start_timestamp) * 1000 async def check_queue(self, send_audio_callback): """ @@ -61,64 +73,80 @@ class AudioRateController: Args: send_audio_callback: 发送音频的回调函数 async def(opus_packet) """ - if self.start_timestamp is None: - self.reset() - while self.queue: item = self.queue[0] item_type = item[0] - if item_type == "audio": + if item_type == "message": + # 消息类型：立即发送，不占用播放时间 + _, message_callback = item + self.queue.popleft() + try: + await message_callback() + except Exception as e: + self.logger.bind(tag=TAG).error(f"发送消息失败: {e}") + raise + + elif item_type == "audio": + if self.start_timestamp is None: + self.start_timestamp = time.monotonic() + _, opus_packet = item - # 计算时间差 - elapsed_ms = self._get_elapsed_ms() - output_ms = self.play_position + # 循环等待直到时间到达 + while True: + # 计算时间差 + elapsed_ms = self._get_elapsed_ms() + output_ms = self.play_position - if elapsed_ms < output_ms: - # 还不到发送时间，计算等待时长 - wait_ms = output_ms - elapsed_ms + if elapsed_ms < output_ms: + # 还不到发送时间，计算等待时长 + wait_ms = output_ms - elapsed_ms - # 等待后继续检查（允许被中断） - try: - await asyncio.sleep(wait_ms / 1000) - except asyncio.CancelledError: - self.logger.bind(tag=TAG).debug("音频发送任务被取消") - raise + # 等待后继续检查（允许被中断） + try: + await asyncio.sleep(wait_ms / 1000) + except asyncio.CancelledError: + self.logger.bind(tag=TAG).debug("音频发送任务被取消") + raise + # 等待结束后重新检查时间（循环回到 while True） + else: + # 时间已到，跳出等待循环 + break - # 继续循环检查（时间可能已到） - continue - - # 时间已到，发送音频 - self.queue.pop(0) + # 时间已到，从队列移除并发送 + self.queue.popleft() self.play_position += self.frame_duration - try: await send_audio_callback(opus_packet) except Exception as e: self.logger.bind(tag=TAG).error(f"发送音频失败: {e}") raise + # 队列处理完后清除事件 + self.queue_empty_event.set() + self.queue_has_data_event.clear() - async def start_sending(self, send_audio_callback, send_message_callback=None): + def start_sending(self, send_audio_callback): """ 启动异步发送任务 Args: send_audio_callback: 发送音频的回调函数 - send_message_callback: 发送消息的回调函数 Returns: asyncio.Task: 发送任务 """ + async def _send_loop(): try: while True: - await self.check_queue(send_audio_callback, send_message_callback) - # 如果队列空了，短暂等待后再检查（避免 busy loop） - await asyncio.sleep(0.01) + # 等待队列数据事件，不轮询等待占用CPU + await self.queue_has_data_event.wait() + + await self.check_queue(send_audio_callback) except asyncio.CancelledError: - self.logger.bind(tag=TAG).info("音频发送循环已停止") + self.logger.bind(tag=TAG).debug("音频发送循环已停止") except Exception as e: self.logger.bind(tag=TAG).error(f"音频发送循环异常: {e}") diff --git a/main/xiaozhi-server/core/utils/cache/config.py b/main/xiaozhi-server/core/utils/cache/config.py index 248c2af7..f85e40bb 100644 --- a/main/xiaozhi-server/core/utils/cache/config.py +++ b/main/xiaozhi-server/core/utils/cache/config.py @@ -19,6 +19,7 @@ class CacheType(Enum): CONFIG = "config" DEVICE_PROMPT = "device_prompt" VOICEPRINT_HEALTH = "voiceprint_health" # 声纹识别健康检查 + AUDIO_DATA = "audio_data" # 音频数据缓存 @dataclass @@ -58,5 +59,8 @@ class CacheConfig: CacheType.VOICEPRINT_HEALTH: cls( strategy=CacheStrategy.TTL, ttl=600, max_size=100 # 10分钟过期 ), + CacheType.AUDIO_DATA: cls( + strategy=CacheStrategy.TTL, ttl=600, max_size=100 # 10分钟过期 + ), } return configs.get(cache_type, cls()) diff --git a/main/xiaozhi-server/core/utils/opus_encoder_utils.py b/main/xiaozhi-server/core/utils/opus_encoder_utils.py index 300b2422..8d603e22 100644 --- a/main/xiaozhi-server/core/utils/opus_encoder_utils.py +++ b/main/xiaozhi-server/core/utils/opus_encoder_utils.py @@ -6,7 +6,6 @@ Opus编码工具类 import logging import traceback import numpy as np -import gc from opuslib_next import Encoder from opuslib_next import constants from typing import Optional, Callable, Any @@ -103,6 +102,9 @@ class OpusEncoderUtils: def _encode(self, frame: np.ndarray) -> Optional[bytes]: """编码一帧音频数据""" try: + # 编码器已释放，跳过编码 + if not hasattr(self, 'encoder') or self.encoder is None: + return None # 将numpy数组转换为bytes frame_bytes = frame.tobytes() # opuslib要求输入字节数必须是channels*2的倍数 diff --git a/main/xiaozhi-server/core/utils/prompt_manager.py b/main/xiaozhi-server/core/utils/prompt_manager.py index 78c63483..70f14d92 100644 --- a/main/xiaozhi-server/core/utils/prompt_manager.py +++ b/main/xiaozhi-server/core/utils/prompt_manager.py @@ -184,10 +184,25 @@ class PromptManager: def update_context_info(self, conn, client_ip: str): """同步更新上下文信息""" try: - # 获取位置信息（使用全局缓存） - local_address = self._get_location_info(client_ip) - # 获取天气信息（使用全局缓存） - self._get_weather_info(conn, local_address) + local_address = "" + if ( + client_ip + and self.base_prompt_template + and ( + "local_address" in self.base_prompt_template + or "weather_info" in self.base_prompt_template + ) + ): + # 获取位置信息（使用全局缓存） + local_address = self._get_location_info(client_ip) + + if ( + self.base_prompt_template + and "weather_info" in self.base_prompt_template + and local_address + ): + # 获取天气信息（使用全局缓存） + self._get_weather_info(conn, local_address) # 获取配置的上下文数据 if hasattr(conn, "device_id") and conn.device_id: diff --git a/main/xiaozhi-server/core/utils/util.py b/main/xiaozhi-server/core/utils/util.py index 37388be8..4547ad8f 100644 --- a/main/xiaozhi-server/core/utils/util.py +++ b/main/xiaozhi-server/core/utils/util.py @@ -4,11 +4,11 @@ import json import copy import wave import socket +import asyncio import requests import subprocess import numpy as np import opuslib_next -import gc from io import BytesIO from core.utils import p3 from pydub import AudioSegment @@ -269,56 +269,82 @@ def audio_to_data_stream( pcm_to_data_stream(raw_data, is_opus, callback) -def audio_to_data(audio_file_path: str, is_opus: bool = True) -> list[bytes]: +async def audio_to_data( + audio_file_path: str, is_opus: bool = True, use_cache: bool = True +) -> list[bytes]: """ 将音频文件转换为Opus/PCM编码的帧列表 Args: audio_file_path: 音频文件路径 is_opus: 是否进行Opus编码 + use_cache: 是否使用缓存 """ - # 获取文件后缀名 - file_type = os.path.splitext(audio_file_path)[1] - if file_type: - file_type = file_type.lstrip(".") - # 读取音频文件，-nostdin 参数：不要从标准输入读取数据，否则FFmpeg会阻塞 - audio = AudioSegment.from_file( - audio_file_path, format=file_type, parameters=["-nostdin"] - ) + from core.utils.cache.manager import cache_manager + from core.utils.cache.config import CacheType - # 转换为单声道/16kHz采样率/16位小端编码（确保与编码器匹配） - audio = audio.set_channels(1).set_frame_rate(16000).set_sample_width(2) + # 生成缓存键，包含文件路径和编码类型 + cache_key = f"{audio_file_path}:{is_opus}" - # 获取原始PCM数据（16位小端） - raw_data = audio.raw_data + # 尝试从缓存获取结果 + if use_cache: + cached_result = cache_manager.get(CacheType.AUDIO_DATA, cache_key) + if cached_result is not None: + return cached_result - # 初始化Opus编码器 - encoder = opuslib_next.Encoder(16000, 1, opuslib_next.APPLICATION_AUDIO) + def _sync_audio_to_data(): + # 获取文件后缀名 + file_type = os.path.splitext(audio_file_path)[1] + if file_type: + file_type = file_type.lstrip(".") + # 读取音频文件，-nostdin 参数：不要从标准输入读取数据，否则FFmpeg会阻塞 + audio = AudioSegment.from_file( + audio_file_path, format=file_type, parameters=["-nostdin"] + ) - # 编码参数 - frame_duration = 60 # 60ms per frame - frame_size = int(16000 * frame_duration / 1000) # 960 samples/frame + # 转换为单声道/16kHz采样率/16位小端编码（确保与编码器匹配） + audio = audio.set_channels(1).set_frame_rate(16000).set_sample_width(2) - datas = [] - # 按帧处理所有音频数据（包括最后一帧可能补零） - for i in range(0, len(raw_data), frame_size * 2): # 16bit=2bytes/sample - # 获取当前帧的二进制数据 - chunk = raw_data[i : i + frame_size * 2] + # 获取原始PCM数据（16位小端） + raw_data = audio.raw_data - # 如果最后一帧不足，补零 - if len(chunk) < frame_size * 2: - chunk += b"\x00" * (frame_size * 2 - len(chunk)) + # 初始化Opus编码器 + encoder = opuslib_next.Encoder(16000, 1, opuslib_next.APPLICATION_AUDIO) - if is_opus: - # 转换为numpy数组处理 - np_frame = np.frombuffer(chunk, dtype=np.int16) - # 编码Opus数据 - frame_data = encoder.encode(np_frame.tobytes(), frame_size) - else: - frame_data = chunk if isinstance(chunk, bytes) else bytes(chunk) + # 编码参数 + frame_duration = 60 # 60ms per frame + frame_size = int(16000 * frame_duration / 1000) # 960 samples/frame - datas.append(frame_data) + datas = [] + # 按帧处理所有音频数据（包括最后一帧可能补零） + for i in range(0, len(raw_data), frame_size * 2): # 16bit=2bytes/sample + # 获取当前帧的二进制数据 + chunk = raw_data[i : i + frame_size * 2] - return datas + # 如果最后一帧不足，补零 + if len(chunk) < frame_size * 2: + chunk += b"\x00" * (frame_size * 2 - len(chunk)) + + if is_opus: + # 转换为numpy数组处理 + np_frame = np.frombuffer(chunk, dtype=np.int16) + # 编码Opus数据 + frame_data = encoder.encode(np_frame.tobytes(), frame_size) + else: + frame_data = chunk if isinstance(chunk, bytes) else bytes(chunk) + + datas.append(frame_data) + + return datas + + loop = asyncio.get_running_loop() + # 在单独的线程中执行同步的音频处理操作 + result = await loop.run_in_executor(None, _sync_audio_to_data) + + # 将结果存入缓存，使用配置中定义的TTL（10分钟） + if use_cache: + cache_manager.set(CacheType.AUDIO_DATA, cache_key, result) + + return result def audio_bytes_to_data_stream( diff --git a/main/xiaozhi-server/plugins_func/functions/get_news_from_chinanews.py b/main/xiaozhi-server/plugins_func/functions/get_news_from_chinanews.py index e5ca0d1f..2267c83c 100644 --- a/main/xiaozhi-server/plugins_func/functions/get_news_from_chinanews.py +++ b/main/xiaozhi-server/plugins_func/functions/get_news_from_chinanews.py @@ -195,7 +195,7 @@ def get_news_from_chinanews( # 否则，获取新闻列表并随机选择一条 # 从配置中获取RSS URL - rss_config = conn.config["plugins"]["get_news_from_chinanews"] + rss_config = conn.config.get("plugins", {}).get("get_news_from_chinanews", {}) default_rss_url = rss_config.get( "default_rss_url", "https://www.chinanews.com.cn/rss/society.xml" ) diff --git a/main/xiaozhi-server/plugins_func/functions/get_news_from_newsnow.py b/main/xiaozhi-server/plugins_func/functions/get_news_from_newsnow.py index 1d60aefd..2bcd9193 100644 --- a/main/xiaozhi-server/plugins_func/functions/get_news_from_newsnow.py +++ b/main/xiaozhi-server/plugins_func/functions/get_news_from_newsnow.py @@ -120,10 +120,10 @@ def fetch_news_from_api(conn, source="thepaper"): """从API获取新闻列表""" try: api_url = f"https://newsnow.busiyi.world/api/s?id={source}" - if conn.config["plugins"].get("get_news_from_newsnow") and conn.config[ - "plugins" - ]["get_news_from_newsnow"].get("url"): - api_url = conn.config["plugins"]["get_news_from_newsnow"]["url"] + source + + news_config = conn.config.get("plugins", {}).get("get_news_from_newsnow", {}) + if news_config.get("url"): + api_url = news_config["url"] + source headers = {"User-Agent": "Mozilla/5.0"} response = requests.get(api_url, headers=headers, timeout=10) diff --git a/main/xiaozhi-server/plugins_func/functions/get_weather.py b/main/xiaozhi-server/plugins_func/functions/get_weather.py index 38770a3f..e95a40d8 100644 --- a/main/xiaozhi-server/plugins_func/functions/get_weather.py +++ b/main/xiaozhi-server/plugins_func/functions/get_weather.py @@ -158,13 +158,10 @@ def parse_weather_info(soup): def get_weather(conn, location: str = None, lang: str = "zh_CN"): from core.utils.cache.manager import cache_manager, CacheType - api_host = conn.config["plugins"]["get_weather"].get( - "api_host", "mj7p3y7naa.re.qweatherapi.com" - ) - api_key = conn.config["plugins"]["get_weather"].get( - "api_key", "a861d0d5e7bf4ee1a83d9a9e4f96d4da" - ) - default_location = conn.config["plugins"]["get_weather"]["default_location"] + weather_config = conn.config.get("plugins", {}).get("get_weather", {}) + api_host = weather_config.get("api_host", "mj7p3y7naa.re.qweatherapi.com") + api_key = weather_config.get("api_key", "a861d0d5e7bf4ee1a83d9a9e4f96d4da") + default_location = weather_config.get("default_location", "广州") client_ip = conn.client_ip # 优先使用用户提供的location参数 diff --git a/main/xiaozhi-server/plugins_func/functions/hass_init.py b/main/xiaozhi-server/plugins_func/functions/hass_init.py index 11cbb7a0..dadb190b 100644 --- a/main/xiaozhi-server/plugins_func/functions/hass_init.py +++ b/main/xiaozhi-server/plugins_func/functions/hass_init.py @@ -11,15 +11,17 @@ def append_devices_to_prompt(conn): "functions", [] ) + # 安全地获取插件配置 + plugins_config = conn.config.get("plugins", {}) config_source = ( "home_assistant" - if conn.config["plugins"].get("home_assistant") + if plugins_config.get("home_assistant") else "hass_get_state" ) if "hass_get_state" in funcs or "hass_set_state" in funcs: prompt = "\n下面是我家智能设备列表（位置，设备名，entity_id），可以通过homeassistant控制\n" - deviceStr = conn.config["plugins"].get(config_source, {}).get("devices", "") + deviceStr = plugins_config.get(config_source, {}).get("devices", "") conn.prompt += prompt + deviceStr + "\n" # 更新提示词 conn.dialogue.update_system_message(conn.prompt) @@ -30,17 +32,17 @@ def initialize_hass_handler(conn): if not conn.load_function_plugin: return ha_config + # 安全地获取插件配置 + plugins_config = conn.config.get("plugins", {}) # 确定配置来源 config_source = ( - "home_assistant" - if conn.config["plugins"].get("home_assistant") - else "hass_get_state" + "home_assistant" if plugins_config.get("home_assistant") else "hass_get_state" ) - if not conn.config["plugins"].get(config_source): + if not plugins_config.get(config_source): return ha_config # 统一获取配置 - plugin_config = conn.config["plugins"][config_source] + plugin_config = plugins_config[config_source] ha_config["base_url"] = plugin_config.get("base_url") ha_config["api_key"] = plugin_config.get("api_key") diff --git a/main/xiaozhi-server/plugins_func/functions/play_music.py b/main/xiaozhi-server/plugins_func/functions/play_music.py index 2cbc4018..be4cf618 100644 --- a/main/xiaozhi-server/plugins_func/functions/play_music.py +++ b/main/xiaozhi-server/plugins_func/functions/play_music.py @@ -118,8 +118,9 @@ def get_music_files(music_dir, music_ext): def initialize_music_handler(conn): global MUSIC_CACHE if MUSIC_CACHE == {}: - if "play_music" in conn.config["plugins"]: - MUSIC_CACHE["music_config"] = conn.config["plugins"]["play_music"] + plugins_config = conn.config.get("plugins", {}) + if "play_music" in plugins_config: + MUSIC_CACHE["music_config"] = plugins_config["play_music"] MUSIC_CACHE["music_dir"] = os.path.abspath( MUSIC_CACHE["music_config"].get("music_dir", "./music") # 默认路径修改 ) diff --git a/main/xiaozhi-server/plugins_func/functions/search_from_ragflow.py b/main/xiaozhi-server/plugins_func/functions/search_from_ragflow.py index d585a762..ec6ac426 100644 --- a/main/xiaozhi-server/plugins_func/functions/search_from_ragflow.py +++ b/main/xiaozhi-server/plugins_func/functions/search_from_ragflow.py @@ -32,9 +32,10 @@ def search_from_ragflow(conn, question=None): else: question = str(question) if question is not None else "" - base_url = conn.config["plugins"]["search_from_ragflow"].get("base_url", "") - api_key = conn.config["plugins"]["search_from_ragflow"].get("api_key", "") - dataset_ids = conn.config["plugins"]["search_from_ragflow"].get("dataset_ids", []) + ragflow_config = conn.config.get("plugins", {}).get("search_from_ragflow", {}) + base_url = ragflow_config.get("base_url", "") + api_key = ragflow_config.get("api_key", "") + dataset_ids = ragflow_config.get("dataset_ids", []) url = base_url + "/api/v1/retrieval" headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"} diff --git a/main/xiaozhi-server/requirements.txt b/main/xiaozhi-server/requirements.txt index 3f62ef4c..919c0f6d 100644 --- a/main/xiaozhi-server/requirements.txt +++ b/main/xiaozhi-server/requirements.txt @@ -13,7 +13,7 @@ pydub==0.25.1 funasr==1.2.7 openai==2.8.1 google-generativeai==0.8.5 -edge_tts==7.2.3 +edge_tts==7.2.6 httpx==0.28.1 aiohttp==3.13.2 aiohttp_cors==0.8.1 diff --git a/main/xiaozhi-server/test/css/test_page.css b/main/xiaozhi-server/test/css/test_page.css index c0564191..b197c60d 100644 --- a/main/xiaozhi-server/test/css/test_page.css +++ b/main/xiaozhi-server/test/css/test_page.css @@ -267,6 +267,7 @@ span.connection-status.llm-emoji { .llm-emoji .status { font-size: 14px !important; padding: 8px 20px !important; + line-height: 1.2 !important; } .emoji-large { @@ -393,7 +394,7 @@ span.connection-status.llm-emoji { /* ==================== 会话记录和日志 ==================== */ .flex-container { display: flex; - margin-top: 10px; + margin-top: 20px; background-color: #f9fafb; } diff --git a/main/xiaozhi-server/test/js/core/audio/player.js b/main/xiaozhi-server/test/js/core/audio/player.js index 459fbb54..bc89d58a 100644 --- a/main/xiaozhi-server/test/js/core/audio/player.js +++ b/main/xiaozhi-server/test/js/core/audio/player.js @@ -249,6 +249,41 @@ export class AudioPlayer { this.playBufferedAudio(); this.startAudioBuffering(); } + + // 获取音频包统计信息 + getAudioStats() { + if (!this.streamingContext) { + return { + pendingDecode: 0, + pendingPlay: 0, + totalPending: 0 + }; + } + + const pendingDecode = this.streamingContext.getPendingDecodeCount(); + const pendingPlay = this.streamingContext.getPendingPlayCount(); + + return { + pendingDecode, // 待解码包数 + pendingPlay, // 待播放包数 + totalPending: pendingDecode + pendingPlay // 总待处理包数 + }; + } + + // 清空所有音频缓冲并停止播放 + clearAllAudio() { + log('AudioPlayer: 清空所有音频', 'info'); + + // 清空接收队列（使用clear方法保持对象引用） + this.queue.clear(); + + // 清空流上下文的所有缓冲 + if (this.streamingContext) { + this.streamingContext.clearAllBuffers(); + } + + log('AudioPlayer: 音频已清空', 'success'); + } } // 创建单例 diff --git a/main/xiaozhi-server/test/js/core/audio/recorder.js b/main/xiaozhi-server/test/js/core/audio/recorder.js index 9f94f895..3fe785f0 100644 --- a/main/xiaozhi-server/test/js/core/audio/recorder.js +++ b/main/xiaozhi-server/test/js/core/audio/recorder.js @@ -240,6 +240,24 @@ export class AudioRecorder { if (this.isRecording) return false; try { + // 检查是否有WebSocketHandler实例 + const { getWebSocketHandler } = await import('../network/websocket.js'); + const wsHandler = getWebSocketHandler(); + + // 如果机器正在说话，发送打断消息 + if (wsHandler && wsHandler.isRemoteSpeaking && wsHandler.currentSessionId) { + const abortMessage = { + session_id: wsHandler.currentSessionId, + type: 'abort', + reason: 'wake_word_detected' + }; + + if (this.websocket && this.websocket.readyState === WebSocket.OPEN) { + this.websocket.send(JSON.stringify(abortMessage)); + log('发送打断消息', 'info'); + } + } + if (!this.initEncoder()) { log('无法启动录音: Opus编码器初始化失败', 'error'); return false; diff --git a/main/xiaozhi-server/test/js/core/audio/stream-context.js b/main/xiaozhi-server/test/js/core/audio/stream-context.js index 9c722505..05b8c1db 100644 --- a/main/xiaozhi-server/test/js/core/audio/stream-context.js +++ b/main/xiaozhi-server/test/js/core/audio/stream-context.js @@ -22,6 +22,7 @@ export class StreamingContext { this.source = null; // 当前音频源 this.totalSamples = 0; // 累积的总样本数 this.lastPlayTime = 0; // 上次播放的时间戳 + this.scheduledEndTime = 0; // 已调度音频的结束时间 } // 缓存音频数组 @@ -31,17 +32,19 @@ export class StreamingContext { // 获取需要处理缓存队列，单线程：在audioBufferQueue一直更新的状态下不会出现安全问题 async getPendingAudioBufferQueue() { - // 原子交换 + 清空 - [this.pendingAudioBufferQueue, this.audioBufferQueue] = [await this.audioBufferQueue.dequeue(), new BlockingQueue()]; + // 等待数据到达并获取 + const data = await this.audioBufferQueue.dequeue(); + // 赋值给待处理队列 + this.pendingAudioBufferQueue = data; } // 获取正在播放已解码的PCM队列，单线程：在activeQueue一直更新的状态下不会出现安全问题 async getQueue(minSamples) { - let TepArray = []; const num = minSamples - this.queue.length > 0 ? minSamples - this.queue.length : 1; - // 原子交换 + 清空 - [TepArray, this.activeQueue] = [await this.activeQueue.dequeue(num), new BlockingQueue()]; - this.queue.push(...TepArray); + + // 等待数据并获取 + const tempArray = await this.activeQueue.dequeue(num); + this.queue.push(...tempArray); } // 将Int16音频数据转换为Float32音频数据 @@ -54,6 +57,57 @@ export class StreamingContext { return float32Data; } + // 获取待解码包数 + getPendingDecodeCount() { + return this.audioBufferQueue.length + this.pendingAudioBufferQueue.length; + } + + // 获取待播放样本数（转换为包数，每包960样本） + getPendingPlayCount() { + // 计算已在队列中的样本 + const queuedSamples = this.activeQueue.length + this.queue.length; + + // 计算已调度但未播放的样本（在Web Audio缓冲区中） + let scheduledSamples = 0; + if (this.playing && this.scheduledEndTime) { + const currentTime = this.audioContext.currentTime; + const remainingTime = Math.max(0, this.scheduledEndTime - currentTime); + scheduledSamples = Math.floor(remainingTime * this.sampleRate); + } + + const totalSamples = queuedSamples + scheduledSamples; + return Math.ceil(totalSamples / 960); + } + + // 清空所有音频缓冲 + clearAllBuffers() { + log('清空所有音频缓冲', 'info'); + + // 清空所有队列（使用clear方法保持对象引用） + this.audioBufferQueue.clear(); + this.pendingAudioBufferQueue = []; + this.activeQueue.clear(); + this.queue = []; + + // 停止当前播放的音频源 + if (this.source) { + try { + this.source.stop(); + this.source.disconnect(); + } catch (e) { + // 忽略已经停止的错误 + } + this.source = null; + } + + // 重置状态 + this.playing = false; + this.scheduledEndTime = this.audioContext.currentTime; + this.totalSamples = 0; + + log('音频缓冲已清空', 'success'); + } + // 将Opus数据解码为PCM async decodeOpusFrames() { if (!this.opusDecoder) { @@ -97,7 +151,7 @@ export class StreamingContext { // 开始播放音频 async startPlaying() { - let scheduledEndTime = this.audioContext.currentTime; // 跟踪已调度音频的结束时间 + this.scheduledEndTime = this.audioContext.currentTime; // 跟踪已调度音频的结束时间 while (true) { // 初始缓冲：等待足够的样本再开始播放 @@ -126,7 +180,7 @@ export class StreamingContext { // 精确调度播放时间 const currentTime = this.audioContext.currentTime; - const startTime = Math.max(scheduledEndTime, currentTime); + const startTime = Math.max(this.scheduledEndTime, currentTime); // 直接连接到输出 this.source.connect(this.audioContext.destination); @@ -136,7 +190,7 @@ export class StreamingContext { // 更新下一个音频块的调度时间 const duration = audioBuffer.duration; - scheduledEndTime = startTime + duration; + this.scheduledEndTime = startTime + duration; this.lastPlayTime = startTime; // 如果队列中数据不足，等待新数据 diff --git a/main/xiaozhi-server/test/js/core/network/websocket.js b/main/xiaozhi-server/test/js/core/network/websocket.js index b62c710b..a605e6f1 100644 --- a/main/xiaozhi-server/test/js/core/network/websocket.js +++ b/main/xiaozhi-server/test/js/core/network/websocket.js @@ -123,7 +123,12 @@ export class WebSocketHandler { } else if (message.state === 'sentence_end') { log(`语音段结束: ${message.text}`, 'info'); } else if (message.state === 'stop') { - log('服务器语音传输结束', 'info'); + log('服务器语音传输结束，清空所有音频缓冲', 'info'); + + // 清空所有音频缓冲并停止播放 + const audioPlayer = getAudioPlayer(); + audioPlayer.clearAllAudio(); + this.isRemoteSpeaking = false; if (this.onRecordButtonStateChange) { this.onRecordButtonStateChange(false); diff --git a/main/xiaozhi-server/test/js/ui/controller.js b/main/xiaozhi-server/test/js/ui/controller.js index 8fc9dd31..3e4f6563 100644 --- a/main/xiaozhi-server/test/js/ui/controller.js +++ b/main/xiaozhi-server/test/js/ui/controller.js @@ -2,6 +2,7 @@ import { loadConfig, saveConfig } from '../config/manager.js'; import { getAudioRecorder } from '../core/audio/recorder.js'; import { getWebSocketHandler } from '../core/network/websocket.js'; +import { getAudioPlayer } from '../core/audio/player.js'; // UI控制器类 export class UIController { @@ -9,6 +10,7 @@ export class UIController { this.isEditing = false; this.visualizerCanvas = null; this.visualizerContext = null; + this.audioStatsTimer = null; } // 初始化 @@ -18,6 +20,7 @@ export class UIController { this.initVisualizer(); this.initEventListeners(); + this.startAudioStatsMonitor(); loadConfig(); } @@ -86,17 +89,20 @@ export class UIController { const sessionStatus = document.getElementById('sessionStatus'); if (!sessionStatus) return; + // 保留背景元素 + const bgHtml = ''; + if (isSpeaking === null) { // 离线状态 - sessionStatus.innerHTML = '😶 小智离线中'; + sessionStatus.innerHTML = bgHtml + '😶 小智离线中'; sessionStatus.className = 'status offline'; } else if (isSpeaking) { // 说话中 - sessionStatus.innerHTML = '😶 小智说话中'; + sessionStatus.innerHTML = bgHtml + '😶 小智说话中'; sessionStatus.className = 'status speaking'; } else { // 聆听中 - sessionStatus.innerHTML = '😶 小智聆听中'; + sessionStatus.innerHTML = bgHtml + '😶 小智聆听中'; sessionStatus.className = 'status listening'; } } @@ -110,8 +116,72 @@ export class UIController { let currentText = sessionStatus.textContent; // 移除现有的表情符号 currentText = currentText.replace(/[\u{1F300}-\u{1F9FF}]|[\u{2600}-\u{26FF}]|[\u{2700}-\u{27BF}]/gu, '').trim(); + + // 保留背景元素 + const bgHtml = ''; + // 使用 innerHTML 添加带样式的表情 - sessionStatus.innerHTML = `${emoji} ${currentText}`; + sessionStatus.innerHTML = bgHtml + `${emoji} ${currentText}`; + } + + // 更新音频统计信息 + updateAudioStats() { + const audioPlayer = getAudioPlayer(); + const stats = audioPlayer.getAudioStats(); + + const sessionStatus = document.getElementById('sessionStatus'); + const sessionStatusBg = document.getElementById('sessionStatusBg'); + + // 只在说话状态下显示背景进度 + if (sessionStatus && sessionStatus.classList.contains('speaking') && sessionStatusBg) { + if (stats.pendingPlay > 0) { + // 计算进度：5包=50%，10包及以上=100% + let percentage; + if (stats.pendingPlay >= 10) { + percentage = 100; + } else { + percentage = (stats.pendingPlay / 10) * 100; + } + + sessionStatusBg.style.width = `${percentage}%`; + + // 根据缓冲量改变背景颜色 + if (stats.pendingPlay < 5) { + // 缓冲不足：橙红色半透明 + sessionStatusBg.style.background = 'linear-gradient(90deg, rgba(255, 152, 0, 0.25), rgba(255, 87, 34, 0.25))'; + } else if (stats.pendingPlay < 10) { + // 一般：黄绿色半透明 + sessionStatusBg.style.background = 'linear-gradient(90deg, rgba(205, 220, 57, 0.25), rgba(76, 175, 80, 0.25))'; + } else { + // 充足：绿蓝色半透明 + sessionStatusBg.style.background = 'linear-gradient(90deg, rgba(76, 175, 80, 0.25), rgba(33, 150, 243, 0.25))'; + } + } else { + // 没有缓冲，隐藏背景 + sessionStatusBg.style.width = '0%'; + } + } else { + // 非说话状态，隐藏背景 + if (sessionStatusBg) { + sessionStatusBg.style.width = '0%'; + } + } + } + + // 启动音频统计监控 + startAudioStatsMonitor() { + // 每100ms更新一次音频统计 + this.audioStatsTimer = setInterval(() => { + this.updateAudioStats(); + }, 100); + } + + // 停止音频统计监控 + stopAudioStatsMonitor() { + if (this.audioStatsTimer) { + clearInterval(this.audioStatsTimer); + this.audioStatsTimer = null; + } } // 绘制音频可视化效果 diff --git a/main/xiaozhi-server/test/js/utils/blocking-queue.js b/main/xiaozhi-server/test/js/utils/blocking-queue.js index 31a43872..738a3e75 100644 --- a/main/xiaozhi-server/test/js/utils/blocking-queue.js +++ b/main/xiaozhi-server/test/js/utils/blocking-queue.js @@ -95,4 +95,9 @@ export default class BlockingQueue { get length() { return this.#items.length; } + + /* 清空队列（保持对象引用，不影响等待者） */ + clear() { + this.#items.length = 0; + } } \ No newline at end of file diff --git a/main/xiaozhi-server/test/test_page.html b/main/xiaozhi-server/test/test_page.html index 97af77ac..f3f583c9 100644 --- a/main/xiaozhi-server/test/test_page.html +++ b/main/xiaozhi-server/test/test_page.html @@ -91,6 +91,7 @@ +
@@ -113,9 +114,12 @@
-
+
- 😶 小智离线中 + + + 😶 小智离线中 +