Merge pull request #1476 from xinnan-tech/fix-manager-agent-vllm

修复智控台保存时保存不了视觉模型的bug

README.md

@@ -121,6 +121,33 @@
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="实时打断" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="拍照识物品" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="多指令任务" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -143,8 +170,8 @@
 #### 🚀 部署方式选择
 | 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | 
 |---------|------|---------|---------|---------|---------|
-| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
+| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
-| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
+| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
 
 > 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
@@ -159,34 +186,47 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 #### 🚩 配置说明和推荐
 > [!Note]
-> 本项目默认的配置是`入门全免费`设置，如果想效果更优，推荐使用`全流式配置`。
+> 本项目默认的配置是`入门全免费`设置，如果想效果更优，推荐使用`流式配置`。
 > 
-> 本项目自`0.5.2`版本，已支持整个生命周期全流式，相比`0.5`版本以前，响应速度提升约`2.5秒`
+> 本项目自`0.5.2`版本，已支持使用流式配置，相比`0.5`版本以前，响应速度提升约`2.5秒`
 
-
+| 模块名称 | 入门全免费设置 | 流式配置 |
-| 模块名称 | 入门全免费设置 | 全流式配置 |
+|:---:|:---:|:---:|
-|---------|---------|------|
+| ASR(语音识别) | FunASR(本地) | 👍DoubaoStreamASR(火山流式语音识别) |
-| ASR(语音识别) | FunASR(本地) | ✅DoubaoASR(火山流式语音识别) |
+| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
-| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | ✅DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
+| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
-| TTS(语音合成) | EdgeTTS(微软语音) | ✅HuoshanDoubleStreamTTS(火山双流式语音合成) |
+| TTS(语音合成) | EdgeTTS(微软语音) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) |
 | Intent(意图识别) | function_call(函数调用) | ✅function_call(函数调用) |
 | Memory(记忆功能) | mem_local_short(本地短期记忆） | ✅mem_local_short（本地短期记忆） |
 
+#### 🔧 测试工具
+本项目提供以下测试工具，帮助您验证系统和选择合适的模型：
+
+| 工具名称 | 位置 | 使用方法 | 功能说明 |
+|:---:|:---|:---:|:---:|
+| 音频交互测试工具 | 位于`main/xiaozhi-server/test/test_page.html` | 使用谷歌浏览器直接打开 | 测试音频播放和接收功能，验证Python端音频处理是否正常 |
+| 模型响应测试工具1 | 位于`main/xiaozhi-server/performance_tester.py` | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、TTS(语音合成)三个核心模块的响应速度 |
+| 模型响应测试工具2 | 位于`main/xiaozhi-server/performance_tester_vllm.py` | 执行 `python performance_tester_vllm.py` | 测试VLLM(视觉模型)的响应速度 |
+
+> 💡 提示：测试模型速度时，只会测试配置了密钥的模型。
+
 ---
 ## 功能清单 ✨
 ### 已实现 ✅
 
 | 功能模块 | 描述 |
-|---------|------|
+|:---:|:---|
-| 通信协议 | 基于 `xiaozhi-esp32` 协议，通过 WebSocket 实现数据交互 |
+| 核心服务架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
-| 对话交互 | 支持唤醒对话、手动对话及实时打断。长时间无对话时自动休眠 |
+| 语音交互系统 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
-| 意图识别 | 支持使用LLM意图识别、function call函数调用，减少硬编码意图判断 |
+| 智能对话系统 | 支持多种LLM(大语言模型)，实现智能对话 |
-| 多语言识别 | 支持国语、粤语、英语、日语、韩语（默认使用 FunASR） |
+| 视觉感知系统 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
-| LLM 模块 | 支持灵活切换 LLM 模块，默认使用 ChatGLMLLM，也可选用阿里百炼、DeepSeek、Ollama 等接口 |
+| 意图识别系统 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
-| TTS 模块 | 支持 EdgeTTS（默认）、火山引擎豆包 TTS 等多种 TTS 接口，满足语音合成需求 |
+| 记忆系统 | 支持本地短期记忆、mem0ai接口记忆，具备记忆总结功能 |
-| 记忆功能 | 支持超长记忆、本地总结记忆、无记忆三种模式，满足不同场景需求 |
+| IOT/MCP控制协议 | 支持设备注册管理、智能控制接口，同时支持IOT、MCP控制协议 |
-| IOT功能 | 支持管理注册设备IOT功能，支持基于对话上下文语境下的智能物联网控制 |
+| 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理 |
-| 智控台 | 提供Web管理界面，支持智能体管理、用户管理、系统配置等功能，方便管理员和用户进行管理 |
+| 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
+| 部署支持 | 支持Docker部署和本地部署，提供完整的配置文件管理 |
+| 插件系统 | 支持功能插件扩展、自定义插件开发和插件热加载 |
 
 ### 正在开发 🚧
 
@@ -223,6 +263,16 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 ---
 
+### VLLM 视觉模型
+
+| 使用方式 | 支持平台 | 免费平台 |
+|:---:|:---:|:---:|
+| openai 接口调用 | 阿里百炼、智谱ChatGLMVLLM | 智谱ChatGLMVLLM |
+
+实际上，任何支持 openai 接口调用的 VLLM 均可接入使用。
+
+---
+
 ### TTS 语音合成
 
 | 使用方式 | 支持平台 | 免费平台 |

README_en.md

@@ -6,14 +6,14 @@
 This project provides backend services for the open-source smart hardware project
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
 Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
-Helping you quickly set up your Xiaozhi server
+Helps you quickly set up your Xiaozhi server
 </p>
 
 <p align="center">
 <a href="./README.md">中文</a>
 · <a href="./docs/FAQ.md">FAQ</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
-· <a href="./README_ed.md#deployment-documentation">Deployment Guide</a>
+· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
 </p>
 <p align="center">
@@ -50,7 +50,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
          <picture>
-           <img alt="Xiaozhi esp32 connecting to custom backend model" src="docs/images/demo1.png" />
+           <img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
          </picture>
         </a>
     </td>
@@ -64,14 +64,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
          <picture>
-           <img alt="Cantonese communication" src="docs/images/demo3.png" />
+           <img alt="Using Cantonese" src="docs/images/demo3.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
          <picture>
-           <img alt="Home appliance control" src="docs/images/demo5.png" />
+           <img alt="Control home appliances" src="docs/images/demo5.png" />
          </picture>
         </a>
     </td>
@@ -94,7 +94,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
          <picture>
-           <img alt="Music playback" src="docs/images/demo7.png" />
+           <img alt="Play music" src="docs/images/demo7.png" />
          </picture>
         </a>
     </td>
@@ -108,7 +108,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
          <picture>
-           <img alt="IOT device control" src="docs/images/demo9.png" />
+           <img alt="IOT command control" src="docs/images/demo9.png" />
          </picture>
         </a>
     </td>
@@ -120,6 +120,33 @@ Want to see it in action? Check out these videos 🎥
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="Real-time interruption" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="Photo recognition" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="Multi-command tasks" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -137,91 +164,102 @@ It is recommended that users prioritize service providers with relevant business
 
 ![Banners](docs/images/banner2.png)
 
-This project offers two deployment methods. Please choose based on your specific needs:
+This project provides two deployment methods. Please choose according to your specific needs:
 
 #### 🚀 Deployment Method Selection
+| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial | 
+|---------|------|---------|---------|---------|---------|
+| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - | 
+| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) | 
 
-| Deployment Method | Features | Use Case | Docker Deployment Guide | Source Code Deployment Guide |
+> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
-|---------|------|---------|---------|---------|
-| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database required | [Docker Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [Local Source Code Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)|
-| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience |[Docker Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [Local Source Code Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) |
-
-> 💡 Note: Below is the test platform deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data cleared daily
 
 ```
-Control Panel: https://2662r3426b.vicp.fun
+Control Panel Address: https://2662r3426b.vicp.fun
 
 Service Test Tool: https://2662r3426b.vicp.fun/test/
-OTA Interface: https://2662r3426b.vicp.fun/xiaozhi/ota/
+OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
-Websocket Interface: wss://2662r3426b.vicp.fun/xiaozhi/v1/
+Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ```
 
+#### 🚩 Configuration Description and Recommendations
+> [!Note]
+> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
+> 
+> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
+
+| Module Name | Entry Level Free Settings | Full Streaming Configuration |
+|---------|---------|------|
+| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
+| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
+| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
+| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
+| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
+| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
+
 ---
 ## Feature List ✨
-
 ### Implemented ✅
 
 | Feature Module | Description |
 |---------|------|
 | Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
-| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of inactivity |
+| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
 | Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
 | Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
-| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, also supports Ali Bailing, DeepSeek, Ollama, etc. |
+| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
-| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces for speech synthesis |
+| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
-| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes for different scenarios |
+| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
-| IOT Function | Supports managing registered device IOT functionality, intelligent IoT control based on dialogue context |
+| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
-| Control Panel | Provides web management interface, supports agent management, user management, system configuration, etc. |
+| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
 
 ### In Development 🚧
 
 To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
 
-If you're a software developer, here's an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
+If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
 
 ---
 
 ## Product Ecosystem 👬
-Xiaozhi is an ecosystem. When using this product, you might want to check out other excellent projects in this ecosystem:
+Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
 
-| Project Name | Project Link | Description |
+| Project Name | Project Address | Project Description |
 |:---------------------|:--------|:--------|
-| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue |
+| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
-| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | A Python-based AI client that allows you to experience Xiaozhi AI functionality through code without physical hardware |
+| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
-| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | A Java-based open-source project providing complete backend service solutions |
+| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
 
 ---
 
-## Supported Platforms/Components 📋
+## Supported Platforms/Components List 📋
 
 ### LLM Language Models
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| openai API | Ali Bailing, Volcano Engine Doubao, DeepSeek, ChatGLM, Gemini | ChatGLM, Gemini |
+| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
-| ollama API | Ollama | - |
+| ollama interface call | Ollama | - |
-| dify API | Dify | - |
+| dify interface call | Dify | - |
-| fastgpt API | Fastgpt | - |
+| fastgpt interface call | Fastgpt | - |
-| coze API | Coze | - |
+| coze interface call | Coze | - |
 
-Actually, any LLM supporting openai API calls can be integrated.
+In fact, any LLM that supports openai interface calls can be integrated and used.
-
----
 
 ### TTS Speech Synthesis
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| API Calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Aliyun TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
+| API Call | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
 | Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
 
 ---
 
 ### VAD Voice Activity Detection
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:---:|:---------:|:----:|:----:|:--:|
-| VAD | SileroVAD | Local Use | Free | |
+| VAD | SileroVAD | Local Usage | Free | |
 
 ---
 
@@ -229,26 +267,26 @@ Actually, any LLM supporting openai API calls can be integrated.
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| Local Use | FunASR, SherpaASR | FunASR, SherpaASR |
+| Local Usage | FunASR, SherpaASR | FunASR, SherpaASR |
-| API Calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+| API Call | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
 
 ---
 
 ### Memory Storage
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:---------------:|:----:|:---------:|:--:|
-| Memory | mem0ai | API Calls | 1000 calls/month quota | |
+| Memory | mem0ai | API Call | 1000 calls/month quota | |
 | Memory | mem_local_short | Local Summary | Free | |
 
 ---
 
 ### Intent Recognition
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:-------------:|:----:|:-------:|:---------------------:|
-| Intent | intent_llm | API Calls | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
+| Intent | intent_llm | API Call | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
-| Intent | function_call | API Calls | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
+| Intent | function_call | API Call | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
 
 ---
 
@@ -257,10 +295,10 @@ Actually, any LLM supporting openai API calls can be integrated.
 | Logo | Project/Company | Description |
 |:---:|:---:|:---|
 | <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
-| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for developing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing comprehensive technical documentation support for this project |
+| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for establishing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing full-chain technical documentation support for this project |
-| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
+| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing the function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
-| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering product user experience with their design experience serving over a thousand enterprises |
+| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering the product user experience with their design experience serving over a thousand enterprises |
-| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
+| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening the visual system of this project, ensuring consistency and extensibility of the overall design style in multi-scenario applications |
 
 
 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
@@ -270,4 +308,4 @@ Actually, any LLM supporting openai API calls can be integrated.
    <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
    <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
  </picture>
-</a> 
+</a>

docs/FAQ.md

@@ -108,7 +108,11 @@ VAD:
 
 参考教程[阿里云短信集成指南](./ali-sms-integration.md)
 
-### 9、更多问题，可联系我们反馈 💬
+### 9、如何开启视觉模型实现拍照识物 📷
+
+参考教程[视觉模型使用指南](./mcp-vision-integration.md)
+
+### 10、更多问题，可联系我们反馈 💬
 
 可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
 

docs/dev-ops-integration.md

@@ -0,0 +1,163 @@
+# 全模块源码部署自动升级方法
+
+本教程是方便全模块源码部署的爱好者，如何通过自动命令，自动拉取源码，自动编译，自动启动端口运行。实现最高效率的升级系统。
+
+本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。
+
+# 开始条件
+- 你的电脑/服务器是linux操作系统
+- 你已经跑通了整个流程
+- 你喜欢跟进最新功能，但是觉得每次手动部署有点麻烦，期待有一个自动更新的方法
+
+第二个条件必须满足，因为本教程所涉及的某些文件，JDK、Node.js环境、Conda环境等，是需要你跑通整个流程才有的，如果你没有跑通，当我讲到某个文件的时候，你可能就不知道什么意思。
+
+# 教程效果
+- 解决国内不能拉取最新项目源码问题
+- 自动拉取代码编译前端文件
+- 自动拉取代码编译java文件，自动杀掉8002端口，自动启动8002端口
+- 自动拉取python代码，自动杀掉8000端口，自动启动8000端口
+
+# 第一步 选好你的项目目录
+
+例如，我规划了我的项目目录是，这是一个新建的空白的目录，如果你不想出错，可以和我一样
+```
+/home/system/xiaozhi
+```
+
+# 第二步 克隆本项目
+此刻，先要执行第一句话，拉取源码，这句命令适用于国内网络的服务器和电脑，无需翻墙
+
+```
+cd /home/system/xiaozhi
+git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-server.git
+```
+
+执行完后，你的项目目录会多了一个文件夹`xiaozhi-esp32-server`，这个就是项目的源码
+
+# 第三步 复制基础的文件
+
+如果你之前已经跑通了整个流程，对funasr的模型文件`xiaozhi-server/models/SenseVoiceSmall/model.pt`和你的私有配置文件`xiaozhi-server/data/.config.yaml`这两个文件不会陌生。
+
+此刻你需要把`model.pt`文件复制到新的目录去，你可以这样
+```
+cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
+cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
+```
+
+# 第四步 建立三个自动编译文件
+
+## 4.1 自动编译mananger-web模块
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8001.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git fetch --all
+git reset --hard
+git pull origin main
+
+
+cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web
+npm install
+npm run build
+rm -rf /home/system/xiaozhi/manager-web
+mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web/dist /home/system/xiaozhi/manager-web
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8001.sh
+```
+执行完后，继续往下
+
+## 4.2 自动编译运行manager-api模块
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8002.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git pull origin main
+
+
+cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api
+rm -rf target
+mvn clean package -Dmaven.test.skip=true
+cd /home/system/xiaozhi/
+
+# 查找占用8002端口的进程号
+PID=$(sudo netstat -tulnp | grep 8002 | awk '{print $7}' | cut -d'/' -f1)
+
+rm -rf /home/system/xiaozhi/xiaozhi-esp32-api.jar
+mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api/target/xiaozhi-esp32-api.jar /home/system/xiaozhi/xiaozhi-esp32-api.jar
+
+# 检查是否找到进程号
+if [ -z "$PID" ]; then
+  echo "没有找到占用8002端口的进程"
+else
+  echo "找到占用8002端口的进程，进程号为: $PID"
+  # 杀掉进程
+  kill -9 $PID
+  kill -9 $PID
+  echo "已杀掉进程 $PID"
+fi
+
+nohup java -jar xiaozhi-esp32-api.jar --spring.profiles.active=dev &
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8002.sh
+```
+执行完后，继续往下
+
+## 4.3 自动编译运行Python项目
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8000.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git pull origin main
+
+# 查找占用8000端口的进程号
+PID=$(sudo netstat -tulnp | grep 8000 | awk '{print $7}' | cut -d'/' -f1)
+
+# 检查是否找到进程号
+if [ -z "$PID" ]; then
+  echo "没有找到占用8000端口的进程"
+else
+  echo "找到占用8000端口的进程，进程号为: $PID"
+  # 杀掉进程
+  kill -9 $PID
+  kill -9 $PID
+  echo "已杀掉进程 $PID"
+fi
+cd main/xiaozhi-server
+pip install -r requirements.txt
+nohup python app.py >/dev/null &
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8000.sh
+```
+执行完后，继续往下
+
+# 日常更新
+
+以上的脚本都建立好后，日常更新，我们只要依次执行以下命令就可以做到自动更新和启动
+
+```
+# 进入pyhton环境
+conda activate xiaozhi-esp32-server
+cd /home/system/xiaozhi
+# 更新并启动Java程序
+./update_8001.sh
+# 更新web程序
+./update_8002.sh
+# 更新并启动python程序
+./update_8000.sh
+# 查看Java日志
+tail -f nohup.out
+# 查看Python日志
+tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server.log
+```
+
+# 注意事项
+测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)

docs/homeassistant-integration.md

@@ -202,7 +202,15 @@ change_role;get_weather;get_news;play_music;hass_get_state;hass_set_state
 
 #### 2. 配置小智开源服务端MCP配置信息
 
-切换到小智开源服务端`xiaozhi-esp32-server`的`mcp_server_settings.json`文件内，在`"mcpServers"`的括号内添加以下内容：
+
+进入`data`目录，找到`.mcp_server_settings.json`文件。
+
+如果你的`data`目录下没有`.mcp_server_settings.json`文件，
+- 请把在`xiaozhi-server`文件夹根目录的`mcp_server_settings.json`文件复制到`data`目录下，并重命名为`.mcp_server_settings.json`
+- 或[下载这个文件](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/main/xiaozhi-server/mcp_server_settings.json)，下载到`data`目录下，并重命名为`.mcp_server_settings.json`
+
+
+修改`"mcpServers"`里的这部分的内容：
 
 ```json
 "Home Assistant": {

docs/mcp-vision-integration.md

@@ -0,0 +1,171 @@
+# 视觉模型使用指南
+本教程分为两部分：
+- 第一部分：单模块运行xiaozhi-server开启视觉模型
+- 第二部分：全模块运行时，如何开启视觉模型
+
+开启视觉模型前，你需要准备三件事：
+- 你需要准备一台带摄像头的设备，而且这台设备已经在虾哥仓库里，实现了调用摄像头功能。例如`立创·实战派ESP32-S3开发板`
+- 你设备固件的版本升级到1.6.6及以上
+- 你已经成功跑通基础对话模块
+
+## 单模块运行xiaozhi-server开启视觉模型
+
+### 第一步确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose.yml`是否放了`8003`端口，如果没有就更新最新的`docker-compose.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步选择你的视觉模型
+打开你的`data/.config.yaml`文件，设置你的`selected_module.VLLM`设置为某个视觉模型。目前我们已经支持`openai`类型接口的视觉模型。`ChatGLMVLLM`就是其中一款兼容`openai`的模型。
+
+```
+selected_module:
+  VAD: ..
+  ASR: ..
+  LLM: ..
+  VLLM: ChatGLMVLLM
+  TTS: ..
+  Memory: ..
+  Intent: ..
+```
+
+假设我们使用`ChatGLMVLLM`作为视觉模型，那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+在你的配置文件中，增加这个配置，如果已经有了这个配置，就设置好你的api_key。
+
+```
+VLLM:
+  ChatGLMVLLM:
+    api_key: 你的api_key
+```
+
+### 第三步启动xiaozhi-server服务
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - OTA接口是           http://192.168.4.7:8003/xiaozhi/ota/
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。
+
+
+## 全模块运行时，如何开启视觉模型
+
+### 第一步 确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose_all.yml`是否映射了`8003`端口，如果没有就更新最新的`docker-compose_all.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步 确认你配置文件
+
+打开你的`data/.config.yaml`文件，确认一下你的配置文件的结构，是否和`data/config_from_api.yaml`一样。如果不一样，或缺少某项，请补齐。
+
+### 第三步 配置视觉模型密钥
+
+那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+登录`智控台`，顶部菜单点击`模型配置`，在左侧栏点击`视觉打语言模型`，找到`VLLM_ChatGLMVLLM`，点击修改按钮，在弹框中，在`API密钥`输入你密钥，点击保存。
+
+保存成功后，去到你需要测试的智能体哪里，点击`配置角色`，在打开的内容里，查看`视觉大语言模型(VLLM)`是否选择了刚才的视觉模型。点击保存。
+
+### 第三步 启动xiaozhi-server模块
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。

main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java

@@ -227,7 +227,7 @@ public interface Constant {
     /**
      * 版本号
      */
-    public static final String VERSION = "0.5.2";
+    public static final String VERSION = "0.5.4";
 
     /**
      * 无效固件URL

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java

@@ -171,6 +171,9 @@ public class AgentController {
         if (dto.getLlmModelId() != null) {
             existingEntity.setLlmModelId(dto.getLlmModelId());
         }
+        if (dto.getVllmModelId() != null) {
+            existingEntity.setVllmModelId(dto.getVllmModelId());
+        }
         if (dto.getTtsModelId() != null) {
             existingEntity.setTtsModelId(dto.getTtsModelId());
         }

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentDTO.java

@@ -27,6 +27,9 @@ public class AgentDTO {
     @Schema(description = "大语言模型名称", example = "llm_model_01")
     private String llmModelName;
 
+    @Schema(description = "视觉模型名称", example = "vllm_model_01")
+    private String vllmModelName;
+
     @Schema(description = "记忆模型ID", example = "mem_model_01")
     private String memModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentUpdateDTO.java

@@ -30,6 +30,9 @@ public class AgentUpdateDTO implements Serializable {
     @Schema(description = "大语言模型标识", example = "llm_model_02", required = false)
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识", example = "vllm_model_02", required = false)
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识", example = "tts_model_02", required = false)
     private String ttsModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentEntity.java

@@ -36,6 +36,9 @@ public class AgentEntity {
     @Schema(description = "大语言模型标识")
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识")
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识")
     private String ttsModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentTemplateEntity.java

@@ -49,6 +49,11 @@ public class AgentTemplateEntity implements Serializable {
      */
     private String llmModelId;
 
+    /**
+     * VLLM模型标识
+     */
+    private String vllmModelId;
+
     /**
      * 语音合成模型标识
      */

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentServiceImpl.java

@@ -102,6 +102,9 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
             // 获取 LLM 模型名称
             dto.setLlmModelName(modelConfigService.getModelNameById(agent.getLlmModelId()));
 
+            // 获取 VLLM 模型名称
+            dto.setVllmModelName(modelConfigService.getModelNameById(agent.getVllmModelId()));
+
             // 获取记忆模型名称
             dto.setMemModelId(agent.getMemModelId());
 

main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java

@@ -72,6 +72,7 @@ public class ConfigServiceImpl implements ConfigService {
                 null,
                 null,
                 null,
+                null,
                 result,
                 isCache);
 
@@ -140,6 +141,7 @@ public class ConfigServiceImpl implements ConfigService {
                 agent.getVadModelId(),
                 agent.getAsrModelId(),
                 agent.getLlmModelId(),
+                agent.getVllmModelId(),
                 agent.getTtsModelId(),
                 agent.getMemModelId(),
                 agent.getIntentModelId(),
@@ -241,6 +243,7 @@ public class ConfigServiceImpl implements ConfigService {
             String vadModelId,
             String asrModelId,
             String llmModelId,
+            String vllmModelId,
             String ttsModelId,
             String memModelId,
             String intentModelId,
@@ -248,8 +251,8 @@ public class ConfigServiceImpl implements ConfigService {
             boolean isCache) {
         Map<String, String> selectedModule = new HashMap<>();
 
-        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM" };
+        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM", "VLLM" };
-        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId };
+        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId, vllmModelId };
         String intentLLMModelId = null;
         String memLocalShortLLMModelId = null;
 

main/manager-api/src/main/resources/db/changelog/202506010920.sql

@@ -0,0 +1,29 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_VLLM_openai';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_VLLM_openai', 'VLLM', 'openai', 'OpenAI接口', '[{"key":"base_url","label":"基础URL","type":"string"},{"key":"model_name","label":"模型名称","type":"string"},{"key":"api_key","label":"API密钥","type":"string"}]', 9, 1, NOW(), 1, NOW());
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_ChatGLMVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_ChatGLMVLLM', 'VLLM', 'ChatGLMVLLM', '智谱视觉AI', 1, 1, '{\"type\": \"openai\", \"model_name\": \"glm-4v-flash\", \"base_url\": \"https://open.bigmodel.cn/api/paas/v4/\", \"api_key\": \"你的api_key\"}', NULL, NULL, 1, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bigmodel.cn/usercenter/proj-mgmt/apikeys',
+`remark` = '智谱视觉AI配置说明：
+1. 访问 https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_ChatGLMVLLM';
+
+
+-- 添加参数
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (113, 'server.http_port', '8003', 'number', 1, 'http服务的端口，用于启动视觉分析接口');
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (114, 'server.vision_explain', 'null', 'string', 1, '视觉分析接口地址，用于下发到设备，多个用;分隔');
+
+-- 智能体表增加VLLM模型配置
+ALTER TABLE `ai_agent` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;
+
+-- 智能体模版表增加VLLM模型配置
+ALTER TABLE `ai_agent_template` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;

main/manager-api/src/main/resources/db/changelog/202506031639.sql

@@ -0,0 +1,45 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_DoubaoStreamASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_DoubaoStreamASR', 'ASR', 'doubao_stream', '火山引擎语音识别(流式)', '[{"key":"appid","label":"应用ID","type":"string"},{"key":"access_token","label":"访问令牌","type":"string"},{"key":"cluster","label":"集群","type":"string"},{"key":"boosting_table_name","label":"热词文件名称","type":"string"},{"key":"correct_table_name","label":"替换词文件名称","type":"string"},{"key":"output_dir","label":"输出目录","type":"string"}]', 3, 1, NOW(), 1, NOW());
+
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'ASR_DoubaoStreamASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_DoubaoStreamASR', 'ASR', 'DoubaoStreamASR', '豆包语音识别(流式)', 0, 1, '{\"type\": \"doubao_stream\", \"appid\": \"\", \"access_token\": \"\", \"cluster\": \"volcengine_input_common\", \"output_dir\": \"tmp/\"}', NULL, NULL, 3, NULL, NULL, NULL, NULL);
+
+
+-- 更新豆包ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.volcengine.com/speech/app',
+`remark` = '豆包ASR配置说明：
+1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
+2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
+3. 需要在火山引擎控制台创建应用并获取appid和access_token
+4. 支持中文语音识别
+5. 需要网络连接
+6. 输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://console.volcengine.com/speech/app
+2. 创建新应用
+3. 获取appid和access_token
+4. 填入配置文件中
+如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
+' WHERE `id` = 'ASR_DoubaoASR';
+
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.volcengine.com/speech/app',
+`remark` = '豆包ASR配置说明：
+1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
+2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
+3. 需要在火山引擎控制台创建应用并获取appid和access_token
+4. 支持中文语音识别
+5. 需要网络连接
+6. 输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://console.volcengine.com/speech/app
+2. 创建新应用
+3. 获取appid和access_token
+4. 填入配置文件中
+如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
+' WHERE `id` = 'ASR_DoubaoStreamASR';

main/manager-api/src/main/resources/db/changelog/202506032232.sql

@@ -0,0 +1,14 @@
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_QwenVLVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_QwenVLVLLM', 'VLLM', 'QwenVLVLLM', '千问视觉模型', 0, 1, '{\"type\": \"openai\", \"model_name\": \"qwen2.5-vl-3b-instruct\", \"base_url\": \"https://dashscope.aliyuncs.com/compatible-mode/v1\", \"api_key\": \"你的api_key\"}', NULL, NULL, 2, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bailian.console.aliyun.com/?tab=api#/api/?type=model&url=https%3A%2F%2Fhelp.aliyun.com%2Fdocument_detail%2F2845564.html&renderType=iframe',
+`remark` = '千问视觉模型配置说明：
+1. 访问 https://bailian.console.aliyun.com/?tab=model#/api-key
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_QwenVLVLLM';
+
+-- 删除参数，这两个参数已挪至python配置文件
+delete from `sys_params` where id  in (113,114);

main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml

@@ -169,4 +169,25 @@ databaseChangeLog:
       changes:
         - sqlFile:
             encoding: utf8
-            path: classpath:db/changelog/202505271414.sql
+            path: classpath:db/changelog/202505271414.sql
+  - changeSet:
+      id: 202506010920
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506010920.sql
+  - changeSet:
+      id: 202506031639
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506031639.sql
+  - changeSet:
+      id: 202506032232
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506032232.sql

main/manager-web/src/components/DeviceItem.vue

@@ -14,10 +14,10 @@
       </div>
     </div>
     <div class="device-name">
-      设备型号：{{ device.ttsModelName }}
+      语言模型：{{ device.llmModelName }}
     </div>
     <div class="device-name">
-      音色模型：{{ device.ttsVoiceName }}
+      音色模型：{{ device.ttsModelName }} ({{ device.ttsVoiceName }})
     </div>
     <div style="display: flex;gap: 10px;align-items: center;">
       <div class="settings-btn" @click="handleConfigure">

main/manager-web/src/views/ModelConfig.vue

@@ -30,6 +30,9 @@
           <el-menu-item index="llm">
             <span class="menu-text">大语言模型</span>
           </el-menu-item>
+          <el-menu-item index="vllm">
+            <span class="menu-text">视觉大模型</span>
+          </el-menu-item>
           <el-menu-item index="intent">
             <span class="menu-text">意图识别</span>
           </el-menu-item>
@@ -173,6 +176,7 @@ export default {
         vad: '语言活动检测模型(VAD)',
         asr: '语音识别模型(ASR)',
         llm: '大语言模型（LLM）',
+        vllm: '视觉大模型（VLLM）',
         intent: '意图识别模型(Intent)',
         tts: '语音合成模型(TTS)',
         memory: '记忆模型(Memory)'

main/manager-web/src/views/roleConfig.vue

@@ -64,7 +64,27 @@
                     </el-form-item>
                   </div>
                   <div class="form-column">
-                    <el-form-item v-for="(model, index) in models" :key="`model-${index}`" :label="model.label"
+                    <div class="model-row">
+                      <el-form-item label="语音活动检测(VAD)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.vadModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('VAD', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['VAD']"
+                              :key="`option-vad-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                      <el-form-item label="语音识别(ASR)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.asrModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('ASR', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['ASR']"
+                              :key="`option-asr-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                    </div>
+                    <el-form-item v-for="(model, index) in models.slice(2)" :key="`model-${index}`" :label="model.label"
                       class="model-item">
                       <div class="model-select-wrapper">
                         <el-select v-model="form.model[model.key]" filterable placeholder="请选择" class="form-select"
@@ -148,6 +168,7 @@ export default {
           vadModelId: "",
           asrModelId: "",
           llmModelId: "",
+          vllmModelId: "",
           memModelId: "",
           intentModelId: "",
         }
@@ -156,6 +177,7 @@ export default {
         { label: '语音活动检测(VAD)', key: 'vadModelId', type: 'VAD' },
         { label: '语音识别(ASR)', key: 'asrModelId', type: 'ASR' },
         { label: '大语言模型(LLM)', key: 'llmModelId', type: 'LLM' },
+        { label: '视觉大模型(VLLM)', key: 'vllmModelId', type: 'VLLM' },
         { label: '意图识别(Intent)', key: 'intentModelId', type: 'Intent' },
         { label: '记忆(Memory)', key: 'memModelId', type: 'Memory' },
         { label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' },
@@ -189,6 +211,7 @@ export default {
         asrModelId: this.form.model.asrModelId,
         vadModelId: this.form.model.vadModelId,
         llmModelId: this.form.model.llmModelId,
+        vllmModelId: this.form.model.vllmModelId,
         ttsModelId: this.form.model.ttsModelId,
         ttsVoiceId: this.form.ttsVoiceId,
         chatHistoryConf: this.form.chatHistoryConf,
@@ -236,6 +259,7 @@ export default {
             vadModelId: "",
             asrModelId: "",
             llmModelId: "",
+            vllmModelId: "",
             memModelId: "",
             intentModelId: "",
           }
@@ -289,6 +313,7 @@ export default {
           vadModelId: templateData.vadModelId || this.form.model.vadModelId,
           asrModelId: templateData.asrModelId || this.form.model.asrModelId,
           llmModelId: templateData.llmModelId || this.form.model.llmModelId,
+          vllmModelId: templateData.vllmModelId || this.form.model.vllmModelId,
           memModelId: templateData.memModelId || this.form.model.memModelId,
           intentModelId: templateData.intentModelId || this.form.model.intentModelId
         }
@@ -305,6 +330,7 @@ export default {
               vadModelId: data.data.vadModelId,
               asrModelId: data.data.asrModelId,
               llmModelId: data.data.llmModelId,
+              vllmModelId: data.data.vllmModelId,
               memModelId: data.data.memModelId,
               intentModelId: data.data.intentModelId
             }
@@ -587,6 +613,25 @@ export default {
   width: 100%;
 }
 
+.model-row {
+  display: flex;
+  gap: 20px;
+  margin-bottom: 6px;
+}
+
+.model-row .model-item {
+  flex: 1;
+  margin-bottom: 0;
+}
+
+.model-row .el-form-item__label {
+  font-size: 12px !important;
+  color: #3d4566 !important;
+  font-weight: 400;
+  line-height: 22px;
+  padding-bottom: 2px;
+}
+
 .function-icons {
   display: flex;
   align-items: center;

main/xiaozhi-server/app.py

@@ -1,11 +1,12 @@
 import sys
+import uuid
 import signal
 import asyncio
 from aioconsole import ainput
 from config.settings import load_config
 from config.logger import setup_logging
 from core.utils.util import get_local_ip
-from core.ota_server import SimpleOtaServer
+from core.http_server import SimpleHttpServer
 from core.websocket_server import WebSocketServer
 from core.utils.util import check_ffmpeg_installed
 
@@ -45,25 +46,37 @@ async def main():
     check_ffmpeg_installed()
     config = load_config()
 
+    # 默认使用manager-api的secret作为auth_key
+    # 如果secret为空，则生成随机密钥
+    # auth_key用于jwt认证，比如视觉分析接口的jwt认证
+    auth_key = config.get("manager-api", {}).get("secret", "")
+    if not auth_key or len(auth_key) == 0 or "你" in auth_key:
+        auth_key = str(uuid.uuid4().hex)
+    config["server"]["auth_key"] = auth_key
+
     # 添加 stdin 监控任务
     stdin_task = asyncio.create_task(monitor_stdin())
 
     # 启动 WebSocket 服务器
     ws_server = WebSocketServer(config)
     ws_task = asyncio.create_task(ws_server.start())
-    ota_task = None
+    # 启动 Simple http 服务器
+    ota_server = SimpleHttpServer(config)
+    ota_task = asyncio.create_task(ota_server.start())
 
     read_config_from_api = config.get("read_config_from_api", False)
+    port = int(config["server"].get("http_port", 8003))
     if not read_config_from_api:
-        # 启动 Simple OTA 服务器
-        ota_server = SimpleOtaServer(config)
-        ota_task = asyncio.create_task(ota_server.start())
-
         logger.bind(tag=TAG).info(
             "OTA接口是\t\thttp://{}:{}/xiaozhi/ota/",
             get_local_ip(),
-            config["server"]["ota_port"],
+            port,
         )
+    logger.bind(tag=TAG).info(
+        "视觉分析接口是\thttp://{}:{}/mcp/vision/explain",
+        get_local_ip(),
+        port,
+    )
 
     # 获取WebSocket配置，使用安全的默认值
     websocket_port = 8000

main/xiaozhi-server/config.yaml

@@ -2,6 +2,7 @@
 # 然后你想修改覆盖修改什么配置，就修改【.config.yaml】文件，而不是修改【config.yaml】文件
 # 系统会优先读取【data/.config.yaml】文件的配置，如果【.config.yaml】文件里的配置不存在，系统会自动去读取【config.yaml】文件的配置。
 # 这样做，可以最简化配置，保护您的密钥安全。
+# 如果你使用了智控台，那么以下所有配置，都不会生效，请在智控台中修改配置
 
 # #####################################################################################
 # #############################以下是服务器基本运行配置####################################
@@ -9,13 +10,21 @@ server:
   # 服务器监听地址和端口(Server listening address and port)
   ip: 0.0.0.0
   port: 8000
-  # OTA接口的端口号
+  # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-  ota_port: 8002
+  http_port: 8003
   # 这个websocket配置是指ota接口向设备发送的websocket地址
-  # 如果按默认的写法，ota接口会自动生成websocket地址。这个地址你可以直接用浏览器访问ota接口确认一下
+  # 如果按默认的写法，ota接口会自动生成websocket地址，并输出在启动日志里，这个地址你可以直接用浏览器访问ota接口确认一下
   # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
-  # 所以如果你使用docker部署或使用公网部署时，请设置正确的websocket地址
+  # 所以如果你使用docker部署时，将websocket设置成局域网地址
+  # 如果你使用公网部署时，将vwebsocket设置成公网地址
   websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
   # OTA返回信息时区偏移量
   timezone_offset: +8
   # 认证配置
@@ -160,6 +169,8 @@ selected_module:
   ASR: FunASR
   # 将根据配置名称对应的type调用实际的LLM适配器
   LLM: ChatGLMLLM
+  # 视觉语言大模型
+  VLLM: ChatGLMVLLM
   # TTS将根据配置名称对应的type调用实际的TTS适配器
   TTS: EdgeTTS
   # 记忆模块，默认不开启记忆；如果想使用超长记忆，推荐使用mem0ai；如果注重隐私，请使用本地的mem_local_short
@@ -253,6 +264,8 @@ ASR:
   DoubaoASR:
     # 可以在这里申请相关Key等信息
     # https://console.volcengine.com/speech/app
+    # DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
+    # 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
     type: doubao
     appid: 你的火山引擎语音合成服务appid
     access_token: 你的火山引擎语音合成服务access_token
@@ -261,6 +274,20 @@ ASR:
     boosting_table_name: （选填）你的热词文件名称
     correct_table_name: （选填）你的替换词文件名称
     output_dir: tmp/
+  DoubaoStreamASR:
+    # 可以在这里申请相关Key等信息
+    # https://console.volcengine.com/speech/app
+    # DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
+    # 开通地址https://console.volcengine.com/speech/service/10011
+    # 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
+    type: doubao_stream
+    appid: 你的火山引擎语音合成服务appid
+    access_token: 你的火山引擎语音合成服务access_token
+    cluster: volcengine_input_common
+    # 热词、替换词使用流程：https://www.volcengine.com/docs/6561/155738
+    boosting_table_name: （选填）你的热词文件名称
+    correct_table_name: （选填）你的替换词文件名称
+    output_dir: tmp/
   TencentASR:
     # token申请地址：https://console.cloud.tencent.com/cam/capi
     # 免费领取资源：https://console.cloud.tencent.com/asr/resourcebundle
@@ -432,6 +459,21 @@ LLM:
     # Xinference服务地址和模型名称
     model_name: qwen2.5:3b-AWQ  # 使用的小模型名称，用于意图识别
     base_url: http://localhost:9997  # Xinference服务地址
+# VLLM配置（视觉语言大模型）
+VLLM:
+  ChatGLMVLLM:
+    type: openai
+    # glm-4v-flash是智谱免费AI的视觉模型，需要先在智谱AI平台创建API密钥并获取api_key
+    # 可在这里找到你的api key https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+    model_name: glm-4v-flash  # 智谱AI的视觉模型
+    url: https://open.bigmodel.cn/api/paas/v4/
+    api_key: 你的api_key
+  QwenVLVLLM:
+    type: openai
+    model_name: qwen2.5-vl-3b-instruct
+    url: https://dashscope.aliyuncs.com/compatible-mode/v1
+    # 可在这里找到你的api key https://bailian.console.aliyun.com/?apiKey=1#/api-key
+    api_key: 你的api_key
 TTS:
   # 当前支持的type为edge、doubao，可自行适配
   EdgeTTS:

main/xiaozhi-server/config/config_loader.py

@@ -59,10 +59,14 @@ def get_config_from_api(config):
         "url": config["manager-api"].get("url", ""),
         "secret": config["manager-api"].get("secret", ""),
     }
+    # server的配置以本地为准
     if config.get("server"):
         config_data["server"] = {
             "ip": config["server"].get("ip", ""),
             "port": config["server"].get("port", ""),
+            "http_port": config["server"].get("http_port", ""),
+            "vision_explain": config["server"].get("vision_explain", ""),
+            "auth_key": config["server"].get("auth_key", ""),
         }
     return config_data
 

main/xiaozhi-server/config/logger.py

@@ -4,7 +4,7 @@ from loguru import logger
 from config.config_loader import load_config
 from config.settings import check_config_file
 
-SERVER_VERSION = "0.5.2"
+SERVER_VERSION = "0.5.4"
 _logger_initialized = False
 
 

main/xiaozhi-server/config_from_api.yaml

@@ -8,6 +8,15 @@
 server:
   ip: 0.0.0.0
   port: 8000
+  # http服务的端口，用于视觉分析接口
+  http_port: 8003
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
 manager-api:
   # 你的manager-api的地址，最好使用局域网ip
   url: http://127.0.0.1:8002/xiaozhi

main/xiaozhi-server/core/api/base_handler.py

@@ -0,0 +1,16 @@
+from aiohttp import web
+from config.logger import setup_logging
+
+
+class BaseHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/api/ota_handler.py

@@ -1,18 +1,15 @@
 import json
 import time
-import asyncio
 from aiohttp import web
-from config.logger import setup_logging
 from core.utils.util import get_local_ip
-from core.utils.modules_initialize import initialize_modules
+from core.api.base_handler import BaseHandler
 
 TAG = __name__
 
 
-class SimpleOtaServer:
+class OTAHandler(BaseHandler):
     def __init__(self, config: dict):
-        self.config = config
+        super().__init__(config)
-        self.logger = setup_logging()
 
     def _get_websocket_url(self, local_ip: str, port: int) -> str:
         """获取websocket地址
@@ -25,41 +22,15 @@ class SimpleOtaServer:
             str: websocket地址
         """
         server_config = self.config["server"]
-        websocket_config = server_config.get("websocket")
+        websocket_config = server_config.get("websocket", "")
 
-        if websocket_config and "你" not in websocket_config:
+        if "你的" not in websocket_config:
             return websocket_config
         else:
             return f"ws://{local_ip}:{port}/xiaozhi/v1/"
 
-    async def start(self):
+    async def handle_post(self, request):
-        server_config = self.config["server"]
+        """处理 OTA POST 请求"""
-        host = server_config.get("ip", "0.0.0.0")
-        port = int(server_config.get("ota_port"))
-
-        if port:
-            app = web.Application()
-            # 添加路由
-            app.add_routes(
-                [
-                    web.get("/xiaozhi/ota/", self._handle_ota_get_request),
-                    web.post("/xiaozhi/ota/", self._handle_ota_request),
-                    web.options("/xiaozhi/ota/", self._handle_ota_request),
-                ]
-            )
-
-            # 运行服务
-            runner = web.AppRunner(app)
-            await runner.setup()
-            site = web.TCPSite(runner, host, port)
-            await site.start()
-
-            # 保持服务运行
-            while True:
-                await asyncio.sleep(3600)  # 每隔 1 小时检查一次
-
-    async def _handle_ota_request(self, request):
-        """处理 /xiaozhi/ota/ 的 POST 请求"""
         try:
             data = await request.text()
             self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}")
@@ -75,11 +46,9 @@ class SimpleOtaServer:
             data_json = json.loads(data)
 
             server_config = self.config["server"]
-            host = server_config.get("ip", "0.0.0.0")
             port = int(server_config.get("port", 8000))
             local_ip = get_local_ip()
 
-            # OTA基础信息
             return_json = {
                 "server_time": {
                     "timestamp": int(round(time.time() * 1000)),
@@ -104,16 +73,11 @@ class SimpleOtaServer:
                 content_type="application/json",
             )
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response
 
-    async def _handle_ota_get_request(self, request):
+    async def handle_get(self, request):
-        """处理 /xiaozhi/ota/ 的 GET 请求"""
+        """处理 OTA GET 请求"""
         try:
             server_config = self.config["server"]
             local_ip = get_local_ip()
@@ -125,10 +89,5 @@ class SimpleOtaServer:
             self.logger.bind(tag=TAG).error(f"OTA GET请求异常: {e}")
             response = web.Response(text="OTA接口异常", content_type="text/plain")
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response

main/xiaozhi-server/core/api/vision_handler.py

@@ -0,0 +1,184 @@
+import json
+import copy
+from aiohttp import web
+from config.logger import setup_logging
+from core.utils.util import get_vision_url, is_valid_image_file
+from core.utils.vllm import create_instance
+from config.config_loader import get_private_config_from_api
+from core.utils.auth import AuthToken
+import base64
+from typing import Tuple, Optional
+
+TAG = __name__
+
+# 设置最大文件大小为5MB
+MAX_FILE_SIZE = 5 * 1024 * 1024
+
+
+class VisionHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        # 初始化认证工具
+        self.auth = AuthToken(config["server"]["auth_key"])
+
+    def _create_error_response(self, message: str) -> dict:
+        """创建统一的错误响应格式"""
+        return {"success": False, "message": message}
+
+    def _verify_auth_token(self, request) -> Tuple[bool, Optional[str]]:
+        """验证认证token"""
+        auth_header = request.headers.get("Authorization", "")
+        if not auth_header.startswith("Bearer "):
+            return False, None
+
+        token = auth_header[7:]  # 移除"Bearer "前缀
+        return self.auth.verify_token(token)
+
+    async def handle_post(self, request):
+        """处理 MCP Vision POST 请求"""
+        response = None  # 初始化response变量
+        try:
+            # 验证token
+            is_valid, token_device_id = self._verify_auth_token(request)
+            if not is_valid:
+                response = web.Response(
+                    text=json.dumps(
+                        self._create_error_response("无效的认证token或token已过期")
+                    ),
+                    content_type="application/json",
+                    status=401,
+                )
+                return response
+
+            # 获取请求头信息
+            device_id = request.headers.get("Device-Id", "")
+            client_id = request.headers.get("Client-Id", "")
+            if device_id != token_device_id:
+                return web.Response(
+                    text=json.dumps(self._create_error_response("设备ID与token不匹配")),
+                    content_type="application/json",
+                    status=401,
+                )
+            # 解析multipart/form-data请求
+            reader = await request.multipart()
+
+            # 读取question字段
+            question_field = await reader.next()
+            if question_field is None:
+                raise ValueError("缺少问题字段")
+            question = await question_field.text()
+            self.logger.bind(tag=TAG).debug(f"Question: {question}")
+
+            # 读取图片文件
+            image_field = await reader.next()
+            if image_field is None:
+                raise ValueError("缺少图片文件")
+
+            # 读取图片数据
+            image_data = await image_field.read()
+            if not image_data:
+                raise ValueError("图片数据为空")
+
+            # 检查文件大小
+            if len(image_data) > MAX_FILE_SIZE:
+                raise ValueError(
+                    f"图片大小超过限制，最大允许{MAX_FILE_SIZE/1024/1024}MB"
+                )
+
+            # 检查文件格式
+            if not is_valid_image_file(image_data):
+                raise ValueError(
+                    "不支持的文件格式，请上传有效的图片文件（支持JPEG、PNG、GIF、BMP、TIFF、WEBP格式）"
+                )
+
+            # 将图片转换为base64编码
+            image_base64 = base64.b64encode(image_data).decode("utf-8")
+
+            # 如果开启了智控台，则从智控台获取模型配置
+            current_config = copy.deepcopy(self.config)
+            read_config_from_api = current_config.get("read_config_from_api", False)
+            if read_config_from_api:
+                current_config = get_private_config_from_api(
+                    current_config,
+                    device_id,
+                    client_id,
+                )
+
+            select_vllm_module = current_config["selected_module"].get("VLLM")
+            if not select_vllm_module:
+                raise ValueError("您还未设置默认的视觉分析模块")
+
+            vllm_type = (
+                select_vllm_module
+                if "type" not in current_config["VLLM"][select_vllm_module]
+                else current_config["VLLM"][select_vllm_module]["type"]
+            )
+
+            if not vllm_type:
+                raise ValueError(f"无法找到VLLM模块对应的供应器{vllm_type}")
+
+            vllm = create_instance(
+                vllm_type, current_config["VLLM"][select_vllm_module]
+            )
+
+            result = vllm.response(question, image_base64)
+
+            return_json = {
+                "success": True,
+                "result": result,
+            }
+
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except ValueError as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response(str(e))
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response("处理请求时发生错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            if response:
+                self._add_cors_headers(response)
+            return response
+
+    async def handle_get(self, request):
+        """处理 MCP Vision GET 请求"""
+        try:
+            vision_explain = get_vision_url(self.config)
+            if vision_explain and len(vision_explain) > 0 and "null" != vision_explain:
+                message = (
+                    f"MCP Vision 接口运行正常，视觉解释接口地址是：{vision_explain}"
+                )
+            else:
+                message = "MCP Vision 接口运行不正常，请打开data目录下的.config.yaml文件，找到【server.vision_explain】，设置好地址"
+
+            response = web.Response(text=message, content_type="text/plain")
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision GET请求异常: {e}")
+            return_json = self._create_error_response("服务器内部错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            self._add_cors_headers(response)
+            return response
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/connection.py

@@ -35,7 +35,6 @@ from plugins_func.loadplugins import auto_import_modules
 from plugins_func.register import Action, ActionResponse
 from core.auth import AuthMiddleware, AuthenticationError
 from config.config_loader import get_private_config_from_api
-from core.handle.receiveAudioHandle import handleAudioMessage
 from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType
 from config.logger import setup_logging, build_module_string, update_module_string
 from config.manage_api_client import DeviceNotFoundException, DeviceBindException
@@ -81,6 +80,7 @@ class ConnectionHandler:
         self.welcome_msg = None
         self.max_output_size = 0
         self.chat_history_conf = 0
+        self.audio_format = "opus"
 
         # 客户端状态相关
         self.client_abort = False
@@ -117,7 +117,10 @@ class ConnectionHandler:
         self.client_voice_stop = False
 
         # asr相关变量
+        # 因为实际部署时可能会用到公共的本地ASR，不能把变量暴露给公共ASR
+        # 所以涉及到ASR的变量，需要在这里定义，属于connection的私有变量
         self.asr_audio = []
+        self.asr_audio_queue = queue.Queue()
 
         # llm相关变量
         self.llm_finish_task = True
@@ -146,7 +149,6 @@ class ConnectionHandler:
             int(self.config.get("close_connection_no_voice_time", 120)) + 60
         )  # 在原来第一道关闭的基础上加60秒，进行二道关闭
 
-        self.audio_format = "opus"
         # {"mcp":true} 表示启用MCP功能
         self.features = None
 
@@ -254,7 +256,11 @@ class ConnectionHandler:
         if isinstance(message, str):
             await handleTextMessage(self, message)
         elif isinstance(message, bytes):
-            await handleAudioMessage(self, message)
+            if self.vad is None:
+                return
+            if self.asr is None:
+                return
+            self.asr_audio_queue.put(message)
 
     async def handle_restart(self, message):
         """处理服务器重启请求"""
@@ -478,6 +484,8 @@ class ConnectionHandler:
             self.memory = modules["memory"]
 
     def _initialize_memory(self):
+        if self.memory is None:
+            return
         """初始化记忆模块"""
         self.memory.init_memory(
             role_id=self.device_id,
@@ -518,6 +526,8 @@ class ConnectionHandler:
                 self.logger.bind(tag=TAG).info("使用主LLM作为意图识别模型")
 
     def _initialize_intent(self):
+        if self.intent is None:
+            return
         self.intent_type = self.config["Intent"][
             self.config["selected_module"]["Intent"]
         ]["type"]
@@ -708,15 +718,24 @@ class ConnectionHandler:
                 # 处理Server端MCP工具调用
                 if self.mcp_manager.is_mcp_tool(function_name):
                     result = self._handle_mcp_tool_call(function_call_data)
-                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(function_name):
+                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(
-                    # 如果是小智端MCP工具调用 
+                    function_name
+                ):
+                    # 如果是小智端MCP工具调用
                     self.logger.bind(tag=TAG).debug(
                         f"调用小智端MCP工具: {function_name}, 参数: {function_arguments}"
                     )
                     try:
-                        result = asyncio.run_coroutine_threadsafe(call_mcp_tool(self, self.mcp_client, function_name, function_arguments), self.loop).result()
+                        result = asyncio.run_coroutine_threadsafe(
+                            call_mcp_tool(
+                                self, self.mcp_client, function_name, function_arguments
+                            ),
+                            self.loop,
+                        ).result()
                         self.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
-                        result = ActionResponse(action=Action.REQLLM, result=result, response="")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result=result, response=""
+                        )
                     except Exception as e:
                         self.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
                         result = ActionResponse(

main/xiaozhi-server/core/handle/abortHandle.py

@@ -1,6 +1,4 @@
 import json
-import queue
-from config.logger import setup_logging
 
 TAG = __name__
 

main/xiaozhi-server/core/handle/helloHandle.py

@@ -4,10 +4,14 @@ import json
 import random
 import shutil
 import asyncio
-from core.handle.mcpHandle import MCPClient, send_mcp_initialize_message, send_mcp_tools_list_request
 from core.handle.sendAudioHandle import send_stt_message
 from core.utils.util import remove_punctuation_and_length
 from core.providers.tts.dto.dto import ContentType, InterfaceType
+from core.handle.mcpHandle import (
+    MCPClient,
+    send_mcp_initialize_message,
+    send_mcp_tools_list_request,
+)
 
 
 TAG = __name__
@@ -29,8 +33,6 @@ async def handleHelloMessage(conn, msg_json):
         format = audio_params.get("format")
         conn.logger.bind(tag=TAG).info(f"客户端音频格式: {format}")
         conn.audio_format = format
-        if conn.asr is not None:
-            conn.asr.set_audio_format(format)
         conn.welcome_msg["audio_params"] = audio_params
     features = msg_json.get("features")
     if features:
@@ -61,6 +63,8 @@ async def checkWakeupWords(conn, text):
     """检查是否是唤醒词"""
     _, filtered_text = remove_punctuation_and_length(text)
     if filtered_text in conn.config.get("wakeup_words"):
+        # 设置刚刚被唤醒的标志
+        conn.just_woken_up = True
         await send_stt_message(conn, text)
 
         file = getWakeupWordFile(WAKEUP_CONFIG["file_name"])
@@ -70,6 +74,8 @@ async def checkWakeupWords(conn, text):
         text_hello = WAKEUP_CONFIG["text"]
         if not text_hello:
             text_hello = text
+        if conn.tts is None:
+            return False
         conn.tts.tts_one_sentence(
             conn, ContentType.FILE, content_file=file, content_detail=text_hello
         )

main/xiaozhi-server/core/handle/intentHandler.py

@@ -96,6 +96,7 @@ async def process_intent_result(conn, intent_result, original_text):
             }
 
             await send_stt_message(conn, original_text)
+            conn.client_abort = False
 
             # 使用executor执行函数调用和结果处理
             def process_function_call():

main/xiaozhi-server/core/handle/mcpHandle.py

@@ -1,6 +1,8 @@
 import json
 import asyncio
 from concurrent.futures import Future
+from core.utils.util import get_vision_url
+from core.utils.auth import AuthToken
 
 TAG = __name__
 
@@ -205,6 +207,20 @@ async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
 
 async def send_mcp_initialize_message(conn):
     """发送MCP初始化消息"""
+
+    vision_url = get_vision_url(conn.config)
+
+    # 密钥生成token
+    auth = AuthToken(conn.config["server"]["auth_key"])
+    token = auth.generate_token(conn.headers.get("device-id"))
+
+    vision = {
+        "url": vision_url,
+        "token": token,
+    }
+
+    conn.logger.bind(tag=TAG).info(f"视觉服务信息: {vision}")
+
     payload = {
         "jsonrpc": "2.0",
         "id": 1,  # mcpInitializeID
@@ -214,6 +230,7 @@ async def send_mcp_initialize_message(conn):
             "capabilities": {
                 "roots": {"listChanged": True},
                 "sampling": {},
+                "vision": vision,
             },
             "clientInfo": {
                 "name": "XiaozhiClient",

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -3,6 +3,7 @@ from core.handle.intentHandler import handle_user_intent
 from core.utils.output_counter import check_device_output_limit
 from core.handle.abortHandle import handleAbortMessage
 import time
+import asyncio
 from core.handle.sendAudioHandle import SentenceType
 from core.utils.util import audio_to_data
 
@@ -10,21 +11,28 @@ TAG = __name__
 
 
 async def handleAudioMessage(conn, audio):
-    if conn.vad is None:
-        conn.logger.bind(tag=TAG).warning("VAD模块未初始化，继续等待")
-        return
-    if conn.asr is None or not hasattr(conn.asr, "conn") or conn.asr.conn is None:
-        conn.logger.bind(tag=TAG).warning("ASR模块未初始化或通道未就绪，继续等待")
-        return
     # 当前片段是否有人说话
     have_voice = conn.vad.is_vad(conn, audio)
+
+    # 如果设备刚刚被唤醒，短暂忽略VAD检测
+    if hasattr(conn, "just_woken_up") and conn.just_woken_up:
+        have_voice = False
+        # 设置一个短暂延迟后恢复VAD检测
+        asyncio.create_task(resume_vad_detection(conn))
+
     if have_voice:
         if conn.client_is_speaking:
             await handleAbortMessage(conn)
     # 设备长时间空闲检测，用于say goodbye
     await no_voice_close_connect(conn, have_voice)
     # 接收音频
-    await conn.asr.receive_audio(audio, have_voice)
+    await conn.asr.receive_audio(conn, audio, have_voice)
+
+
+async def resume_vad_detection(conn):
+    # 等待2秒后恢复VAD检测
+    await asyncio.sleep(2)
+    conn.just_woken_up = False
 
 
 async def startToChat(conn, text):

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -89,8 +89,7 @@ async def sendAudio(conn, audios, pre_buffer=True):
     # 播放剩余音频帧
     for opus_packet in remaining_audios:
         if conn.client_abort:
-            conn.client_abort = False
+            break
-            return
 
         # 每分钟重置一次计时器
         if time.perf_counter() - last_reset_time > 60:

main/xiaozhi-server/core/handle/textHandle.py

@@ -61,6 +61,7 @@ async def handleTextMessage(conn, message):
                         await send_tts_message(conn, "stop", None)
                         conn.client_is_speaking = False
                     elif is_wakeup_words:
+                        conn.just_woken_up = True
                         # 上报纯文字数据（复用ASR上报功能，但不提供音频数据）
                         enqueue_asr_report(conn, "嘿，你好呀", [])
                         await startToChat(conn, "嘿，你好呀")
@@ -78,7 +79,9 @@ async def handleTextMessage(conn, message):
         elif msg_json["type"] == "mcp":
             conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message}")
             if "payload" in msg_json:
-                asyncio.create_task(handle_mcp_message(conn, conn.mcp_client, msg_json["payload"]))
+                asyncio.create_task(
+                    handle_mcp_message(conn, conn.mcp_client, msg_json["payload"])
+                )
         elif msg_json["type"] == "server":
             # 记录日志时过滤敏感信息
             conn.logger.bind(tag=TAG).info(

main/xiaozhi-server/core/http_server.py

@@ -0,0 +1,71 @@
+import asyncio
+from aiohttp import web
+from config.logger import setup_logging
+from core.api.ota_handler import OTAHandler
+from core.api.vision_handler import VisionHandler
+
+TAG = __name__
+
+
+class SimpleHttpServer:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        self.ota_handler = OTAHandler(config)
+        self.vision_handler = VisionHandler(config)
+
+    def _get_websocket_url(self, local_ip: str, port: int) -> str:
+        """获取websocket地址
+
+        Args:
+            local_ip: 本地IP地址
+            port: 端口号
+
+        Returns:
+            str: websocket地址
+        """
+        server_config = self.config["server"]
+        websocket_config = server_config.get("websocket")
+
+        if websocket_config and "你" not in websocket_config:
+            return websocket_config
+        else:
+            return f"ws://{local_ip}:{port}/xiaozhi/v1/"
+
+    async def start(self):
+        server_config = self.config["server"]
+        host = server_config.get("ip", "0.0.0.0")
+        port = int(server_config.get("http_port", 8003))
+
+        if port:
+            app = web.Application()
+
+            read_config_from_api = server_config.get("read_config_from_api", False)
+
+            if not read_config_from_api:
+                # 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
+                app.add_routes(
+                    [
+                        web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
+                        web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
+                        web.options("/xiaozhi/ota/", self.ota_handler.handle_post),
+                    ]
+                )
+            # 添加路由
+            app.add_routes(
+                [
+                    web.get("/mcp/vision/explain", self.vision_handler.handle_get),
+                    web.post("/mcp/vision/explain", self.vision_handler.handle_post),
+                    web.options("/mcp/vision/explain", self.vision_handler.handle_post),
+                ]
+            )
+
+            # 运行服务
+            runner = web.AppRunner(app)
+            await runner.setup()
+            site = web.TCPSite(runner, host, port)
+            await site.start()
+
+            # 保持服务运行
+            while True:
+                await asyncio.sleep(3600)  # 每隔 1 小时检查一次

main/xiaozhi-server/core/mcp/MCPClient.py

@@ -92,11 +92,21 @@ class MCPClient:
                         args=self.config.get("args", []),
                         env=env,
                     )
-                    stdio_r, stdio_w = await stack.enter_async_context(stdio_client(params))
+                    stdio_r, stdio_w = await stack.enter_async_context(
+                        stdio_client(params)
+                    )
                     read_stream, write_stream = stdio_r, stdio_w
                 # 建立SSEClient
                 elif "url" in self.config:
-                    sse_r, sse_w = await stack.enter_async_context(sse_client(self.config["url"]))
+                    if "API_ACCESS_TOKEN" in self.config:
+                        headers = {
+                            "Authorization": f"Bearer {self.config['API_ACCESS_TOKEN']}"
+                        }
+                    else:
+                        headers = {}
+                    sse_r, sse_w = await stack.enter_async_context(
+                        sse_client(self.config["url"], headers=headers)
+                    )
                     read_stream, write_stream = sse_r, sse_w
 
                 else:

main/xiaozhi-server/core/providers/asr/aliyun.py

@@ -213,7 +213,7 @@ class ASRProvider(ASRProviderBase):
             return None
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if self._is_token_expired():
@@ -223,7 +223,7 @@ class ASRProvider(ASRProviderBase):
         file_path = None
         try:
             # 解码Opus为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/baidu.py

@@ -30,7 +30,7 @@ class ASRProvider(ASRProviderBase):
         os.makedirs(self.output_dir, exist_ok=True)
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if not opus_data:
@@ -45,7 +45,7 @@ class ASRProvider(ASRProviderBase):
                 return None, file_path
 
             # 将Opus音频数据解码为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/base.py

@@ -1,15 +1,19 @@
 import os
-import time
+import wave
 import copy
 import uuid
-import wave
+import queue
+import asyncio
+import traceback
+import threading
 import opuslib_next
 from abc import ABC, abstractmethod
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
-from core.utils.util import remove_punctuation_and_length
-from core.handle.reportHandle import enqueue_asr_report
 from core.handle.receiveAudioHandle import startToChat
+from core.handle.reportHandle import enqueue_asr_report
+from core.utils.util import remove_punctuation_and_length
+from core.handle.receiveAudioHandle import handleAudioMessage
 
 TAG = __name__
 logger = setup_logging()
@@ -17,53 +21,75 @@ logger = setup_logging()
 
 class ASRProviderBase(ABC):
     def __init__(self):
-        self.audio_format = "opus"
+        pass
-        self.conn = None
 
     # 打开音频通道
     # 这里默认是非流式的处理方式
     # 流式处理方式请在子类中重写
     async def open_audio_channels(self, conn):
-        self.conn = conn
+        # tts 消化线程
+        conn.asr_priority_thread = threading.Thread(
+            target=self.asr_text_priority_thread, args=(conn,), daemon=True
+        )
+        conn.asr_priority_thread.start()
+
+    # 有序处理ASR音频
+    def asr_text_priority_thread(self, conn):
+        while not conn.stop_event.is_set():
+            try:
+                message = conn.asr_audio_queue.get(timeout=1)
+                future = asyncio.run_coroutine_threadsafe(
+                    handleAudioMessage(conn, message),
+                    conn.loop,
+                )
+                future.result()
+            except queue.Empty:
+                continue
+            except Exception as e:
+                logger.bind(tag=TAG).error(
+                    f"处理ASR文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
+                )
+                continue
 
     # 接收音频
     # 这里默认是非流式的处理方式
     # 流式处理方式请在子类中重写
-    async def receive_audio(self, audio, audio_have_voice):
+    async def receive_audio(self, conn, audio, audio_have_voice):
-        if (
+        if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
-            self.conn.client_listen_mode == "auto"
-            or self.conn.client_listen_mode == "realtime"
-        ):
             have_voice = audio_have_voice
         else:
-            have_voice = self.conn.client_have_voice
+            have_voice = conn.client_have_voice
         # 如果本次没有声音，本段也没声音，就把声音丢弃了
-        self.conn.asr_audio.append(audio)
+        conn.asr_audio.append(audio)
-        if have_voice == False and self.conn.client_have_voice == False:
+        if have_voice == False and conn.client_have_voice == False:
-            self.conn.asr_audio = self.conn.asr_audio[-10:]
+            conn.asr_audio = conn.asr_audio[-10:]
             return
 
         # 如果本段有声音，且已经停止了
-        if self.conn.client_voice_stop:
+        if conn.client_voice_stop:
-            asr_audio_task = copy.deepcopy(self.conn.asr_audio)
+            asr_audio_task = copy.deepcopy(conn.asr_audio)
-            self.conn.asr_audio.clear()
+            conn.asr_audio.clear()
 
             # 音频太短了，无法识别
-            self.conn.reset_vad_states()
+            conn.reset_vad_states()
             if len(asr_audio_task) > 15:
-                await self.handle_voice_stop(asr_audio_task)
+                await self.handle_voice_stop(conn, asr_audio_task)
 
     # 处理语音停止
-    async def handle_voice_stop(self, asr_audio_task):
+    async def handle_voice_stop(self, conn, asr_audio_task):
         raw_text, _ = await self.speech_to_text(
-            asr_audio_task, self.conn.session_id
+            asr_audio_task, conn.session_id, conn.audio_format
         )  # 确保ASR模块返回原始文本
-        self.conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+        conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
         text_len, _ = remove_punctuation_and_length(raw_text)
+        self.stop_ws_connection()
         if text_len > 0:
             # 使用自定义模块进行上报
-            await startToChat(self.conn, raw_text)
+            await startToChat(conn, raw_text)
-            enqueue_asr_report(self.conn, raw_text, asr_audio_task)
+            enqueue_asr_report(conn, raw_text, asr_audio_task)
+
+    def stop_ws_connection(self):
+        pass
 
     def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
         """PCM数据保存为WAV文件"""
@@ -81,15 +107,11 @@ class ASRProviderBase(ABC):
 
     @abstractmethod
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         pass
 
-    def set_audio_format(self, format: str) -> None:
-        """设置音频格式"""
-        self.audio_format = format
-
     @staticmethod
     def decode_opus(opus_data: List[bytes]) -> bytes:
         """将Opus音频数据解码为PCM数据"""

main/xiaozhi-server/core/providers/asr/doubao.py

@@ -1,534 +1,267 @@
+import time
+import os
+import uuid
 import json
 import gzip
-import uuid
-import asyncio
 import websockets
-import opuslib_next
-from core.providers.asr.base import ASRProviderBase
 from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.providers.asr.base import ASRProviderBase
 from core.providers.asr.dto.dto import InterfaceType
-import threading
+
 
 TAG = __name__
 logger = setup_logging()
 
 CLIENT_FULL_REQUEST = 0b0001
 CLIENT_AUDIO_ONLY_REQUEST = 0b0010
+
+NO_SEQUENCE = 0b0000
+NEG_SEQUENCE = 0b0010
+
 SERVER_FULL_RESPONSE = 0b1001
 SERVER_ACK = 0b1011
 SERVER_ERROR_RESPONSE = 0b1111
-NO_SEQUENCE = 0b0000
+
-NEG_SEQUENCE = 0b0010
+NO_SERIALIZATION = 0b0000
-JSON_SERIALIZATION = 0b0001
+JSON = 0b0001
-GZIP_COMPRESSION = 0b0001
+THRIFT = 0b0011
-PROTOCOL_VERSION = 0b0001
+CUSTOM_TYPE = 0b1111
+NO_COMPRESSION = 0b0000
+GZIP = 0b0001
+CUSTOM_COMPRESSION = 0b1111
+
+
+def parse_response(res):
+    """
+    protocol_version(4 bits), header_size(4 bits),
+    message_type(4 bits), message_type_specific_flags(4 bits)
+    serialization_method(4 bits) message_compression(4 bits)
+    reserved （8bits) 保留字段
+    header_extensions 扩展头(大小等于 8 * 4 * (header_size - 1) )
+    payload 类似与http 请求体
+    """
+    protocol_version = res[0] >> 4
+    header_size = res[0] & 0x0F
+    message_type = res[1] >> 4
+    message_type_specific_flags = res[1] & 0x0F
+    serialization_method = res[2] >> 4
+    message_compression = res[2] & 0x0F
+    reserved = res[3]
+    header_extensions = res[4 : header_size * 4]
+    payload = res[header_size * 4 :]
+    result = {}
+    payload_msg = None
+    payload_size = 0
+    if message_type == SERVER_FULL_RESPONSE:
+        payload_size = int.from_bytes(payload[:4], "big", signed=True)
+        payload_msg = payload[4:]
+    elif message_type == SERVER_ACK:
+        seq = int.from_bytes(payload[:4], "big", signed=True)
+        result["seq"] = seq
+        if len(payload) >= 8:
+            payload_size = int.from_bytes(payload[4:8], "big", signed=False)
+            payload_msg = payload[8:]
+    elif message_type == SERVER_ERROR_RESPONSE:
+        code = int.from_bytes(payload[:4], "big", signed=False)
+        result["code"] = code
+        payload_size = int.from_bytes(payload[4:8], "big", signed=False)
+        payload_msg = payload[8:]
+    if payload_msg is None:
+        return result
+    if message_compression == GZIP:
+        payload_msg = gzip.decompress(payload_msg)
+    if serialization_method == JSON:
+        payload_msg = json.loads(str(payload_msg, "utf-8"))
+    elif serialization_method != NO_SERIALIZATION:
+        payload_msg = str(payload_msg, "utf-8")
+    result["payload_msg"] = payload_msg
+    result["payload_size"] = payload_size
+    return result
 
 
 class ASRProvider(ASRProviderBase):
-    def __init__(self, config, delete_audio_file):
+    def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
-        self.interface_type = InterfaceType.STREAM
+        self.interface_type = InterfaceType.NON_STREAM
-        self.config = config
+        self.appid = config.get("appid")
-        self.text = ""
-        self.max_retries = 3
-        self.retry_delay = 2  # 重试延迟秒数
-        self.recv_lock = asyncio.Lock()  # 添加接收锁
-        self.reconnect_lock = asyncio.Lock()  # 添加重连锁
-        self.last_reconnect_time = 0  # 上次重连时间
-        self.reconnect_cooldown = 1  # 增加重连冷却时间到10秒
-        self.reconnect_count = 0  # 当前重连次数
-        self.max_reconnect_count = 3  # 减少最大重连次数到3次
-        self.asr_thread = None  # ASR监听线程
-        self.thread_lock = threading.Lock()  # 线程管理锁
-        self.is_reconnecting = False  # 添加重连状态标志
-
-        # 添加会话管理相关属性
-        self._session_lock = asyncio.Lock()  # 会话操作的并发锁
-        self._current_session_id = None  # 当前会话ID
-        self._session_started = False  # 会话是否已开始
-        self._session_finished = False  # 会话是否已结束
-        self._session_close_event = asyncio.Event()  # 添加会话关闭事件
-
-        self.appid = str(config.get("appid"))
         self.cluster = config.get("cluster")
         self.access_token = config.get("access_token")
         self.boosting_table_name = config.get("boosting_table_name", "")
         self.correct_table_name = config.get("correct_table_name", "")
-        self.output_dir = config.get("output_dir", "temp/")
+        self.output_dir = config.get("output_dir")
         self.delete_audio_file = delete_audio_file
 
-        self.ws_url = "wss://openspeech.bytedance.com/api/v2/asr"
+        self.host = "openspeech.bytedance.com"
-        self.uid = config.get("uid", "streaming_asr_service")
+        self.ws_url = f"wss://{self.host}/api/v2/asr"
-        self.workflow = config.get(
+        self.success_code = 1000
-            "workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
+        self.seg_duration = 15000
-        )
-        self.result_type = config.get("result_type", "single")
-        self.format = config.get("format", "raw")
-        self.codec = config.get("codec", "pcm")
-        self.rate = config.get("sample_rate", 16000)
-        self.language = config.get("language", "zh-CN")
-        self.bits = config.get("bits", 16)
-        self.channel = config.get("channel", 1)
-        self.auth_method = config.get("auth_method", "token")
-        self.secret = config.get("secret", "access_secret")
-        self.decoder = opuslib_next.Decoder(16000, 1)
-        self.asr_ws = None
-        self.forward_task = None
-        self.conn = None
 
-    ###################################################################################
+        # 确保输出目录存在
-    # 豆包流式ASR重写父类的方法--开始
+        os.makedirs(self.output_dir, exist_ok=True)
-    ###################################################################################
-    async def open_audio_channels(self, conn):
-        await super().open_audio_channels(conn)
 
-        async with self._session_lock:
+    @staticmethod
-            # 如果正在重连，等待重连完成
+    def _generate_header(
-            if self.is_reconnecting:
+        message_type=CLIENT_FULL_REQUEST, message_type_specific_flags=NO_SEQUENCE
-                logger.bind(tag=TAG).info("等待当前重连完成...")
+    ) -> bytearray:
-                await self._session_close_event.wait()
+        """Generate protocol header."""
-                self._session_close_event.clear()
+        header = bytearray()
+        header_size = 1
+        header.append((0b0001 << 4) | header_size)  # Protocol version
+        header.append((message_type << 4) | message_type_specific_flags)
+        header.append((0b0001 << 4) | 0b0001)  # JSON serialization & GZIP compression
+        header.append(0x00)  # reserved
+        return header
 
-            # 如果已有会话未结束，先关闭它
+    def _construct_request(self, reqid) -> dict:
-            if self._session_started and not self._session_finished:
+        """Construct the request payload."""
-                logger.bind(tag=TAG).warning(
+        return {
-                    f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
-                )
-                if self.asr_ws is not None:
-                    try:
-                        await self.asr_ws.close()
-                    except Exception as e:
-                        logger.bind(tag=TAG).warning(f"关闭旧连接时发生错误: {e}")
-                    finally:
-                        self.asr_ws = None
-                        self._session_finished = True
-                        self._session_close_event.set()
-
-            # 重置会话状态
-            self._current_session_id = str(uuid.uuid4())
-            self._session_started = True
-            self._session_finished = False
-            self.is_reconnecting = True
-
-            try:
-                retry_count = 0
-                while retry_count < self.max_retries:
-                    try:
-                        headers = (
-                            self.token_auth() if self.auth_method == "token" else None
-                        )
-                        self.asr_ws = await websockets.connect(
-                            self.ws_url,
-                            additional_headers=headers,
-                            max_size=1000000000,
-                            ping_interval=None,
-                            ping_timeout=None,
-                            close_timeout=10,
-                        )
-
-                        # 发送初始化请求
-                        request_params = self.construct_request(
-                            self._current_session_id
-                        )
-                        try:
-                            payload_bytes = str.encode(json.dumps(request_params))
-                            payload_bytes = gzip.compress(payload_bytes)
-                            full_client_request = self.generate_header()
-                            full_client_request.extend(
-                                (len(payload_bytes)).to_bytes(4, "big")
-                            )
-                            full_client_request.extend(payload_bytes)
-                            await self.asr_ws.send(full_client_request)
-                        except Exception as e:
-                            logger.bind(tag=TAG).error(f"发送初始化请求失败: {e}")
-                            raise e
-
-                        # 等待初始化响应
-                        try:
-                            init_res = await self.asr_ws.recv()
-                            self.parse_response(init_res)
-                        except Exception as e:
-                            logger.bind(tag=TAG).error(f"ASR服务初始化失败: {e}")
-                            raise e
-
-                        # 启动接收ASR结果的异步任务
-                        with self.thread_lock:
-                            if (
-                                self.asr_thread is None
-                                or not self.asr_thread.is_alive()
-                            ):
-                                logger.bind(tag=TAG).info("创建新的ASR监听线程...")
-                                self.asr_thread = threading.Thread(
-                                    target=self._start_monitor_asr_response_thread,
-                                    daemon=True,
-                                )
-                                self.asr_thread.start()
-                                # 等待一小段时间确保线程启动
-                                await asyncio.sleep(0.1)
-                                if not self.asr_thread.is_alive():
-                                    logger.bind(tag=TAG).error("ASR监听线程启动失败")
-                                    raise Exception("ASR监听线程启动失败")
-                                logger.bind(tag=TAG).info("ASR监听线程已启动")
-                        return
-
-                    except websockets.exceptions.WebSocketException as e:
-                        retry_count += 1
-                        if retry_count < self.max_retries:
-                            logger.bind(tag=TAG).warning(
-                                f"WebSocket连接失败，正在进行第{retry_count}次重试: {e}"
-                            )
-                            await asyncio.sleep(self.retry_delay)
-                        else:
-                            logger.bind(tag=TAG).warning(
-                                f"WebSocket连接失败，已达到最大重试次数: {e}"
-                            )
-                            raise
-                    except Exception as e:
-                        logger.bind(tag=TAG).error(f"WebSocket连接发生未知错误: {e}")
-                        raise
-            finally:
-                self.is_reconnecting = False
-                self._session_close_event.set()
-
-    async def receive_audio(self, audio, _):
-        if not isinstance(audio, bytes):
-            return
-
-        try:
-            # 解码opus得到PCM数据
-            pcm_frame = self.decoder.decode(audio, 960)
-            payload = gzip.compress(pcm_frame)
-            audio_request = bytearray(self.generate_audio_default_header())
-            audio_request.extend(len(payload).to_bytes(4, "big"))
-            audio_request.extend(payload)
-            if self.asr_ws:
-                await self.asr_ws.send(audio_request)
-        except Exception as e:
-            logger.bind(tag=TAG).debug(f"发送音频数据时发生错误: {e}")
-
-    ###################################################################################
-    # 豆包流式ASR重写父类的方法--结束
-    ###################################################################################
-
-    def construct_request(self, reqid):
-        req = {
             "app": {
-                "appid": self.appid,
+                "appid": f"{self.appid}",
                 "cluster": self.cluster,
                 "token": self.access_token,
             },
-            "user": {"uid": self.uid},
+            "user": {
+                "uid": str(uuid.uuid4()),
+            },
             "request": {
                 "reqid": reqid,
-                "workflow": self.workflow,
+                "show_utterances": False,
-                "show_utterances": True,
-                "result_type": self.result_type,
                 "sequence": 1,
                 "boosting_table_name": self.boosting_table_name,
                 "correct_table_name": self.correct_table_name,
             },
             "audio": {
-                "format": self.format,
+                "format": "raw",
-                "codec": self.codec,
+                "rate": 16000,
-                "rate": self.rate,
+                "language": "zh-CN",
-                "language": self.language,
+                "bits": 16,
-                "bits": self.bits,
+                "channel": 1,
-                "channel": self.channel,
+                "codec": "raw",
             },
         }
-        return req
 
-    def token_auth(self):
+    async def _send_request(
-        return {"Authorization": f"Bearer; {self.access_token}"}
+        self, audio_data: List[bytes], segment_size: int
-
+    ) -> Optional[str]:
-    def generate_header(
+        """Send request to Volcano ASR service."""
-        self,
-        version=PROTOCOL_VERSION,
-        message_type=CLIENT_FULL_REQUEST,
-        message_type_specific_flags=NO_SEQUENCE,
-        serial_method=JSON_SERIALIZATION,
-        compression_type=GZIP_COMPRESSION,
-        reserved_data=0x00,
-        extension_header: bytes = b"",
-    ):
-        """
-        生成协议头：
-         - 第1字节：高4位：协议版本，低4位：头部大小（单位 4 字节）
-         - 第2字节：高4位：消息类型，低4位：消息类型特定标志
-         - 第3字节：高4位：序列化方式，低4位：压缩方式
-         - 第4字节：保留字段
-         - 后续：扩展头（如果有）
-        """
-        header = bytearray()
-        header_size = int(len(extension_header) / 4) + 1
-        header.append((version << 4) | header_size)
-        header.append((message_type << 4) | message_type_specific_flags)
-        header.append((serial_method << 4) | compression_type)
-        header.append(reserved_data)
-        header.extend(extension_header)
-        return header
-
-    def generate_full_default_header(self):
-        # full client request 默认头
-        return self.generate_header(
-            version=PROTOCOL_VERSION,
-            message_type=CLIENT_FULL_REQUEST,
-            message_type_specific_flags=NO_SEQUENCE,
-            serial_method=JSON_SERIALIZATION,
-            compression_type=GZIP_COMPRESSION,
-        )
-
-    def generate_audio_default_header(self):
-        # 普通音频片段请求
-        return self.generate_header(
-            version=PROTOCOL_VERSION,
-            message_type=CLIENT_AUDIO_ONLY_REQUEST,
-            message_type_specific_flags=NO_SEQUENCE,
-            serial_method=JSON_SERIALIZATION,
-            compression_type=GZIP_COMPRESSION,
-        )
-
-    def generate_last_audio_default_header(self):
-        # 最后一个音频片段标志
-        return self.generate_header(
-            version=PROTOCOL_VERSION,
-            message_type=CLIENT_AUDIO_ONLY_REQUEST,
-            message_type_specific_flags=NEG_SEQUENCE,  # 用 NEG_SEQUENCE 表示结束
-            serial_method=JSON_SERIALIZATION,
-            compression_type=GZIP_COMPRESSION,
-        )
-
-    def _start_monitor_asr_response_thread(self):
-        # 初始化链接
         try:
-            with self.thread_lock:
+            auth_header = {"Authorization": "Bearer; {}".format(self.access_token)}
-                if self.conn is None or self.conn.loop is None:
+            async with websockets.connect(
-                    logger.bind(tag=TAG).error(
+                self.ws_url, additional_headers=auth_header
-                        "无法启动ASR监听线程：conn或loop未初始化"
+            ) as websocket:
-                    )
+                # Prepare request data
-                    return
+                request_params = self._construct_request(str(uuid.uuid4()))
+                payload_bytes = str.encode(json.dumps(request_params))
+                payload_bytes = gzip.compress(payload_bytes)
+                full_client_request = self._generate_header()
+                full_client_request.extend(
+                    (len(payload_bytes)).to_bytes(4, "big")
+                )  # payload size(4 bytes)
+                full_client_request.extend(payload_bytes)  # payload
 
-                try:
+                # Send header and metadata
-                    logger.bind(tag=TAG).info("开始启动ASR监听...")
+                # full_client_request
-                    asyncio.run_coroutine_threadsafe(
+                await websocket.send(full_client_request)
-                        self._forward_asr_results(), loop=self.conn.loop
+                res = await websocket.recv()
-                    )
+                result = parse_response(res)
-                    logger.bind(tag=TAG).info("ASR监听已启动")
+                if (
-                except Exception as e:
+                    "payload_msg" in result
-                    logger.bind(tag=TAG).error(f"启动ASR监听线程失败: {e}")
+                    and result["payload_msg"]["code"] != self.success_code
-        except Exception as e:
+                ):
-            logger.bind(tag=TAG).error(f"ASR监听线程发生未预期的错误: {e}")
+                    logger.bind(tag=TAG).error(f"ASR error: {result}")
+                    return None
 
-    async def _forward_asr_results(self):
+                for seq, (chunk, last) in enumerate(
-        try:
+                    self.slice_data(audio_data, segment_size), 1
-            while not self.conn.stop_event.is_set():
+                ):
-                try:
+                    if last:
-                    if self.asr_ws is None:
+                        audio_only_request = self._generate_header(
-                        # 检查是否需要重连
+                            message_type=CLIENT_AUDIO_ONLY_REQUEST,
-                        async with self.reconnect_lock:
+                            message_type_specific_flags=NEG_SEQUENCE,
-                            current_time = asyncio.get_event_loop().time()
-                            if (
-                                current_time - self.last_reconnect_time
-                                < self.reconnect_cooldown
-                            ):
-                                await asyncio.sleep(1)
-                                continue
-
-                            if self.reconnect_count >= self.max_reconnect_count:
-                                logger.bind(tag=TAG).error(
-                                    "达到最大重连次数限制，停止重连"
-                                )
-                                await asyncio.sleep(self.reconnect_cooldown)
-                                self.reconnect_count = 0
-                                continue
-
-                            self.last_reconnect_time = current_time
-                            self.reconnect_count += 1
-                            logger.bind(tag=TAG).info(
-                                f"尝试重新连接ASR服务... (第{self.reconnect_count}次)"
-                            )
-                            await self.open_audio_channels(self.conn)
-                            continue
-
-                    # 使用锁来确保同一时间只有一个协程在接收数据
-                    async with self.recv_lock:
-                        response = await self.asr_ws.recv()
-                        result = self.parse_response(response)
-
-                    # 检查是否需要重连
-                    if result.get("need_reconnect", False):
-                        logger.bind(tag=TAG).info(
-                            "检测到需要重连的错误，准备重新连接..."
                         )
-                        if self.asr_ws is not None:
+                    else:
-                            try:
+                        audio_only_request = self._generate_header(
-                                await self.asr_ws.close()
+                            message_type=CLIENT_AUDIO_ONLY_REQUEST
-                            except Exception as e:
+                        )
-                                logger.bind(tag=TAG).warning(
+                    payload_bytes = gzip.compress(chunk)
-                                    f"关闭旧连接时发生错误: {e}"
+                    audio_only_request.extend(
-                                )
+                        (len(payload_bytes)).to_bytes(4, "big")
-                            finally:
+                    )  # payload size(4 bytes)
-                                self.asr_ws = None
+                    audio_only_request.extend(payload_bytes)  # payload
-                        continue
+                    # Send audio data
+                    await websocket.send(audio_only_request)
 
-                    if "payload_msg" in result:
+                # Receive response
-                        if "result" in result["payload_msg"]:
+                response = await websocket.recv()
-                            # 检查是否有utterances并且definite为True
+                result = parse_response(response)
-                            utterances = result["payload_msg"]["result"][0].get(
-                                "utterances", []
-                            )
-                            for utterance in utterances:
-                                if utterance.get("definite", False):
-                                    self.text = utterance["text"]
-                                    await self.handle_voice_stop(None)
-                                    break
 
-                except websockets.ConnectionClosed:
+                if (
-                    logger.bind(tag=TAG).debug("ASR服务连接已关闭，准备重连...")
+                    "payload_msg" in result
-                    # 确保关闭旧连接
+                    and result["payload_msg"]["code"] == self.success_code
-                    if self.asr_ws is not None:
+                ):
-                        try:
+                    if len(result["payload_msg"]["result"]) > 0:
-                            await self.asr_ws.close()
+                        return result["payload_msg"]["result"][0]["text"]
-                        except Exception as e:
+                    return None
-                            logger.bind(tag=TAG).warning(f"关闭旧连接时发生错误: {e}")
+                else:
-                        finally:
+                    logger.bind(tag=TAG).error(f"ASR error: {result}")
-                            self.asr_ws = None
+                    return None
-
-                    # 等待冷却时间
-                    await asyncio.sleep(self.reconnect_cooldown)
-                    continue
-
-                except Exception as e:
-                    if not self.conn.stop_event.is_set():
-                        logger.bind(tag=TAG).error(f"ASR监听发生错误: {e}")
-                        await asyncio.sleep(self.retry_delay)
-                        continue
 
         except Exception as e:
-            logger.bind(tag=TAG).error(f"ASR监听线程发生错误: {e}")
+            logger.bind(tag=TAG).error(f"ASR request failed: {e}", exc_info=True)
-            # 确保在发生严重错误时也能继续尝试重连
+            return None
-            if not self.conn.stop_event.is_set():
-                await asyncio.sleep(self.retry_delay)
-                await self._forward_asr_results()  # 递归重试
 
-    async def speech_to_text(self, opus_data, session_id):
+    @staticmethod
-        result = self.text
+    def slice_data(data: bytes, chunk_size: int) -> (list, bool):
-        self.text = ""  # 清空text
-        return result, None
-
-    def parse_response(self, res: bytes) -> dict:
         """
-        解析 ASR 服务返回的二进制响应。
+        slice data
-        根据协议格式解析头部和 payload，若采用 GZIP 压缩则先解压，再根据 JSON 反序列化。
+        :param data: wav data
+        :param chunk_size: the segment size in one request
+        :return: segment data, last flag
         """
-        protocol_version = res[0] >> 4
+        data_len = len(data)
-        header_size = res[0] & 0x0F
+        offset = 0
-        message_type = res[1] >> 4
+        while offset + chunk_size < data_len:
-        serialization_method = res[2] >> 4
+            yield data[offset : offset + chunk_size], False
-        message_compression = res[2] & 0x0F
+            offset += chunk_size
-        payload = res[header_size * 4 :]
-        result = {}
-        payload_msg = None
-        payload_size = 0
-
-        if message_type == SERVER_FULL_RESPONSE:
-            payload_size = int.from_bytes(payload[:4], "big", signed=True)
-            payload_msg = payload[4:]
-        elif message_type == SERVER_ACK:
-            seq = int.from_bytes(payload[:4], "big", signed=True)
-            result["seq"] = seq
-            if len(payload) >= 8:
-                payload_size = int.from_bytes(payload[4:8], "big", signed=False)
-                payload_msg = payload[8:]
-        elif message_type == SERVER_ERROR_RESPONSE:
-            code = int.from_bytes(payload[:4], "big", signed=False)
-            result["code"] = code
-            payload_size = int.from_bytes(payload[4:8], "big", signed=False)
-            payload_msg = payload[8:]
-
-        if payload_msg is None:
-            return result
-        if message_compression == GZIP_COMPRESSION:
-            payload_msg = gzip.decompress(payload_msg)
-        if serialization_method == JSON_SERIALIZATION:
-            payload_msg = json.loads(payload_msg.decode("utf-8"))
         else:
-            payload_msg = payload_msg.decode("utf-8")
+            yield data[offset:data_len], True
-        result["payload_msg"] = payload_msg
-        result["payload_size"] = payload_size
 
-        # 错误码处理
+    async def speech_to_text(
-        if "code" in result:
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
-            error_code = result["code"]
+    ) -> Tuple[Optional[str], Optional[str]]:
-            error_message = ""
+        """将语音数据转换为文本"""
 
-            if error_code == 1000:
+        file_path = None
-                error_message = "成功"
+        try:
-            elif error_code == 1001:
+            # 合并所有opus数据包
-                error_message = "请求参数无效：请求参数缺失必需字段/字段值无效/重复请求"
+            if audio_format == "pcm":
-            elif error_code == 1002:
+                pcm_data = opus_data
-                error_message = "无访问权限：token无效/过期/无权访问指定服务"
-            elif error_code == 1003:
-                error_message = "访问超频：当前appid访问QPS超出设定阈值"
-            elif error_code == 1004:
-                error_message = "访问超额：当前appid访问次数超出限制"
-            elif error_code == 1005:
-                error_message = "服务器繁忙：服务过载，无法处理当前请求"
-            elif error_code == 1010:
-                error_message = "音频过长：音频数据时长超出阈值"
-            elif error_code == 1011:
-                error_message = "音频过大：音频数据大小超出阈值"
-            elif error_code == 1012:
-                error_message = "音频格式无效：音频header有误/无法进行音频解码"
-            elif error_code == 1013:
-                error_message = "音频静音：音频未识别出任何文本结果"
-            elif error_code >= 1020 and error_code <= 1022:
-                error_message = "识别相关错误：需要重连"
-                if error_code == 1020:
-                    error_message = "识别等待超时：等待下一包就绪超时"
-                elif error_code == 1021:
-                    error_message = "识别处理超时：识别处理过程超时"
-                elif error_code == 1022:
-                    error_message = "识别错误：识别过程中发生错误"
             else:
-                error_message = "未知错误：未归类错误"
+                pcm_data = self.decode_opus(opus_data)
+            combined_pcm_data = b"".join(pcm_data)
 
-            logger.bind(tag=TAG).debug(
+            # 判断是否保存为WAV文件
-                f"ASR错误: {error_message} (错误码: {error_code})"
+            if self.delete_audio_file:
-            )
+                pass
+            else:
+                file_path = self.save_audio_to_file(pcm_data, session_id)
 
-            # 如果是识别相关错误，标记需要重连
+            # 直接使用PCM数据
-            if error_code >= 1020 or error_code == 1001:
+            # 计算分段大小 (单声道, 16bit, 16kHz采样率)
-                result["need_reconnect"] = True
+            size_per_sec = 1 * 2 * 16000  # nchannels * sampwidth * framerate
+            segment_size = int(size_per_sec * self.seg_duration / 1000)
 
-        return result
+            # 语音识别
-
+            start_time = time.time()
-    async def close_session(self):
+            text = await self._send_request(combined_pcm_data, segment_size)
-        """关闭当前会话"""
+            if text:
-        async with self._session_lock:
+                logger.bind(tag=TAG).debug(
-            if not self._session_started:
+                    f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {text}"
-                logger.bind(tag=TAG).warning("尝试关闭未开始的会话")
-                return
-
-            if self._session_finished:
-                logger.bind(tag=TAG).warning(
-                    f"会话 {self._current_session_id} 已经关闭"
                 )
-                return
+                return text, file_path
+            return "", file_path
 
-            try:
+        except Exception as e:
-                if self.asr_ws is not None:
+            logger.bind(tag=TAG).error(f"语音识别失败: {e}", exc_info=True)
-                    await self.asr_ws.close()
+            return "", file_path
-            except Exception as e:
-                logger.bind(tag=TAG).warning(f"关闭WebSocket连接时发生错误: {e}")
-            finally:
-                self.asr_ws = None
-                self._session_finished = True
-                self._session_started = False
-                self._current_session_id = None
-                # 重置重连计数
-                self.reconnect_count = 0
-
-    async def close(self):
-        """资源清理方法"""
-        await self.close_session()

main/xiaozhi-server/core/providers/asr/doubao_stream.py

@@ -0,0 +1,344 @@
+import json
+import gzip
+import uuid
+import asyncio
+import websockets
+import opuslib_next
+from core.providers.asr.base import ASRProviderBase
+from config.logger import setup_logging
+from core.providers.asr.dto.dto import InterfaceType
+
+TAG = __name__
+logger = setup_logging()
+
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__()
+        self.interface_type = InterfaceType.STREAM
+        self.config = config
+        self.text = ""
+        self.max_retries = 3
+        self.retry_delay = 2
+        self.decoder = opuslib_next.Decoder(16000, 1)
+        self.asr_ws = None
+        self.forward_task = None
+        self.is_processing = False  # 添加处理状态标志
+
+        # 配置参数
+        self.appid = str(config.get("appid"))
+        self.cluster = config.get("cluster")
+        self.access_token = config.get("access_token")
+        self.boosting_table_name = config.get("boosting_table_name", "")
+        self.correct_table_name = config.get("correct_table_name", "")
+        self.output_dir = config.get("output_dir", "tmp/")
+        self.delete_audio_file = delete_audio_file
+
+        # 火山引擎ASR配置
+        self.ws_url = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel"
+        self.uid = config.get("uid", "streaming_asr_service")
+        self.workflow = config.get(
+            "workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
+        )
+        self.result_type = config.get("result_type", "single")
+        self.format = config.get("format", "pcm")
+        self.codec = config.get("codec", "pcm")
+        self.rate = config.get("sample_rate", 16000)
+        self.language = config.get("language", "zh-CN")
+        self.bits = config.get("bits", 16)
+        self.channel = config.get("channel", 1)
+        self.auth_method = config.get("auth_method", "token")
+        self.secret = config.get("secret", "access_secret")
+
+    async def open_audio_channels(self, conn):
+        await super().open_audio_channels(conn)
+
+    async def receive_audio(self, conn, audio, audio_have_voice):
+        conn.asr_audio.append(audio)
+        conn.asr_audio = conn.asr_audio[-10:]
+
+        # 如果本次有声音，且之前没有建立连接
+        if audio_have_voice and self.asr_ws is None and not self.is_processing:
+            try:
+                self.is_processing = True
+                # 建立新的WebSocket连接
+                headers = self.token_auth() if self.auth_method == "token" else None
+                logger.bind(tag=TAG).info(f"正在连接ASR服务，headers: {headers}")
+
+                self.asr_ws = await websockets.connect(
+                    self.ws_url,
+                    additional_headers=headers,
+                    max_size=1000000000,
+                    ping_interval=None,
+                    ping_timeout=None,
+                    close_timeout=10,
+                )
+
+                # 发送初始化请求
+                request_params = self.construct_request(str(uuid.uuid4()))
+                try:
+                    payload_bytes = str.encode(json.dumps(request_params))
+                    payload_bytes = gzip.compress(payload_bytes)
+                    full_client_request = self.generate_header()
+                    full_client_request.extend((len(payload_bytes)).to_bytes(4, "big"))
+                    full_client_request.extend(payload_bytes)
+
+                    logger.bind(tag=TAG).info(f"发送初始化请求: {request_params}")
+                    await self.asr_ws.send(full_client_request)
+
+                    # 等待初始化响应
+                    init_res = await self.asr_ws.recv()
+                    result = self.parse_response(init_res)
+                    logger.bind(tag=TAG).info(f"收到初始化响应: {result}")
+
+                    # 检查初始化响应
+                    if "code" in result and result["code"] != 1000:
+                        error_msg = f"ASR服务初始化失败: {result.get('payload_msg', {}).get('message', '未知错误')}"
+                        if "payload_msg" in result:
+                            error_msg += f"\n详细错误信息: {json.dumps(result['payload_msg'], ensure_ascii=False)}"
+                        logger.bind(tag=TAG).error(error_msg)
+                        raise Exception(error_msg)
+
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"发送初始化请求失败: {str(e)}")
+                    if hasattr(e, "__cause__") and e.__cause__:
+                        logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                    raise e
+
+                # 启动接收ASR结果的异步任务
+                self.forward_task = asyncio.create_task(self._forward_asr_results(conn))
+
+                # 发送缓存的音频数据
+                if conn.asr_audio and len(conn.asr_audio) > 0:
+                    for cached_audio in conn.asr_audio[-10:]:
+                        try:
+                            pcm_frame = self.decoder.decode(cached_audio, 960)
+                            payload = gzip.compress(pcm_frame)
+                            audio_request = bytearray(
+                                self.generate_audio_default_header()
+                            )
+                            audio_request.extend(len(payload).to_bytes(4, "big"))
+                            audio_request.extend(payload)
+                            await self.asr_ws.send(audio_request)
+                        except Exception as e:
+                            logger.bind(tag=TAG).info(
+                                f"发送缓存音频数据时发生错误: {e}"
+                            )
+
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"建立ASR连接失败: {str(e)}")
+                if hasattr(e, "__cause__") and e.__cause__:
+                    logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                if self.asr_ws:
+                    await self.asr_ws.close()
+                    self.asr_ws = None
+                self.is_processing = False
+                return
+
+        # 发送当前音频数据
+        if self.asr_ws and self.is_processing:
+            try:
+                pcm_frame = self.decoder.decode(audio, 960)
+                payload = gzip.compress(pcm_frame)
+                audio_request = bytearray(self.generate_audio_default_header())
+                audio_request.extend(len(payload).to_bytes(4, "big"))
+                audio_request.extend(payload)
+                await self.asr_ws.send(audio_request)
+            except Exception as e:
+                logger.bind(tag=TAG).info(f"发送音频数据时发生错误: {e}")
+
+    async def _forward_asr_results(self, conn):
+        try:
+            while self.asr_ws and not conn.stop_event.is_set():
+                try:
+                    response = await self.asr_ws.recv()
+                    result = self.parse_response(response)
+                    logger.bind(tag=TAG).debug(f"收到ASR结果: {result}")
+
+                    if "payload_msg" in result:
+                        payload = result["payload_msg"]
+                        if "result" in payload:
+                            utterances = payload["result"].get("utterances", [])
+                            # 检查duration和空文本的情况
+                            if (
+                                payload.get("audio_info", {}).get("duration", 0) > 2000
+                                and not utterances
+                                and not payload["result"].get("text")
+                            ):
+                                logger.bind(tag=TAG).error(f"识别文本：空")
+                                self.text = ""
+                                conn.reset_vad_states()
+                                await self.handle_voice_stop(conn, None)
+                                break
+
+                            for utterance in utterances:
+                                if utterance.get("definite", False):
+                                    self.text = utterance["text"]
+                                    logger.bind(tag=TAG).info(
+                                        f"识别到文本: {self.text}"
+                                    )
+                                    conn.reset_vad_states()
+                                    await self.handle_voice_stop(conn, None)
+                                    break
+                        elif "error" in payload:
+                            error_msg = payload.get("error", "未知错误")
+                            logger.bind(tag=TAG).error(f"ASR服务返回错误: {error_msg}")
+                            break
+
+                except websockets.ConnectionClosed:
+                    logger.bind(tag=TAG).info("ASR服务连接已关闭")
+                    self.is_processing = False
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"处理ASR结果时发生错误: {str(e)}")
+                    if hasattr(e, "__cause__") and e.__cause__:
+                        logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                    self.is_processing = False
+                    break
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"ASR结果转发任务发生错误: {str(e)}")
+            if hasattr(e, "__cause__") and e.__cause__:
+                logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+        finally:
+            if self.asr_ws:
+                await self.asr_ws.close()
+                self.asr_ws = None
+            self.is_processing = False
+
+    def stop_ws_connection(self):
+        if self.asr_ws:
+            asyncio.create_task(self.asr_ws.close())
+            self.asr_ws = None
+        self.is_processing = False
+
+    def construct_request(self, reqid):
+        req = {
+            "app": {
+                "appid": self.appid,
+                "cluster": self.cluster,
+                "token": self.access_token,
+            },
+            "user": {"uid": self.uid},
+            "request": {
+                "reqid": reqid,
+                "workflow": self.workflow,
+                "show_utterances": True,
+                "result_type": self.result_type,
+                "sequence": 1,
+                "boosting_table_name": self.boosting_table_name,
+                "correct_table_name": self.correct_table_name,
+                "end_window_size": 200,
+            },
+            "audio": {
+                "format": self.format,
+                "codec": self.codec,
+                "rate": self.rate,
+                "language": self.language,
+                "bits": self.bits,
+                "channel": self.channel,
+                "sample_rate": self.rate,
+            },
+        }
+        logger.bind(tag=TAG).debug(
+            f"构造请求参数: {json.dumps(req, ensure_ascii=False)}"
+        )
+        return req
+
+    def token_auth(self):
+        return {
+            "X-Api-App-Key": self.appid,
+            "X-Api-Access-Key": self.access_token,
+            "X-Api-Resource-Id": "volc.bigasr.sauc.duration",
+            "X-Api-Connect-Id": str(uuid.uuid4()),
+            "Host": "openspeech.bytedance.com",
+        }
+
+    def generate_header(
+        self,
+        version=0x01,
+        message_type=0x01,
+        message_type_specific_flags=0x00,
+        serial_method=0x01,
+        compression_type=0x01,
+        reserved_data=0x00,
+        extension_header: bytes = b"",
+    ):
+        header = bytearray()
+        header_size = int(len(extension_header) / 4) + 1
+        header.append((version << 4) | header_size)
+        header.append((message_type << 4) | message_type_specific_flags)
+        header.append((serial_method << 4) | compression_type)
+        header.append(reserved_data)
+        header.extend(extension_header)
+        return header
+
+    def generate_audio_default_header(self):
+        return self.generate_header(
+            version=0x01,
+            message_type=0x02,
+            message_type_specific_flags=0x00,
+            serial_method=0x01,
+            compression_type=0x01,
+        )
+
+    def generate_last_audio_default_header(self):
+        return self.generate_header(
+            version=0x01,
+            message_type=0x02,
+            message_type_specific_flags=0x02,
+            serial_method=0x01,
+            compression_type=0x01,
+        )
+
+    def parse_response(self, res: bytes) -> dict:
+        try:
+            # 检查响应长度
+            if len(res) < 4:
+                logger.bind(tag=TAG).error(f"响应数据长度不足: {len(res)}")
+                return {"error": "响应数据长度不足"}
+
+            # 获取消息头
+            header = res[:4]
+            message_type = header[1] >> 4
+
+            # 如果是错误响应
+            if message_type == 0x0F:  # SERVER_ERROR_RESPONSE
+                code = int.from_bytes(header[4:8], "big", signed=False)
+                error_msg = res[8:].decode("utf-8")
+                return {"code": code, "error": error_msg}
+
+            # 获取JSON数据（跳过12字节头部）
+            try:
+                json_data = res[12:].decode("utf-8")
+                result = json.loads(json_data)
+                logger.bind(tag=TAG).debug(f"成功解析JSON响应: {result}")
+                return {"payload_msg": result}
+            except (UnicodeDecodeError, json.JSONDecodeError) as e:
+                logger.bind(tag=TAG).error(f"JSON解析失败: {str(e)}")
+                logger.bind(tag=TAG).error(f"原始数据: {res}")
+                raise
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"解析响应失败: {str(e)}")
+            logger.bind(tag=TAG).error(f"原始响应数据: {res.hex()}")
+            raise
+
+    async def speech_to_text(self, opus_data, session_id, audio_format):
+        result = self.text
+        self.text = ""  # 清空text
+        return result, None
+
+    async def close(self):
+        """资源清理方法"""
+        if self.asr_ws:
+            await self.asr_ws.close()
+            self.asr_ws = None
+        if self.forward_task:
+            self.forward_task.cancel()
+            try:
+                await self.forward_task
+            except asyncio.CancelledError:
+                pass
+            self.forward_task = None
+        self.is_processing = False

main/xiaozhi-server/core/providers/asr/fun_local.py

@@ -54,7 +54,7 @@ class ASRProvider(ASRProviderBase):
             )
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """语音转文本主处理逻辑"""
         file_path = None
@@ -63,7 +63,7 @@ class ASRProvider(ASRProviderBase):
         while retry_count < MAX_RETRIES:
             try:
                 # 合并所有opus数据包
-                if self.audio_format == "pcm":
+                if audio_format == "pcm":
                     pcm_data = opus_data
                 else:
                     pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/fun_server.py

@@ -100,7 +100,7 @@ class ASRProvider(ASRProviderBase):
         logger.bind(tag=TAG).debug(f"Sent end message: {end_message}")
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """
         Convert speech data to text using FunASR.
@@ -109,7 +109,7 @@ class ASRProvider(ASRProviderBase):
         :return: Tuple containing recognized text and optional timestamp.
         """
         file_path = None
-        if self.audio_format == "pcm":
+        if audio_format == "pcm":
             pcm_data = opus_data
         else:
             pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/sherpa_onnx_local.py

@@ -109,14 +109,14 @@ class ASRProvider(ASRProviderBase):
             return samples_float32, f.getframerate()
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """语音转文本主处理逻辑"""
         file_path = None
         try:
             # 保存音频文件
             start_time = time.time()
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/tencent.py

@@ -32,7 +32,7 @@ class ASRProvider(ASRProviderBase):
         os.makedirs(self.output_dir, exist_ok=True)
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if not opus_data:
@@ -47,7 +47,7 @@ class ASRProvider(ASRProviderBase):
                 return None, file_path
 
             # 将Opus音频数据解码为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py

@@ -122,6 +122,8 @@ class IntentProvider(IntentProviderBase):
     async def detect_intent(self, conn, dialogue_history: List[Dict], text: str) -> str:
         if not self.llm:
             raise ValueError("LLM provider not set")
+        if conn.func_handler is None:
+            return '{"function_call": {"name": "continue_chat"}}'
 
         # 记录整体开始时间
         total_start_time = time.time()
@@ -148,9 +150,8 @@ class IntentProvider(IntentProviderBase):
         self.clean_cache()
 
         if self.promot == "":
-            if hasattr(conn, "func_handler"):
+            functions = conn.func_handler.get_functions()
-                functions = conn.func_handler.get_functions()
+            self.promot = self.get_intent_system_prompt(functions)
-                self.promot = self.get_intent_system_prompt(functions)
 
         music_config = initialize_music_handler(conn)
         music_file_names = music_config["music_file_names"]

main/xiaozhi-server/core/providers/tts/base.py

@@ -52,6 +52,7 @@ class TTSProviderBase(ABC):
         )
         self.first_sentence_punctuations = (
             "，",
+            "～",
             "~",
             "、",
             ",",
@@ -178,6 +179,9 @@ class TTSProviderBase(ABC):
         while not self.conn.stop_event.is_set():
             try:
                 message = self.tts_text_queue.get(timeout=1)
+                if self.conn.client_abort:
+                    logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                    continue
                 if message.sentence_type == SentenceType.FIRST:
                     # 初始化参数
                     self.tts_stop_request = False

main/xiaozhi-server/core/providers/tts/huoshan_double_stream.py

@@ -3,14 +3,13 @@ import uuid
 import json
 import queue
 import asyncio
-import threading
 import traceback
 import websockets
-import time
 from config.logger import setup_logging
 from core.utils import opus_encoder_utils
 from core.utils.util import check_model_key
 from core.providers.tts.base import TTSProviderBase
+from core.handle.abortHandle import handleAbortMessage
 from core.providers.tts.dto.dto import SentenceType, ContentType, InterfaceType
 
 TAG = __name__
@@ -139,7 +138,7 @@ class Response:
 class TTSProvider(TTSProviderBase):
     def __init__(self, config, delete_audio_file):
         super().__init__(config, delete_audio_file)
-        self.ws = None  # 初始化ws属性
+        self.ws = None
         self.interface_type = InterfaceType.DUAL_STREAM
         self.appId = config.get("appid")
         self.access_token = config.get("access_token")
@@ -154,91 +153,55 @@ class TTSProvider(TTSProviderBase):
         self.authorization = config.get("authorization")
         self.header = {"Authorization": f"{self.authorization}{self.access_token}"}
         self.enable_two_way = True
-        self.start_connection_flag = False
         self.tts_text = ""
-        # 合成文字语音后，播放的音频文件列表
         self.before_stop_play_files = []
         self.opus_encoder = opus_encoder_utils.OpusEncoderUtils(
             sample_rate=16000, channels=1, frame_size_ms=60
         )
         check_model_key("TTS", self.access_token)
 
-        # 添加会话状态控制
-        self._session_lock = asyncio.Lock()  # 会话操作的并发锁
-        self._current_session_id = None  # 当前会话ID
-        self._session_started = False  # 会话是否已开始
-        self._session_finished = False  # 会话是否已结束
-        self._connection_ready = False  # 连接是否就绪
-        self._reconnect_attempts = 0  # 重连尝试次数
-        self._max_reconnect_attempts = 3  # 最大重连次数
-
-    ###################################################################################
-    # 火山双流式TTS重写父类的方法--开始
-    ###################################################################################
-
     async def open_audio_channels(self, conn):
         try:
             await super().open_audio_channels(conn)
-            await self._ensure_connection()
-            tts_priority = threading.Thread(
-                target=self._start_monitor_tts_response_thread, daemon=True
-            )
-            tts_priority.start()
         except Exception as e:
             logger.bind(tag=TAG).error(f"Failed to open audio channels: {str(e)}")
             self.ws = None
             raise
 
     async def _ensure_connection(self):
-        """确保WebSocket连接可用"""
+        """建立新的WebSocket连接"""
         try:
-            if self.ws is None:
+            logger.bind(tag=TAG).info("开始建立新连接...")
-                logger.bind(tag=TAG).info("WebSocket连接不存在，开始建立新连接...")
+            ws_header = {
-                ws_header = {
+                "X-Api-App-Key": self.appId,
-                    "X-Api-App-Key": self.appId,
+                "X-Api-Access-Key": self.access_token,
-                    "X-Api-Access-Key": self.access_token,
+                "X-Api-Resource-Id": self.resource_id,
-                    "X-Api-Resource-Id": self.resource_id,
+                "X-Api-Connect-Id": uuid.uuid4(),
-                    "X-Api-Connect-Id": uuid.uuid4(),
+            }
-                }
+            self.ws = await websockets.connect(
-                self.ws = await websockets.connect(
+                self.ws_url, additional_headers=ws_header, max_size=1000000000
-                    self.ws_url, additional_headers=ws_header, max_size=1000000000
+            )
-                )
+            logger.bind(tag=TAG).info("WebSocket连接建立成功")
-                self._connection_ready = True
+            return self.ws
-                self._reconnect_attempts = 0
-                logger.bind(tag=TAG).info("WebSocket连接建立成功")
-            else:
-                # 尝试发送ping来检查连接是否还活着
-                try:
-                    logger.bind(tag=TAG).debug("检查WebSocket连接状态...")
-                    pong_waiter = await self.ws.ping()
-                    await asyncio.wait_for(pong_waiter, timeout=1.0)
-                    logger.bind(tag=TAG).debug("WebSocket连接状态正常")
-                except (asyncio.TimeoutError, websockets.ConnectionClosed):
-                    # 如果ping失败，重新建立连接
-                    logger.bind(tag=TAG).warning("WebSocket连接已断开，准备重新连接...")
-                    try:
-                        await self.ws.close()
-                    except:
-                        pass
-                    self.ws = None
-                    self._connection_ready = False
-                    # 重新建立连接
-                    await self._ensure_connection()
         except Exception as e:
-            logger.bind(tag=TAG).error(f"确保连接失败: {str(e)}")
+            logger.bind(tag=TAG).error(f"建立连接失败: {str(e)}")
-            self._connection_ready = False
             self.ws = None
             raise
 
     def tts_text_priority_thread(self):
+        """火山引擎双流式TTS的文本处理线程"""
         logger.bind(tag=TAG).info("TTS文本处理线程启动")
         while not self.conn.stop_event.is_set():
             try:
                 logger.bind(tag=TAG).debug("等待TTS文本队列消息...")
                 message = self.tts_text_queue.get(timeout=1)
-                logger.bind(tag=TAG).info(
+                logger.bind(tag=TAG).debug(
                     f"收到TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
                 )
+                if self.conn.client_abort:
+                    logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                    continue
+
                 if message.sentence_type == SentenceType.FIRST:
                     # 初始化参数
                     try:
@@ -253,13 +216,12 @@ class TTSProvider(TTSProviderBase):
                         logger.bind(tag=TAG).info("TTS会话启动成功")
                     except Exception as e:
                         logger.bind(tag=TAG).error(f"启动TTS会话失败: {str(e)}")
-                        # 直接跳过当前消息，不重新入队
-                        time.sleep(1)
                         continue
+
                 elif ContentType.TEXT == message.content_type:
                     if message.content_detail:
                         try:
-                            logger.bind(tag=TAG).info(
+                            logger.bind(tag=TAG).debug(
                                 f"开始发送TTS文本: {message.content_detail}"
                             )
                             future = asyncio.run_coroutine_threadsafe(
@@ -267,12 +229,11 @@ class TTSProvider(TTSProviderBase):
                                 loop=self.conn.loop,
                             )
                             future.result()
-                            logger.bind(tag=TAG).info("TTS文本发送成功")
+                            logger.bind(tag=TAG).debug("TTS文本发送成功")
                         except Exception as e:
                             logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
-                            # 直接跳过当前消息，不重新入队
-                            time.sleep(1)
                             continue
+
                 elif ContentType.FILE == message.content_type:
                     logger.bind(tag=TAG).info(
                         f"添加音频文件到待播放列表: {message.content_file}"
@@ -289,11 +250,8 @@ class TTSProvider(TTSProviderBase):
                             loop=self.conn.loop,
                         )
                         future.result()
-                        logger.bind(tag=TAG).info("TTS会话结束成功")
                     except Exception as e:
                         logger.bind(tag=TAG).error(f"结束TTS会话失败: {str(e)}")
-                        # 直接跳过当前消息，不重新入队
-                        time.sleep(1)
                         continue
 
             except queue.Empty:
@@ -302,103 +260,204 @@ class TTSProvider(TTSProviderBase):
                 logger.bind(tag=TAG).error(
                     f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
                 )
-                # 如果是WebSocket连接关闭错误，等待一段时间后继续
-                if "non-exist session" in str(e):
-                    time.sleep(1)
                 continue
 
     async def text_to_speak(self, text, _):
         """发送文本到TTS服务"""
         try:
-            # 确保WebSocket连接可用
+            # 建立新连接
-            if not self._connection_ready or self.ws is None:
+            if self.ws is None:
-                logger.bind(tag=TAG).warning("WebSocket连接不可用，尝试重新连接...")
+                await handleAbortMessage(self.conn)
-                await self._ensure_connection()
+                logger.bind(tag=TAG).error(f"WebSocket连接不存在，终止发送文本")
-
+                return
             # 发送文本
             await self.send_text(self.speaker, text, self.conn.sentence_id)
             return
         except Exception as e:
             logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
-            # 如果是连接问题，尝试重新连接
+            if self.ws:
-            if isinstance(e, websockets.ConnectionClosed):
+                try:
-                self._connection_ready = False
+                    await self.ws.close()
+                except:
+                    pass
                 self.ws = None
-                await self._handle_connection_error()
             raise
 
-    ###################################################################################
+    async def start_session(self, session_id):
-    # 火山双流式TTS重写父类的方法--结束
+        logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
-    ###################################################################################
+        try:
-    def _start_monitor_tts_response_thread(self):
+            # 建立新连接
-        # 初始化链接
+            await self._ensure_connection()
-        asyncio.run_coroutine_threadsafe(
+
-            self._start_monitor_tts_response(), loop=self.conn.loop
+            # 启动监听任务
-        )
+            self._monitor_task = asyncio.create_task(self._start_monitor_tts_response())
+
+            header = Header(
+                message_type=FULL_CLIENT_REQUEST,
+                message_type_specific_flags=MsgTypeFlagWithEvent,
+                serial_method=JSON,
+            ).as_bytes()
+            optional = Optional(
+                event=EVENT_StartSession, sessionId=session_id
+            ).as_bytes()
+            payload = self.get_payload_bytes(
+                event=EVENT_StartSession, speaker=self.speaker
+            )
+            await self.send_event(header, optional, payload)
+            logger.bind(tag=TAG).info("会话启动请求已发送")
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
+            # 确保清理资源
+            if hasattr(self, "_monitor_task"):
+                try:
+                    self._monitor_task.cancel()
+                    await self._monitor_task
+                except:
+                    pass
+                self._monitor_task = None
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+            raise
+
+    async def finish_session(self, session_id):
+        logger.bind(tag=TAG).info(f"关闭会话～～{session_id}")
+        try:
+            if self.ws:
+                header = Header(
+                    message_type=FULL_CLIENT_REQUEST,
+                    message_type_specific_flags=MsgTypeFlagWithEvent,
+                    serial_method=JSON,
+                ).as_bytes()
+                optional = Optional(
+                    event=EVENT_FinishSession, sessionId=session_id
+                ).as_bytes()
+                payload = str.encode("{}")
+                await self.send_event(header, optional, payload)
+                logger.bind(tag=TAG).info("会话结束请求已发送")
+
+                # 等待监听任务完成
+                if hasattr(self, "_monitor_task"):
+                    try:
+                        await self._monitor_task
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(
+                            f"等待监听任务完成时发生错误: {str(e)}"
+                        )
+                    finally:
+                        self._monitor_task = None
+
+                # 关闭连接
+                await self.close()
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
+            # 确保清理资源
+            if hasattr(self, "_monitor_task"):
+                try:
+                    self._monitor_task.cancel()
+                    await self._monitor_task
+                except:
+                    pass
+                self._monitor_task = None
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+            raise
+
+    async def close(self):
+        """资源清理方法"""
+        if self.ws:
+            try:
+                await self.ws.close()
+            except:
+                pass
+            self.ws = None
 
     async def _start_monitor_tts_response(self):
+        """监听TTS响应"""
         opus_datas_cache = []
-        # 添加标志来区分是否是第一句话
         is_first_sentence = True
-        while not self.conn.stop_event.is_set():
+        try:
-            try:
+            while not self.conn.stop_event.is_set():
-                # 确保 `recv()` 运行在同一个 event loop
+                try:
-                msg = await self.ws.recv()
+                    # 确保 `recv()` 运行在同一个 event loop
-                res = self.parser_response(msg)
+                    msg = await self.ws.recv()
-                self.print_response(res, "send_text res:")
+                    res = self.parser_response(msg)
+                    self.print_response(res, "send_text res:")
 
-                if res.optional.event == EVENT_TTSSentenceStart:
+                    # 检查客户端是否中止
-                    json_data = json.loads(res.payload.decode("utf-8"))
+                    if self.conn.client_abort:
-                    self.tts_text = json_data.get("text", "")
+                        logger.bind(tag=TAG).info("收到打断信息，终止监听TTS响应")
-                    logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
+                        break
-                    self.tts_audio_queue.put((SentenceType.FIRST, [], self.tts_text))
+
-                    opus_datas_cache = []
+                    if res.optional.event == EVENT_TTSSentenceStart:
-                elif (
+                        json_data = json.loads(res.payload.decode("utf-8"))
-                    res.optional.event == EVENT_TTSResponse
+                        self.tts_text = json_data.get("text", "")
-                    and res.header.message_type == AUDIO_ONLY_RESPONSE
+                        logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
-                ):
-                    logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
-                    opus_datas = self.wav_to_opus_data_audio_raw(res.payload)
-                    logger.bind(tag=TAG).debug(
-                        f"推送数据到队列里面帧数～～{len(opus_datas)}"
-                    )
-                    if is_first_sentence:
-                        # 第一句话直接发送
                         self.tts_audio_queue.put(
-                            (SentenceType.MIDDLE, opus_datas, self.tts_text)
+                            (SentenceType.FIRST, [], self.tts_text)
                         )
-                    else:
+                        opus_datas_cache = []
-                        # 后续句子缓存
+                    elif (
-                        opus_datas_cache = opus_datas_cache + opus_datas
+                        res.optional.event == EVENT_TTSResponse
-                elif res.optional.event == EVENT_TTSSentenceEnd:
+                        and res.header.message_type == AUDIO_ONLY_RESPONSE
-                    logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
+                    ):
-                    if not is_first_sentence:
+                        logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
-                        # 只有非第一句话才发送缓存的数据
+                        opus_datas = self.wav_to_opus_data_audio_raw(res.payload)
-                        self.tts_audio_queue.put(
+                        logger.bind(tag=TAG).debug(
-                            (SentenceType.MIDDLE, opus_datas_cache, self.tts_text)
+                            f"推送数据到队列里面帧数～～{len(opus_datas)}"
                         )
-                    # 第一句话结束后，将标志设置为False
+                        if is_first_sentence:
-                    is_first_sentence = False
+                            # 第一句话直接发送
-                elif res.optional.event == EVENT_SessionFinished:
-                    logger.bind(tag=TAG).debug(f"会话结束～～")
-                    for tts_file, text in self.before_stop_play_files:
-                        if tts_file and os.path.exists(tts_file):
-                            audio_datas = self._process_audio_file(tts_file)
                             self.tts_audio_queue.put(
-                                (SentenceType.MIDDLE, audio_datas, text)
+                                (SentenceType.MIDDLE, opus_datas, self.tts_text)
                             )
-                    self.before_stop_play_files.clear()
+                        else:
-                    self.tts_audio_queue.put((SentenceType.LAST, [], None))
+                            # 后续句子缓存
-
+                            opus_datas_cache = opus_datas_cache + opus_datas
-                    opus_datas_cache = []
+                    elif res.optional.event == EVENT_TTSSentenceEnd:
-                    is_first_sentence = True
+                        logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
-                    continue
+                        if not is_first_sentence:
-            except websockets.ConnectionClosed:
+                            # 只有非第一句话才发送缓存的数据
-                break  # 连接关闭时退出监听
+                            self.tts_audio_queue.put(
-            except Exception as e:
+                                (SentenceType.MIDDLE, opus_datas_cache, self.tts_text)
-                logger.bind(tag=TAG).error(f"Error in _start_monitor_tts_response: {e}")
+                            )
-                traceback.print_exc()
+                        # 第一句话结束后，将标志设置为False
-                continue
+                        is_first_sentence = False
+                    elif res.optional.event == EVENT_SessionFinished:
+                        logger.bind(tag=TAG).debug(f"会话结束～～")
+                        for tts_file, text in self.before_stop_play_files:
+                            if tts_file and os.path.exists(tts_file):
+                                audio_datas = self._process_audio_file(tts_file)
+                                self.tts_audio_queue.put(
+                                    (SentenceType.MIDDLE, audio_datas, text)
+                                )
+                        self.before_stop_play_files.clear()
+                        self.tts_audio_queue.put((SentenceType.LAST, [], None))
+                        break
+                except websockets.ConnectionClosed:
+                    logger.bind(tag=TAG).warning("WebSocket连接已关闭")
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(
+                        f"Error in _start_monitor_tts_response: {e}"
+                    )
+                    traceback.print_exc()
+                    break
+        finally:
+            # 确保清理资源
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
 
     async def send_event(
         self, header: bytes, optional: bytes | None = None, payload: bytes = None
@@ -413,11 +472,8 @@ class TTSProvider(TTSProviderBase):
                 full_client_request.extend(payload)
             await self.ws.send(full_client_request)
         except websockets.ConnectionClosed:
-            if await self._handle_connection_error():
+            logger.bind(tag=TAG).error(f"ConnectionClosed")
-                # 重连成功后重试发送
+            raise
-                await self.ws.send(full_client_request)
-            else:
-                raise
 
     async def send_text(self, speaker: str, text: str, session_id):
         header = Header(
@@ -540,200 +596,6 @@ class TTSProvider(TTSProviderBase):
             )
         )
 
-    async def finish_connection(self):
-        header = Header(
-            message_type=FULL_CLIENT_REQUEST,
-            message_type_specific_flags=MsgTypeFlagWithEvent,
-            serial_method=JSON,
-        ).as_bytes()
-        optional = Optional(event=EVENT_FinishConnection).as_bytes()
-        payload = str.encode("{}")
-        await self.send_event(header, optional, payload)
-        return
-
-    async def start_session(self, session_id):
-        logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
-        try:
-            async with self._session_lock:
-                try:
-                    # 确保连接可用
-                    logger.bind(tag=TAG).info("检查WebSocket连接状态...")
-                    await asyncio.wait_for(self._ensure_connection(), timeout=5)
-
-                    # 如果已有会话未结束，先关闭它
-                    if self._session_started and not self._session_finished:
-                        logger.bind(tag=TAG).warning(
-                            f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
-                        )
-                        try:
-                            await asyncio.wait_for(
-                                self.finish_session(self._current_session_id), timeout=5
-                            )
-                        except Exception as e:
-                            logger.bind(tag=TAG).error(f"关闭旧会话失败: {str(e)}")
-                            # 强制重置会话状态
-                            self._session_started = False
-                            self._session_finished = True
-                            self._current_session_id = None
-
-                    # 重置会话状态
-                    self._current_session_id = session_id
-                    self._session_started = True
-                    self._session_finished = False
-                    logger.bind(tag=TAG).info(
-                        f"会话状态已更新 - 开始: {self._session_started}, 结束: {self._session_finished}"
-                    )
-
-                    header = Header(
-                        message_type=FULL_CLIENT_REQUEST,
-                        message_type_specific_flags=MsgTypeFlagWithEvent,
-                        serial_method=JSON,
-                    ).as_bytes()
-                    optional = Optional(
-                        event=EVENT_StartSession, sessionId=session_id
-                    ).as_bytes()
-                    payload = self.get_payload_bytes(
-                        event=EVENT_StartSession, speaker=self.speaker
-                    )
-                    await asyncio.wait_for(
-                        self.send_event(header, optional, payload), timeout=5
-                    )
-                    logger.bind(tag=TAG).info("会话启动请求已发送")
-                except Exception as e:
-                    logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
-                    self._session_started = False
-                    self._session_finished = True
-                    self._current_session_id = None
-                    raise
-        except asyncio.TimeoutError:
-            logger.bind(tag=TAG).error(f"启动会话超时: {session_id}")
-            # 超时后强制重置会话状态
-            self._session_started = False
-            self._session_finished = True
-            self._current_session_id = None
-            # 尝试关闭WebSocket连接
-            if self.ws:
-                try:
-                    await self.ws.close()
-                except:
-                    pass
-                self.ws = None
-        except Exception as e:
-            logger.bind(tag=TAG).error(f"启动会话时发生未知错误: {str(e)}")
-            # 发生未知错误时也重置会话状态
-            self._session_started = False
-            self._session_finished = True
-            self._current_session_id = None
-
-    async def finish_session(self, session_id):
-        logger.bind(tag=TAG).info(f"关闭会话～～{session_id}")
-        try:
-            async with self._session_lock:
-                try:
-                    # 检查会话状态
-                    if not self._session_started:
-                        logger.bind(tag=TAG).warning(
-                            f"尝试关闭未开始的会话 {session_id}"
-                        )
-                        return
-
-                    if self._session_finished:
-                        logger.bind(tag=TAG).warning(f"会话 {session_id} 已经关闭")
-                        return
-
-                    if self._current_session_id != session_id:
-                        logger.bind(tag=TAG).warning(
-                            f"尝试关闭错误的会话 {session_id}，当前会话为 {self._current_session_id}"
-                        )
-                        # 即使会话ID不匹配，也尝试关闭当前会话
-                        if self._current_session_id:
-                            session_id = self._current_session_id
-
-                    # 确保WebSocket连接可用
-                    if self.ws is None:
-                        logger.bind(tag=TAG).warning(
-                            "WebSocket连接不存在，尝试重新连接..."
-                        )
-                        await asyncio.wait_for(self._ensure_connection(), timeout=5)
-
-                    header = Header(
-                        message_type=FULL_CLIENT_REQUEST,
-                        message_type_specific_flags=MsgTypeFlagWithEvent,
-                        serial_method=JSON,
-                    ).as_bytes()
-                    optional = Optional(
-                        event=EVENT_FinishSession, sessionId=session_id
-                    ).as_bytes()
-                    payload = str.encode("{}")
-                    await asyncio.wait_for(
-                        self.send_event(header, optional, payload), timeout=5
-                    )
-                    logger.bind(tag=TAG).info("会话结束请求已发送")
-
-                    # 更新会话状态
-                    self._session_finished = True
-                    self._session_started = False
-                    self._current_session_id = None
-                    logger.bind(tag=TAG).info(
-                        "会话状态已更新 - 开始: False, 结束: True"
-                    )
-                except Exception as e:
-                    logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
-                    # 即使发生错误，也要重置会话状态
-                    self._session_finished = True
-                    self._session_started = False
-                    self._current_session_id = None
-                    raise
-        except asyncio.TimeoutError:
-            logger.bind(tag=TAG).error(f"关闭会话超时: {session_id}")
-            # 超时后强制重置会话状态
-            self._session_finished = True
-            self._session_started = False
-            self._current_session_id = None
-            # 尝试关闭WebSocket连接
-            if self.ws:
-                try:
-                    await self.ws.close()
-                except:
-                    pass
-                self.ws = None
-        except Exception as e:
-            logger.bind(tag=TAG).error(f"关闭会话时发生未知错误: {str(e)}")
-            # 发生未知错误时也重置会话状态
-            self._session_finished = True
-            self._session_started = False
-            self._current_session_id = None
-
-    async def reset(self):
-        # 关闭之前的对话
-        if self.start_connection_flag:
-            await self.finish_connection()
-            self.start_connection_flag = False
-        await self.start_connection()
-        self.start_connection_flag = True
-
-    async def close(self):
-        """资源清理方法"""
-        await self.finish_connection()
-        await self.ws.close()
-
     def wav_to_opus_data_audio_raw(self, raw_data_var, is_end=False):
         opus_datas = self.opus_encoder.encode_pcm_to_opus(raw_data_var, is_end)
         return opus_datas
-
-    async def _handle_connection_error(self):
-        """处理连接错误"""
-        if self._reconnect_attempts < self._max_reconnect_attempts:
-            self._reconnect_attempts += 1
-            logger.bind(tag=TAG).warning(
-                f"尝试重新连接 (第{self._reconnect_attempts}次)"
-            )
-            try:
-                await self._ensure_connection()
-                return True
-            except Exception as e:
-                logger.bind(tag=TAG).error(f"重新连接失败: {str(e)}")
-                return False
-        else:
-            logger.bind(tag=TAG).error("达到最大重连次数，放弃重连")
-            return False

main/xiaozhi-server/core/providers/vllm/base.py

@@ -0,0 +1,12 @@
+from abc import ABC, abstractmethod
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class VLLMProviderBase(ABC):
+    @abstractmethod
+    def response(self, question, base64_image):
+        """VLLM response generator"""
+        pass

main/xiaozhi-server/core/providers/vllm/openai.py

@@ -0,0 +1,65 @@
+import openai
+import json
+from config.logger import setup_logging
+from core.utils.util import check_model_key
+from core.providers.vllm.base import VLLMProviderBase
+
+TAG = __name__
+logger = setup_logging()
+
+
+class VLLMProvider(VLLMProviderBase):
+    def __init__(self, config):
+        self.model_name = config.get("model_name")
+        self.api_key = config.get("api_key")
+        if "base_url" in config:
+            self.base_url = config.get("base_url")
+        else:
+            self.base_url = config.get("url")
+
+        param_defaults = {
+            "max_tokens": (500, int),
+            "temperature": (0.7, lambda x: round(float(x), 1)),
+            "top_p": (1.0, lambda x: round(float(x), 1)),
+        }
+
+        for param, (default, converter) in param_defaults.items():
+            value = config.get(param)
+            try:
+                setattr(
+                    self,
+                    param,
+                    converter(value) if value not in (None, "") else default,
+                )
+            except (ValueError, TypeError):
+                setattr(self, param, default)
+
+        check_model_key("VLLM", self.api_key)
+        self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
+
+    def response(self, question, base64_image):
+        try:
+            messages = [
+                {
+                    "role": "user",
+                    "content": [
+                        {"type": "text", "text": question},
+                        {
+                            "type": "image_url",
+                            "image_url": {
+                                "url": f"data:image/jpeg;base64,{base64_image}"
+                            },
+                        },
+                    ],
+                }
+            ]
+
+            response = self.client.chat.completions.create(
+                model=self.model_name, messages=messages, stream=False
+            )
+
+            return response.choices[0].message.content
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"Error in response generation: {e}")
+            raise

main/xiaozhi-server/core/utils/auth.py

@@ -0,0 +1,43 @@
+import jwt
+import time
+from datetime import datetime, timedelta, timezone
+from typing import Optional, Tuple
+
+
+class AuthToken:
+    def __init__(self, secret_key: str):
+        self.secret_key = secret_key
+
+    def generate_token(self, device_id: str) -> str:
+        """
+        生成JWT token
+        :param device_id: 设备ID
+        :return: JWT token字符串
+        """
+        # 设置过期时间为1小时后
+        expire_time = datetime.now(timezone.utc) + timedelta(hours=1)
+
+        # 创建payload
+        payload = {"device_id": device_id, "exp": expire_time.timestamp()}
+
+        # 使用JWT进行编码
+        token = jwt.encode(payload, self.secret_key, algorithm="HS256")
+        return token
+
+    def verify_token(self, token: str) -> Tuple[bool, Optional[str]]:
+        """
+        验证token
+        :param token: JWT token字符串
+        :return: (是否有效, 设备ID)
+        """
+        try:
+            # 解码token
+            payload = jwt.decode(token, self.secret_key, algorithms=["HS256"])
+
+            # 检查是否过期
+            if payload["exp"] < time.time():
+                return False, None
+
+            return True, payload["device_id"]
+        except jwt.InvalidTokenError:
+            return False, None

main/xiaozhi-server/core/utils/util.py

@@ -882,3 +882,51 @@ def filter_sensitive_info(config: dict) -> dict:
         return filtered
 
     return _filter_dict(copy.deepcopy(config))
+
+
+def get_vision_url(config: dict) -> str:
+    """获取 vision URL
+
+    Args:
+        config: 配置字典
+
+    Returns:
+        str: vision URL
+    """
+    server_config = config["server"]
+    vision_explain = server_config.get("vision_explain", "")
+    if "你的" in vision_explain:
+        local_ip = get_local_ip()
+        port = int(server_config.get("http_port", 8003))
+        vision_explain = f"http://{local_ip}:{port}/mcp/vision/explain"
+    return vision_explain
+
+
+def is_valid_image_file(file_data: bytes) -> bool:
+    """
+    检查文件数据是否为有效的图片格式
+
+    Args:
+        file_data: 文件的二进制数据
+
+    Returns:
+        bool: 如果是有效的图片格式返回True，否则返回False
+    """
+    # 常见图片格式的魔数（文件头）
+    image_signatures = {
+        b"\xff\xd8\xff": "JPEG",
+        b"\x89PNG\r\n\x1a\n": "PNG",
+        b"GIF87a": "GIF",
+        b"GIF89a": "GIF",
+        b"BM": "BMP",
+        b"II*\x00": "TIFF",
+        b"MM\x00*": "TIFF",
+        b"RIFF": "WEBP",
+    }
+
+    # 检查文件头是否匹配任何已知的图片格式
+    for signature in image_signatures:
+        if file_data.startswith(signature):
+            return True
+
+    return False

main/xiaozhi-server/core/utils/vllm.py

@@ -0,0 +1,23 @@
+import os
+import sys
+
+# 添加项目根目录到Python路径
+current_dir = os.path.dirname(os.path.abspath(__file__))
+project_root = os.path.abspath(os.path.join(current_dir, "..", ".."))
+sys.path.insert(0, project_root)
+
+from config.logger import setup_logging
+import importlib
+
+logger = setup_logging()
+
+
+def create_instance(class_name, *args, **kwargs):
+    # 创建LLM实例
+    if os.path.exists(os.path.join("core", "providers", "vllm", f"{class_name}.py")):
+        lib_name = f"core.providers.vllm.{class_name}"
+        if lib_name not in sys.modules:
+            sys.modules[lib_name] = importlib.import_module(f"{lib_name}")
+        return sys.modules[lib_name].VLLMProvider(*args, **kwargs)
+
+    raise ValueError(f"不支持的VLLM类型: {class_name}，请检查该配置的type是否设置正确")

main/xiaozhi-server/docker-compose.yml

@@ -13,8 +13,8 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
-      # ota服务端
+      # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-      - "8002:8002"
+      - "8003:8003"
     volumes:
       # 配置文件目录
       - ./data:/opt/xiaozhi-esp32-server/data

main/xiaozhi-server/docker-compose_all.yml

@@ -15,6 +15,8 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
+      # http服务的端口，用于视觉分析接口
+      - "8003:8003"
     security_opt:
       - seccomp:unconfined
     environment:

main/xiaozhi-server/performance_tester.py

@@ -140,7 +140,9 @@ class AsyncPerformanceTester:
 
             print(f"🎵 测试 STT: {stt_name}")
 
-            text, _ = await stt.speech_to_text([self.test_wav_list[0]], "1")
+            text, _ = await stt.speech_to_text(
+                [self.test_wav_list[0]], "1", stt.audio_format
+            )
 
             if text is None:
                 print(f"❌ {stt_name} 连接失败")
@@ -151,7 +153,7 @@ class AsyncPerformanceTester:
 
             for i, sentence in enumerate(self.test_wav_list, 1):
                 start = time.time()
-                text, _ = await stt.speech_to_text([sentence], "1")
+                text, _ = await stt.speech_to_text([sentence], "1", stt.audio_format)
                 duration = time.time() - start
                 total_time += duration
 

main/xiaozhi-server/performance_tester_vllm.py

@@ -0,0 +1,189 @@
+import time
+import asyncio
+import logging
+import statistics
+import base64
+from typing import Dict
+from tabulate import tabulate
+from config.settings import load_config
+from core.utils.vllm import create_instance
+
+# 设置全局日志级别为WARNING，抑制INFO级别日志
+logging.basicConfig(level=logging.WARNING)
+
+
+class AsyncVisionPerformanceTester:
+    def __init__(self):
+        self.config = load_config()
+        self.test_images = [
+            "../../docs/images/demo1.png",
+            "../../docs/images/demo2.png",
+        ]
+        self.test_questions = [
+            "这张图片里有什么？",
+            "请详细描述这张图片的内容",
+        ]
+
+        # 加载测试图片
+        self.results = {"vllm": {}}
+
+    async def _test_vllm(self, vllm_name: str, config: Dict) -> Dict:
+        """异步测试单个视觉大模型性能"""
+        try:
+            # 检查API密钥配置
+            if "api_key" in config and any(
+                x in config["api_key"] for x in ["你的", "placeholder", "sk-xxx"]
+            ):
+                print(f"⏭️  VLLM {vllm_name} 未配置api_key，已跳过")
+                return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+            # 获取实际类型（兼容旧配置）
+            module_type = config.get("type", vllm_name)
+            vllm = create_instance(module_type, config)
+
+            print(f"🖼️ 测试 VLLM: {vllm_name}")
+
+            # 创建所有测试任务
+            test_tasks = []
+            for question in self.test_questions:
+                for image in self.test_images:
+                    test_tasks.append(
+                        self._test_single_vision(vllm_name, vllm, question, image)
+                    )
+
+            # 并发执行所有测试
+            test_results = await asyncio.gather(*test_tasks)
+
+            # 处理结果
+            valid_results = [r for r in test_results if r is not None]
+            if not valid_results:
+                print(f"⚠️  {vllm_name} 无有效数据，可能配置错误")
+                return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+            response_times = [r["response_time"] for r in valid_results]
+
+            # 过滤异常数据
+            mean = statistics.mean(response_times)
+            stdev = statistics.stdev(response_times) if len(response_times) > 1 else 0
+            filtered_times = [t for t in response_times if t <= mean + 3 * stdev]
+
+            if len(filtered_times) < len(test_tasks) * 0.5:
+                print(f"⚠️  {vllm_name} 有效数据不足，可能网络不稳定")
+                return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+            return {
+                "name": vllm_name,
+                "type": "vllm",
+                "avg_response": sum(response_times) / len(response_times),
+                "std_response": (
+                    statistics.stdev(response_times) if len(response_times) > 1 else 0
+                ),
+                "errors": 0,
+            }
+
+        except Exception as e:
+            print(f"⚠️ VLLM {vllm_name} 测试失败: {str(e)}")
+            return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+    async def _test_single_vision(
+        self, vllm_name: str, vllm, question: str, image: str
+    ) -> Dict:
+        """测试单个视觉问题的性能"""
+        try:
+            print(f"📝 {vllm_name} 开始测试: {question[:20]}...")
+            start_time = time.time()
+
+            # 读取图片并转换为base64
+            with open(image, "rb") as image_file:
+                image_data = image_file.read()
+                image_base64 = base64.b64encode(image_data).decode("utf-8")
+
+            # 直接获取响应
+            response = vllm.response(question, image_base64)
+            response_time = time.time() - start_time
+            print(f"✓ {vllm_name} 完成响应: {response_time:.3f}s")
+
+            return {
+                "name": vllm_name,
+                "type": "vllm",
+                "response_time": response_time,
+            }
+        except Exception as e:
+            print(f"⚠️ {vllm_name} 测试失败: {str(e)}")
+            return None
+
+    def _print_results(self):
+        """打印测试结果"""
+        vllm_table = []
+        for name, data in self.results["vllm"].items():
+            if data["errors"] == 0:
+                stability = data["std_response"] / data["avg_response"]
+                vllm_table.append(
+                    [
+                        name,
+                        f"{data['avg_response']:.3f}秒",
+                        f"{stability:.3f}",
+                    ]
+                )
+
+        if vllm_table:
+            print("\n视觉大模型性能排行:\n")
+            print(
+                tabulate(
+                    vllm_table,
+                    headers=["模型名称", "响应耗时", "稳定性"],
+                    tablefmt="github",
+                    colalign=("left", "right", "right"),
+                    disable_numparse=True,
+                )
+            )
+        else:
+            print("\n⚠️ 没有可用的视觉大模型进行测试。")
+
+    async def run(self):
+        """执行全量异步测试"""
+        print("🔍 开始筛选可用视觉大模型...")
+
+        if not self.test_images:
+            print(f"\n⚠️  {self.image_root} 路径下没有图片文件，无法进行测试")
+            return
+
+        # 创建所有测试任务
+        all_tasks = []
+
+        # VLLM测试任务
+        if self.config.get("VLLM") is not None:
+            for vllm_name, config in self.config.get("VLLM", {}).items():
+                if "api_key" in config and any(
+                    x in config["api_key"] for x in ["你的", "placeholder", "sk-xxx"]
+                ):
+                    print(f"⏭️  VLLM {vllm_name} 未配置api_key，已跳过")
+                    continue
+                print(f"🖼️ 添加VLLM测试任务: {vllm_name}")
+                all_tasks.append(self._test_vllm(vllm_name, config))
+
+        print(f"\n✅ 找到 {len(all_tasks)} 个可用视觉大模型")
+        print(f"✅ 使用 {len(self.test_images)} 张测试图片")
+        print(f"✅ 使用 {len(self.test_questions)} 个测试问题")
+        print("\n⏳ 开始并发测试所有模型...\n")
+
+        # 并发执行所有测试任务
+        all_results = await asyncio.gather(*all_tasks, return_exceptions=True)
+
+        # 处理结果
+        for result in all_results:
+            if isinstance(result, dict) and result["errors"] == 0:
+                self.results["vllm"][result["name"]] = result
+
+        # 打印结果
+        print("\n📊 生成测试报告...")
+        self._print_results()
+
+
+async def main():
+    tester = AsyncVisionPerformanceTester()
+    await tester.run()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

main/xiaozhi-server/requirements.txt

@@ -21,7 +21,7 @@ cozepy==0.12.0
 mem0ai==0.1.62
 bs4==0.0.2
 modelscope==1.23.2
-sherpa_onnx==1.11.0
+sherpa_onnx==1.12.0
 mcp==1.8.1
 cnlunar==0.2.0
 PySocks==1.7.1
@@ -31,3 +31,4 @@ chardet==5.2.0
 aioconsole==0.8.1
 markitdown==0.1.1
 mcp-proxy==0.6.0
+PyJWT==2.8.0