Merge pull request #1476 from xinnan-tech/fix-manager-agent-vllm

修复智控台保存时保存不了视觉模型的bug

README.md

@@ -121,6 +121,33 @@
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="实时打断" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="拍照识物品" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="多指令任务" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -141,11 +168,10 @@
 本项目提供两种部署方式，请根据您的具体需求选择：
 
 #### 🚀 部署方式选择
-
+| 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | 
-| 部署方式 | 特点 | 适用场景 | Docker部署文档 | 源码部署文档 | 视频教程 | 
 |---------|------|---------|---------|---------|---------|
-| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [Docker只运行Server](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [本地源码只运行Server](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| - | 
+| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
-| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[Docker运行全模块](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [本地源码运行全模块](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [本地源码视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
+| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
 
 > 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
@@ -158,22 +184,49 @@ OTA接口地址: https://2662r3426b.vicp.fun/xiaozhi/ota/
 Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ```
 
+#### 🚩 配置说明和推荐
+> [!Note]
+> 本项目默认的配置是`入门全免费`设置，如果想效果更优，推荐使用`流式配置`。
+> 
+> 本项目自`0.5.2`版本，已支持使用流式配置，相比`0.5`版本以前，响应速度提升约`2.5秒`
+
+| 模块名称 | 入门全免费设置 | 流式配置 |
+|:---:|:---:|:---:|
+| ASR(语音识别) | FunASR(本地) | 👍DoubaoStreamASR(火山流式语音识别) |
+| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
+| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
+| TTS(语音合成) | EdgeTTS(微软语音) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) |
+| Intent(意图识别) | function_call(函数调用) | ✅function_call(函数调用) |
+| Memory(记忆功能) | mem_local_short(本地短期记忆） | ✅mem_local_short（本地短期记忆） |
+
+#### 🔧 测试工具
+本项目提供以下测试工具，帮助您验证系统和选择合适的模型：
+
+| 工具名称 | 位置 | 使用方法 | 功能说明 |
+|:---:|:---|:---:|:---:|
+| 音频交互测试工具 | 位于`main/xiaozhi-server/test/test_page.html` | 使用谷歌浏览器直接打开 | 测试音频播放和接收功能，验证Python端音频处理是否正常 |
+| 模型响应测试工具1 | 位于`main/xiaozhi-server/performance_tester.py` | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、TTS(语音合成)三个核心模块的响应速度 |
+| 模型响应测试工具2 | 位于`main/xiaozhi-server/performance_tester_vllm.py` | 执行 `python performance_tester_vllm.py` | 测试VLLM(视觉模型)的响应速度 |
+
+> 💡 提示：测试模型速度时，只会测试配置了密钥的模型。
+
 ---
 ## 功能清单 ✨
-
 ### 已实现 ✅
 
 | 功能模块 | 描述 |
-|---------|------|
+|:---:|:---|
-| 通信协议 | 基于 `xiaozhi-esp32` 协议，通过 WebSocket 实现数据交互 |
+| 核心服务架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
-| 对话交互 | 支持唤醒对话、手动对话及实时打断。长时间无对话时自动休眠 |
+| 语音交互系统 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
-| 意图识别 | 支持使用LLM意图识别、function call函数调用，减少硬编码意图判断 |
+| 智能对话系统 | 支持多种LLM(大语言模型)，实现智能对话 |
-| 多语言识别 | 支持国语、粤语、英语、日语、韩语（默认使用 FunASR） |
+| 视觉感知系统 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
-| LLM 模块 | 支持灵活切换 LLM 模块，默认使用 ChatGLMLLM，也可选用阿里百炼、DeepSeek、Ollama 等接口 |
+| 意图识别系统 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
-| TTS 模块 | 支持 EdgeTTS（默认）、火山引擎豆包 TTS 等多种 TTS 接口，满足语音合成需求 |
+| 记忆系统 | 支持本地短期记忆、mem0ai接口记忆，具备记忆总结功能 |
-| 记忆功能 | 支持超长记忆、本地总结记忆、无记忆三种模式，满足不同场景需求 |
+| IOT/MCP控制协议 | 支持设备注册管理、智能控制接口，同时支持IOT、MCP控制协议 |
-| IOT功能 | 支持管理注册设备IOT功能，支持基于对话上下文语境下的智能物联网控制 |
+| 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理 |
-| 智控台 | 提供Web管理界面，支持智能体管理、用户管理、系统配置等功能，方便管理员和用户进行管理 |
+| 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
+| 部署支持 | 支持Docker部署和本地部署，提供完整的配置文件管理 |
+| 插件系统 | 支持功能插件扩展、自定义插件开发和插件热加载 |
 
 ### 正在开发 🚧
 
@@ -210,6 +263,16 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 ---
 
+### VLLM 视觉模型
+
+| 使用方式 | 支持平台 | 免费平台 |
+|:---:|:---:|:---:|
+| openai 接口调用 | 阿里百炼、智谱ChatGLMVLLM | 智谱ChatGLMVLLM |
+
+实际上，任何支持 openai 接口调用的 VLLM 均可接入使用。
+
+---
+
 ### TTS 语音合成
 
 | 使用方式 | 支持平台 | 免费平台 |

README_en.md

@@ -6,14 +6,14 @@
 This project provides backend services for the open-source smart hardware project
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
 Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
-Helping you quickly set up your Xiaozhi server
+Helps you quickly set up your Xiaozhi server
 </p>
 
 <p align="center">
 <a href="./README.md">中文</a>
 · <a href="./docs/FAQ.md">FAQ</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
-· <a href="./README_ed.md#deployment-documentation">Deployment Guide</a>
+· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
 </p>
 <p align="center">
@@ -50,7 +50,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
          <picture>
-           <img alt="Xiaozhi esp32 connecting to custom backend model" src="docs/images/demo1.png" />
+           <img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
          </picture>
         </a>
     </td>
@@ -64,14 +64,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
          <picture>
-           <img alt="Cantonese communication" src="docs/images/demo3.png" />
+           <img alt="Using Cantonese" src="docs/images/demo3.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
          <picture>
-           <img alt="Home appliance control" src="docs/images/demo5.png" />
+           <img alt="Control home appliances" src="docs/images/demo5.png" />
          </picture>
         </a>
     </td>
@@ -94,7 +94,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
          <picture>
-           <img alt="Music playback" src="docs/images/demo7.png" />
+           <img alt="Play music" src="docs/images/demo7.png" />
          </picture>
         </a>
     </td>
@@ -108,7 +108,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
          <picture>
-           <img alt="IOT device control" src="docs/images/demo9.png" />
+           <img alt="IOT command control" src="docs/images/demo9.png" />
          </picture>
         </a>
     </td>
@@ -120,6 +120,33 @@ Want to see it in action? Check out these videos 🎥
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="Real-time interruption" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="Photo recognition" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="Multi-command tasks" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -137,91 +164,102 @@ It is recommended that users prioritize service providers with relevant business
 
 ![Banners](docs/images/banner2.png)
 
-This project offers two deployment methods. Please choose based on your specific needs:
+This project provides two deployment methods. Please choose according to your specific needs:
 
 #### 🚀 Deployment Method Selection
+| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial | 
+|---------|------|---------|---------|---------|---------|
+| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - | 
+| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) | 
 
-| Deployment Method | Features | Use Case | Docker Deployment Guide | Source Code Deployment Guide |
+> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
-|---------|------|---------|---------|---------|
-| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database required | [Docker Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [Local Source Code Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)|
-| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience |[Docker Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [Local Source Code Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) |
-
-> 💡 Note: Below is the test platform deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data cleared daily
 
 ```
-Control Panel: https://2662r3426b.vicp.fun
+Control Panel Address: https://2662r3426b.vicp.fun
 
 Service Test Tool: https://2662r3426b.vicp.fun/test/
-OTA Interface: https://2662r3426b.vicp.fun/xiaozhi/ota/
+OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
-Websocket Interface: wss://2662r3426b.vicp.fun/xiaozhi/v1/
+Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ```
 
+#### 🚩 Configuration Description and Recommendations
+> [!Note]
+> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
+> 
+> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
+
+| Module Name | Entry Level Free Settings | Full Streaming Configuration |
+|---------|---------|------|
+| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
+| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
+| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
+| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
+| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
+| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
+
 ---
 ## Feature List ✨
-
 ### Implemented ✅
 
 | Feature Module | Description |
 |---------|------|
 | Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
-| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of inactivity |
+| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
 | Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
 | Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
-| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, also supports Ali Bailing, DeepSeek, Ollama, etc. |
+| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
-| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces for speech synthesis |
+| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
-| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes for different scenarios |
+| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
-| IOT Function | Supports managing registered device IOT functionality, intelligent IoT control based on dialogue context |
+| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
-| Control Panel | Provides web management interface, supports agent management, user management, system configuration, etc. |
+| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
 
 ### In Development 🚧
 
 To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
 
-If you're a software developer, here's an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
+If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
 
 ---
 
 ## Product Ecosystem 👬
-Xiaozhi is an ecosystem. When using this product, you might want to check out other excellent projects in this ecosystem:
+Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
 
-| Project Name | Project Link | Description |
+| Project Name | Project Address | Project Description |
 |:---------------------|:--------|:--------|
-| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue |
+| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
-| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | A Python-based AI client that allows you to experience Xiaozhi AI functionality through code without physical hardware |
+| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
-| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | A Java-based open-source project providing complete backend service solutions |
+| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
 
 ---
 
-## Supported Platforms/Components 📋
+## Supported Platforms/Components List 📋
 
 ### LLM Language Models
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| openai API | Ali Bailing, Volcano Engine Doubao, DeepSeek, ChatGLM, Gemini | ChatGLM, Gemini |
+| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
-| ollama API | Ollama | - |
+| ollama interface call | Ollama | - |
-| dify API | Dify | - |
+| dify interface call | Dify | - |
-| fastgpt API | Fastgpt | - |
+| fastgpt interface call | Fastgpt | - |
-| coze API | Coze | - |
+| coze interface call | Coze | - |
 
-Actually, any LLM supporting openai API calls can be integrated.
+In fact, any LLM that supports openai interface calls can be integrated and used.
-
----
 
 ### TTS Speech Synthesis
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| API Calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Aliyun TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
+| API Call | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
 | Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
 
 ---
 
 ### VAD Voice Activity Detection
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:---:|:---------:|:----:|:----:|:--:|
-| VAD | SileroVAD | Local Use | Free | |
+| VAD | SileroVAD | Local Usage | Free | |
 
 ---
 
@@ -229,26 +267,26 @@ Actually, any LLM supporting openai API calls can be integrated.
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| Local Use | FunASR, SherpaASR | FunASR, SherpaASR |
+| Local Usage | FunASR, SherpaASR | FunASR, SherpaASR |
-| API Calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+| API Call | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
 
 ---
 
 ### Memory Storage
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:---------------:|:----:|:---------:|:--:|
-| Memory | mem0ai | API Calls | 1000 calls/month quota | |
+| Memory | mem0ai | API Call | 1000 calls/month quota | |
 | Memory | mem_local_short | Local Summary | Free | |
 
 ---
 
 ### Intent Recognition
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:-------------:|:----:|:-------:|:---------------------:|
-| Intent | intent_llm | API Calls | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
+| Intent | intent_llm | API Call | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
-| Intent | function_call | API Calls | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
+| Intent | function_call | API Call | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
 
 ---
 
@@ -257,10 +295,10 @@ Actually, any LLM supporting openai API calls can be integrated.
 | Logo | Project/Company | Description |
 |:---:|:---:|:---|
 | <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
-| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for developing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing comprehensive technical documentation support for this project |
+| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for establishing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing full-chain technical documentation support for this project |
-| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
+| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing the function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
-| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering product user experience with their design experience serving over a thousand enterprises |
+| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering the product user experience with their design experience serving over a thousand enterprises |
-| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
+| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening the visual system of this project, ensuring consistency and extensibility of the overall design style in multi-scenario applications |
 
 
 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">

docs/FAQ.md

@@ -108,7 +108,11 @@ VAD:
 
 参考教程[阿里云短信集成指南](./ali-sms-integration.md)
 
-### 9、更多问题，可联系我们反馈 💬
+### 9、如何开启视觉模型实现拍照识物 📷
+
+参考教程[视觉模型使用指南](./mcp-vision-integration.md)
+
+### 10、更多问题，可联系我们反馈 💬
 
 可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
 

docs/dev-ops-integration.md

@@ -0,0 +1,163 @@
+# 全模块源码部署自动升级方法
+
+本教程是方便全模块源码部署的爱好者，如何通过自动命令，自动拉取源码，自动编译，自动启动端口运行。实现最高效率的升级系统。
+
+本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。
+
+# 开始条件
+- 你的电脑/服务器是linux操作系统
+- 你已经跑通了整个流程
+- 你喜欢跟进最新功能，但是觉得每次手动部署有点麻烦，期待有一个自动更新的方法
+
+第二个条件必须满足，因为本教程所涉及的某些文件，JDK、Node.js环境、Conda环境等，是需要你跑通整个流程才有的，如果你没有跑通，当我讲到某个文件的时候，你可能就不知道什么意思。
+
+# 教程效果
+- 解决国内不能拉取最新项目源码问题
+- 自动拉取代码编译前端文件
+- 自动拉取代码编译java文件，自动杀掉8002端口，自动启动8002端口
+- 自动拉取python代码，自动杀掉8000端口，自动启动8000端口
+
+# 第一步 选好你的项目目录
+
+例如，我规划了我的项目目录是，这是一个新建的空白的目录，如果你不想出错，可以和我一样
+```
+/home/system/xiaozhi
+```
+
+# 第二步 克隆本项目
+此刻，先要执行第一句话，拉取源码，这句命令适用于国内网络的服务器和电脑，无需翻墙
+
+```
+cd /home/system/xiaozhi
+git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-server.git
+```
+
+执行完后，你的项目目录会多了一个文件夹`xiaozhi-esp32-server`，这个就是项目的源码
+
+# 第三步 复制基础的文件
+
+如果你之前已经跑通了整个流程，对funasr的模型文件`xiaozhi-server/models/SenseVoiceSmall/model.pt`和你的私有配置文件`xiaozhi-server/data/.config.yaml`这两个文件不会陌生。
+
+此刻你需要把`model.pt`文件复制到新的目录去，你可以这样
+```
+cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
+cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
+```
+
+# 第四步 建立三个自动编译文件
+
+## 4.1 自动编译mananger-web模块
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8001.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git fetch --all
+git reset --hard
+git pull origin main
+
+
+cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web
+npm install
+npm run build
+rm -rf /home/system/xiaozhi/manager-web
+mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web/dist /home/system/xiaozhi/manager-web
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8001.sh
+```
+执行完后，继续往下
+
+## 4.2 自动编译运行manager-api模块
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8002.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git pull origin main
+
+
+cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api
+rm -rf target
+mvn clean package -Dmaven.test.skip=true
+cd /home/system/xiaozhi/
+
+# 查找占用8002端口的进程号
+PID=$(sudo netstat -tulnp | grep 8002 | awk '{print $7}' | cut -d'/' -f1)
+
+rm -rf /home/system/xiaozhi/xiaozhi-esp32-api.jar
+mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api/target/xiaozhi-esp32-api.jar /home/system/xiaozhi/xiaozhi-esp32-api.jar
+
+# 检查是否找到进程号
+if [ -z "$PID" ]; then
+  echo "没有找到占用8002端口的进程"
+else
+  echo "找到占用8002端口的进程，进程号为: $PID"
+  # 杀掉进程
+  kill -9 $PID
+  kill -9 $PID
+  echo "已杀掉进程 $PID"
+fi
+
+nohup java -jar xiaozhi-esp32-api.jar --spring.profiles.active=dev &
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8002.sh
+```
+执行完后，继续往下
+
+## 4.3 自动编译运行Python项目
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8000.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git pull origin main
+
+# 查找占用8000端口的进程号
+PID=$(sudo netstat -tulnp | grep 8000 | awk '{print $7}' | cut -d'/' -f1)
+
+# 检查是否找到进程号
+if [ -z "$PID" ]; then
+  echo "没有找到占用8000端口的进程"
+else
+  echo "找到占用8000端口的进程，进程号为: $PID"
+  # 杀掉进程
+  kill -9 $PID
+  kill -9 $PID
+  echo "已杀掉进程 $PID"
+fi
+cd main/xiaozhi-server
+pip install -r requirements.txt
+nohup python app.py >/dev/null &
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8000.sh
+```
+执行完后，继续往下
+
+# 日常更新
+
+以上的脚本都建立好后，日常更新，我们只要依次执行以下命令就可以做到自动更新和启动
+
+```
+# 进入pyhton环境
+conda activate xiaozhi-esp32-server
+cd /home/system/xiaozhi
+# 更新并启动Java程序
+./update_8001.sh
+# 更新web程序
+./update_8002.sh
+# 更新并启动python程序
+./update_8000.sh
+# 查看Java日志
+tail -f nohup.out
+# 查看Python日志
+tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server.log
+```
+
+# 注意事项
+测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)

docs/homeassistant-integration.md

@@ -202,7 +202,15 @@ change_role;get_weather;get_news;play_music;hass_get_state;hass_set_state
 
 #### 2. 配置小智开源服务端MCP配置信息
 
-切换到小智开源服务端`xiaozhi-esp32-server`的`mcp_server_settings.json`文件内，在`"mcpServers"`的括号内添加以下内容：
+
+进入`data`目录，找到`.mcp_server_settings.json`文件。
+
+如果你的`data`目录下没有`.mcp_server_settings.json`文件，
+- 请把在`xiaozhi-server`文件夹根目录的`mcp_server_settings.json`文件复制到`data`目录下，并重命名为`.mcp_server_settings.json`
+- 或[下载这个文件](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/main/xiaozhi-server/mcp_server_settings.json)，下载到`data`目录下，并重命名为`.mcp_server_settings.json`
+
+
+修改`"mcpServers"`里的这部分的内容：
 
 ```json
 "Home Assistant": {

docs/mcp-vision-integration.md

@@ -0,0 +1,171 @@
+# 视觉模型使用指南
+本教程分为两部分：
+- 第一部分：单模块运行xiaozhi-server开启视觉模型
+- 第二部分：全模块运行时，如何开启视觉模型
+
+开启视觉模型前，你需要准备三件事：
+- 你需要准备一台带摄像头的设备，而且这台设备已经在虾哥仓库里，实现了调用摄像头功能。例如`立创·实战派ESP32-S3开发板`
+- 你设备固件的版本升级到1.6.6及以上
+- 你已经成功跑通基础对话模块
+
+## 单模块运行xiaozhi-server开启视觉模型
+
+### 第一步确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose.yml`是否放了`8003`端口，如果没有就更新最新的`docker-compose.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步选择你的视觉模型
+打开你的`data/.config.yaml`文件，设置你的`selected_module.VLLM`设置为某个视觉模型。目前我们已经支持`openai`类型接口的视觉模型。`ChatGLMVLLM`就是其中一款兼容`openai`的模型。
+
+```
+selected_module:
+  VAD: ..
+  ASR: ..
+  LLM: ..
+  VLLM: ChatGLMVLLM
+  TTS: ..
+  Memory: ..
+  Intent: ..
+```
+
+假设我们使用`ChatGLMVLLM`作为视觉模型，那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+在你的配置文件中，增加这个配置，如果已经有了这个配置，就设置好你的api_key。
+
+```
+VLLM:
+  ChatGLMVLLM:
+    api_key: 你的api_key
+```
+
+### 第三步启动xiaozhi-server服务
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - OTA接口是           http://192.168.4.7:8003/xiaozhi/ota/
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。
+
+
+## 全模块运行时，如何开启视觉模型
+
+### 第一步 确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose_all.yml`是否映射了`8003`端口，如果没有就更新最新的`docker-compose_all.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步 确认你配置文件
+
+打开你的`data/.config.yaml`文件，确认一下你的配置文件的结构，是否和`data/config_from_api.yaml`一样。如果不一样，或缺少某项，请补齐。
+
+### 第三步 配置视觉模型密钥
+
+那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+登录`智控台`，顶部菜单点击`模型配置`，在左侧栏点击`视觉打语言模型`，找到`VLLM_ChatGLMVLLM`，点击修改按钮，在弹框中，在`API密钥`输入你密钥，点击保存。
+
+保存成功后，去到你需要测试的智能体哪里，点击`配置角色`，在打开的内容里，查看`视觉大语言模型(VLLM)`是否选择了刚才的视觉模型。点击保存。
+
+### 第三步 启动xiaozhi-server模块
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。

main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java

@@ -227,7 +227,7 @@ public interface Constant {
     /**
      * 版本号
      */
-    public static final String VERSION = "0.5.1";
+    public static final String VERSION = "0.5.4";
 
     /**
      * 无效固件URL

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java

@@ -171,6 +171,9 @@ public class AgentController {
         if (dto.getLlmModelId() != null) {
             existingEntity.setLlmModelId(dto.getLlmModelId());
         }
+        if (dto.getVllmModelId() != null) {
+            existingEntity.setVllmModelId(dto.getVllmModelId());
+        }
         if (dto.getTtsModelId() != null) {
             existingEntity.setTtsModelId(dto.getTtsModelId());
         }

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentDTO.java

@@ -27,6 +27,9 @@ public class AgentDTO {
     @Schema(description = "大语言模型名称", example = "llm_model_01")
     private String llmModelName;
 
+    @Schema(description = "视觉模型名称", example = "vllm_model_01")
+    private String vllmModelName;
+
     @Schema(description = "记忆模型ID", example = "mem_model_01")
     private String memModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentUpdateDTO.java

@@ -30,6 +30,9 @@ public class AgentUpdateDTO implements Serializable {
     @Schema(description = "大语言模型标识", example = "llm_model_02", required = false)
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识", example = "vllm_model_02", required = false)
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识", example = "tts_model_02", required = false)
     private String ttsModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentEntity.java

@@ -36,6 +36,9 @@ public class AgentEntity {
     @Schema(description = "大语言模型标识")
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识")
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识")
     private String ttsModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentTemplateEntity.java

@@ -49,6 +49,11 @@ public class AgentTemplateEntity implements Serializable {
      */
     private String llmModelId;
 
+    /**
+     * VLLM模型标识
+     */
+    private String vllmModelId;
+
     /**
      * 语音合成模型标识
      */

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentServiceImpl.java

@@ -102,6 +102,9 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
             // 获取 LLM 模型名称
             dto.setLlmModelName(modelConfigService.getModelNameById(agent.getLlmModelId()));
 
+            // 获取 VLLM 模型名称
+            dto.setVllmModelName(modelConfigService.getModelNameById(agent.getVllmModelId()));
+
             // 获取记忆模型名称
             dto.setMemModelId(agent.getMemModelId());
 

main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java

@@ -72,6 +72,7 @@ public class ConfigServiceImpl implements ConfigService {
                 null,
                 null,
                 null,
+                null,
                 result,
                 isCache);
 
@@ -140,6 +141,7 @@ public class ConfigServiceImpl implements ConfigService {
                 agent.getVadModelId(),
                 agent.getAsrModelId(),
                 agent.getLlmModelId(),
+                agent.getVllmModelId(),
                 agent.getTtsModelId(),
                 agent.getMemModelId(),
                 agent.getIntentModelId(),
@@ -241,6 +243,7 @@ public class ConfigServiceImpl implements ConfigService {
             String vadModelId,
             String asrModelId,
             String llmModelId,
+            String vllmModelId,
             String ttsModelId,
             String memModelId,
             String intentModelId,
@@ -248,8 +251,8 @@ public class ConfigServiceImpl implements ConfigService {
             boolean isCache) {
         Map<String, String> selectedModule = new HashMap<>();
 
-        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM" };
+        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM", "VLLM" };
-        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId };
+        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId, vllmModelId };
         String intentLLMModelId = null;
         String memLocalShortLLMModelId = null;
 

main/manager-api/src/main/resources/db/changelog/202506010920.sql

@@ -0,0 +1,29 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_VLLM_openai';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_VLLM_openai', 'VLLM', 'openai', 'OpenAI接口', '[{"key":"base_url","label":"基础URL","type":"string"},{"key":"model_name","label":"模型名称","type":"string"},{"key":"api_key","label":"API密钥","type":"string"}]', 9, 1, NOW(), 1, NOW());
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_ChatGLMVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_ChatGLMVLLM', 'VLLM', 'ChatGLMVLLM', '智谱视觉AI', 1, 1, '{\"type\": \"openai\", \"model_name\": \"glm-4v-flash\", \"base_url\": \"https://open.bigmodel.cn/api/paas/v4/\", \"api_key\": \"你的api_key\"}', NULL, NULL, 1, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bigmodel.cn/usercenter/proj-mgmt/apikeys',
+`remark` = '智谱视觉AI配置说明：
+1. 访问 https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_ChatGLMVLLM';
+
+
+-- 添加参数
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (113, 'server.http_port', '8003', 'number', 1, 'http服务的端口，用于启动视觉分析接口');
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (114, 'server.vision_explain', 'null', 'string', 1, '视觉分析接口地址，用于下发到设备，多个用;分隔');
+
+-- 智能体表增加VLLM模型配置
+ALTER TABLE `ai_agent` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;
+
+-- 智能体模版表增加VLLM模型配置
+ALTER TABLE `ai_agent_template` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;

main/manager-api/src/main/resources/db/changelog/202506031639.sql

@@ -0,0 +1,45 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_DoubaoStreamASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_DoubaoStreamASR', 'ASR', 'doubao_stream', '火山引擎语音识别(流式)', '[{"key":"appid","label":"应用ID","type":"string"},{"key":"access_token","label":"访问令牌","type":"string"},{"key":"cluster","label":"集群","type":"string"},{"key":"boosting_table_name","label":"热词文件名称","type":"string"},{"key":"correct_table_name","label":"替换词文件名称","type":"string"},{"key":"output_dir","label":"输出目录","type":"string"}]', 3, 1, NOW(), 1, NOW());
+
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'ASR_DoubaoStreamASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_DoubaoStreamASR', 'ASR', 'DoubaoStreamASR', '豆包语音识别(流式)', 0, 1, '{\"type\": \"doubao_stream\", \"appid\": \"\", \"access_token\": \"\", \"cluster\": \"volcengine_input_common\", \"output_dir\": \"tmp/\"}', NULL, NULL, 3, NULL, NULL, NULL, NULL);
+
+
+-- 更新豆包ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.volcengine.com/speech/app',
+`remark` = '豆包ASR配置说明：
+1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
+2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
+3. 需要在火山引擎控制台创建应用并获取appid和access_token
+4. 支持中文语音识别
+5. 需要网络连接
+6. 输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://console.volcengine.com/speech/app
+2. 创建新应用
+3. 获取appid和access_token
+4. 填入配置文件中
+如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
+' WHERE `id` = 'ASR_DoubaoASR';
+
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.volcengine.com/speech/app',
+`remark` = '豆包ASR配置说明：
+1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
+2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
+3. 需要在火山引擎控制台创建应用并获取appid和access_token
+4. 支持中文语音识别
+5. 需要网络连接
+6. 输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://console.volcengine.com/speech/app
+2. 创建新应用
+3. 获取appid和access_token
+4. 填入配置文件中
+如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
+' WHERE `id` = 'ASR_DoubaoStreamASR';

main/manager-api/src/main/resources/db/changelog/202506032232.sql

@@ -0,0 +1,14 @@
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_QwenVLVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_QwenVLVLLM', 'VLLM', 'QwenVLVLLM', '千问视觉模型', 0, 1, '{\"type\": \"openai\", \"model_name\": \"qwen2.5-vl-3b-instruct\", \"base_url\": \"https://dashscope.aliyuncs.com/compatible-mode/v1\", \"api_key\": \"你的api_key\"}', NULL, NULL, 2, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bailian.console.aliyun.com/?tab=api#/api/?type=model&url=https%3A%2F%2Fhelp.aliyun.com%2Fdocument_detail%2F2845564.html&renderType=iframe',
+`remark` = '千问视觉模型配置说明：
+1. 访问 https://bailian.console.aliyun.com/?tab=model#/api-key
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_QwenVLVLLM';
+
+-- 删除参数，这两个参数已挪至python配置文件
+delete from `sys_params` where id  in (113,114);

main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml

@@ -170,3 +170,24 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202505271414.sql
+  - changeSet:
+      id: 202506010920
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506010920.sql
+  - changeSet:
+      id: 202506031639
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506031639.sql
+  - changeSet:
+      id: 202506032232
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506032232.sql

main/manager-web/src/components/DeviceItem.vue

@@ -14,10 +14,10 @@
       </div>
     </div>
     <div class="device-name">
-      设备型号：{{ device.ttsModelName }}
+      语言模型：{{ device.llmModelName }}
     </div>
     <div class="device-name">
-      音色模型：{{ device.ttsVoiceName }}
+      音色模型：{{ device.ttsModelName }} ({{ device.ttsVoiceName }})
     </div>
     <div style="display: flex;gap: 10px;align-items: center;">
       <div class="settings-btn" @click="handleConfigure">

main/manager-web/src/views/ModelConfig.vue

@@ -30,6 +30,9 @@
           <el-menu-item index="llm">
             <span class="menu-text">大语言模型</span>
           </el-menu-item>
+          <el-menu-item index="vllm">
+            <span class="menu-text">视觉大模型</span>
+          </el-menu-item>
           <el-menu-item index="intent">
             <span class="menu-text">意图识别</span>
           </el-menu-item>
@@ -173,6 +176,7 @@ export default {
         vad: '语言活动检测模型(VAD)',
         asr: '语音识别模型(ASR)',
         llm: '大语言模型（LLM）',
+        vllm: '视觉大模型（VLLM）',
         intent: '意图识别模型(Intent)',
         tts: '语音合成模型(TTS)',
         memory: '记忆模型(Memory)'

main/manager-web/src/views/roleConfig.vue

@@ -64,7 +64,27 @@
                     </el-form-item>
                   </div>
                   <div class="form-column">
-                    <el-form-item v-for="(model, index) in models" :key="`model-${index}`" :label="model.label"
+                    <div class="model-row">
+                      <el-form-item label="语音活动检测(VAD)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.vadModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('VAD', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['VAD']"
+                              :key="`option-vad-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                      <el-form-item label="语音识别(ASR)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.asrModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('ASR', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['ASR']"
+                              :key="`option-asr-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                    </div>
+                    <el-form-item v-for="(model, index) in models.slice(2)" :key="`model-${index}`" :label="model.label"
                       class="model-item">
                       <div class="model-select-wrapper">
                         <el-select v-model="form.model[model.key]" filterable placeholder="请选择" class="form-select"
@@ -148,6 +168,7 @@ export default {
           vadModelId: "",
           asrModelId: "",
           llmModelId: "",
+          vllmModelId: "",
           memModelId: "",
           intentModelId: "",
         }
@@ -156,6 +177,7 @@ export default {
         { label: '语音活动检测(VAD)', key: 'vadModelId', type: 'VAD' },
         { label: '语音识别(ASR)', key: 'asrModelId', type: 'ASR' },
         { label: '大语言模型(LLM)', key: 'llmModelId', type: 'LLM' },
+        { label: '视觉大模型(VLLM)', key: 'vllmModelId', type: 'VLLM' },
         { label: '意图识别(Intent)', key: 'intentModelId', type: 'Intent' },
         { label: '记忆(Memory)', key: 'memModelId', type: 'Memory' },
         { label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' },
@@ -189,6 +211,7 @@ export default {
         asrModelId: this.form.model.asrModelId,
         vadModelId: this.form.model.vadModelId,
         llmModelId: this.form.model.llmModelId,
+        vllmModelId: this.form.model.vllmModelId,
         ttsModelId: this.form.model.ttsModelId,
         ttsVoiceId: this.form.ttsVoiceId,
         chatHistoryConf: this.form.chatHistoryConf,
@@ -236,6 +259,7 @@ export default {
             vadModelId: "",
             asrModelId: "",
             llmModelId: "",
+            vllmModelId: "",
             memModelId: "",
             intentModelId: "",
           }
@@ -289,6 +313,7 @@ export default {
           vadModelId: templateData.vadModelId || this.form.model.vadModelId,
           asrModelId: templateData.asrModelId || this.form.model.asrModelId,
           llmModelId: templateData.llmModelId || this.form.model.llmModelId,
+          vllmModelId: templateData.vllmModelId || this.form.model.vllmModelId,
           memModelId: templateData.memModelId || this.form.model.memModelId,
           intentModelId: templateData.intentModelId || this.form.model.intentModelId
         }
@@ -305,6 +330,7 @@ export default {
               vadModelId: data.data.vadModelId,
               asrModelId: data.data.asrModelId,
               llmModelId: data.data.llmModelId,
+              vllmModelId: data.data.vllmModelId,
               memModelId: data.data.memModelId,
               intentModelId: data.data.intentModelId
             }
@@ -587,6 +613,25 @@ export default {
   width: 100%;
 }
 
+.model-row {
+  display: flex;
+  gap: 20px;
+  margin-bottom: 6px;
+}
+
+.model-row .model-item {
+  flex: 1;
+  margin-bottom: 0;
+}
+
+.model-row .el-form-item__label {
+  font-size: 12px !important;
+  color: #3d4566 !important;
+  font-weight: 400;
+  line-height: 22px;
+  padding-bottom: 2px;
+}
+
 .function-icons {
   display: flex;
   align-items: center;

main/xiaozhi-server/app.py

@@ -1,13 +1,14 @@
-import asyncio
 import sys
+import uuid
 import signal
+import asyncio
+from aioconsole import ainput
 from config.settings import load_config
-from core.websocket_server import WebSocketServer
-from core.ota_server import SimpleOtaServer
-from core.utils.util import check_ffmpeg_installed
 from config.logger import setup_logging
 from core.utils.util import get_local_ip
-from aioconsole import ainput
+from core.http_server import SimpleHttpServer
+from core.websocket_server import WebSocketServer
+from core.utils.util import check_ffmpeg_installed
 
 TAG = __name__
 logger = setup_logging()
@@ -45,25 +46,37 @@ async def main():
     check_ffmpeg_installed()
     config = load_config()
 
+    # 默认使用manager-api的secret作为auth_key
+    # 如果secret为空，则生成随机密钥
+    # auth_key用于jwt认证，比如视觉分析接口的jwt认证
+    auth_key = config.get("manager-api", {}).get("secret", "")
+    if not auth_key or len(auth_key) == 0 or "你" in auth_key:
+        auth_key = str(uuid.uuid4().hex)
+    config["server"]["auth_key"] = auth_key
+
     # 添加 stdin 监控任务
     stdin_task = asyncio.create_task(monitor_stdin())
 
     # 启动 WebSocket 服务器
     ws_server = WebSocketServer(config)
     ws_task = asyncio.create_task(ws_server.start())
-    ota_task = None
+    # 启动 Simple http 服务器
+    ota_server = SimpleHttpServer(config)
+    ota_task = asyncio.create_task(ota_server.start())
 
     read_config_from_api = config.get("read_config_from_api", False)
+    port = int(config["server"].get("http_port", 8003))
     if not read_config_from_api:
-        # 启动 Simple OTA 服务器
-        ota_server = SimpleOtaServer(config)
-        ota_task = asyncio.create_task(ota_server.start())
-
         logger.bind(tag=TAG).info(
             "OTA接口是\t\thttp://{}:{}/xiaozhi/ota/",
             get_local_ip(),
-            config["server"]["ota_port"],
+            port,
         )
+    logger.bind(tag=TAG).info(
+        "视觉分析接口是\thttp://{}:{}/mcp/vision/explain",
+        get_local_ip(),
+        port,
+    )
 
     # 获取WebSocket配置，使用安全的默认值
     websocket_port = 8000

main/xiaozhi-server/config.yaml

@@ -2,6 +2,7 @@
 # 然后你想修改覆盖修改什么配置，就修改【.config.yaml】文件，而不是修改【config.yaml】文件
 # 系统会优先读取【data/.config.yaml】文件的配置，如果【.config.yaml】文件里的配置不存在，系统会自动去读取【config.yaml】文件的配置。
 # 这样做，可以最简化配置，保护您的密钥安全。
+# 如果你使用了智控台，那么以下所有配置，都不会生效，请在智控台中修改配置
 
 # #####################################################################################
 # #############################以下是服务器基本运行配置####################################
@@ -9,13 +10,21 @@ server:
   # 服务器监听地址和端口(Server listening address and port)
   ip: 0.0.0.0
   port: 8000
-  # OTA接口的端口号
+  # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-  ota_port: 8002
+  http_port: 8003
   # 这个websocket配置是指ota接口向设备发送的websocket地址
-  # 如果按默认的写法，ota接口会自动生成websocket地址。这个地址你可以直接用浏览器访问ota接口确认一下
+  # 如果按默认的写法，ota接口会自动生成websocket地址，并输出在启动日志里，这个地址你可以直接用浏览器访问ota接口确认一下
   # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
-  # 所以如果你使用docker部署或使用公网部署时，请设置正确的websocket地址
+  # 所以如果你使用docker部署时，将websocket设置成局域网地址
+  # 如果你使用公网部署时，将vwebsocket设置成公网地址
   websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
   # OTA返回信息时区偏移量
   timezone_offset: +8
   # 认证配置
@@ -160,6 +169,8 @@ selected_module:
   ASR: FunASR
   # 将根据配置名称对应的type调用实际的LLM适配器
   LLM: ChatGLMLLM
+  # 视觉语言大模型
+  VLLM: ChatGLMVLLM
   # TTS将根据配置名称对应的type调用实际的TTS适配器
   TTS: EdgeTTS
   # 记忆模块，默认不开启记忆；如果想使用超长记忆，推荐使用mem0ai；如果注重隐私，请使用本地的mem_local_short
@@ -253,6 +264,8 @@ ASR:
   DoubaoASR:
     # 可以在这里申请相关Key等信息
     # https://console.volcengine.com/speech/app
+    # DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
+    # 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
     type: doubao
     appid: 你的火山引擎语音合成服务appid
     access_token: 你的火山引擎语音合成服务access_token
@@ -261,6 +274,20 @@ ASR:
     boosting_table_name: （选填）你的热词文件名称
     correct_table_name: （选填）你的替换词文件名称
     output_dir: tmp/
+  DoubaoStreamASR:
+    # 可以在这里申请相关Key等信息
+    # https://console.volcengine.com/speech/app
+    # DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
+    # 开通地址https://console.volcengine.com/speech/service/10011
+    # 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
+    type: doubao_stream
+    appid: 你的火山引擎语音合成服务appid
+    access_token: 你的火山引擎语音合成服务access_token
+    cluster: volcengine_input_common
+    # 热词、替换词使用流程：https://www.volcengine.com/docs/6561/155738
+    boosting_table_name: （选填）你的热词文件名称
+    correct_table_name: （选填）你的替换词文件名称
+    output_dir: tmp/
   TencentASR:
     # token申请地址：https://console.cloud.tencent.com/cam/capi
     # 免费领取资源：https://console.cloud.tencent.com/asr/resourcebundle
@@ -432,6 +459,21 @@ LLM:
     # Xinference服务地址和模型名称
     model_name: qwen2.5:3b-AWQ  # 使用的小模型名称，用于意图识别
     base_url: http://localhost:9997  # Xinference服务地址
+# VLLM配置（视觉语言大模型）
+VLLM:
+  ChatGLMVLLM:
+    type: openai
+    # glm-4v-flash是智谱免费AI的视觉模型，需要先在智谱AI平台创建API密钥并获取api_key
+    # 可在这里找到你的api key https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+    model_name: glm-4v-flash  # 智谱AI的视觉模型
+    url: https://open.bigmodel.cn/api/paas/v4/
+    api_key: 你的api_key
+  QwenVLVLLM:
+    type: openai
+    model_name: qwen2.5-vl-3b-instruct
+    url: https://dashscope.aliyuncs.com/compatible-mode/v1
+    # 可在这里找到你的api key https://bailian.console.aliyun.com/?apiKey=1#/api-key
+    api_key: 你的api_key
 TTS:
   # 当前支持的type为edge、doubao，可自行适配
   EdgeTTS:

main/xiaozhi-server/config/config_loader.py

@@ -59,10 +59,14 @@ def get_config_from_api(config):
         "url": config["manager-api"].get("url", ""),
         "secret": config["manager-api"].get("secret", ""),
     }
+    # server的配置以本地为准
     if config.get("server"):
         config_data["server"] = {
             "ip": config["server"].get("ip", ""),
             "port": config["server"].get("port", ""),
+            "http_port": config["server"].get("http_port", ""),
+            "vision_explain": config["server"].get("vision_explain", ""),
+            "auth_key": config["server"].get("auth_key", ""),
         }
     return config_data
 

main/xiaozhi-server/config/logger.py

@@ -4,7 +4,7 @@ from loguru import logger
 from config.config_loader import load_config
 from config.settings import check_config_file
 
-SERVER_VERSION = "0.5.1"
+SERVER_VERSION = "0.5.4"
 _logger_initialized = False
 
 
@@ -48,7 +48,11 @@ def setup_logging():
 
     # 第一次初始化时配置日志
     if not _logger_initialized:
-        logger.configure(extra={"selected_module": log_config.get("selected_module", "00000000000000")})  # 新增配置
+        logger.configure(
+            extra={
+                "selected_module": log_config.get("selected_module", "00000000000000")
+            }
+        )  # 新增配置
         log_format = log_config.get(
             "log_format",
             "<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
@@ -62,7 +66,9 @@ def setup_logging():
         log_format = log_format.replace("{version}", SERVER_VERSION)
         log_format = log_format.replace("{selected_module}", selected_module_str)
         log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace("{selected_module}", selected_module_str)
+        log_format_file = log_format_file.replace(
+            "{selected_module}", selected_module_str
+        )
 
         log_level = log_config.get("log_level", "INFO")
         log_dir = log_config.get("log_dir", "tmp")
@@ -116,15 +122,25 @@ def update_module_string(selected_module_str):
         log_format = log_format.replace("{version}", SERVER_VERSION)
         log_format = log_format.replace("{selected_module}", selected_module_str)
         log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace("{selected_module}", selected_module_str)
+        log_format_file = log_format_file.replace(
+            "{selected_module}", selected_module_str
+        )
 
         logger.remove()
-        logger.add(sys.stdout, format=log_format, level=log_config.get("log_level", "INFO"), filter=formatter)
         logger.add(
-            os.path.join(log_config.get("log_dir", "tmp"), log_config.get("log_file", "server.log")),
+            sys.stdout,
+            format=log_format,
+            level=log_config.get("log_level", "INFO"),
+            filter=formatter,
+        )
+        logger.add(
+            os.path.join(
+                log_config.get("log_dir", "tmp"),
+                log_config.get("log_file", "server.log"),
+            ),
             format=log_format_file,
             level=log_config.get("log_level", "INFO"),
-            filter=formatter
+            filter=formatter,
         )
 
     except Exception as e:

main/xiaozhi-server/config_from_api.yaml

@@ -8,6 +8,15 @@
 server:
   ip: 0.0.0.0
   port: 8000
+  # http服务的端口，用于视觉分析接口
+  http_port: 8003
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
 manager-api:
   # 你的manager-api的地址，最好使用局域网ip
   url: http://127.0.0.1:8002/xiaozhi

main/xiaozhi-server/core/api/base_handler.py

@@ -0,0 +1,16 @@
+from aiohttp import web
+from config.logger import setup_logging
+
+
+class BaseHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/api/ota_handler.py

@@ -1,18 +1,15 @@
 import json
 import time
-import asyncio
 from aiohttp import web
-from config.logger import setup_logging
+from core.utils.util import get_local_ip
-from core.connection import ConnectionHandler
+from core.api.base_handler import BaseHandler
-from core.utils.util import get_local_ip, initialize_modules
 
 TAG = __name__
 
 
-class SimpleOtaServer:
+class OTAHandler(BaseHandler):
     def __init__(self, config: dict):
-        self.config = config
+        super().__init__(config)
-        self.logger = setup_logging()
 
     def _get_websocket_url(self, local_ip: str, port: int) -> str:
         """获取websocket地址
@@ -25,41 +22,15 @@ class SimpleOtaServer:
             str: websocket地址
         """
         server_config = self.config["server"]
-        websocket_config = server_config.get("websocket")
+        websocket_config = server_config.get("websocket", "")
 
-        if websocket_config and "你" not in websocket_config:
+        if "你的" not in websocket_config:
             return websocket_config
         else:
             return f"ws://{local_ip}:{port}/xiaozhi/v1/"
 
-    async def start(self):
+    async def handle_post(self, request):
-        server_config = self.config["server"]
+        """处理 OTA POST 请求"""
-        host = server_config.get("ip", "0.0.0.0")
-        port = int(server_config.get("ota_port"))
-
-        if port:
-            app = web.Application()
-            # 添加路由
-            app.add_routes(
-                [
-                    web.get("/xiaozhi/ota/", self._handle_ota_get_request),
-                    web.post("/xiaozhi/ota/", self._handle_ota_request),
-                    web.options("/xiaozhi/ota/", self._handle_ota_request),
-                ]
-            )
-
-            # 运行服务
-            runner = web.AppRunner(app)
-            await runner.setup()
-            site = web.TCPSite(runner, host, port)
-            await site.start()
-
-            # 保持服务运行
-            while True:
-                await asyncio.sleep(3600)  # 每隔 1 小时检查一次
-
-    async def _handle_ota_request(self, request):
-        """处理 /xiaozhi/ota/ 的 POST 请求"""
         try:
             data = await request.text()
             self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}")
@@ -75,11 +46,9 @@ class SimpleOtaServer:
             data_json = json.loads(data)
 
             server_config = self.config["server"]
-            host = server_config.get("ip", "0.0.0.0")
             port = int(server_config.get("port", 8000))
             local_ip = get_local_ip()
 
-            # OTA基础信息
             return_json = {
                 "server_time": {
                     "timestamp": int(round(time.time() * 1000)),
@@ -104,16 +73,11 @@ class SimpleOtaServer:
                 content_type="application/json",
             )
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response
 
-    async def _handle_ota_get_request(self, request):
+    async def handle_get(self, request):
-        """处理 /xiaozhi/ota/ 的 GET 请求"""
+        """处理 OTA GET 请求"""
         try:
             server_config = self.config["server"]
             local_ip = get_local_ip()
@@ -125,10 +89,5 @@ class SimpleOtaServer:
             self.logger.bind(tag=TAG).error(f"OTA GET请求异常: {e}")
             response = web.Response(text="OTA接口异常", content_type="text/plain")
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response

main/xiaozhi-server/core/api/vision_handler.py

@@ -0,0 +1,184 @@
+import json
+import copy
+from aiohttp import web
+from config.logger import setup_logging
+from core.utils.util import get_vision_url, is_valid_image_file
+from core.utils.vllm import create_instance
+from config.config_loader import get_private_config_from_api
+from core.utils.auth import AuthToken
+import base64
+from typing import Tuple, Optional
+
+TAG = __name__
+
+# 设置最大文件大小为5MB
+MAX_FILE_SIZE = 5 * 1024 * 1024
+
+
+class VisionHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        # 初始化认证工具
+        self.auth = AuthToken(config["server"]["auth_key"])
+
+    def _create_error_response(self, message: str) -> dict:
+        """创建统一的错误响应格式"""
+        return {"success": False, "message": message}
+
+    def _verify_auth_token(self, request) -> Tuple[bool, Optional[str]]:
+        """验证认证token"""
+        auth_header = request.headers.get("Authorization", "")
+        if not auth_header.startswith("Bearer "):
+            return False, None
+
+        token = auth_header[7:]  # 移除"Bearer "前缀
+        return self.auth.verify_token(token)
+
+    async def handle_post(self, request):
+        """处理 MCP Vision POST 请求"""
+        response = None  # 初始化response变量
+        try:
+            # 验证token
+            is_valid, token_device_id = self._verify_auth_token(request)
+            if not is_valid:
+                response = web.Response(
+                    text=json.dumps(
+                        self._create_error_response("无效的认证token或token已过期")
+                    ),
+                    content_type="application/json",
+                    status=401,
+                )
+                return response
+
+            # 获取请求头信息
+            device_id = request.headers.get("Device-Id", "")
+            client_id = request.headers.get("Client-Id", "")
+            if device_id != token_device_id:
+                return web.Response(
+                    text=json.dumps(self._create_error_response("设备ID与token不匹配")),
+                    content_type="application/json",
+                    status=401,
+                )
+            # 解析multipart/form-data请求
+            reader = await request.multipart()
+
+            # 读取question字段
+            question_field = await reader.next()
+            if question_field is None:
+                raise ValueError("缺少问题字段")
+            question = await question_field.text()
+            self.logger.bind(tag=TAG).debug(f"Question: {question}")
+
+            # 读取图片文件
+            image_field = await reader.next()
+            if image_field is None:
+                raise ValueError("缺少图片文件")
+
+            # 读取图片数据
+            image_data = await image_field.read()
+            if not image_data:
+                raise ValueError("图片数据为空")
+
+            # 检查文件大小
+            if len(image_data) > MAX_FILE_SIZE:
+                raise ValueError(
+                    f"图片大小超过限制，最大允许{MAX_FILE_SIZE/1024/1024}MB"
+                )
+
+            # 检查文件格式
+            if not is_valid_image_file(image_data):
+                raise ValueError(
+                    "不支持的文件格式，请上传有效的图片文件（支持JPEG、PNG、GIF、BMP、TIFF、WEBP格式）"
+                )
+
+            # 将图片转换为base64编码
+            image_base64 = base64.b64encode(image_data).decode("utf-8")
+
+            # 如果开启了智控台，则从智控台获取模型配置
+            current_config = copy.deepcopy(self.config)
+            read_config_from_api = current_config.get("read_config_from_api", False)
+            if read_config_from_api:
+                current_config = get_private_config_from_api(
+                    current_config,
+                    device_id,
+                    client_id,
+                )
+
+            select_vllm_module = current_config["selected_module"].get("VLLM")
+            if not select_vllm_module:
+                raise ValueError("您还未设置默认的视觉分析模块")
+
+            vllm_type = (
+                select_vllm_module
+                if "type" not in current_config["VLLM"][select_vllm_module]
+                else current_config["VLLM"][select_vllm_module]["type"]
+            )
+
+            if not vllm_type:
+                raise ValueError(f"无法找到VLLM模块对应的供应器{vllm_type}")
+
+            vllm = create_instance(
+                vllm_type, current_config["VLLM"][select_vllm_module]
+            )
+
+            result = vllm.response(question, image_base64)
+
+            return_json = {
+                "success": True,
+                "result": result,
+            }
+
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except ValueError as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response(str(e))
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response("处理请求时发生错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            if response:
+                self._add_cors_headers(response)
+            return response
+
+    async def handle_get(self, request):
+        """处理 MCP Vision GET 请求"""
+        try:
+            vision_explain = get_vision_url(self.config)
+            if vision_explain and len(vision_explain) > 0 and "null" != vision_explain:
+                message = (
+                    f"MCP Vision 接口运行正常，视觉解释接口地址是：{vision_explain}"
+                )
+            else:
+                message = "MCP Vision 接口运行不正常，请打开data目录下的.config.yaml文件，找到【server.vision_explain】，设置好地址"
+
+            response = web.Response(text=message, content_type="text/plain")
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision GET请求异常: {e}")
+            return_json = self._create_error_response("服务器内部错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            self._add_cors_headers(response)
+            return response
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/connection.py

@@ -10,27 +10,31 @@ import threading
 import traceback
 import subprocess
 import websockets
+from core.handle.mcpHandle import call_mcp_tool
 from core.utils.util import (
     extract_json_from_string,
-    initialize_modules,
     check_vad_update,
     check_asr_update,
     filter_sensitive_info,
-    initialize_tts,
 )
 from typing import Dict, Any
 from core.mcp.manager import MCPManager
+from core.utils.modules_initialize import (
+    initialize_modules,
+    initialize_tts,
+    initialize_asr,
+)
 from core.handle.reportHandle import report
 from core.providers.tts.default import DefaultTTS
 from concurrent.futures import ThreadPoolExecutor
 from core.utils.dialogue import Message, Dialogue
+from core.providers.asr.dto.dto import InterfaceType
 from core.handle.textHandle import handleTextMessage
 from core.handle.functionHandler import FunctionHandler
 from plugins_func.loadplugins import auto_import_modules
 from plugins_func.register import Action, ActionResponse
 from core.auth import AuthMiddleware, AuthenticationError
 from config.config_loader import get_private_config_from_api
-from core.handle.receiveAudioHandle import handleAudioMessage
 from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType
 from config.logger import setup_logging, build_module_string, update_module_string
 from config.manage_api_client import DeviceNotFoundException, DeviceBindException
@@ -76,9 +80,11 @@ class ConnectionHandler:
         self.welcome_msg = None
         self.max_output_size = 0
         self.chat_history_conf = 0
+        self.audio_format = "opus"
 
         # 客户端状态相关
         self.client_abort = False
+        self.client_is_speaking = False
         self.client_listen_mode = "auto"
 
         # 线程任务相关
@@ -111,8 +117,10 @@ class ConnectionHandler:
         self.client_voice_stop = False
 
         # asr相关变量
+        # 因为实际部署时可能会用到公共的本地ASR，不能把变量暴露给公共ASR
+        # 所以涉及到ASR的变量，需要在这里定义，属于connection的私有变量
         self.asr_audio = []
-        self.asr_server_receive = True
+        self.asr_audio_queue = queue.Queue()
 
         # llm相关变量
         self.llm_finish_task = True
@@ -141,7 +149,8 @@ class ConnectionHandler:
             int(self.config.get("close_connection_no_voice_time", 120)) + 60
         )  # 在原来第一道关闭的基础上加60秒，进行二道关闭
 
-        self.audio_format = "opus"
+        # {"mcp":true} 表示启用MCP功能
+        self.features = None
 
     async def handle_connection(self, ws):
         try:
@@ -247,7 +256,11 @@ class ConnectionHandler:
         if isinstance(message, str):
             await handleTextMessage(self, message)
         elif isinstance(message, bytes):
-            await handleAudioMessage(self, message)
+            if self.vad is None:
+                return
+            if self.asr is None:
+                return
+            self.asr_audio_queue.put(message)
 
     async def handle_restart(self, message):
         """处理服务器重启请求"""
@@ -315,10 +328,14 @@ class ConnectionHandler:
             if self.vad is None:
                 self.vad = self._vad
             if self.asr is None:
-                self.asr = self._asr
+                self.asr = self._initialize_asr()
+            # 打开语音识别通道
+            asyncio.run_coroutine_threadsafe(
+                self.asr.open_audio_channels(self), self.loop
+            )
             if self.tts is None:
                 self.tts = self._initialize_tts()
-            # 使用事件循环运行异步方法
+            # 打开语音合成通道
             asyncio.run_coroutine_threadsafe(
                 self.tts.open_audio_channels(self), self.loop
             )
@@ -356,6 +373,19 @@ class ConnectionHandler:
 
         return tts
 
+    def _initialize_asr(self):
+        """初始化ASR"""
+        if self._asr.interface_type == InterfaceType.LOCAL:
+            # 如果公共ASR是本地服务，则直接返回
+            # 因为本地一个实例ASR，可以被多个连接共享
+            asr = self._asr
+        else:
+            # 如果公共ASR是远程服务，则初始化一个新实例
+            # 因为远程ASR，涉及到websocket连接和接收线程，需要每个连接一个实例
+            asr = initialize_asr(self.config)
+
+        return asr
+
     def _initialize_private_config(self):
         """如果是从配置文件获取，则进行二次实例化"""
         if not self.read_config_from_api:
@@ -454,6 +484,8 @@ class ConnectionHandler:
             self.memory = modules["memory"]
 
     def _initialize_memory(self):
+        if self.memory is None:
+            return
         """初始化记忆模块"""
         self.memory.init_memory(
             role_id=self.device_id,
@@ -494,6 +526,8 @@ class ConnectionHandler:
                 self.logger.bind(tag=TAG).info("使用主LLM作为意图识别模型")
 
     def _initialize_intent(self):
+        if self.intent is None:
+            return
         self.intent_type = self.config["Intent"][
             self.config["selected_module"]["Intent"]
         ]["type"]
@@ -558,6 +592,12 @@ class ConnectionHandler:
         functions = None
         if self.intent_type == "function_call" and hasattr(self, "func_handler"):
             functions = self.func_handler.get_functions()
+        if hasattr(self, "mcp_client"):
+            mcp_tools = self.mcp_client.get_available_tools()
+            if mcp_tools is not None and len(mcp_tools) > 0:
+                if functions is None:
+                    functions = []
+                functions.extend(mcp_tools)
         response_message = []
 
         try:
@@ -595,8 +635,10 @@ class ConnectionHandler:
         function_arguments = ""
         content_arguments = ""
         text_index = 0
-
+        self.client_abort = False
         for response in llm_responses:
+            if self.client_abort:
+                break
             if functions is not None:
                 content, tools_call = response
                 if "content" in response:
@@ -622,9 +664,6 @@ class ConnectionHandler:
             if content is not None and len(content) > 0:
                 if not tool_call_flag:
                     response_message.append(content)
-                    if self.client_abort:
-                        break
-
                     if text_index == 0:
                         self.tts.tts_text_queue.put(
                             TTSMessageDTO(
@@ -676,9 +715,32 @@ class ConnectionHandler:
                     "arguments": function_arguments,
                 }
 
-                # 处理MCP工具调用
+                # 处理Server端MCP工具调用
                 if self.mcp_manager.is_mcp_tool(function_name):
                     result = self._handle_mcp_tool_call(function_call_data)
+                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(
+                    function_name
+                ):
+                    # 如果是小智端MCP工具调用
+                    self.logger.bind(tag=TAG).debug(
+                        f"调用小智端MCP工具: {function_name}, 参数: {function_arguments}"
+                    )
+                    try:
+                        result = asyncio.run_coroutine_threadsafe(
+                            call_mcp_tool(
+                                self, self.mcp_client, function_name, function_arguments
+                            ),
+                            self.loop,
+                        ).result()
+                        self.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result=result, response=""
+                        )
+                    except Exception as e:
+                        self.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result="MCP工具调用失败", response=""
+                        )
                 else:
                     # 处理系统函数
                     result = self.func_handler.handle_llm_function_call(
@@ -801,9 +863,7 @@ class ConnectionHandler:
                 item = self.report_queue.get(timeout=1)
                 if item is None:  # 检测毒丸对象
                     break
-
                 type, text, audio_data, report_time = item
-
                 try:
                     # 检查线程池状态
                     if self.executor is None:
@@ -833,8 +893,8 @@ class ConnectionHandler:
             self.report_queue.task_done()
 
     def clearSpeakStatus(self):
+        self.client_is_speaking = False
         self.logger.bind(tag=TAG).debug(f"清除服务端讲话状态")
-        self.asr_server_receive = True
 
     async def close(self, ws=None):
         """资源清理方法"""

main/xiaozhi-server/core/handle/abortHandle.py

@@ -1,6 +1,4 @@
 import json
-import queue
-from config.logger import setup_logging
 
 TAG = __name__
 

main/xiaozhi-server/core/handle/functionHandler.py

@@ -61,7 +61,7 @@ class FunctionHandler:
         self.function_registry.register_function("plugin_loader")
         self.function_registry.register_function("get_time")
         self.function_registry.register_function("get_lunar")
-        self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
+        # self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
 
     def register_config_functions(self):
         """注册配置中的函数,可以不同客户端使用不同的配置"""

main/xiaozhi-server/core/handle/helloHandle.py

@@ -7,6 +7,11 @@ import asyncio
 from core.handle.sendAudioHandle import send_stt_message
 from core.utils.util import remove_punctuation_and_length
 from core.providers.tts.dto.dto import ContentType, InterfaceType
+from core.handle.mcpHandle import (
+    MCPClient,
+    send_mcp_initialize_message,
+    send_mcp_tools_list_request,
+)
 
 
 TAG = __name__
@@ -28,9 +33,18 @@ async def handleHelloMessage(conn, msg_json):
         format = audio_params.get("format")
         conn.logger.bind(tag=TAG).info(f"客户端音频格式: {format}")
         conn.audio_format = format
-        if conn.asr is not None:
-            conn.asr.set_audio_format(format)
         conn.welcome_msg["audio_params"] = audio_params
+    features = msg_json.get("features")
+    if features:
+        conn.logger.bind(tag=TAG).info(f"客户端特性: {features}")
+        conn.features = features
+        if features.get("mcp"):
+            conn.logger.bind(tag=TAG).info("客户端支持MCP")
+            conn.mcp_client = MCPClient()
+            # 发送初始化
+            asyncio.create_task(send_mcp_initialize_message(conn))
+            # 发送mcp消息，获取tools列表
+            asyncio.create_task(send_mcp_tools_list_request(conn))
 
     await conn.websocket.send(json.dumps(conn.welcome_msg))
 
@@ -49,6 +63,8 @@ async def checkWakeupWords(conn, text):
     """检查是否是唤醒词"""
     _, filtered_text = remove_punctuation_and_length(text)
     if filtered_text in conn.config.get("wakeup_words"):
+        # 设置刚刚被唤醒的标志
+        conn.just_woken_up = True
         await send_stt_message(conn, text)
 
         file = getWakeupWordFile(WAKEUP_CONFIG["file_name"])
@@ -58,6 +74,8 @@ async def checkWakeupWords(conn, text):
         text_hello = WAKEUP_CONFIG["text"]
         if not text_hello:
             text_hello = text
+        if conn.tts is None:
+            return False
         conn.tts.tts_one_sentence(
             conn, ContentType.FILE, content_file=file, content_detail=text_hello
         )
@@ -92,7 +110,13 @@ async def wakeupWordsResponse(conn):
 
     """唤醒词响应"""
     wakeup_word = random.choice(WAKEUP_CONFIG["words"])
-    result = conn.llm.response_no_stream(conn.config["prompt"], wakeup_word)
+    question = (
+        "此刻用户正在和你说```"
+        + wakeup_word
+        + "```。\n请你根据以上用户的内容，进行简短回复，文字内容控制在15个字以内。\n"
+        + "请勿对这条内容本身进行任何解释和回应，仅返回对用户的内容的回复。"
+    )
+    result = conn.llm.response_no_stream(conn.config["prompt"], question)
     if result is None or result == "":
         return
     tts_file = await asyncio.to_thread(conn.tts.to_tts, result)

main/xiaozhi-server/core/handle/intentHandler.py

@@ -1,4 +1,3 @@
-from config.logger import setup_logging
 import json
 import uuid
 from core.handle.sendAudioHandle import send_stt_message
@@ -97,6 +96,7 @@ async def process_intent_result(conn, intent_result, original_text):
             }
 
             await send_stt_message(conn, original_text)
+            conn.client_abort = False
 
             # 使用executor执行函数调用和结果处理
             def process_function_call():
@@ -146,4 +146,3 @@ async def process_intent_result(conn, intent_result, original_text):
 def speak_txt(conn, text):
     conn.tts.tts_one_sentence(conn, ContentType.TEXT, content_detail=text)
     conn.dialogue.put(Message(role="assistant", content=text))
- 

main/xiaozhi-server/core/handle/mcpHandle.py

@@ -0,0 +1,374 @@
+import json
+import asyncio
+from concurrent.futures import Future
+from core.utils.util import get_vision_url
+from core.utils.auth import AuthToken
+
+TAG = __name__
+
+
+class MCPClient:
+    """MCPClient，用于管理MCP状态和工具"""
+
+    def __init__(self):
+        self.tools = {}  # Dictionary for O(1) lookup
+        self.ready = False
+        self.call_results = {}  # To store Futures for tool call responses
+        self.next_id = 1
+        self.lock = asyncio.Lock()
+        self._cached_available_tools = None  # Cache for get_available_tools
+
+    def has_tool(self, name: str) -> bool:
+        return name in self.tools
+
+    def get_available_tools(self) -> list:
+        # Check if the cache is valid
+        if self._cached_available_tools is not None:
+            return self._cached_available_tools
+
+        # If cache is not valid, regenerate the list
+        result = []
+        for tool_name, tool_data in self.tools.items():
+            function_def = {
+                "name": tool_data["name"],
+                "description": tool_data["description"],
+                "parameters": {
+                    "type": tool_data["inputSchema"].get("type", "object"),
+                    "properties": tool_data["inputSchema"].get("properties", {}),
+                    "required": tool_data["inputSchema"].get("required", []),
+                },
+            }
+            result.append({"type": "function", "function": function_def})
+
+        self._cached_available_tools = result  # Store the generated list in cache
+        return result
+
+    async def is_ready(self) -> bool:
+        async with self.lock:
+            return self.ready
+
+    async def set_ready(self, status: bool):
+        async with self.lock:
+            self.ready = status
+
+    async def add_tool(self, tool_data: dict):
+        async with self.lock:
+            self.tools[tool_data["name"]] = tool_data
+            self._cached_available_tools = (
+                None  # Invalidate the cache when a tool is added
+            )
+
+    async def get_next_id(self) -> int:
+        async with self.lock:
+            current_id = self.next_id
+            self.next_id += 1
+            return current_id
+
+    async def register_call_result_future(self, id: int, future: Future):
+        async with self.lock:
+            self.call_results[id] = future
+
+    async def resolve_call_result(self, id: int, result: any):
+        async with self.lock:
+            if id in self.call_results:
+                future = self.call_results.pop(id)
+                if not future.done():
+                    future.set_result(result)
+
+    async def reject_call_result(self, id: int, exception: Exception):
+        async with self.lock:
+            if id in self.call_results:
+                future = self.call_results.pop(id)
+                if not future.done():
+                    future.set_exception(exception)
+
+    async def cleanup_call_result(self, id: int):
+        async with self.lock:
+            if id in self.call_results:
+                self.call_results.pop(id)
+
+
+async def send_mcp_message(conn, payload: dict):
+    """Helper to send MCP messages, encapsulating common logic."""
+    if not conn.features.get("mcp"):
+        conn.logger.bind(tag=TAG).warning("客户端不支持MCP，无法发送MCP消息")
+        return
+
+    message = json.dumps({"type": "mcp", "payload": payload})
+
+    try:
+        await conn.websocket.send(message)
+        conn.logger.bind(tag=TAG).info(f"成功发送MCP消息: {message}")
+    except Exception as e:
+        conn.logger.bind(tag=TAG).error(f"发送MCP消息失败: {e}")
+
+
+async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
+    """处理MCP消息,包括初始化、工具列表和工具调用响应等"""
+    conn.logger.bind(tag=TAG).info(f"处理MCP消息: {payload}")
+
+    if not isinstance(payload, dict):
+        conn.logger.bind(tag=TAG).error("MCP消息缺少payload字段或格式错误")
+        return
+
+    # Handle result
+    if "result" in payload:
+        result = payload["result"]
+        msg_id = int(payload.get("id", 0))
+
+        # Check for tool call response first
+        if msg_id in mcp_client.call_results:
+            conn.logger.bind(tag=TAG).debug(
+                f"收到工具调用响应，ID: {msg_id}, 结果: {result}"
+            )
+            await mcp_client.resolve_call_result(msg_id, result)
+            return
+
+        if msg_id == 1:  # mcpInitializeID
+            conn.logger.bind(tag=TAG).debug("收到MCP初始化响应")
+            server_info = result.get("serverInfo")
+            if isinstance(server_info, dict):
+                name = server_info.get("name")
+                version = server_info.get("version")
+                conn.logger.bind(tag=TAG).info(
+                    f"客户端MCP服务器信息: name={name}, version={version}"
+                )
+            await send_mcp_tools_list_request(
+                conn
+            )  # After initialization, request tool list
+            return
+
+        elif msg_id == 2:  # mcpToolsListID
+            conn.logger.bind(tag=TAG).debug("收到MCP工具列表响应")
+            if isinstance(result, dict) and "tools" in result:
+                tools_data = result["tools"]
+                if not isinstance(tools_data, list):
+                    conn.logger.bind(tag=TAG).error("工具列表格式错误")
+                    return
+
+                conn.logger.bind(tag=TAG).info(
+                    f"客户端设备支持的工具数量: {len(tools_data)}"
+                )
+
+                for i, tool in enumerate(tools_data):
+                    if not isinstance(tool, dict):
+                        continue
+
+                    name = tool.get("name", "")
+                    description = tool.get("description", "")
+                    input_schema = {"type": "object", "properties": {}, "required": []}
+
+                    if "inputSchema" in tool and isinstance(tool["inputSchema"], dict):
+                        schema = tool["inputSchema"]
+                        input_schema["type"] = schema.get("type", "object")
+                        input_schema["properties"] = schema.get("properties", {})
+                        input_schema["required"] = [
+                            s for s in schema.get("required", []) if isinstance(s, str)
+                        ]
+
+                    new_tool = {
+                        "name": name,
+                        "description": description,
+                        "inputSchema": input_schema,
+                    }
+                    await mcp_client.add_tool(new_tool)
+                    conn.logger.bind(tag=TAG).debug(f"客户端工具 #{i+1}: {name}")
+
+                next_cursor = result.get("nextCursor", "")
+                if next_cursor:
+                    conn.logger.bind(tag=TAG).info(
+                        f"有更多工具，nextCursor: {next_cursor}"
+                    )
+                    await send_mcp_tools_list_continue_request(conn, next_cursor)
+                else:
+                    await mcp_client.set_ready(True)
+                    conn.logger.bind(tag=TAG).info("所有工具已获取，MCP客户端准备就绪")
+            return
+
+    # Handle method calls (requests from the client)
+    elif "method" in payload:
+        method = payload["method"]
+        conn.logger.bind(tag=TAG).info(f"收到MCP客户端请求: {method}")
+
+    elif "error" in payload:
+        error_data = payload["error"]
+        error_msg = error_data.get("message", "未知错误")
+        conn.logger.bind(tag=TAG).error(f"收到MCP错误响应: {error_msg}")
+
+        msg_id = int(payload.get("id", 0))
+        if msg_id in mcp_client.call_results:
+            await mcp_client.reject_call_result(
+                msg_id, Exception(f"MCP错误: {error_msg}")
+            )
+
+
+# --- Outgoing MCP Messages ---
+
+
+async def send_mcp_initialize_message(conn):
+    """发送MCP初始化消息"""
+
+    vision_url = get_vision_url(conn.config)
+
+    # 密钥生成token
+    auth = AuthToken(conn.config["server"]["auth_key"])
+    token = auth.generate_token(conn.headers.get("device-id"))
+
+    vision = {
+        "url": vision_url,
+        "token": token,
+    }
+
+    conn.logger.bind(tag=TAG).info(f"视觉服务信息: {vision}")
+
+    payload = {
+        "jsonrpc": "2.0",
+        "id": 1,  # mcpInitializeID
+        "method": "initialize",
+        "params": {
+            "protocolVersion": "2024-11-05",
+            "capabilities": {
+                "roots": {"listChanged": True},
+                "sampling": {},
+                "vision": vision,
+            },
+            "clientInfo": {
+                "name": "XiaozhiClient",
+                "version": "1.0.0",
+            },
+        },
+    }
+    conn.logger.bind(tag=TAG).info("发送MCP初始化消息")
+    await send_mcp_message(conn, payload)
+
+
+async def send_mcp_tools_list_request(conn):
+    """发送MCP工具列表请求"""
+    payload = {
+        "jsonrpc": "2.0",
+        "id": 2,  # mcpToolsListID
+        "method": "tools/list",
+    }
+    conn.logger.bind(tag=TAG).debug("发送MCP工具列表请求")
+    await send_mcp_message(conn, payload)
+
+
+async def send_mcp_tools_list_continue_request(conn, cursor: str):
+    """发送带有cursor的MCP工具列表请求"""
+    payload = {
+        "jsonrpc": "2.0",
+        "id": 2,  # mcpToolsListID (same ID for continuation)
+        "method": "tools/list",
+        "params": {"cursor": cursor},
+    }
+    conn.logger.bind(tag=TAG).info(f"发送带cursor的MCP工具列表请求: {cursor}")
+    await send_mcp_message(conn, payload)
+
+
+async def call_mcp_tool(
+    conn, mcp_client: MCPClient, tool_name: str, args: str = "{}", timeout: int = 30
+):
+    """
+    调用指定的工具，并等待响应
+    """
+    if not await mcp_client.is_ready():
+        raise RuntimeError("MCP客户端尚未准备就绪")
+
+    if not mcp_client.has_tool(tool_name):
+        raise ValueError(f"工具 {tool_name} 不存在")
+
+    tool_call_id = await mcp_client.get_next_id()
+    result_future = asyncio.Future()
+    await mcp_client.register_call_result_future(tool_call_id, result_future)
+
+    # 处理参数
+    try:
+        if isinstance(args, str):
+            # 确保字符串是有效的JSON
+            if not args.strip():
+                arguments = {}
+            else:
+                try:
+                    # 尝试直接解析
+                    arguments = json.loads(args)
+                except json.JSONDecodeError:
+                    # 如果解析失败，尝试合并多个JSON对象
+                    try:
+                        # 使用正则表达式匹配所有JSON对象
+                        import re
+
+                        json_objects = re.findall(r"\{[^{}]*\}", args)
+                        if len(json_objects) > 1:
+                            # 合并所有JSON对象
+                            merged_dict = {}
+                            for json_str in json_objects:
+                                try:
+                                    obj = json.loads(json_str)
+                                    if isinstance(obj, dict):
+                                        merged_dict.update(obj)
+                                except json.JSONDecodeError:
+                                    continue
+                            if merged_dict:
+                                arguments = merged_dict
+                            else:
+                                raise ValueError(f"无法解析任何有效的JSON对象: {args}")
+                        else:
+                            raise ValueError(f"参数JSON解析失败: {args}")
+                    except Exception as e:
+                        conn.logger.bind(tag=TAG).error(
+                            f"参数JSON解析失败: {str(e)}, 原始参数: {args}"
+                        )
+                        raise ValueError(f"参数JSON解析失败: {str(e)}")
+        elif isinstance(args, dict):
+            arguments = args
+        else:
+            raise ValueError(f"参数类型错误，期望字符串或字典，实际类型: {type(args)}")
+
+        # 确保参数是字典类型
+        if not isinstance(arguments, dict):
+            raise ValueError(f"参数必须是字典类型，实际类型: {type(arguments)}")
+
+    except Exception as e:
+        if not isinstance(e, ValueError):
+            raise ValueError(f"参数处理失败: {str(e)}")
+        raise e
+
+    payload = {
+        "jsonrpc": "2.0",
+        "id": tool_call_id,
+        "method": "tools/call",
+        "params": {"name": tool_name, "arguments": arguments},
+    }
+
+    conn.logger.bind(tag=TAG).info(
+        f"发送客户端mcp工具调用请求: {tool_name}，参数: {args}"
+    )
+    await send_mcp_message(conn, payload)
+
+    try:
+        # Wait for response or timeout
+        raw_result = await asyncio.wait_for(result_future, timeout=timeout)
+        conn.logger.bind(tag=TAG).info(
+            f"客户端mcp工具调用 {tool_name} 成功，原始结果: {raw_result}"
+        )
+
+        if isinstance(raw_result, dict):
+            if raw_result.get("isError") is True:
+                error_msg = raw_result.get(
+                    "error", "工具调用返回错误，但未提供具体错误信息"
+                )
+                raise RuntimeError(f"工具调用错误: {error_msg}")
+
+            content = raw_result.get("content")
+            if isinstance(content, list) and len(content) > 0:
+                if isinstance(content[0], dict) and "text" in content[0]:
+                    # 直接返回文本内容，不进行JSON解析
+                    return content[0]["text"]
+        # 如果结果不是预期的格式，将其转换为字符串
+        return str(raw_result)
+    except asyncio.TimeoutError:
+        await mcp_client.cleanup_call_result(tool_call_id)
+        raise TimeoutError("工具调用请求超时")
+    except Exception as e:
+        await mcp_client.cleanup_call_result(tool_call_id)
+        raise e

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -1,10 +1,9 @@
-import time
-import copy
-from core.utils.util import remove_punctuation_and_length
 from core.handle.sendAudioHandle import send_stt_message
 from core.handle.intentHandler import handle_user_intent
 from core.utils.output_counter import check_device_output_limit
-from core.handle.reportHandle import enqueue_asr_report
+from core.handle.abortHandle import handleAbortMessage
+import time
+import asyncio
 from core.handle.sendAudioHandle import SentenceType
 from core.utils.util import audio_to_data
 
@@ -12,47 +11,28 @@ TAG = __name__
 
 
 async def handleAudioMessage(conn, audio):
-    if conn.vad is None:
+    # 当前片段是否有人说话
-        return
+    have_voice = conn.vad.is_vad(conn, audio)
-    if not conn.asr_server_receive:
-        conn.logger.bind(tag=TAG).debug(f"前期数据处理中，暂停接收")
-        return
-    if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
-        have_voice = conn.vad.is_vad(conn, audio)
-    else:
-        have_voice = conn.client_have_voice
 
-    # 如果本次没有声音，本段也没声音，就把声音丢弃了
+    # 如果设备刚刚被唤醒，短暂忽略VAD检测
-    if have_voice == False and conn.client_have_voice == False:
+    if hasattr(conn, "just_woken_up") and conn.just_woken_up:
-        await no_voice_close_connect(conn)
+        have_voice = False
-        conn.asr_audio.append(audio)
+        # 设置一个短暂延迟后恢复VAD检测
-        conn.asr_audio = conn.asr_audio[
+        asyncio.create_task(resume_vad_detection(conn))
-            -10:
+
-        ]  # 保留最新的10帧音频内容，解决ASR句首丢字问题
+    if have_voice:
-        return
+        if conn.client_is_speaking:
-    conn.client_no_voice_last_time = 0.0
+            await handleAbortMessage(conn)
-    conn.asr_audio.append(audio)
+    # 设备长时间空闲检测，用于say goodbye
-    # 如果本段有声音，且已经停止了
+    await no_voice_close_connect(conn, have_voice)
-    if conn.client_voice_stop:
+    # 接收音频
-        conn.client_abort = False
+    await conn.asr.receive_audio(conn, audio, have_voice)
-        conn.asr_server_receive = False
+
-        # 音频太短了，无法识别
+
-        if len(conn.asr_audio) < 15:
+async def resume_vad_detection(conn):
-            conn.asr_server_receive = True
+    # 等待2秒后恢复VAD检测
-        else:
+    await asyncio.sleep(2)
-            raw_text, _ = await conn.asr.speech_to_text(
+    conn.just_woken_up = False
-                conn.asr_audio, conn.session_id
-            )  # 确保ASR模块返回原始文本
-            conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
-            text_len, _ = remove_punctuation_and_length(raw_text)
-            if text_len > 0:
-                # 使用自定义模块进行上报
-                await startToChat(conn, raw_text)
-                enqueue_asr_report(conn, raw_text, copy.deepcopy(conn.asr_audio))
-            else:
-                conn.asr_server_receive = True
-        conn.asr_audio.clear()
-        conn.reset_vad_states()
 
 
 async def startToChat(conn, text):
@@ -67,13 +47,14 @@ async def startToChat(conn, text):
         ):
             await max_out_size(conn)
             return
+    if conn.client_is_speaking:
+        await handleAbortMessage(conn)
 
     # 首先进行意图分析
     intent_handled = await handle_user_intent(conn, text)
 
     if intent_handled:
         # 如果意图已被处理，不再进行聊天
-        conn.asr_server_receive = True
         return
 
     # 意图未被处理，继续常规聊天流程
@@ -81,7 +62,10 @@ async def startToChat(conn, text):
     conn.executor.submit(conn.chat, text)
 
 
-async def no_voice_close_connect(conn):
+async def no_voice_close_connect(conn, have_voice):
+    if have_voice:
+        conn.client_no_voice_last_time = 0.0
+        return
     if conn.client_no_voice_last_time == 0.0:
         conn.client_no_voice_last_time = time.time() * 1000
     else:
@@ -95,7 +79,6 @@ async def no_voice_close_connect(conn):
         ):
             conn.close_after_chat = True
             conn.client_abort = False
-            conn.asr_server_receive = False
             end_prompt = conn.config.get("end_prompt", {})
             if end_prompt and end_prompt.get("enable", True) is False:
                 conn.logger.bind(tag=TAG).info("结束对话，无需发送结束提示语")
@@ -103,7 +86,7 @@ async def no_voice_close_connect(conn):
                 return
             prompt = end_prompt.get("prompt")
             if not prompt:
-                prompt = "请你以“时间过得真快”未来头，用富有感情、依依不舍的话来结束这场对话吧。！"
+                prompt = "请你以```时间过得真快```未来头，用富有感情、依依不舍的话来结束这场对话吧。！"
             await startToChat(conn, prompt)
 
 

main/xiaozhi-server/core/handle/reportHandle.py

@@ -8,6 +8,7 @@ TTS上报功能已集成到ConnectionHandler类中。
 
 具体实现请参考core/connection.py中的相关代码。
 """
+
 import time
 
 import opuslib_next
@@ -145,4 +146,4 @@ def enqueue_asr_report(conn, text, opus_data):
                 f"ASR数据已加入上报队列: {conn.device_id}, 不上报音频"
             )
     except Exception as e:
-        conn.logger.bind(tag=TAG).error(f"加入ASR上报队列失败: {text}, {e}")
+        conn.logger.bind(tag=TAG).debug(f"加入ASR上报队列失败: {text}, {e}")

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -62,6 +62,7 @@ async def sendAudioMessage(conn, sentenceType, audios, text):
     # 发送结束消息（如果是最后一个文本）
     if conn.llm_finish_task and sentenceType == SentenceType.LAST:
         await send_tts_message(conn, "stop", None)
+        conn.client_is_speaking = False
         if conn.close_after_chat:
             await conn.close()
 
@@ -88,7 +89,7 @@ async def sendAudio(conn, audios, pre_buffer=True):
     # 播放剩余音频帧
     for opus_packet in remaining_audios:
         if conn.client_abort:
-            return
+            break
 
         # 每分钟重置一次计时器
         if time.perf_counter() - last_reset_time > 60:
@@ -131,9 +132,15 @@ async def send_tts_message(conn, state, text=None):
 
 
 async def send_stt_message(conn, text):
+    end_prompt_str = conn.config.get("end_prompt", {}).get("prompt")
+    if end_prompt_str and end_prompt_str == text:
+        await send_tts_message(conn, "start")
+        return
+
     """发送 STT 状态消息"""
     stt_text = get_string_no_punctuation_or_emoji(text)
     await conn.websocket.send(
         json.dumps({"type": "stt", "text": stt_text, "session_id": conn.session_id})
     )
+    conn.client_is_speaking = True
     await send_tts_message(conn, "start")

main/xiaozhi-server/core/handle/textHandle.py

@@ -1,6 +1,7 @@
 import json
 from core.handle.abortHandle import handleAbortMessage
 from core.handle.helloHandle import handleHelloMessage
+from core.handle.mcpHandle import handle_mcp_message
 from core.utils.util import remove_punctuation_and_length, filter_sensitive_info
 from core.handle.receiveAudioHandle import startToChat, handleAudioMessage
 from core.handle.sendAudioHandle import send_stt_message, send_tts_message
@@ -41,12 +42,13 @@ async def handleTextMessage(conn, message):
                 if len(conn.asr_audio) > 0:
                     await handleAudioMessage(conn, b"")
             elif msg_json["state"] == "detect":
-                conn.asr_server_receive = False
                 conn.client_have_voice = False
                 conn.asr_audio.clear()
                 if "text" in msg_json:
                     original_text = msg_json["text"]  # 保留原始文本
-                    filtered_len, filtered_text = remove_punctuation_and_length(original_text)
+                    filtered_len, filtered_text = remove_punctuation_and_length(
+                        original_text
+                    )
 
                     # 识别是否是唤醒词
                     is_wakeup_words = filtered_text in conn.config.get("wakeup_words")
@@ -57,7 +59,9 @@ async def handleTextMessage(conn, message):
                         # 如果是唤醒词，且关闭了唤醒词回复，就不用回答
                         await send_stt_message(conn, original_text)
                         await send_tts_message(conn, "stop", None)
+                        conn.client_is_speaking = False
                     elif is_wakeup_words:
+                        conn.just_woken_up = True
                         # 上报纯文字数据（复用ASR上报功能，但不提供音频数据）
                         enqueue_asr_report(conn, "嘿，你好呀", [])
                         await startToChat(conn, "嘿，你好呀")
@@ -72,6 +76,12 @@ async def handleTextMessage(conn, message):
                 asyncio.create_task(handleIotDescriptors(conn, msg_json["descriptors"]))
             if "states" in msg_json:
                 asyncio.create_task(handleIotStatus(conn, msg_json["states"]))
+        elif msg_json["type"] == "mcp":
+            conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message}")
+            if "payload" in msg_json:
+                asyncio.create_task(
+                    handle_mcp_message(conn, conn.mcp_client, msg_json["payload"])
+                )
         elif msg_json["type"] == "server":
             # 记录日志时过滤敏感信息
             conn.logger.bind(tag=TAG).info(
@@ -151,5 +161,7 @@ async def handleTextMessage(conn, message):
             # 重启服务器
             elif msg_json["action"] == "restart":
                 await conn.handle_restart(msg_json)
+        else:
+            conn.logger.bind(tag=TAG).error(f"收到未知类型消息：{message}")
     except json.JSONDecodeError:
         await conn.websocket.send(message)

main/xiaozhi-server/core/http_server.py

@@ -0,0 +1,71 @@
+import asyncio
+from aiohttp import web
+from config.logger import setup_logging
+from core.api.ota_handler import OTAHandler
+from core.api.vision_handler import VisionHandler
+
+TAG = __name__
+
+
+class SimpleHttpServer:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        self.ota_handler = OTAHandler(config)
+        self.vision_handler = VisionHandler(config)
+
+    def _get_websocket_url(self, local_ip: str, port: int) -> str:
+        """获取websocket地址
+
+        Args:
+            local_ip: 本地IP地址
+            port: 端口号
+
+        Returns:
+            str: websocket地址
+        """
+        server_config = self.config["server"]
+        websocket_config = server_config.get("websocket")
+
+        if websocket_config and "你" not in websocket_config:
+            return websocket_config
+        else:
+            return f"ws://{local_ip}:{port}/xiaozhi/v1/"
+
+    async def start(self):
+        server_config = self.config["server"]
+        host = server_config.get("ip", "0.0.0.0")
+        port = int(server_config.get("http_port", 8003))
+
+        if port:
+            app = web.Application()
+
+            read_config_from_api = server_config.get("read_config_from_api", False)
+
+            if not read_config_from_api:
+                # 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
+                app.add_routes(
+                    [
+                        web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
+                        web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
+                        web.options("/xiaozhi/ota/", self.ota_handler.handle_post),
+                    ]
+                )
+            # 添加路由
+            app.add_routes(
+                [
+                    web.get("/mcp/vision/explain", self.vision_handler.handle_get),
+                    web.post("/mcp/vision/explain", self.vision_handler.handle_post),
+                    web.options("/mcp/vision/explain", self.vision_handler.handle_post),
+                ]
+            )
+
+            # 运行服务
+            runner = web.AppRunner(app)
+            await runner.setup()
+            site = web.TCPSite(runner, host, port)
+            await site.start()
+
+            # 保持服务运行
+            while True:
+                await asyncio.sleep(3600)  # 每隔 1 小时检查一次

main/xiaozhi-server/core/mcp/MCPClient.py

@@ -92,11 +92,21 @@ class MCPClient:
                         args=self.config.get("args", []),
                         env=env,
                     )
-                    stdio_r, stdio_w = await stack.enter_async_context(stdio_client(params))
+                    stdio_r, stdio_w = await stack.enter_async_context(
+                        stdio_client(params)
+                    )
                     read_stream, write_stream = stdio_r, stdio_w
                 # 建立SSEClient
                 elif "url" in self.config:
-                    sse_r, sse_w = await stack.enter_async_context(sse_client(self.config["url"]))
+                    if "API_ACCESS_TOKEN" in self.config:
+                        headers = {
+                            "Authorization": f"Bearer {self.config['API_ACCESS_TOKEN']}"
+                        }
+                    else:
+                        headers = {}
+                    sse_r, sse_w = await stack.enter_async_context(
+                        sse_client(self.config["url"], headers=headers)
+                    )
                     read_stream, write_stream = sse_r, sse_w
 
                 else:

main/xiaozhi-server/core/providers/asr/aliyun.py

@@ -2,9 +2,6 @@ import http.client
 import json
 import asyncio
 from typing import Optional, Tuple, List
-import opuslib_next
-import wave
-import io
 import os
 import uuid
 import hmac
@@ -16,6 +13,7 @@ import time
 from datetime import datetime
 from config.logger import setup_logging
 from core.providers.asr.base import ASRProviderBase
+from core.providers.asr.dto.dto import InterfaceType
 
 TAG = __name__
 logger = setup_logging()
@@ -92,6 +90,7 @@ class AccessToken:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         """阿里云ASR初始化"""
         # 新增空值判断逻辑
         self.access_key_id = config.get("access_key_id")
@@ -165,21 +164,6 @@ class ASRProvider(ASRProviderBase):
         request += "&enable_voice_detection=false"
         return request
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)  # 单声道
-            wf.setsampwidth(2)  # 16-bit
-            wf.setframerate(self.sample_rate)
-            wf.writeframes(b"".join(pcm_data))
-
-        logger.bind(tag=TAG).debug(f"音频文件已保存至: {file_path}")
-        return file_path
-
     async def _send_request(self, pcm_data: bytes) -> Optional[str]:
         """发送请求到阿里云ASR服务"""
         try:
@@ -229,7 +213,7 @@ class ASRProvider(ASRProviderBase):
             return None
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if self._is_token_expired():
@@ -239,7 +223,7 @@ class ASRProvider(ASRProviderBase):
         file_path = None
         try:
             # 解码Opus为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/baidu.py

@@ -1,18 +1,10 @@
-import base64
-import hashlib
-import hmac
-import json
 import time
-from datetime import datetime, timezone
 import os
-import uuid
 from typing import Optional, Tuple, List
-import wave
-import opuslib_next
-
 from aip import AipSpeech
 from core.providers.asr.base import ASRProviderBase
 from config.logger import setup_logging
+from core.providers.asr.dto.dto import InterfaceType
 
 TAG = __name__
 logger = setup_logging()
@@ -21,6 +13,7 @@ logger = setup_logging()
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool = True):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.app_id = config.get("app_id")
         self.api_key = config.get("api_key")
         self.secret_key = config.get("secret_key")
@@ -36,22 +29,8 @@ class ASRProvider(ASRProviderBase):
         # 确保输出目录存在
         os.makedirs(self.output_dir, exist_ok=True)
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if not opus_data:
@@ -66,7 +45,7 @@ class ASRProvider(ASRProviderBase):
                 return None, file_path
 
             # 将Opus音频数据解码为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/base.py

@@ -1,7 +1,19 @@
-from abc import ABC, abstractmethod
+import os
-from typing import Optional, Tuple, List
+import wave
+import copy
+import uuid
+import queue
+import asyncio
+import traceback
+import threading
 import opuslib_next
+from abc import ABC, abstractmethod
 from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.handle.receiveAudioHandle import startToChat
+from core.handle.reportHandle import enqueue_asr_report
+from core.utils.util import remove_punctuation_and_length
+from core.handle.receiveAudioHandle import handleAudioMessage
 
 TAG = __name__
 logger = setup_logging()
@@ -9,24 +21,97 @@ logger = setup_logging()
 
 class ASRProviderBase(ABC):
     def __init__(self):
-        self.audio_format = "opus"
+        pass
+
+    # 打开音频通道
+    # 这里默认是非流式的处理方式
+    # 流式处理方式请在子类中重写
+    async def open_audio_channels(self, conn):
+        # tts 消化线程
+        conn.asr_priority_thread = threading.Thread(
+            target=self.asr_text_priority_thread, args=(conn,), daemon=True
+        )
+        conn.asr_priority_thread.start()
+
+    # 有序处理ASR音频
+    def asr_text_priority_thread(self, conn):
+        while not conn.stop_event.is_set():
+            try:
+                message = conn.asr_audio_queue.get(timeout=1)
+                future = asyncio.run_coroutine_threadsafe(
+                    handleAudioMessage(conn, message),
+                    conn.loop,
+                )
+                future.result()
+            except queue.Empty:
+                continue
+            except Exception as e:
+                logger.bind(tag=TAG).error(
+                    f"处理ASR文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
+                )
+                continue
+
+    # 接收音频
+    # 这里默认是非流式的处理方式
+    # 流式处理方式请在子类中重写
+    async def receive_audio(self, conn, audio, audio_have_voice):
+        if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
+            have_voice = audio_have_voice
+        else:
+            have_voice = conn.client_have_voice
+        # 如果本次没有声音，本段也没声音，就把声音丢弃了
+        conn.asr_audio.append(audio)
+        if have_voice == False and conn.client_have_voice == False:
+            conn.asr_audio = conn.asr_audio[-10:]
+            return
+
+        # 如果本段有声音，且已经停止了
+        if conn.client_voice_stop:
+            asr_audio_task = copy.deepcopy(conn.asr_audio)
+            conn.asr_audio.clear()
+
+            # 音频太短了，无法识别
+            conn.reset_vad_states()
+            if len(asr_audio_task) > 15:
+                await self.handle_voice_stop(conn, asr_audio_task)
+
+    # 处理语音停止
+    async def handle_voice_stop(self, conn, asr_audio_task):
+        raw_text, _ = await self.speech_to_text(
+            asr_audio_task, conn.session_id, conn.audio_format
+        )  # 确保ASR模块返回原始文本
+        conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+        text_len, _ = remove_punctuation_and_length(raw_text)
+        self.stop_ws_connection()
+        if text_len > 0:
+            # 使用自定义模块进行上报
+            await startToChat(conn, raw_text)
+            enqueue_asr_report(conn, raw_text, asr_audio_task)
+
+    def stop_ws_connection(self):
+        pass
 
-    @abstractmethod
     def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
         """PCM数据保存为WAV文件"""
-        pass
+        module_name = __name__.split(".")[-1]
+        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
+        file_path = os.path.join(self.output_dir, file_name)
+
+        with wave.open(file_path, "wb") as wf:
+            wf.setnchannels(1)
+            wf.setsampwidth(2)  # 2 bytes = 16-bit
+            wf.setframerate(16000)
+            wf.writeframes(b"".join(pcm_data))
+
+        return file_path
 
     @abstractmethod
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         pass
 
-    def set_audio_format(self, format: str) -> None:
-        """设置音频格式"""
-        self.audio_format = format
-
     @staticmethod
     def decode_opus(opus_data: List[bytes]) -> bytes:
         """将Opus音频数据解码为PCM数据"""

main/xiaozhi-server/core/providers/asr/doubao.py

@@ -1,17 +1,14 @@
 import time
-import io
-import wave
 import os
-from typing import Optional, Tuple, List
 import uuid
-import websockets
 import json
 import gzip
-
+import websockets
-import opuslib_next
-from core.providers.asr.base import ASRProviderBase
-
 from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.providers.asr.base import ASRProviderBase
+from core.providers.asr.dto.dto import InterfaceType
+
 
 TAG = __name__
 logger = setup_logging()
@@ -86,6 +83,7 @@ def parse_response(res):
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.appid = config.get("appid")
         self.cluster = config.get("cluster")
         self.access_token = config.get("access_token")
@@ -102,20 +100,6 @@ class ASRProvider(ASRProviderBase):
         # 确保输出目录存在
         os.makedirs(self.output_dir, exist_ok=True)
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     @staticmethod
     def _generate_header(
         message_type=CLIENT_FULL_REQUEST, message_type_specific_flags=NO_SEQUENCE
@@ -244,14 +228,14 @@ class ASRProvider(ASRProviderBase):
             yield data[offset:data_len], True
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
 
         file_path = None
         try:
             # 合并所有opus数据包
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/doubao_stream.py

@@ -0,0 +1,344 @@
+import json
+import gzip
+import uuid
+import asyncio
+import websockets
+import opuslib_next
+from core.providers.asr.base import ASRProviderBase
+from config.logger import setup_logging
+from core.providers.asr.dto.dto import InterfaceType
+
+TAG = __name__
+logger = setup_logging()
+
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__()
+        self.interface_type = InterfaceType.STREAM
+        self.config = config
+        self.text = ""
+        self.max_retries = 3
+        self.retry_delay = 2
+        self.decoder = opuslib_next.Decoder(16000, 1)
+        self.asr_ws = None
+        self.forward_task = None
+        self.is_processing = False  # 添加处理状态标志
+
+        # 配置参数
+        self.appid = str(config.get("appid"))
+        self.cluster = config.get("cluster")
+        self.access_token = config.get("access_token")
+        self.boosting_table_name = config.get("boosting_table_name", "")
+        self.correct_table_name = config.get("correct_table_name", "")
+        self.output_dir = config.get("output_dir", "tmp/")
+        self.delete_audio_file = delete_audio_file
+
+        # 火山引擎ASR配置
+        self.ws_url = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel"
+        self.uid = config.get("uid", "streaming_asr_service")
+        self.workflow = config.get(
+            "workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
+        )
+        self.result_type = config.get("result_type", "single")
+        self.format = config.get("format", "pcm")
+        self.codec = config.get("codec", "pcm")
+        self.rate = config.get("sample_rate", 16000)
+        self.language = config.get("language", "zh-CN")
+        self.bits = config.get("bits", 16)
+        self.channel = config.get("channel", 1)
+        self.auth_method = config.get("auth_method", "token")
+        self.secret = config.get("secret", "access_secret")
+
+    async def open_audio_channels(self, conn):
+        await super().open_audio_channels(conn)
+
+    async def receive_audio(self, conn, audio, audio_have_voice):
+        conn.asr_audio.append(audio)
+        conn.asr_audio = conn.asr_audio[-10:]
+
+        # 如果本次有声音，且之前没有建立连接
+        if audio_have_voice and self.asr_ws is None and not self.is_processing:
+            try:
+                self.is_processing = True
+                # 建立新的WebSocket连接
+                headers = self.token_auth() if self.auth_method == "token" else None
+                logger.bind(tag=TAG).info(f"正在连接ASR服务，headers: {headers}")
+
+                self.asr_ws = await websockets.connect(
+                    self.ws_url,
+                    additional_headers=headers,
+                    max_size=1000000000,
+                    ping_interval=None,
+                    ping_timeout=None,
+                    close_timeout=10,
+                )
+
+                # 发送初始化请求
+                request_params = self.construct_request(str(uuid.uuid4()))
+                try:
+                    payload_bytes = str.encode(json.dumps(request_params))
+                    payload_bytes = gzip.compress(payload_bytes)
+                    full_client_request = self.generate_header()
+                    full_client_request.extend((len(payload_bytes)).to_bytes(4, "big"))
+                    full_client_request.extend(payload_bytes)
+
+                    logger.bind(tag=TAG).info(f"发送初始化请求: {request_params}")
+                    await self.asr_ws.send(full_client_request)
+
+                    # 等待初始化响应
+                    init_res = await self.asr_ws.recv()
+                    result = self.parse_response(init_res)
+                    logger.bind(tag=TAG).info(f"收到初始化响应: {result}")
+
+                    # 检查初始化响应
+                    if "code" in result and result["code"] != 1000:
+                        error_msg = f"ASR服务初始化失败: {result.get('payload_msg', {}).get('message', '未知错误')}"
+                        if "payload_msg" in result:
+                            error_msg += f"\n详细错误信息: {json.dumps(result['payload_msg'], ensure_ascii=False)}"
+                        logger.bind(tag=TAG).error(error_msg)
+                        raise Exception(error_msg)
+
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"发送初始化请求失败: {str(e)}")
+                    if hasattr(e, "__cause__") and e.__cause__:
+                        logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                    raise e
+
+                # 启动接收ASR结果的异步任务
+                self.forward_task = asyncio.create_task(self._forward_asr_results(conn))
+
+                # 发送缓存的音频数据
+                if conn.asr_audio and len(conn.asr_audio) > 0:
+                    for cached_audio in conn.asr_audio[-10:]:
+                        try:
+                            pcm_frame = self.decoder.decode(cached_audio, 960)
+                            payload = gzip.compress(pcm_frame)
+                            audio_request = bytearray(
+                                self.generate_audio_default_header()
+                            )
+                            audio_request.extend(len(payload).to_bytes(4, "big"))
+                            audio_request.extend(payload)
+                            await self.asr_ws.send(audio_request)
+                        except Exception as e:
+                            logger.bind(tag=TAG).info(
+                                f"发送缓存音频数据时发生错误: {e}"
+                            )
+
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"建立ASR连接失败: {str(e)}")
+                if hasattr(e, "__cause__") and e.__cause__:
+                    logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                if self.asr_ws:
+                    await self.asr_ws.close()
+                    self.asr_ws = None
+                self.is_processing = False
+                return
+
+        # 发送当前音频数据
+        if self.asr_ws and self.is_processing:
+            try:
+                pcm_frame = self.decoder.decode(audio, 960)
+                payload = gzip.compress(pcm_frame)
+                audio_request = bytearray(self.generate_audio_default_header())
+                audio_request.extend(len(payload).to_bytes(4, "big"))
+                audio_request.extend(payload)
+                await self.asr_ws.send(audio_request)
+            except Exception as e:
+                logger.bind(tag=TAG).info(f"发送音频数据时发生错误: {e}")
+
+    async def _forward_asr_results(self, conn):
+        try:
+            while self.asr_ws and not conn.stop_event.is_set():
+                try:
+                    response = await self.asr_ws.recv()
+                    result = self.parse_response(response)
+                    logger.bind(tag=TAG).debug(f"收到ASR结果: {result}")
+
+                    if "payload_msg" in result:
+                        payload = result["payload_msg"]
+                        if "result" in payload:
+                            utterances = payload["result"].get("utterances", [])
+                            # 检查duration和空文本的情况
+                            if (
+                                payload.get("audio_info", {}).get("duration", 0) > 2000
+                                and not utterances
+                                and not payload["result"].get("text")
+                            ):
+                                logger.bind(tag=TAG).error(f"识别文本：空")
+                                self.text = ""
+                                conn.reset_vad_states()
+                                await self.handle_voice_stop(conn, None)
+                                break
+
+                            for utterance in utterances:
+                                if utterance.get("definite", False):
+                                    self.text = utterance["text"]
+                                    logger.bind(tag=TAG).info(
+                                        f"识别到文本: {self.text}"
+                                    )
+                                    conn.reset_vad_states()
+                                    await self.handle_voice_stop(conn, None)
+                                    break
+                        elif "error" in payload:
+                            error_msg = payload.get("error", "未知错误")
+                            logger.bind(tag=TAG).error(f"ASR服务返回错误: {error_msg}")
+                            break
+
+                except websockets.ConnectionClosed:
+                    logger.bind(tag=TAG).info("ASR服务连接已关闭")
+                    self.is_processing = False
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"处理ASR结果时发生错误: {str(e)}")
+                    if hasattr(e, "__cause__") and e.__cause__:
+                        logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                    self.is_processing = False
+                    break
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"ASR结果转发任务发生错误: {str(e)}")
+            if hasattr(e, "__cause__") and e.__cause__:
+                logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+        finally:
+            if self.asr_ws:
+                await self.asr_ws.close()
+                self.asr_ws = None
+            self.is_processing = False
+
+    def stop_ws_connection(self):
+        if self.asr_ws:
+            asyncio.create_task(self.asr_ws.close())
+            self.asr_ws = None
+        self.is_processing = False
+
+    def construct_request(self, reqid):
+        req = {
+            "app": {
+                "appid": self.appid,
+                "cluster": self.cluster,
+                "token": self.access_token,
+            },
+            "user": {"uid": self.uid},
+            "request": {
+                "reqid": reqid,
+                "workflow": self.workflow,
+                "show_utterances": True,
+                "result_type": self.result_type,
+                "sequence": 1,
+                "boosting_table_name": self.boosting_table_name,
+                "correct_table_name": self.correct_table_name,
+                "end_window_size": 200,
+            },
+            "audio": {
+                "format": self.format,
+                "codec": self.codec,
+                "rate": self.rate,
+                "language": self.language,
+                "bits": self.bits,
+                "channel": self.channel,
+                "sample_rate": self.rate,
+            },
+        }
+        logger.bind(tag=TAG).debug(
+            f"构造请求参数: {json.dumps(req, ensure_ascii=False)}"
+        )
+        return req
+
+    def token_auth(self):
+        return {
+            "X-Api-App-Key": self.appid,
+            "X-Api-Access-Key": self.access_token,
+            "X-Api-Resource-Id": "volc.bigasr.sauc.duration",
+            "X-Api-Connect-Id": str(uuid.uuid4()),
+            "Host": "openspeech.bytedance.com",
+        }
+
+    def generate_header(
+        self,
+        version=0x01,
+        message_type=0x01,
+        message_type_specific_flags=0x00,
+        serial_method=0x01,
+        compression_type=0x01,
+        reserved_data=0x00,
+        extension_header: bytes = b"",
+    ):
+        header = bytearray()
+        header_size = int(len(extension_header) / 4) + 1
+        header.append((version << 4) | header_size)
+        header.append((message_type << 4) | message_type_specific_flags)
+        header.append((serial_method << 4) | compression_type)
+        header.append(reserved_data)
+        header.extend(extension_header)
+        return header
+
+    def generate_audio_default_header(self):
+        return self.generate_header(
+            version=0x01,
+            message_type=0x02,
+            message_type_specific_flags=0x00,
+            serial_method=0x01,
+            compression_type=0x01,
+        )
+
+    def generate_last_audio_default_header(self):
+        return self.generate_header(
+            version=0x01,
+            message_type=0x02,
+            message_type_specific_flags=0x02,
+            serial_method=0x01,
+            compression_type=0x01,
+        )
+
+    def parse_response(self, res: bytes) -> dict:
+        try:
+            # 检查响应长度
+            if len(res) < 4:
+                logger.bind(tag=TAG).error(f"响应数据长度不足: {len(res)}")
+                return {"error": "响应数据长度不足"}
+
+            # 获取消息头
+            header = res[:4]
+            message_type = header[1] >> 4
+
+            # 如果是错误响应
+            if message_type == 0x0F:  # SERVER_ERROR_RESPONSE
+                code = int.from_bytes(header[4:8], "big", signed=False)
+                error_msg = res[8:].decode("utf-8")
+                return {"code": code, "error": error_msg}
+
+            # 获取JSON数据（跳过12字节头部）
+            try:
+                json_data = res[12:].decode("utf-8")
+                result = json.loads(json_data)
+                logger.bind(tag=TAG).debug(f"成功解析JSON响应: {result}")
+                return {"payload_msg": result}
+            except (UnicodeDecodeError, json.JSONDecodeError) as e:
+                logger.bind(tag=TAG).error(f"JSON解析失败: {str(e)}")
+                logger.bind(tag=TAG).error(f"原始数据: {res}")
+                raise
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"解析响应失败: {str(e)}")
+            logger.bind(tag=TAG).error(f"原始响应数据: {res.hex()}")
+            raise
+
+    async def speech_to_text(self, opus_data, session_id, audio_format):
+        result = self.text
+        self.text = ""  # 清空text
+        return result, None
+
+    async def close(self):
+        """资源清理方法"""
+        if self.asr_ws:
+            await self.asr_ws.close()
+            self.asr_ws = None
+        if self.forward_task:
+            self.forward_task.cancel()
+            try:
+                await self.forward_task
+            except asyncio.CancelledError:
+                pass
+            self.forward_task = None
+        self.is_processing = False

main/xiaozhi-server/core/providers/asr/dto/dto.py

@@ -0,0 +1,9 @@
+from enum import Enum
+from typing import Union, Optional
+
+
+class InterfaceType(Enum):
+    # 接口类型
+    STREAM = "STREAM"  # 流式接口
+    NON_STREAM = "NON_STREAM"  # 非流式接口
+    LOCAL = "LOCAL"  # 本地服务

main/xiaozhi-server/core/providers/asr/fun_local.py

@@ -1,15 +1,14 @@
 import time
-import wave
 import os
 import sys
 import io
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
-import uuid
 from core.providers.asr.base import ASRProviderBase
 from funasr import AutoModel
 from funasr.utils.postprocess_utils import rich_transcription_postprocess
 import shutil
+from core.providers.asr.dto.dto import InterfaceType
 
 TAG = __name__
 logger = setup_logging()
@@ -38,6 +37,7 @@ class CaptureOutput:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.LOCAL
         self.model_dir = config.get("model_dir")
         self.output_dir = config.get("output_dir")  # 修正配置键名
         self.delete_audio_file = delete_audio_file
@@ -53,22 +53,8 @@ class ASRProvider(ASRProviderBase):
                 # device="cuda:0",  # 启用GPU加速
             )
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """语音转文本主处理逻辑"""
         file_path = None
@@ -77,7 +63,7 @@ class ASRProvider(ASRProviderBase):
         while retry_count < MAX_RETRIES:
             try:
                 # 合并所有opus数据包
-                if self.audio_format == "pcm":
+                if audio_format == "pcm":
                     pcm_data = opus_data
                 else:
                     pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/fun_server.py

@@ -1,11 +1,8 @@
 from typing import Optional, Tuple, List
-import opuslib_next
 from core.providers.asr.base import ASRProviderBase
-import os
+from core.providers.asr.dto.dto import InterfaceType
 import ssl
 import json
-import uuid
-import wave
 import websockets
 from config.logger import setup_logging
 import asyncio
@@ -23,6 +20,7 @@ class ASRProvider(ASRProviderBase):
         :param delete_audio_file: Boolean to indicate whether to delete audio files after processing.
         """
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.host = config.get("host", "localhost")
         self.port = config.get("port", 10095)
         self.api_key = config.get("api_key", "none")
@@ -43,20 +41,6 @@ class ASRProvider(ASRProviderBase):
             self.ssl_context.check_hostname = False
             self.ssl_context.verify_mode = ssl.CERT_NONE
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def _receive_responses(self, ws) -> None:
         """
         Asynchronous generator to receive messages from the WebSocket.
@@ -116,7 +100,7 @@ class ASRProvider(ASRProviderBase):
         logger.bind(tag=TAG).debug(f"Sent end message: {end_message}")
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """
         Convert speech data to text using FunASR.
@@ -125,7 +109,7 @@ class ASRProvider(ASRProviderBase):
         :return: Tuple containing recognized text and optional timestamp.
         """
         file_path = None
-        if self.audio_format == "pcm":
+        if audio_format == "pcm":
             pcm_data = opus_data
         else:
             pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/sherpa_onnx_local.py

@@ -5,8 +5,7 @@ import sys
 import io
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
-import uuid
+from core.providers.asr.dto.dto import InterfaceType
-import opuslib_next
 from core.providers.asr.base import ASRProviderBase
 
 import numpy as np
@@ -38,6 +37,7 @@ class CaptureOutput:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.LOCAL
         self.model_dir = config.get("model_dir")
         self.output_dir = config.get("output_dir")
         self.delete_audio_file = delete_audio_file
@@ -84,20 +84,6 @@ class ASRProvider(ASRProviderBase):
                 use_itn=True,
             )
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     def read_wave(self, wave_filename: str) -> Tuple[np.ndarray, int]:
         """
         Args:
@@ -123,14 +109,14 @@ class ASRProvider(ASRProviderBase):
             return samples_float32, f.getframerate()
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """语音转文本主处理逻辑"""
         file_path = None
         try:
             # 保存音频文件
             start_time = time.time()
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/tencent.py

@@ -5,11 +5,8 @@ import json
 import time
 from datetime import datetime, timezone
 import os
-import uuid
 from typing import Optional, Tuple, List
-import wave
+from core.providers.asr.dto.dto import InterfaceType
-import opuslib_next
-
 import requests
 from core.providers.asr.base import ASRProviderBase
 from config.logger import setup_logging
@@ -25,6 +22,7 @@ class ASRProvider(ASRProviderBase):
 
     def __init__(self, config: dict, delete_audio_file: bool = True):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.secret_id = config.get("secret_id")
         self.secret_key = config.get("secret_key")
         self.output_dir = config.get("output_dir")
@@ -33,22 +31,8 @@ class ASRProvider(ASRProviderBase):
         # 确保输出目录存在
         os.makedirs(self.output_dir, exist_ok=True)
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if not opus_data:
@@ -63,7 +47,7 @@ class ASRProvider(ASRProviderBase):
                 return None, file_path
 
             # 将Opus音频数据解码为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py

@@ -122,6 +122,8 @@ class IntentProvider(IntentProviderBase):
     async def detect_intent(self, conn, dialogue_history: List[Dict], text: str) -> str:
         if not self.llm:
             raise ValueError("LLM provider not set")
+        if conn.func_handler is None:
+            return '{"function_call": {"name": "continue_chat"}}'
 
         # 记录整体开始时间
         total_start_time = time.time()
@@ -148,9 +150,8 @@ class IntentProvider(IntentProviderBase):
         self.clean_cache()
 
         if self.promot == "":
-            if hasattr(conn, "func_handler"):
+            functions = conn.func_handler.get_functions()
-                functions = conn.func_handler.get_functions()
+            self.promot = self.get_intent_system_prompt(functions)
-                self.promot = self.get_intent_system_prompt(functions)
 
         music_config = initialize_music_handler(conn)
         music_file_names = music_config["music_file_names"]

main/xiaozhi-server/core/providers/tts/base.py

@@ -52,6 +52,7 @@ class TTSProviderBase(ABC):
         )
         self.first_sentence_punctuations = (
             "，",
+            "～",
             "~",
             "、",
             ",",
@@ -178,6 +179,9 @@ class TTSProviderBase(ABC):
         while not self.conn.stop_event.is_set():
             try:
                 message = self.tts_text_queue.get(timeout=1)
+                if self.conn.client_abort:
+                    logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                    continue
                 if message.sentence_type == SentenceType.FIRST:
                     # 初始化参数
                     self.tts_stop_request = False
@@ -216,6 +220,7 @@ class TTSProviderBase(ABC):
                 logger.bind(tag=TAG).error(
                     f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
                 )
+                continue
 
     def _audio_play_priority_thread(self):
         while not self.conn.stop_event.is_set():

main/xiaozhi-server/core/providers/tts/huoshan_double_stream.py

@@ -3,13 +3,13 @@ import uuid
 import json
 import queue
 import asyncio
-import threading
 import traceback
 import websockets
 from config.logger import setup_logging
 from core.utils import opus_encoder_utils
 from core.utils.util import check_model_key
 from core.providers.tts.base import TTSProviderBase
+from core.handle.abortHandle import handleAbortMessage
 from core.providers.tts.dto.dto import SentenceType, ContentType, InterfaceType
 
 TAG = __name__
@@ -138,6 +138,7 @@ class Response:
 class TTSProvider(TTSProviderBase):
     def __init__(self, config, delete_audio_file):
         super().__init__(config, delete_audio_file)
+        self.ws = None
         self.interface_type = InterfaceType.DUAL_STREAM
         self.appId = config.get("appid")
         self.access_token = config.get("access_token")
@@ -152,73 +153,106 @@ class TTSProvider(TTSProviderBase):
         self.authorization = config.get("authorization")
         self.header = {"Authorization": f"{self.authorization}{self.access_token}"}
         self.enable_two_way = True
-        self.start_connection_flag = False
         self.tts_text = ""
-        # 合成文字语音后，播放的音频文件列表
         self.before_stop_play_files = []
         self.opus_encoder = opus_encoder_utils.OpusEncoderUtils(
             sample_rate=16000, channels=1, frame_size_ms=60
         )
         check_model_key("TTS", self.access_token)
 
-        # 添加会话状态控制
-        self._session_lock = asyncio.Lock()  # 会话操作的并发锁
-        self._current_session_id = None  # 当前会话ID
-        self._session_started = False  # 会话是否已开始
-        self._session_finished = False  # 会话是否已结束
-
-    ###################################################################################
-    # 火山双流式TTS重写父类的方法--开始
-    ###################################################################################
-
     async def open_audio_channels(self, conn):
-        await super().open_audio_channels(conn)
+        try:
-        ws_header = {
+            await super().open_audio_channels(conn)
-            "X-Api-App-Key": self.appId,
+        except Exception as e:
-            "X-Api-Access-Key": self.access_token,
+            logger.bind(tag=TAG).error(f"Failed to open audio channels: {str(e)}")
-            "X-Api-Resource-Id": self.resource_id,
+            self.ws = None
-            "X-Api-Connect-Id": uuid.uuid4(),
+            raise
-        }
+
-        self.ws = await websockets.connect(
+    async def _ensure_connection(self):
-            self.ws_url, additional_headers=ws_header, max_size=1000000000
+        """建立新的WebSocket连接"""
-        )
+        try:
-        tts_priority = threading.Thread(
+            logger.bind(tag=TAG).info("开始建立新连接...")
-            target=self._start_monitor_tts_response_thread, daemon=True
+            ws_header = {
-        )
+                "X-Api-App-Key": self.appId,
-        tts_priority.start()
+                "X-Api-Access-Key": self.access_token,
+                "X-Api-Resource-Id": self.resource_id,
+                "X-Api-Connect-Id": uuid.uuid4(),
+            }
+            self.ws = await websockets.connect(
+                self.ws_url, additional_headers=ws_header, max_size=1000000000
+            )
+            logger.bind(tag=TAG).info("WebSocket连接建立成功")
+            return self.ws
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"建立连接失败: {str(e)}")
+            self.ws = None
+            raise
 
     def tts_text_priority_thread(self):
+        """火山引擎双流式TTS的文本处理线程"""
+        logger.bind(tag=TAG).info("TTS文本处理线程启动")
         while not self.conn.stop_event.is_set():
             try:
+                logger.bind(tag=TAG).debug("等待TTS文本队列消息...")
                 message = self.tts_text_queue.get(timeout=1)
                 logger.bind(tag=TAG).debug(
-                    f"TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name}"
+                    f"收到TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
                 )
+                if self.conn.client_abort:
+                    logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                    continue
+
                 if message.sentence_type == SentenceType.FIRST:
                     # 初始化参数
-                    future = asyncio.run_coroutine_threadsafe(
+                    try:
-                        self.start_session(self.conn.sentence_id), loop=self.conn.loop
+                        logger.bind(tag=TAG).info("开始启动TTS会话...")
-                    )
-                    future.result()
-                    self.tts_audio_first_sentence = True
-                    self.before_stop_play_files.clear()
-                elif ContentType.TEXT == message.content_type:
-                    if message.content_detail:
                         future = asyncio.run_coroutine_threadsafe(
-                            self.text_to_speak(message.content_detail, None),
+                            self.start_session(self.conn.sentence_id),
                             loop=self.conn.loop,
                         )
                         future.result()
+                        self.tts_audio_first_sentence = True
+                        self.before_stop_play_files.clear()
+                        logger.bind(tag=TAG).info("TTS会话启动成功")
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"启动TTS会话失败: {str(e)}")
+                        continue
+
+                elif ContentType.TEXT == message.content_type:
+                    if message.content_detail:
+                        try:
+                            logger.bind(tag=TAG).debug(
+                                f"开始发送TTS文本: {message.content_detail}"
+                            )
+                            future = asyncio.run_coroutine_threadsafe(
+                                self.text_to_speak(message.content_detail, None),
+                                loop=self.conn.loop,
+                            )
+                            future.result()
+                            logger.bind(tag=TAG).debug("TTS文本发送成功")
+                        except Exception as e:
+                            logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
+                            continue
+
                 elif ContentType.FILE == message.content_type:
+                    logger.bind(tag=TAG).info(
+                        f"添加音频文件到待播放列表: {message.content_file}"
+                    )
                     self.before_stop_play_files.append(
                         (message.content_file, message.content_detail)
                     )
 
                 if message.sentence_type == SentenceType.LAST:
-                    future = asyncio.run_coroutine_threadsafe(
+                    try:
-                        self.finish_session(self.conn.sentence_id), loop=self.conn.loop
+                        logger.bind(tag=TAG).info("开始结束TTS会话...")
-                    )
+                        future = asyncio.run_coroutine_threadsafe(
-                    future.result()
+                            self.finish_session(self.conn.sentence_id),
+                            loop=self.conn.loop,
+                        )
+                        future.result()
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"结束TTS会话失败: {str(e)}")
+                        continue
 
             except queue.Empty:
                 continue
@@ -226,96 +260,220 @@ class TTSProvider(TTSProviderBase):
                 logger.bind(tag=TAG).error(
                     f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
                 )
+                continue
 
     async def text_to_speak(self, text, _):
-        # 发送文本
+        """发送文本到TTS服务"""
-        await self.send_text(self.speaker, text, self.conn.sentence_id)
+        try:
-        return
+            # 建立新连接
+            if self.ws is None:
+                await handleAbortMessage(self.conn)
+                logger.bind(tag=TAG).error(f"WebSocket连接不存在，终止发送文本")
+                return
+            # 发送文本
+            await self.send_text(self.speaker, text, self.conn.sentence_id)
+            return
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+            raise
 
-    ###################################################################################
+    async def start_session(self, session_id):
-    # 火山双流式TTS重写父类的方法--结束
+        logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
-    ###################################################################################
+        try:
-    def _start_monitor_tts_response_thread(self):
+            # 建立新连接
-        # 初始化链接
+            await self._ensure_connection()
-        asyncio.run_coroutine_threadsafe(
+
-            self._start_monitor_tts_response(), loop=self.conn.loop
+            # 启动监听任务
-        )
+            self._monitor_task = asyncio.create_task(self._start_monitor_tts_response())
+
+            header = Header(
+                message_type=FULL_CLIENT_REQUEST,
+                message_type_specific_flags=MsgTypeFlagWithEvent,
+                serial_method=JSON,
+            ).as_bytes()
+            optional = Optional(
+                event=EVENT_StartSession, sessionId=session_id
+            ).as_bytes()
+            payload = self.get_payload_bytes(
+                event=EVENT_StartSession, speaker=self.speaker
+            )
+            await self.send_event(header, optional, payload)
+            logger.bind(tag=TAG).info("会话启动请求已发送")
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
+            # 确保清理资源
+            if hasattr(self, "_monitor_task"):
+                try:
+                    self._monitor_task.cancel()
+                    await self._monitor_task
+                except:
+                    pass
+                self._monitor_task = None
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+            raise
+
+    async def finish_session(self, session_id):
+        logger.bind(tag=TAG).info(f"关闭会话～～{session_id}")
+        try:
+            if self.ws:
+                header = Header(
+                    message_type=FULL_CLIENT_REQUEST,
+                    message_type_specific_flags=MsgTypeFlagWithEvent,
+                    serial_method=JSON,
+                ).as_bytes()
+                optional = Optional(
+                    event=EVENT_FinishSession, sessionId=session_id
+                ).as_bytes()
+                payload = str.encode("{}")
+                await self.send_event(header, optional, payload)
+                logger.bind(tag=TAG).info("会话结束请求已发送")
+
+                # 等待监听任务完成
+                if hasattr(self, "_monitor_task"):
+                    try:
+                        await self._monitor_task
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(
+                            f"等待监听任务完成时发生错误: {str(e)}"
+                        )
+                    finally:
+                        self._monitor_task = None
+
+                # 关闭连接
+                await self.close()
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
+            # 确保清理资源
+            if hasattr(self, "_monitor_task"):
+                try:
+                    self._monitor_task.cancel()
+                    await self._monitor_task
+                except:
+                    pass
+                self._monitor_task = None
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+            raise
+
+    async def close(self):
+        """资源清理方法"""
+        if self.ws:
+            try:
+                await self.ws.close()
+            except:
+                pass
+            self.ws = None
 
     async def _start_monitor_tts_response(self):
+        """监听TTS响应"""
         opus_datas_cache = []
-        # 添加标志来区分是否是第一句话
         is_first_sentence = True
-        while not self.conn.stop_event.is_set():
+        try:
-            try:
+            while not self.conn.stop_event.is_set():
-                # 确保 `recv()` 运行在同一个 event loop
+                try:
-                msg = await self.ws.recv()
+                    # 确保 `recv()` 运行在同一个 event loop
-                res = self.parser_response(msg)
+                    msg = await self.ws.recv()
-                self.print_response(res, "send_text res:")
+                    res = self.parser_response(msg)
+                    self.print_response(res, "send_text res:")
 
-                if res.optional.event == EVENT_TTSSentenceStart:
+                    # 检查客户端是否中止
-                    json_data = json.loads(res.payload.decode("utf-8"))
+                    if self.conn.client_abort:
-                    self.tts_text = json_data.get("text", "")
+                        logger.bind(tag=TAG).info("收到打断信息，终止监听TTS响应")
-                    logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
+                        break
-                    self.tts_audio_queue.put((SentenceType.FIRST, [], self.tts_text))
+
-                    opus_datas_cache = []
+                    if res.optional.event == EVENT_TTSSentenceStart:
-                elif (
+                        json_data = json.loads(res.payload.decode("utf-8"))
-                    res.optional.event == EVENT_TTSResponse
+                        self.tts_text = json_data.get("text", "")
-                    and res.header.message_type == AUDIO_ONLY_RESPONSE
+                        logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
-                ):
-                    logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
-                    opus_datas = self.wav_to_opus_data_audio_raw(res.payload)
-                    logger.bind(tag=TAG).debug(
-                        f"推送数据到队列里面帧数～～{len(opus_datas)}"
-                    )
-                    if is_first_sentence:
-                        # 第一句话直接发送
                         self.tts_audio_queue.put(
-                            (SentenceType.MIDDLE, opus_datas, self.tts_text)
+                            (SentenceType.FIRST, [], self.tts_text)
                         )
-                    else:
+                        opus_datas_cache = []
-                        # 后续句子缓存
+                    elif (
-                        opus_datas_cache = opus_datas_cache + opus_datas
+                        res.optional.event == EVENT_TTSResponse
-                elif res.optional.event == EVENT_TTSSentenceEnd:
+                        and res.header.message_type == AUDIO_ONLY_RESPONSE
-                    logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
+                    ):
-                    if not is_first_sentence:
+                        logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
-                        # 只有非第一句话才发送缓存的数据
+                        opus_datas = self.wav_to_opus_data_audio_raw(res.payload)
-                        self.tts_audio_queue.put(
+                        logger.bind(tag=TAG).debug(
-                            (SentenceType.MIDDLE, opus_datas_cache, self.tts_text)
+                            f"推送数据到队列里面帧数～～{len(opus_datas)}"
                         )
-                    # 第一句话结束后，将标志设置为False
+                        if is_first_sentence:
-                    is_first_sentence = False
+                            # 第一句话直接发送
-                elif res.optional.event == EVENT_SessionFinished:
-                    logger.bind(tag=TAG).debug(f"会话结束～～")
-                    for tts_file, text in self.before_stop_play_files:
-                        if tts_file and os.path.exists(tts_file):
-                            audio_datas = self._process_audio_file(tts_file)
                             self.tts_audio_queue.put(
-                                (SentenceType.MIDDLE, audio_datas, text)
+                                (SentenceType.MIDDLE, opus_datas, self.tts_text)
                             )
-                    self.before_stop_play_files.clear()
+                        else:
-                    self.tts_audio_queue.put((SentenceType.LAST, [], None))
+                            # 后续句子缓存
-
+                            opus_datas_cache = opus_datas_cache + opus_datas
-                    opus_datas_cache = []
+                    elif res.optional.event == EVENT_TTSSentenceEnd:
-                    is_first_sentence = True
+                        logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
-                    continue
+                        if not is_first_sentence:
-            except websockets.ConnectionClosed:
+                            # 只有非第一句话才发送缓存的数据
-                break  # 连接关闭时退出监听
+                            self.tts_audio_queue.put(
-            except Exception as e:
+                                (SentenceType.MIDDLE, opus_datas_cache, self.tts_text)
-                logger.bind(tag=TAG).error(f"Error in _start_monitor_tts_response: {e}")
+                            )
-                traceback.print_exc()
+                        # 第一句话结束后，将标志设置为False
-                continue
+                        is_first_sentence = False
+                    elif res.optional.event == EVENT_SessionFinished:
+                        logger.bind(tag=TAG).debug(f"会话结束～～")
+                        for tts_file, text in self.before_stop_play_files:
+                            if tts_file and os.path.exists(tts_file):
+                                audio_datas = self._process_audio_file(tts_file)
+                                self.tts_audio_queue.put(
+                                    (SentenceType.MIDDLE, audio_datas, text)
+                                )
+                        self.before_stop_play_files.clear()
+                        self.tts_audio_queue.put((SentenceType.LAST, [], None))
+                        break
+                except websockets.ConnectionClosed:
+                    logger.bind(tag=TAG).warning("WebSocket连接已关闭")
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(
+                        f"Error in _start_monitor_tts_response: {e}"
+                    )
+                    traceback.print_exc()
+                    break
+        finally:
+            # 确保清理资源
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
 
     async def send_event(
         self, header: bytes, optional: bytes | None = None, payload: bytes = None
     ):
-        full_client_request = bytearray(header)
+        try:
-        if optional is not None:
+            full_client_request = bytearray(header)
-            full_client_request.extend(optional)
+            if optional is not None:
-        if payload is not None:
+                full_client_request.extend(optional)
-            payload_size = len(payload).to_bytes(4, "big", signed=True)
+            if payload is not None:
-            full_client_request.extend(payload_size)
+                payload_size = len(payload).to_bytes(4, "big", signed=True)
-            full_client_request.extend(payload)
+                full_client_request.extend(payload_size)
-        await self.ws.send(full_client_request)
+                full_client_request.extend(payload)
+            await self.ws.send(full_client_request)
+        except websockets.ConnectionClosed:
+            logger.bind(tag=TAG).error(f"ConnectionClosed")
+            raise
 
     async def send_text(self, speaker: str, text: str, session_id):
         header = Header(
@@ -438,92 +596,6 @@ class TTSProvider(TTSProviderBase):
             )
         )
 
-    async def finish_connection(self):
-        header = Header(
-            message_type=FULL_CLIENT_REQUEST,
-            message_type_specific_flags=MsgTypeFlagWithEvent,
-            serial_method=JSON,
-        ).as_bytes()
-        optional = Optional(event=EVENT_FinishConnection).as_bytes()
-        payload = str.encode("{}")
-        await self.send_event(header, optional, payload)
-        return
-
-    async def start_session(self, session_id):
-        logger.bind(tag=TAG).debug(f"开始会话～～{session_id}")
-        async with self._session_lock:
-            # 如果已有会话未结束，先关闭它
-            if self._session_started and not self._session_finished:
-                logger.bind(tag=TAG).warning(
-                    f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
-                )
-                await self.finish_session(self._current_session_id)
-
-            # 重置会话状态
-            self._current_session_id = session_id
-            self._session_started = True
-            self._session_finished = False
-
-            header = Header(
-                message_type=FULL_CLIENT_REQUEST,
-                message_type_specific_flags=MsgTypeFlagWithEvent,
-                serial_method=JSON,
-            ).as_bytes()
-            optional = Optional(
-                event=EVENT_StartSession, sessionId=session_id
-            ).as_bytes()
-            payload = self.get_payload_bytes(
-                event=EVENT_StartSession, speaker=self.speaker
-            )
-            await self.send_event(header, optional, payload)
-
-    async def finish_session(self, session_id):
-        logger.bind(tag=TAG).debug(f"关闭会话～～{session_id}")
-        async with self._session_lock:
-            # 检查会话状态
-            if not self._session_started:
-                logger.bind(tag=TAG).warning(f"尝试关闭未开始的会话 {session_id}")
-                return
-
-            if self._session_finished:
-                logger.bind(tag=TAG).warning(f"会话 {session_id} 已经关闭")
-                return
-
-            if self._current_session_id != session_id:
-                logger.bind(tag=TAG).warning(
-                    f"尝试关闭错误的会话 {session_id}，当前会话为 {self._current_session_id}"
-                )
-                return
-
-            header = Header(
-                message_type=FULL_CLIENT_REQUEST,
-                message_type_specific_flags=MsgTypeFlagWithEvent,
-                serial_method=JSON,
-            ).as_bytes()
-            optional = Optional(
-                event=EVENT_FinishSession, sessionId=session_id
-            ).as_bytes()
-            payload = str.encode("{}")
-            await self.send_event(header, optional, payload)
-
-            # 更新会话状态
-            self._session_finished = True
-            self._session_started = False
-            self._current_session_id = None
-
-    async def reset(self):
-        # 关闭之前的对话
-        if self.start_connection_flag:
-            await self.finish_connection()
-            self.start_connection_flag = False
-        await self.start_connection()
-        self.start_connection_flag = True
-
-    async def close(self):
-        """资源清理方法"""
-        await self.finish_connection()
-        await self.ws.close()
-
     def wav_to_opus_data_audio_raw(self, raw_data_var, is_end=False):
         opus_datas = self.opus_encoder.encode_pcm_to_opus(raw_data_var, is_end)
         return opus_datas

main/xiaozhi-server/core/providers/vllm/base.py

@@ -0,0 +1,12 @@
+from abc import ABC, abstractmethod
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class VLLMProviderBase(ABC):
+    @abstractmethod
+    def response(self, question, base64_image):
+        """VLLM response generator"""
+        pass

main/xiaozhi-server/core/providers/vllm/openai.py

@@ -0,0 +1,65 @@
+import openai
+import json
+from config.logger import setup_logging
+from core.utils.util import check_model_key
+from core.providers.vllm.base import VLLMProviderBase
+
+TAG = __name__
+logger = setup_logging()
+
+
+class VLLMProvider(VLLMProviderBase):
+    def __init__(self, config):
+        self.model_name = config.get("model_name")
+        self.api_key = config.get("api_key")
+        if "base_url" in config:
+            self.base_url = config.get("base_url")
+        else:
+            self.base_url = config.get("url")
+
+        param_defaults = {
+            "max_tokens": (500, int),
+            "temperature": (0.7, lambda x: round(float(x), 1)),
+            "top_p": (1.0, lambda x: round(float(x), 1)),
+        }
+
+        for param, (default, converter) in param_defaults.items():
+            value = config.get(param)
+            try:
+                setattr(
+                    self,
+                    param,
+                    converter(value) if value not in (None, "") else default,
+                )
+            except (ValueError, TypeError):
+                setattr(self, param, default)
+
+        check_model_key("VLLM", self.api_key)
+        self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
+
+    def response(self, question, base64_image):
+        try:
+            messages = [
+                {
+                    "role": "user",
+                    "content": [
+                        {"type": "text", "text": question},
+                        {
+                            "type": "image_url",
+                            "image_url": {
+                                "url": f"data:image/jpeg;base64,{base64_image}"
+                            },
+                        },
+                    ],
+                }
+            ]
+
+            response = self.client.chat.completions.create(
+                model=self.model_name, messages=messages, stream=False
+            )
+
+            return response.choices[0].message.content
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"Error in response generation: {e}")
+            raise

main/xiaozhi-server/core/utils/auth.py

@@ -0,0 +1,43 @@
+import jwt
+import time
+from datetime import datetime, timedelta, timezone
+from typing import Optional, Tuple
+
+
+class AuthToken:
+    def __init__(self, secret_key: str):
+        self.secret_key = secret_key
+
+    def generate_token(self, device_id: str) -> str:
+        """
+        生成JWT token
+        :param device_id: 设备ID
+        :return: JWT token字符串
+        """
+        # 设置过期时间为1小时后
+        expire_time = datetime.now(timezone.utc) + timedelta(hours=1)
+
+        # 创建payload
+        payload = {"device_id": device_id, "exp": expire_time.timestamp()}
+
+        # 使用JWT进行编码
+        token = jwt.encode(payload, self.secret_key, algorithm="HS256")
+        return token
+
+    def verify_token(self, token: str) -> Tuple[bool, Optional[str]]:
+        """
+        验证token
+        :param token: JWT token字符串
+        :return: (是否有效, 设备ID)
+        """
+        try:
+            # 解码token
+            payload = jwt.decode(token, self.secret_key, algorithms=["HS256"])
+
+            # 检查是否过期
+            if payload["exp"] < time.time():
+                return False, None
+
+            return True, payload["device_id"]
+        except jwt.InvalidTokenError:
+            return False, None

main/xiaozhi-server/core/utils/modules_initialize.py

@@ -0,0 +1,128 @@
+from typing import Dict, Any
+from config.logger import setup_logging
+from core.utils import tts, llm, intent, memory, vad, asr
+
+TAG = __name__
+logger = setup_logging()
+
+
+def initialize_modules(
+    logger,
+    config: Dict[str, Any],
+    init_vad=False,
+    init_asr=False,
+    init_llm=False,
+    init_tts=False,
+    init_memory=False,
+    init_intent=False,
+) -> Dict[str, Any]:
+    """
+    初始化所有模块组件
+
+    Args:
+        config: 配置字典
+
+    Returns:
+        Dict[str, Any]: 包含所有初始化后的模块的字典
+    """
+    modules = {}
+
+    # 初始化TTS模块
+    if init_tts:
+        select_tts_module = config["selected_module"]["TTS"]
+        modules["tts"] = initialize_tts(config)
+        logger.bind(tag=TAG).info(f"初始化组件: tts成功 {select_tts_module}")
+
+    # 初始化LLM模块
+    if init_llm:
+        select_llm_module = config["selected_module"]["LLM"]
+        llm_type = (
+            select_llm_module
+            if "type" not in config["LLM"][select_llm_module]
+            else config["LLM"][select_llm_module]["type"]
+        )
+        modules["llm"] = llm.create_instance(
+            llm_type,
+            config["LLM"][select_llm_module],
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: llm成功 {select_llm_module}")
+
+    # 初始化Intent模块
+    if init_intent:
+        select_intent_module = config["selected_module"]["Intent"]
+        intent_type = (
+            select_intent_module
+            if "type" not in config["Intent"][select_intent_module]
+            else config["Intent"][select_intent_module]["type"]
+        )
+        modules["intent"] = intent.create_instance(
+            intent_type,
+            config["Intent"][select_intent_module],
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: intent成功 {select_intent_module}")
+
+    # 初始化Memory模块
+    if init_memory:
+        select_memory_module = config["selected_module"]["Memory"]
+        memory_type = (
+            select_memory_module
+            if "type" not in config["Memory"][select_memory_module]
+            else config["Memory"][select_memory_module]["type"]
+        )
+        modules["memory"] = memory.create_instance(
+            memory_type,
+            config["Memory"][select_memory_module],
+            config.get("summaryMemory", None),
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: memory成功 {select_memory_module}")
+
+    # 初始化VAD模块
+    if init_vad:
+        select_vad_module = config["selected_module"]["VAD"]
+        vad_type = (
+            select_vad_module
+            if "type" not in config["VAD"][select_vad_module]
+            else config["VAD"][select_vad_module]["type"]
+        )
+        modules["vad"] = vad.create_instance(
+            vad_type,
+            config["VAD"][select_vad_module],
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: vad成功 {select_vad_module}")
+
+    # 初始化ASR模块
+    if init_asr:
+        select_asr_module = config["selected_module"]["ASR"]
+        modules["asr"] = initialize_asr(config)
+        logger.bind(tag=TAG).info(f"初始化组件: asr成功 {select_asr_module}")
+    return modules
+
+
+def initialize_tts(config):
+    select_tts_module = config["selected_module"]["TTS"]
+    tts_type = (
+        select_tts_module
+        if "type" not in config["TTS"][select_tts_module]
+        else config["TTS"][select_tts_module]["type"]
+    )
+    new_tts = tts.create_instance(
+        tts_type,
+        config["TTS"][select_tts_module],
+        str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
+    )
+    return new_tts
+
+
+def initialize_asr(config):
+    select_asr_module = config["selected_module"]["ASR"]
+    asr_type = (
+        select_asr_module
+        if "type" not in config["ASR"][select_asr_module]
+        else config["ASR"][select_asr_module]["type"]
+    )
+    new_asr = asr.create_instance(
+        asr_type,
+        config["ASR"][select_asr_module],
+        str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
+    )
+    return new_asr

main/xiaozhi-server/core/utils/util.py

@@ -7,8 +7,6 @@ import numpy as np
 import requests
 import opuslib_next
 from pydub import AudioSegment
-from typing import Dict, Any
-from core.utils import tts, llm, intent, memory, vad, asr
 import copy
 
 TAG = __name__
@@ -245,122 +243,6 @@ def extract_json_from_string(input_string):
     return None
 
 
-def initialize_modules(
-    logger,
-    config: Dict[str, Any],
-    init_vad=False,
-    init_asr=False,
-    init_llm=False,
-    init_tts=False,
-    init_memory=False,
-    init_intent=False,
-) -> Dict[str, Any]:
-    """
-    初始化所有模块组件
-
-    Args:
-        config: 配置字典
-
-    Returns:
-        Dict[str, Any]: 包含所有初始化后的模块的字典
-    """
-    modules = {}
-
-    # 初始化TTS模块
-    if init_tts:
-        select_tts_module = config["selected_module"]["TTS"]
-        modules["tts"] = initialize_tts(config)
-        logger.bind(tag=TAG).info(f"初始化组件: tts成功 {select_tts_module}")
-
-    # 初始化LLM模块
-    if init_llm:
-        select_llm_module = config["selected_module"]["LLM"]
-        llm_type = (
-            select_llm_module
-            if "type" not in config["LLM"][select_llm_module]
-            else config["LLM"][select_llm_module]["type"]
-        )
-        modules["llm"] = llm.create_instance(
-            llm_type,
-            config["LLM"][select_llm_module],
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: llm成功 {select_llm_module}")
-
-    # 初始化Intent模块
-    if init_intent:
-        select_intent_module = config["selected_module"]["Intent"]
-        intent_type = (
-            select_intent_module
-            if "type" not in config["Intent"][select_intent_module]
-            else config["Intent"][select_intent_module]["type"]
-        )
-        modules["intent"] = intent.create_instance(
-            intent_type,
-            config["Intent"][select_intent_module],
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: intent成功 {select_intent_module}")
-
-    # 初始化Memory模块
-    if init_memory:
-        select_memory_module = config["selected_module"]["Memory"]
-        memory_type = (
-            select_memory_module
-            if "type" not in config["Memory"][select_memory_module]
-            else config["Memory"][select_memory_module]["type"]
-        )
-        modules["memory"] = memory.create_instance(
-            memory_type,
-            config["Memory"][select_memory_module],
-            config.get("summaryMemory", None),
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: memory成功 {select_memory_module}")
-
-    # 初始化VAD模块
-    if init_vad:
-        select_vad_module = config["selected_module"]["VAD"]
-        vad_type = (
-            select_vad_module
-            if "type" not in config["VAD"][select_vad_module]
-            else config["VAD"][select_vad_module]["type"]
-        )
-        modules["vad"] = vad.create_instance(
-            vad_type,
-            config["VAD"][select_vad_module],
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: vad成功 {select_vad_module}")
-
-    # 初始化ASR模块
-    if init_asr:
-        select_asr_module = config["selected_module"]["ASR"]
-        asr_type = (
-            select_asr_module
-            if "type" not in config["ASR"][select_asr_module]
-            else config["ASR"][select_asr_module]["type"]
-        )
-        modules["asr"] = asr.create_instance(
-            asr_type,
-            config["ASR"][select_asr_module],
-            str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: asr成功 {select_asr_module}")
-    return modules
-
-
-def initialize_tts(config):
-    select_tts_module = config["selected_module"]["TTS"]
-    tts_type = (
-        select_tts_module
-        if "type" not in config["TTS"][select_tts_module]
-        else config["TTS"][select_tts_module]["type"]
-    )
-    new_tts = tts.create_instance(
-        tts_type,
-        config["TTS"][select_tts_module],
-        str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
-    )
-    return new_tts
-
-
 def analyze_emotion(text):
     """
     分析文本情感并返回对应的emoji名称（支持中英文）
@@ -1000,3 +882,51 @@ def filter_sensitive_info(config: dict) -> dict:
         return filtered
 
     return _filter_dict(copy.deepcopy(config))
+
+
+def get_vision_url(config: dict) -> str:
+    """获取 vision URL
+
+    Args:
+        config: 配置字典
+
+    Returns:
+        str: vision URL
+    """
+    server_config = config["server"]
+    vision_explain = server_config.get("vision_explain", "")
+    if "你的" in vision_explain:
+        local_ip = get_local_ip()
+        port = int(server_config.get("http_port", 8003))
+        vision_explain = f"http://{local_ip}:{port}/mcp/vision/explain"
+    return vision_explain
+
+
+def is_valid_image_file(file_data: bytes) -> bool:
+    """
+    检查文件数据是否为有效的图片格式
+
+    Args:
+        file_data: 文件的二进制数据
+
+    Returns:
+        bool: 如果是有效的图片格式返回True，否则返回False
+    """
+    # 常见图片格式的魔数（文件头）
+    image_signatures = {
+        b"\xff\xd8\xff": "JPEG",
+        b"\x89PNG\r\n\x1a\n": "PNG",
+        b"GIF87a": "GIF",
+        b"GIF89a": "GIF",
+        b"BM": "BMP",
+        b"II*\x00": "TIFF",
+        b"MM\x00*": "TIFF",
+        b"RIFF": "WEBP",
+    }
+
+    # 检查文件头是否匹配任何已知的图片格式
+    for signature in image_signatures:
+        if file_data.startswith(signature):
+            return True
+
+    return False

main/xiaozhi-server/core/utils/vllm.py

@@ -0,0 +1,23 @@
+import os
+import sys
+
+# 添加项目根目录到Python路径
+current_dir = os.path.dirname(os.path.abspath(__file__))
+project_root = os.path.abspath(os.path.join(current_dir, "..", ".."))
+sys.path.insert(0, project_root)
+
+from config.logger import setup_logging
+import importlib
+
+logger = setup_logging()
+
+
+def create_instance(class_name, *args, **kwargs):
+    # 创建LLM实例
+    if os.path.exists(os.path.join("core", "providers", "vllm", f"{class_name}.py")):
+        lib_name = f"core.providers.vllm.{class_name}"
+        if lib_name not in sys.modules:
+            sys.modules[lib_name] = importlib.import_module(f"{lib_name}")
+        return sys.modules[lib_name].VLLMProvider(*args, **kwargs)
+
+    raise ValueError(f"不支持的VLLM类型: {class_name}，请检查该配置的type是否设置正确")

main/xiaozhi-server/core/websocket_server.py

@@ -2,8 +2,9 @@ import asyncio
 import websockets
 from config.logger import setup_logging
 from core.connection import ConnectionHandler
-from core.utils.util import initialize_modules, check_vad_update, check_asr_update
 from config.config_loader import get_config_from_api
+from core.utils.modules_initialize import initialize_modules
+from core.utils.util import check_vad_update, check_asr_update
 
 TAG = __name__
 

main/xiaozhi-server/docker-compose.yml

@@ -13,8 +13,8 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
-      # ota服务端
+      # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-      - "8002:8002"
+      - "8003:8003"
     volumes:
       # 配置文件目录
       - ./data:/opt/xiaozhi-esp32-server/data

main/xiaozhi-server/docker-compose_all.yml

@@ -15,6 +15,8 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
+      # http服务的端口，用于视觉分析接口
+      - "8003:8003"
     security_opt:
       - seccomp:unconfined
     environment:

main/xiaozhi-server/performance_tester.py

@@ -140,7 +140,9 @@ class AsyncPerformanceTester:
 
             print(f"🎵 测试 STT: {stt_name}")
 
-            text, _ = await stt.speech_to_text([self.test_wav_list[0]], "1")
+            text, _ = await stt.speech_to_text(
+                [self.test_wav_list[0]], "1", stt.audio_format
+            )
 
             if text is None:
                 print(f"❌ {stt_name} 连接失败")
@@ -151,7 +153,7 @@ class AsyncPerformanceTester:
 
             for i, sentence in enumerate(self.test_wav_list, 1):
                 start = time.time()
-                text, _ = await stt.speech_to_text([sentence], "1")
+                text, _ = await stt.speech_to_text([sentence], "1", stt.audio_format)
                 duration = time.time() - start
                 total_time += duration
 

main/xiaozhi-server/performance_tester_vllm.py

@@ -0,0 +1,189 @@
+import time
+import asyncio
+import logging
+import statistics
+import base64
+from typing import Dict
+from tabulate import tabulate
+from config.settings import load_config
+from core.utils.vllm import create_instance
+
+# 设置全局日志级别为WARNING，抑制INFO级别日志
+logging.basicConfig(level=logging.WARNING)
+
+
+class AsyncVisionPerformanceTester:
+    def __init__(self):
+        self.config = load_config()
+        self.test_images = [
+            "../../docs/images/demo1.png",
+            "../../docs/images/demo2.png",
+        ]
+        self.test_questions = [
+            "这张图片里有什么？",
+            "请详细描述这张图片的内容",
+        ]
+
+        # 加载测试图片
+        self.results = {"vllm": {}}
+
+    async def _test_vllm(self, vllm_name: str, config: Dict) -> Dict:
+        """异步测试单个视觉大模型性能"""
+        try:
+            # 检查API密钥配置
+            if "api_key" in config and any(
+                x in config["api_key"] for x in ["你的", "placeholder", "sk-xxx"]
+            ):
+                print(f"⏭️  VLLM {vllm_name} 未配置api_key，已跳过")
+                return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+            # 获取实际类型（兼容旧配置）
+            module_type = config.get("type", vllm_name)
+            vllm = create_instance(module_type, config)
+
+            print(f"🖼️ 测试 VLLM: {vllm_name}")
+
+            # 创建所有测试任务
+            test_tasks = []
+            for question in self.test_questions:
+                for image in self.test_images:
+                    test_tasks.append(
+                        self._test_single_vision(vllm_name, vllm, question, image)
+                    )
+
+            # 并发执行所有测试
+            test_results = await asyncio.gather(*test_tasks)
+
+            # 处理结果
+            valid_results = [r for r in test_results if r is not None]
+            if not valid_results:
+                print(f"⚠️  {vllm_name} 无有效数据，可能配置错误")
+                return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+            response_times = [r["response_time"] for r in valid_results]
+
+            # 过滤异常数据
+            mean = statistics.mean(response_times)
+            stdev = statistics.stdev(response_times) if len(response_times) > 1 else 0
+            filtered_times = [t for t in response_times if t <= mean + 3 * stdev]
+
+            if len(filtered_times) < len(test_tasks) * 0.5:
+                print(f"⚠️  {vllm_name} 有效数据不足，可能网络不稳定")
+                return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+            return {
+                "name": vllm_name,
+                "type": "vllm",
+                "avg_response": sum(response_times) / len(response_times),
+                "std_response": (
+                    statistics.stdev(response_times) if len(response_times) > 1 else 0
+                ),
+                "errors": 0,
+            }
+
+        except Exception as e:
+            print(f"⚠️ VLLM {vllm_name} 测试失败: {str(e)}")
+            return {"name": vllm_name, "type": "vllm", "errors": 1}
+
+    async def _test_single_vision(
+        self, vllm_name: str, vllm, question: str, image: str
+    ) -> Dict:
+        """测试单个视觉问题的性能"""
+        try:
+            print(f"📝 {vllm_name} 开始测试: {question[:20]}...")
+            start_time = time.time()
+
+            # 读取图片并转换为base64
+            with open(image, "rb") as image_file:
+                image_data = image_file.read()
+                image_base64 = base64.b64encode(image_data).decode("utf-8")
+
+            # 直接获取响应
+            response = vllm.response(question, image_base64)
+            response_time = time.time() - start_time
+            print(f"✓ {vllm_name} 完成响应: {response_time:.3f}s")
+
+            return {
+                "name": vllm_name,
+                "type": "vllm",
+                "response_time": response_time,
+            }
+        except Exception as e:
+            print(f"⚠️ {vllm_name} 测试失败: {str(e)}")
+            return None
+
+    def _print_results(self):
+        """打印测试结果"""
+        vllm_table = []
+        for name, data in self.results["vllm"].items():
+            if data["errors"] == 0:
+                stability = data["std_response"] / data["avg_response"]
+                vllm_table.append(
+                    [
+                        name,
+                        f"{data['avg_response']:.3f}秒",
+                        f"{stability:.3f}",
+                    ]
+                )
+
+        if vllm_table:
+            print("\n视觉大模型性能排行:\n")
+            print(
+                tabulate(
+                    vllm_table,
+                    headers=["模型名称", "响应耗时", "稳定性"],
+                    tablefmt="github",
+                    colalign=("left", "right", "right"),
+                    disable_numparse=True,
+                )
+            )
+        else:
+            print("\n⚠️ 没有可用的视觉大模型进行测试。")
+
+    async def run(self):
+        """执行全量异步测试"""
+        print("🔍 开始筛选可用视觉大模型...")
+
+        if not self.test_images:
+            print(f"\n⚠️  {self.image_root} 路径下没有图片文件，无法进行测试")
+            return
+
+        # 创建所有测试任务
+        all_tasks = []
+
+        # VLLM测试任务
+        if self.config.get("VLLM") is not None:
+            for vllm_name, config in self.config.get("VLLM", {}).items():
+                if "api_key" in config and any(
+                    x in config["api_key"] for x in ["你的", "placeholder", "sk-xxx"]
+                ):
+                    print(f"⏭️  VLLM {vllm_name} 未配置api_key，已跳过")
+                    continue
+                print(f"🖼️ 添加VLLM测试任务: {vllm_name}")
+                all_tasks.append(self._test_vllm(vllm_name, config))
+
+        print(f"\n✅ 找到 {len(all_tasks)} 个可用视觉大模型")
+        print(f"✅ 使用 {len(self.test_images)} 张测试图片")
+        print(f"✅ 使用 {len(self.test_questions)} 个测试问题")
+        print("\n⏳ 开始并发测试所有模型...\n")
+
+        # 并发执行所有测试任务
+        all_results = await asyncio.gather(*all_tasks, return_exceptions=True)
+
+        # 处理结果
+        for result in all_results:
+            if isinstance(result, dict) and result["errors"] == 0:
+                self.results["vllm"][result["name"]] = result
+
+        # 打印结果
+        print("\n📊 生成测试报告...")
+        self._print_results()
+
+
+async def main():
+    tester = AsyncVisionPerformanceTester()
+    await tester.run()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

main/xiaozhi-server/requirements.txt

@@ -21,7 +21,7 @@ cozepy==0.12.0
 mem0ai==0.1.62
 bs4==0.0.2
 modelscope==1.23.2
-sherpa_onnx==1.11.0
+sherpa_onnx==1.12.0
 mcp==1.8.1
 cnlunar==0.2.0
 PySocks==1.7.1
@@ -31,3 +31,4 @@ chardet==5.2.0
 aioconsole==0.8.1
 markitdown==0.1.1
 mcp-proxy==0.6.0
+PyJWT==2.8.0

main/xiaozhi-server/test/test_page.html

@@ -1484,6 +1484,23 @@
                                 if (message.text && message.text !== '😊') {
                                     addMessage(message.text);
                                 }
+                            }else if (message.type === 'mcp') {
+                                const payload = message.payload || {};
+                                log(`服务器下发: ${JSON.stringify(message)}`, 'info');
+                                if (payload) {
+                                    // 模拟小智客户端行为
+                                    if(payload.method === 'tools/list'){
+                                        const replay_message = JSON.stringify({"session_id":"","type":"mcp","payload":{"jsonrpc":"2.0","id":2,"result":{"tools":[{"name":"self.get_device_status","description":"Provides the real-time information of the device, including the current status of the audio speaker, screen, battery, network, etc.\nUse this tool for: \n1. Answering questions about current condition (e.g. what is the current volume of the audio speaker?)\n2. As the first step to control the device (e.g. turn up / down the volume of the audio speaker, etc.)","inputSchema":{"type":"object","properties":{}}},{"name":"self.audio_speaker.set_volume","description":"Set the volume of the audio speaker. If the current volume is unknown, you must call `self.get_device_status` tool first and then call this tool.","inputSchema":{"type":"object","properties":{"volume":{"type":"integer","minimum":0,"maximum":100}},"required":["volume"]}},{"name":"self.screen.set_brightness","description":"Set the brightness of the screen.","inputSchema":{"type":"object","properties":{"brightness":{"type":"integer","minimum":0,"maximum":100}},"required":["brightness"]}},{"name":"self.screen.set_theme","description":"Set the theme of the screen. The theme can be 'light' or 'dark'.","inputSchema":{"type":"object","properties":{"theme":{"type":"string"}},"required":["theme"]}}]}}})
+                                        websocket.send(replay_message);
+                                        log(`回复MCP消息: ${replay_message}`, 'info');
+                                    } else if(payload.method === 'tools/call'){
+                                        // 模拟回复
+                                        const replay_message = JSON.stringify({"session_id":"9f261599","type":"mcp","payload":{"jsonrpc":"2.0","id": payload.id,"result":{"content":[{"type":"text","text":"true"}],"isError":false}}})
+                                        websocket.send(replay_message);
+                                        log(`回复MCP消息: ${replay_message}`, 'info');
+                                    }
+                                }
+                                
                             } else {
                                 // 未知消息类型
                                 log(`未知消息类型: ${message.type}`, 'info');
@@ -1523,7 +1540,10 @@
                     device_id: config.deviceId,
                     device_name: config.deviceName,
                     device_mac: config.deviceMac,
-                    token: config.token
+                    token: config.token,
+                    features: {
+                        mcp: true
+                    }
                 };
 
                 log('发送hello握手消息', 'info');