Merge pull request #1438 from xinnan-tech/vllm

udpate:添加拍照识物教程

README.md

@@ -121,6 +121,33 @@
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="实时打断" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="拍照识物品" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="多指令任务" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -141,11 +168,10 @@
 本项目提供两种部署方式，请根据您的具体需求选择：
 
 #### 🚀 部署方式选择
-
+| 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | 
-| 部署方式 | 特点 | 适用场景 | Docker部署文档 | 源码部署文档 | 视频教程 | 
 |---------|------|---------|---------|---------|---------|
-| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [Docker只运行Server](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [本地源码只运行Server](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| - | 
+| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
-| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[Docker运行全模块](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [本地源码运行全模块](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [本地源码视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
+| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
 
 > 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
@@ -158,9 +184,23 @@ OTA接口地址: https://2662r3426b.vicp.fun/xiaozhi/ota/
 Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ```
 
+#### 🚩 配置说明和推荐
+> [!Note]
+> 本项目默认的配置是`入门全免费`设置，如果想效果更优，推荐使用`全流式配置`。
+> 
+> 本项目自`0.5.2`版本，已支持整个生命周期全流式，相比`0.5`版本以前，响应速度提升约`2.5秒`
+
+
+| 模块名称 | 入门全免费设置 | 全流式配置 |
+|---------|---------|------|
+| ASR(语音识别) | FunASR(本地) | ✅DoubaoASR(火山流式语音识别) |
+| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | ✅DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
+| TTS(语音合成) | EdgeTTS(微软语音) | ✅HuoshanDoubleStreamTTS(火山双流式语音合成) |
+| Intent(意图识别) | function_call(函数调用) | ✅function_call(函数调用) |
+| Memory(记忆功能) | mem_local_short(本地短期记忆） | ✅mem_local_short（本地短期记忆） |
+
 ---
 ## 功能清单 ✨
-
 ### 已实现 ✅
 
 | 功能模块 | 描述 |

docs/FAQ.md

@@ -108,7 +108,11 @@ VAD:
 
 参考教程[阿里云短信集成指南](./ali-sms-integration.md)
 
-### 9、更多问题，可联系我们反馈 💬
+### 9、如何开启视觉模型实现拍照识物 📷
+
+参考教程[视觉模型使用指南](./mcp-vision-integration.md)
+
+### 10、更多问题，可联系我们反馈 💬
 
 可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
 

docs/mcp-vision-integration.md

@@ -0,0 +1,171 @@
+# 视觉模型使用指南
+本教程分为两部分：
+- 第一部分：单模块运行xiaozhi-server开启视觉模型
+- 第二部分：全模块运行时，如何开启视觉模型
+
+开启视觉模型前，你需要准备三件事：
+- 你需要准备一台带摄像头的设备，而且这台设备已经在虾哥仓库里，实现了调用摄像头功能。例如`立创·实战派ESP32-S3开发板`
+- 你设备固件的版本升级到1.6.6及以上
+- 你已经成功跑通基础对话模块
+
+## 单模块运行xiaozhi-server开启视觉模型
+
+### 第一步确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose.yml`是否放了`8003`端口，如果没有就更新最新的`docker-compose.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步选择你的视觉模型
+打开你的`data/.config.yaml`文件，设置你的`selected_module.VLLM`设置为某个视觉模型。目前我们已经支持`openai`类型接口的视觉模型。`ChatGLMVLLM`就是其中一款兼容`openai`的模型。
+
+```
+selected_module:
+  VAD: ..
+  ASR: ..
+  LLM: ..
+  VLLM: ChatGLMVLLM
+  TTS: ..
+  Memory: ..
+  Intent: ..
+```
+
+假设我们使用`ChatGLMVLLM`作为视觉模型，那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+在你的配置文件中，增加这个配置，如果已经有了这个配置，就设置好你的api_key。
+
+```
+VLLM:
+  ChatGLMVLLM:
+    api_key: 你的api_key
+```
+
+### 第三步启动xiaozhi-server服务
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - OTA接口是           http://192.168.4.7:8003/xiaozhi/ota/
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。
+
+
+## 全模块运行时，如何开启视觉模型
+
+### 第一步 确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose_all.yml`是否映射了`8003`端口，如果没有就更新最新的`docker-compose_all.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步 确认你配置文件
+
+打开你的`data/.config.yaml`文件，确认一下你的配置文件的结构，是否和`data/config_from_api.yaml`一样。如果不一样，或缺少某项，请补齐。
+
+### 第三步 配置视觉模型密钥
+
+那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+登录`智控台`，顶部菜单点击`模型配置`，在左侧栏点击`视觉打语言模型`，找到`VLLM_ChatGLMVLLM`，点击修改按钮，在弹框中，在`API密钥`输入你密钥，点击保存。
+
+保存成功后，去到你需要测试的智能体哪里，点击`配置角色`，在打开的内容里，查看`视觉大语言模型(VLLM)`是否选择了刚才的视觉模型。点击保存。
+
+### 第三步 启动xiaozhi-server模块
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。

main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java

@@ -227,7 +227,7 @@ public interface Constant {
     /**
      * 版本号
      */
-    public static final String VERSION = "0.5.1";
+    public static final String VERSION = "0.5.2";
 
     /**
      * 无效固件URL

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentDTO.java

@@ -27,6 +27,9 @@ public class AgentDTO {
     @Schema(description = "大语言模型名称", example = "llm_model_01")
     private String llmModelName;
 
+    @Schema(description = "视觉模型名称", example = "vllm_model_01")
+    private String vllmModelName;
+
     @Schema(description = "记忆模型ID", example = "mem_model_01")
     private String memModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentEntity.java

@@ -36,6 +36,9 @@ public class AgentEntity {
     @Schema(description = "大语言模型标识")
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识")
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识")
     private String ttsModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentTemplateEntity.java

@@ -49,6 +49,11 @@ public class AgentTemplateEntity implements Serializable {
      */
     private String llmModelId;
 
+    /**
+     * VLLM模型标识
+     */
+    private String vllmModelId;
+
     /**
      * 语音合成模型标识
      */

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentServiceImpl.java

@@ -102,6 +102,9 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
             // 获取 LLM 模型名称
             dto.setLlmModelName(modelConfigService.getModelNameById(agent.getLlmModelId()));
 
+            // 获取 VLLM 模型名称
+            dto.setVllmModelName(modelConfigService.getModelNameById(agent.getVllmModelId()));
+
             // 获取记忆模型名称
             dto.setMemModelId(agent.getMemModelId());
 

main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java

@@ -72,6 +72,7 @@ public class ConfigServiceImpl implements ConfigService {
                 null,
                 null,
                 null,
+                null,
                 result,
                 isCache);
 
@@ -140,6 +141,7 @@ public class ConfigServiceImpl implements ConfigService {
                 agent.getVadModelId(),
                 agent.getAsrModelId(),
                 agent.getLlmModelId(),
+                agent.getVllmModelId(),
                 agent.getTtsModelId(),
                 agent.getMemModelId(),
                 agent.getIntentModelId(),
@@ -241,6 +243,7 @@ public class ConfigServiceImpl implements ConfigService {
             String vadModelId,
             String asrModelId,
             String llmModelId,
+            String vllmModelId,
             String ttsModelId,
             String memModelId,
             String intentModelId,
@@ -248,8 +251,8 @@ public class ConfigServiceImpl implements ConfigService {
             boolean isCache) {
         Map<String, String> selectedModule = new HashMap<>();
 
-        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM" };
+        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM", "VLLM" };
-        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId };
+        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId, vllmModelId };
         String intentLLMModelId = null;
         String memLocalShortLLMModelId = null;
 

main/manager-api/src/main/resources/db/changelog/202506010920.sql

@@ -0,0 +1,29 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_VLLM_openai';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_VLLM_openai', 'VLLM', 'openai', 'OpenAI接口', '[{"key":"base_url","label":"基础URL","type":"string"},{"key":"model_name","label":"模型名称","type":"string"},{"key":"api_key","label":"API密钥","type":"string"}]', 9, 1, NOW(), 1, NOW());
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_ChatGLMVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_ChatGLMVLLM', 'VLLM', 'ChatGLMVLLM', '智谱视觉AI', 1, 1, '{\"type\": \"openai\", \"model_name\": \"glm-4v-flash\", \"base_url\": \"https://open.bigmodel.cn/api/paas/v4/\", \"api_key\": \"你的api_key\"}', NULL, NULL, 1, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bigmodel.cn/usercenter/proj-mgmt/apikeys',
+`remark` = '智谱视觉AI配置说明：
+1. 访问 https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_ChatGLMVLLM';
+
+
+-- 添加参数
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (113, 'server.http_port', '8003', 'number', 1, 'http服务的端口，用于启动视觉分析接口');
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (114, 'server.vision_explain', 'null', 'string', 1, '视觉分析接口地址，用于下发到设备，多个用;分隔');
+
+-- 智能体表增加VLLM模型配置
+ALTER TABLE `ai_agent` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;
+
+-- 智能体模版表增加VLLM模型配置
+ALTER TABLE `ai_agent_template` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;

main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml

@@ -170,3 +170,10 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202505271414.sql
+  - changeSet:
+      id: 202506010920
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506010920.sql

main/manager-web/src/components/DeviceItem.vue

@@ -14,10 +14,10 @@
       </div>
     </div>
     <div class="device-name">
-      设备型号：{{ device.ttsModelName }}
+      语言模型：{{ device.llmModelName }}
     </div>
     <div class="device-name">
-      音色模型：{{ device.ttsVoiceName }}
+      音色模型：{{ device.ttsModelName }} ({{ device.ttsVoiceName }})
     </div>
     <div style="display: flex;gap: 10px;align-items: center;">
       <div class="settings-btn" @click="handleConfigure">

main/manager-web/src/views/ModelConfig.vue

@@ -30,6 +30,9 @@
           <el-menu-item index="llm">
             <span class="menu-text">大语言模型</span>
           </el-menu-item>
+          <el-menu-item index="vllm">
+            <span class="menu-text">视觉大语言模型</span>
+          </el-menu-item>
           <el-menu-item index="intent">
             <span class="menu-text">意图识别</span>
           </el-menu-item>
@@ -173,6 +176,7 @@ export default {
         vad: '语言活动检测模型(VAD)',
         asr: '语音识别模型(ASR)',
         llm: '大语言模型（LLM）',
+        vllm: '视觉大语言模型（VLLM）',
         intent: '意图识别模型(Intent)',
         tts: '语音合成模型(TTS)',
         memory: '记忆模型(Memory)'

main/manager-web/src/views/roleConfig.vue

@@ -64,7 +64,27 @@
                     </el-form-item>
                   </div>
                   <div class="form-column">
-                    <el-form-item v-for="(model, index) in models" :key="`model-${index}`" :label="model.label"
+                    <div class="model-row">
+                      <el-form-item label="语音活动检测(VAD)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.vadModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('VAD', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['VAD']"
+                              :key="`option-vad-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                      <el-form-item label="语音识别(ASR)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.asrModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('ASR', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['ASR']"
+                              :key="`option-asr-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                    </div>
+                    <el-form-item v-for="(model, index) in models.slice(2)" :key="`model-${index}`" :label="model.label"
                       class="model-item">
                       <div class="model-select-wrapper">
                         <el-select v-model="form.model[model.key]" filterable placeholder="请选择" class="form-select"
@@ -148,6 +168,7 @@ export default {
           vadModelId: "",
           asrModelId: "",
           llmModelId: "",
+          vllmModelId: "",
           memModelId: "",
           intentModelId: "",
         }
@@ -156,6 +177,7 @@ export default {
         { label: '语音活动检测(VAD)', key: 'vadModelId', type: 'VAD' },
         { label: '语音识别(ASR)', key: 'asrModelId', type: 'ASR' },
         { label: '大语言模型(LLM)', key: 'llmModelId', type: 'LLM' },
+        { label: '视觉大语言模型(VLLM)', key: 'vllmModelId', type: 'VLLM' },
         { label: '意图识别(Intent)', key: 'intentModelId', type: 'Intent' },
         { label: '记忆(Memory)', key: 'memModelId', type: 'Memory' },
         { label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' },
@@ -189,6 +211,7 @@ export default {
         asrModelId: this.form.model.asrModelId,
         vadModelId: this.form.model.vadModelId,
         llmModelId: this.form.model.llmModelId,
+        vllmModelId: this.form.model.vllmModelId,
         ttsModelId: this.form.model.ttsModelId,
         ttsVoiceId: this.form.ttsVoiceId,
         chatHistoryConf: this.form.chatHistoryConf,
@@ -236,6 +259,7 @@ export default {
             vadModelId: "",
             asrModelId: "",
             llmModelId: "",
+            vllmModelId: "",
             memModelId: "",
             intentModelId: "",
           }
@@ -289,6 +313,7 @@ export default {
           vadModelId: templateData.vadModelId || this.form.model.vadModelId,
           asrModelId: templateData.asrModelId || this.form.model.asrModelId,
           llmModelId: templateData.llmModelId || this.form.model.llmModelId,
+          vllmModelId: templateData.vllmModelId || this.form.model.vllmModelId,
           memModelId: templateData.memModelId || this.form.model.memModelId,
           intentModelId: templateData.intentModelId || this.form.model.intentModelId
         }
@@ -305,6 +330,7 @@ export default {
               vadModelId: data.data.vadModelId,
               asrModelId: data.data.asrModelId,
               llmModelId: data.data.llmModelId,
+              vllmModelId: data.data.vllmModelId,
               memModelId: data.data.memModelId,
               intentModelId: data.data.intentModelId
             }
@@ -587,6 +613,25 @@ export default {
   width: 100%;
 }
 
+.model-row {
+  display: flex;
+  gap: 20px;
+  margin-bottom: 6px;
+}
+
+.model-row .model-item {
+  flex: 1;
+  margin-bottom: 0;
+}
+
+.model-row .el-form-item__label {
+  font-size: 12px !important;
+  color: #3d4566 !important;
+  font-weight: 400;
+  line-height: 22px;
+  padding-bottom: 2px;
+}
+
 .function-icons {
   display: flex;
   align-items: center;

main/xiaozhi-server/app.py

@@ -1,13 +1,14 @@
-import asyncio
 import sys
+import uuid
 import signal
+import asyncio
+from aioconsole import ainput
 from config.settings import load_config
-from core.websocket_server import WebSocketServer
-from core.ota_server import SimpleOtaServer
-from core.utils.util import check_ffmpeg_installed
 from config.logger import setup_logging
 from core.utils.util import get_local_ip
-from aioconsole import ainput
+from core.http_server import SimpleHttpServer
+from core.websocket_server import WebSocketServer
+from core.utils.util import check_ffmpeg_installed
 
 TAG = __name__
 logger = setup_logging()
@@ -45,25 +46,32 @@ async def main():
     check_ffmpeg_installed()
     config = load_config()
 
+    # 生成随机密钥并添加到配置中
+    config["server"]["auth_key"] = str(uuid.uuid4().hex)
+
     # 添加 stdin 监控任务
     stdin_task = asyncio.create_task(monitor_stdin())
 
     # 启动 WebSocket 服务器
     ws_server = WebSocketServer(config)
     ws_task = asyncio.create_task(ws_server.start())
-    ota_task = None
+    # 启动 Simple http 服务器
+    ota_server = SimpleHttpServer(config)
+    ota_task = asyncio.create_task(ota_server.start())
 
     read_config_from_api = config.get("read_config_from_api", False)
+    port = int(config["server"].get("http_port", 8003))
     if not read_config_from_api:
-        # 启动 Simple OTA 服务器
-        ota_server = SimpleOtaServer(config)
-        ota_task = asyncio.create_task(ota_server.start())
-
         logger.bind(tag=TAG).info(
             "OTA接口是\t\thttp://{}:{}/xiaozhi/ota/",
             get_local_ip(),
-            config["server"]["ota_port"],
+            port,
         )
+    logger.bind(tag=TAG).info(
+        "视觉分析接口是\thttp://{}:{}/mcp/vision/explain",
+        get_local_ip(),
+        port,
+    )
 
     # 获取WebSocket配置，使用安全的默认值
     websocket_port = 8000

main/xiaozhi-server/config.yaml

@@ -2,6 +2,7 @@
 # 然后你想修改覆盖修改什么配置，就修改【.config.yaml】文件，而不是修改【config.yaml】文件
 # 系统会优先读取【data/.config.yaml】文件的配置，如果【.config.yaml】文件里的配置不存在，系统会自动去读取【config.yaml】文件的配置。
 # 这样做，可以最简化配置，保护您的密钥安全。
+# 如果你使用了智控台，那么以下所有配置，都不会生效，请在智控台中修改配置
 
 # #####################################################################################
 # #############################以下是服务器基本运行配置####################################
@@ -9,13 +10,21 @@ server:
   # 服务器监听地址和端口(Server listening address and port)
   ip: 0.0.0.0
   port: 8000
-  # OTA接口的端口号
+  # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-  ota_port: 8002
+  http_port: 8003
   # 这个websocket配置是指ota接口向设备发送的websocket地址
-  # 如果按默认的写法，ota接口会自动生成websocket地址。这个地址你可以直接用浏览器访问ota接口确认一下
+  # 如果按默认的写法，ota接口会自动生成websocket地址，并输出在启动日志里，这个地址你可以直接用浏览器访问ota接口确认一下
   # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
-  # 所以如果你使用docker部署或使用公网部署时，请设置正确的websocket地址
+  # 所以如果你使用docker部署时，将websocket设置成局域网地址
+  # 如果你使用公网部署时，将vwebsocket设置成公网地址
   websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
   # OTA返回信息时区偏移量
   timezone_offset: +8
   # 认证配置
@@ -160,6 +169,8 @@ selected_module:
   ASR: FunASR
   # 将根据配置名称对应的type调用实际的LLM适配器
   LLM: ChatGLMLLM
+  # 视觉语言大模型
+  VLLM: ChatGLMVLLM
   # TTS将根据配置名称对应的type调用实际的TTS适配器
   TTS: EdgeTTS
   # 记忆模块，默认不开启记忆；如果想使用超长记忆，推荐使用mem0ai；如果注重隐私，请使用本地的mem_local_short
@@ -432,6 +443,15 @@ LLM:
     # Xinference服务地址和模型名称
     model_name: qwen2.5:3b-AWQ  # 使用的小模型名称，用于意图识别
     base_url: http://localhost:9997  # Xinference服务地址
+# VLLM配置（视觉语言大模型）
+VLLM:
+  ChatGLMVLLM:
+    type: openai
+    # glm-4v-flash是智谱免费AI的视觉模型，需要先在智谱AI平台创建API密钥并获取api_key
+    # 可在这里找到你的api key https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+    model_name: glm-4v-flash  # 智谱AI的视觉模型
+    url: https://open.bigmodel.cn/api/paas/v4/
+    api_key: 你的api_key
 TTS:
   # 当前支持的type为edge、doubao，可自行适配
   EdgeTTS:

main/xiaozhi-server/config/config_loader.py

@@ -59,10 +59,14 @@ def get_config_from_api(config):
         "url": config["manager-api"].get("url", ""),
         "secret": config["manager-api"].get("secret", ""),
     }
+    # server的配置以本地为准
     if config.get("server"):
         config_data["server"] = {
             "ip": config["server"].get("ip", ""),
             "port": config["server"].get("port", ""),
+            "http_port": config["server"].get("http_port", ""),
+            "vision_explain": config["server"].get("vision_explain", ""),
+            "auth_key": config["server"].get("auth_key", ""),
         }
     return config_data
 

main/xiaozhi-server/config/logger.py

@@ -4,7 +4,7 @@ from loguru import logger
 from config.config_loader import load_config
 from config.settings import check_config_file
 
-SERVER_VERSION = "0.5.1"
+SERVER_VERSION = "0.5.2"
 _logger_initialized = False
 
 
@@ -48,7 +48,11 @@ def setup_logging():
 
     # 第一次初始化时配置日志
     if not _logger_initialized:
-        logger.configure(extra={"selected_module": log_config.get("selected_module", "00000000000000")})  # 新增配置
+        logger.configure(
+            extra={
+                "selected_module": log_config.get("selected_module", "00000000000000")
+            }
+        )  # 新增配置
         log_format = log_config.get(
             "log_format",
             "<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
@@ -62,7 +66,9 @@ def setup_logging():
         log_format = log_format.replace("{version}", SERVER_VERSION)
         log_format = log_format.replace("{selected_module}", selected_module_str)
         log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace("{selected_module}", selected_module_str)
+        log_format_file = log_format_file.replace(
+            "{selected_module}", selected_module_str
+        )
 
         log_level = log_config.get("log_level", "INFO")
         log_dir = log_config.get("log_dir", "tmp")
@@ -116,15 +122,25 @@ def update_module_string(selected_module_str):
         log_format = log_format.replace("{version}", SERVER_VERSION)
         log_format = log_format.replace("{selected_module}", selected_module_str)
         log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace("{selected_module}", selected_module_str)
+        log_format_file = log_format_file.replace(
+            "{selected_module}", selected_module_str
+        )
 
         logger.remove()
-        logger.add(sys.stdout, format=log_format, level=log_config.get("log_level", "INFO"), filter=formatter)
         logger.add(
-            os.path.join(log_config.get("log_dir", "tmp"), log_config.get("log_file", "server.log")),
+            sys.stdout,
+            format=log_format,
+            level=log_config.get("log_level", "INFO"),
+            filter=formatter,
+        )
+        logger.add(
+            os.path.join(
+                log_config.get("log_dir", "tmp"),
+                log_config.get("log_file", "server.log"),
+            ),
             format=log_format_file,
             level=log_config.get("log_level", "INFO"),
-            filter=formatter
+            filter=formatter,
         )
 
     except Exception as e:

main/xiaozhi-server/config_from_api.yaml

@@ -8,6 +8,15 @@
 server:
   ip: 0.0.0.0
   port: 8000
+  # http服务的端口，用于视觉分析接口
+  http_port: 8003
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
 manager-api:
   # 你的manager-api的地址，最好使用局域网ip
   url: http://127.0.0.1:8002/xiaozhi

main/xiaozhi-server/core/api/base_handler.py

@@ -0,0 +1,16 @@
+from aiohttp import web
+from config.logger import setup_logging
+
+
+class BaseHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/api/ota_handler.py

@@ -1,18 +1,15 @@
 import json
 import time
-import asyncio
 from aiohttp import web
-from config.logger import setup_logging
+from core.utils.util import get_local_ip
-from core.connection import ConnectionHandler
+from core.api.base_handler import BaseHandler
-from core.utils.util import get_local_ip, initialize_modules
 
 TAG = __name__
 
 
-class SimpleOtaServer:
+class OTAHandler(BaseHandler):
     def __init__(self, config: dict):
-        self.config = config
+        super().__init__(config)
-        self.logger = setup_logging()
 
     def _get_websocket_url(self, local_ip: str, port: int) -> str:
         """获取websocket地址
@@ -25,41 +22,15 @@ class SimpleOtaServer:
             str: websocket地址
         """
         server_config = self.config["server"]
-        websocket_config = server_config.get("websocket")
+        websocket_config = server_config.get("websocket", "")
 
-        if websocket_config and "你" not in websocket_config:
+        if "你的" not in websocket_config:
             return websocket_config
         else:
             return f"ws://{local_ip}:{port}/xiaozhi/v1/"
 
-    async def start(self):
+    async def handle_post(self, request):
-        server_config = self.config["server"]
+        """处理 OTA POST 请求"""
-        host = server_config.get("ip", "0.0.0.0")
-        port = int(server_config.get("ota_port"))
-
-        if port:
-            app = web.Application()
-            # 添加路由
-            app.add_routes(
-                [
-                    web.get("/xiaozhi/ota/", self._handle_ota_get_request),
-                    web.post("/xiaozhi/ota/", self._handle_ota_request),
-                    web.options("/xiaozhi/ota/", self._handle_ota_request),
-                ]
-            )
-
-            # 运行服务
-            runner = web.AppRunner(app)
-            await runner.setup()
-            site = web.TCPSite(runner, host, port)
-            await site.start()
-
-            # 保持服务运行
-            while True:
-                await asyncio.sleep(3600)  # 每隔 1 小时检查一次
-
-    async def _handle_ota_request(self, request):
-        """处理 /xiaozhi/ota/ 的 POST 请求"""
         try:
             data = await request.text()
             self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}")
@@ -75,11 +46,9 @@ class SimpleOtaServer:
             data_json = json.loads(data)
 
             server_config = self.config["server"]
-            host = server_config.get("ip", "0.0.0.0")
             port = int(server_config.get("port", 8000))
             local_ip = get_local_ip()
 
-            # OTA基础信息
             return_json = {
                 "server_time": {
                     "timestamp": int(round(time.time() * 1000)),
@@ -104,16 +73,11 @@ class SimpleOtaServer:
                 content_type="application/json",
             )
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response
 
-    async def _handle_ota_get_request(self, request):
+    async def handle_get(self, request):
-        """处理 /xiaozhi/ota/ 的 GET 请求"""
+        """处理 OTA GET 请求"""
         try:
             server_config = self.config["server"]
             local_ip = get_local_ip()
@@ -125,10 +89,5 @@ class SimpleOtaServer:
             self.logger.bind(tag=TAG).error(f"OTA GET请求异常: {e}")
             response = web.Response(text="OTA接口异常", content_type="text/plain")
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response

main/xiaozhi-server/core/api/vision_handler.py

@@ -0,0 +1,181 @@
+import json
+import copy
+from aiohttp import web
+from config.logger import setup_logging
+from core.utils.util import get_vision_url, is_valid_image_file
+from core.utils.vllm import create_instance
+from config.config_loader import get_private_config_from_api
+from core.utils.auth import AuthToken
+import base64
+from typing import Tuple, Optional
+
+TAG = __name__
+
+# 设置最大文件大小为5MB
+MAX_FILE_SIZE = 5 * 1024 * 1024
+
+
+class VisionHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        # 初始化认证工具
+        self.auth = AuthToken(config["server"]["auth_key"])
+
+    def _create_error_response(self, message: str) -> dict:
+        """创建统一的错误响应格式"""
+        return {"success": False, "message": message}
+
+    def _verify_auth_token(self, request) -> Tuple[bool, Optional[str]]:
+        """验证认证token"""
+        auth_header = request.headers.get("Authorization", "")
+        if not auth_header.startswith("Bearer "):
+            return False, None
+
+        token = auth_header[7:]  # 移除"Bearer "前缀
+        return self.auth.verify_token(token)
+
+    async def handle_post(self, request):
+        """处理 MCP Vision POST 请求"""
+        try:
+            # 验证token
+            is_valid, token_device_id = self._verify_auth_token(request)
+            if not is_valid:
+                return web.Response(
+                    text=json.dumps(
+                        self._create_error_response("无效的认证token或token已过期")
+                    ),
+                    content_type="application/json",
+                    status=401,
+                )
+
+            # 获取请求头信息
+            device_id = request.headers.get("Device-Id", "")
+            client_id = request.headers.get("Client-Id", "")
+            if device_id != token_device_id:
+                return web.Response(
+                    text=json.dumps(self._create_error_response("设备ID与token不匹配")),
+                    content_type="application/json",
+                    status=401,
+                )
+            # 解析multipart/form-data请求
+            reader = await request.multipart()
+
+            # 读取question字段
+            question_field = await reader.next()
+            if question_field is None:
+                raise ValueError("缺少问题字段")
+            question = await question_field.text()
+            self.logger.bind(tag=TAG).debug(f"Question: {question}")
+
+            # 读取图片文件
+            image_field = await reader.next()
+            if image_field is None:
+                raise ValueError("缺少图片文件")
+
+            # 读取图片数据
+            image_data = await image_field.read()
+            if not image_data:
+                raise ValueError("图片数据为空")
+
+            # 检查文件大小
+            if len(image_data) > MAX_FILE_SIZE:
+                raise ValueError(
+                    f"图片大小超过限制，最大允许{MAX_FILE_SIZE/1024/1024}MB"
+                )
+
+            # 检查文件格式
+            if not is_valid_image_file(image_data):
+                raise ValueError(
+                    "不支持的文件格式，请上传有效的图片文件（支持JPEG、PNG、GIF、BMP、TIFF、WEBP格式）"
+                )
+
+            # 将图片转换为base64编码
+            image_base64 = base64.b64encode(image_data).decode("utf-8")
+
+            # 如果开启了智控台，则从智控台获取模型配置
+            current_config = copy.deepcopy(self.config)
+            read_config_from_api = current_config.get("read_config_from_api", False)
+            if read_config_from_api:
+                current_config = get_private_config_from_api(
+                    current_config,
+                    device_id,
+                    client_id,
+                )
+
+            select_vllm_module = current_config["selected_module"].get("VLLM")
+            if not select_vllm_module:
+                raise ValueError("您还未设置默认的视觉分析模块")
+
+            vllm_type = (
+                select_vllm_module
+                if "type" not in current_config["VLLM"][select_vllm_module]
+                else current_config["VLLM"][select_vllm_module]["type"]
+            )
+
+            if not vllm_type:
+                raise ValueError(f"无法找到VLLM模块对应的供应器{vllm_type}")
+
+            vllm = create_instance(
+                vllm_type, current_config["VLLM"][select_vllm_module]
+            )
+
+            response = vllm.response(question, image_base64)
+
+            return_json = {
+                "success": True,
+                "result": response,
+            }
+
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except ValueError as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response(str(e))
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response("处理请求时发生错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            self._add_cors_headers(response)
+            return response
+
+    async def handle_get(self, request):
+        """处理 MCP Vision GET 请求"""
+        try:
+            vision_explain = get_vision_url(self.config)
+            if vision_explain and len(vision_explain) > 0 and "null" != vision_explain:
+                message = (
+                    f"MCP Vision 接口运行正常，视觉解释接口地址是：{vision_explain}"
+                )
+            else:
+                message = "MCP Vision 接口运行不正常，请打开data目录下的.config.yaml文件，找到【server.vision_explain】，设置好地址"
+
+            response = web.Response(text=message, content_type="text/plain")
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision GET请求异常: {e}")
+            return_json = self._create_error_response("服务器内部错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            self._add_cors_headers(response)
+            return response
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/connection.py

@@ -10,20 +10,25 @@ import threading
 import traceback
 import subprocess
 import websockets
+from core.handle.mcpHandle import call_mcp_tool
 from core.utils.util import (
     extract_json_from_string,
-    initialize_modules,
     check_vad_update,
     check_asr_update,
     filter_sensitive_info,
-    initialize_tts,
 )
 from typing import Dict, Any
 from core.mcp.manager import MCPManager
+from core.utils.modules_initialize import (
+    initialize_modules,
+    initialize_tts,
+    initialize_asr,
+)
 from core.handle.reportHandle import report
 from core.providers.tts.default import DefaultTTS
 from concurrent.futures import ThreadPoolExecutor
 from core.utils.dialogue import Message, Dialogue
+from core.providers.asr.dto.dto import InterfaceType
 from core.handle.textHandle import handleTextMessage
 from core.handle.functionHandler import FunctionHandler
 from plugins_func.loadplugins import auto_import_modules
@@ -79,6 +84,7 @@ class ConnectionHandler:
 
         # 客户端状态相关
         self.client_abort = False
+        self.client_is_speaking = False
         self.client_listen_mode = "auto"
 
         # 线程任务相关
@@ -112,7 +118,6 @@ class ConnectionHandler:
 
         # asr相关变量
         self.asr_audio = []
-        self.asr_server_receive = True
 
         # llm相关变量
         self.llm_finish_task = True
@@ -142,6 +147,8 @@ class ConnectionHandler:
         )  # 在原来第一道关闭的基础上加60秒，进行二道关闭
 
         self.audio_format = "opus"
+        # {"mcp":true} 表示启用MCP功能
+        self.features = None
 
     async def handle_connection(self, ws):
         try:
@@ -315,10 +322,14 @@ class ConnectionHandler:
             if self.vad is None:
                 self.vad = self._vad
             if self.asr is None:
-                self.asr = self._asr
+                self.asr = self._initialize_asr()
+            # 打开语音识别通道
+            asyncio.run_coroutine_threadsafe(
+                self.asr.open_audio_channels(self), self.loop
+            )
             if self.tts is None:
                 self.tts = self._initialize_tts()
-            # 使用事件循环运行异步方法
+            # 打开语音合成通道
             asyncio.run_coroutine_threadsafe(
                 self.tts.open_audio_channels(self), self.loop
             )
@@ -356,6 +367,19 @@ class ConnectionHandler:
 
         return tts
 
+    def _initialize_asr(self):
+        """初始化ASR"""
+        if self._asr.interface_type == InterfaceType.LOCAL:
+            # 如果公共ASR是本地服务，则直接返回
+            # 因为本地一个实例ASR，可以被多个连接共享
+            asr = self._asr
+        else:
+            # 如果公共ASR是远程服务，则初始化一个新实例
+            # 因为远程ASR，涉及到websocket连接和接收线程，需要每个连接一个实例
+            asr = initialize_asr(self.config)
+
+        return asr
+
     def _initialize_private_config(self):
         """如果是从配置文件获取，则进行二次实例化"""
         if not self.read_config_from_api:
@@ -454,6 +478,8 @@ class ConnectionHandler:
             self.memory = modules["memory"]
 
     def _initialize_memory(self):
+        if self.memory is None:
+            return
         """初始化记忆模块"""
         self.memory.init_memory(
             role_id=self.device_id,
@@ -494,6 +520,8 @@ class ConnectionHandler:
                 self.logger.bind(tag=TAG).info("使用主LLM作为意图识别模型")
 
     def _initialize_intent(self):
+        if self.intent is None:
+            return
         self.intent_type = self.config["Intent"][
             self.config["selected_module"]["Intent"]
         ]["type"]
@@ -558,6 +586,12 @@ class ConnectionHandler:
         functions = None
         if self.intent_type == "function_call" and hasattr(self, "func_handler"):
             functions = self.func_handler.get_functions()
+        if hasattr(self, "mcp_client"):
+            mcp_tools = self.mcp_client.get_available_tools()
+            if mcp_tools is not None and len(mcp_tools) > 0:
+                if functions is None:
+                    functions = []
+                functions.extend(mcp_tools)
         response_message = []
 
         try:
@@ -595,8 +629,10 @@ class ConnectionHandler:
         function_arguments = ""
         content_arguments = ""
         text_index = 0
-
+        self.client_abort = False
         for response in llm_responses:
+            if self.client_abort:
+                break
             if functions is not None:
                 content, tools_call = response
                 if "content" in response:
@@ -622,9 +658,6 @@ class ConnectionHandler:
             if content is not None and len(content) > 0:
                 if not tool_call_flag:
                     response_message.append(content)
-                    if self.client_abort:
-                        break
-
                     if text_index == 0:
                         self.tts.tts_text_queue.put(
                             TTSMessageDTO(
@@ -676,9 +709,32 @@ class ConnectionHandler:
                     "arguments": function_arguments,
                 }
 
-                # 处理MCP工具调用
+                # 处理Server端MCP工具调用
                 if self.mcp_manager.is_mcp_tool(function_name):
                     result = self._handle_mcp_tool_call(function_call_data)
+                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(
+                    function_name
+                ):
+                    # 如果是小智端MCP工具调用
+                    self.logger.bind(tag=TAG).debug(
+                        f"调用小智端MCP工具: {function_name}, 参数: {function_arguments}"
+                    )
+                    try:
+                        result = asyncio.run_coroutine_threadsafe(
+                            call_mcp_tool(
+                                self, self.mcp_client, function_name, function_arguments
+                            ),
+                            self.loop,
+                        ).result()
+                        self.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result=result, response=""
+                        )
+                    except Exception as e:
+                        self.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result="MCP工具调用失败", response=""
+                        )
                 else:
                     # 处理系统函数
                     result = self.func_handler.handle_llm_function_call(
@@ -801,9 +857,7 @@ class ConnectionHandler:
                 item = self.report_queue.get(timeout=1)
                 if item is None:  # 检测毒丸对象
                     break
-
                 type, text, audio_data, report_time = item
-
                 try:
                     # 检查线程池状态
                     if self.executor is None:
@@ -833,8 +887,8 @@ class ConnectionHandler:
             self.report_queue.task_done()
 
     def clearSpeakStatus(self):
+        self.client_is_speaking = False
         self.logger.bind(tag=TAG).debug(f"清除服务端讲话状态")
-        self.asr_server_receive = True
 
     async def close(self, ws=None):
         """资源清理方法"""

main/xiaozhi-server/core/handle/functionHandler.py

@@ -61,7 +61,7 @@ class FunctionHandler:
         self.function_registry.register_function("plugin_loader")
         self.function_registry.register_function("get_time")
         self.function_registry.register_function("get_lunar")
-        self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
+        # self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
 
     def register_config_functions(self):
         """注册配置中的函数,可以不同客户端使用不同的配置"""

main/xiaozhi-server/core/handle/helloHandle.py

@@ -7,6 +7,11 @@ import asyncio
 from core.handle.sendAudioHandle import send_stt_message
 from core.utils.util import remove_punctuation_and_length
 from core.providers.tts.dto.dto import ContentType, InterfaceType
+from core.handle.mcpHandle import (
+    MCPClient,
+    send_mcp_initialize_message,
+    send_mcp_tools_list_request,
+)
 
 
 TAG = __name__
@@ -31,6 +36,17 @@ async def handleHelloMessage(conn, msg_json):
         if conn.asr is not None:
             conn.asr.set_audio_format(format)
         conn.welcome_msg["audio_params"] = audio_params
+    features = msg_json.get("features")
+    if features:
+        conn.logger.bind(tag=TAG).info(f"客户端特性: {features}")
+        conn.features = features
+        if features.get("mcp"):
+            conn.logger.bind(tag=TAG).info("客户端支持MCP")
+            conn.mcp_client = MCPClient()
+            # 发送初始化
+            asyncio.create_task(send_mcp_initialize_message(conn))
+            # 发送mcp消息，获取tools列表
+            asyncio.create_task(send_mcp_tools_list_request(conn))
 
     await conn.websocket.send(json.dumps(conn.welcome_msg))
 
@@ -92,7 +108,13 @@ async def wakeupWordsResponse(conn):
 
     """唤醒词响应"""
     wakeup_word = random.choice(WAKEUP_CONFIG["words"])
-    result = conn.llm.response_no_stream(conn.config["prompt"], wakeup_word)
+    question = (
+        "此刻用户正在和你说```"
+        + wakeup_word
+        + "```。\n请你根据以上用户的内容，进行简短回复，文字内容控制在15个字以内。\n"
+        + "请勿对这条内容本身进行任何解释和回应，仅返回对用户的内容的回复。"
+    )
+    result = conn.llm.response_no_stream(conn.config["prompt"], question)
     if result is None or result == "":
         return
     tts_file = await asyncio.to_thread(conn.tts.to_tts, result)

main/xiaozhi-server/core/handle/intentHandler.py

@@ -1,4 +1,3 @@
-from config.logger import setup_logging
 import json
 import uuid
 from core.handle.sendAudioHandle import send_stt_message
@@ -146,4 +145,3 @@ async def process_intent_result(conn, intent_result, original_text):
 def speak_txt(conn, text):
     conn.tts.tts_one_sentence(conn, ContentType.TEXT, content_detail=text)
     conn.dialogue.put(Message(role="assistant", content=text))
- 

main/xiaozhi-server/core/handle/mcpHandle.py

@@ -0,0 +1,374 @@
+import json
+import asyncio
+from concurrent.futures import Future
+from core.utils.util import get_vision_url
+from core.utils.auth import AuthToken
+
+TAG = __name__
+
+
+class MCPClient:
+    """MCPClient，用于管理MCP状态和工具"""
+
+    def __init__(self):
+        self.tools = {}  # Dictionary for O(1) lookup
+        self.ready = False
+        self.call_results = {}  # To store Futures for tool call responses
+        self.next_id = 1
+        self.lock = asyncio.Lock()
+        self._cached_available_tools = None  # Cache for get_available_tools
+
+    def has_tool(self, name: str) -> bool:
+        return name in self.tools
+
+    def get_available_tools(self) -> list:
+        # Check if the cache is valid
+        if self._cached_available_tools is not None:
+            return self._cached_available_tools
+
+        # If cache is not valid, regenerate the list
+        result = []
+        for tool_name, tool_data in self.tools.items():
+            function_def = {
+                "name": tool_data["name"],
+                "description": tool_data["description"],
+                "parameters": {
+                    "type": tool_data["inputSchema"].get("type", "object"),
+                    "properties": tool_data["inputSchema"].get("properties", {}),
+                    "required": tool_data["inputSchema"].get("required", []),
+                },
+            }
+            result.append({"type": "function", "function": function_def})
+
+        self._cached_available_tools = result  # Store the generated list in cache
+        return result
+
+    async def is_ready(self) -> bool:
+        async with self.lock:
+            return self.ready
+
+    async def set_ready(self, status: bool):
+        async with self.lock:
+            self.ready = status
+
+    async def add_tool(self, tool_data: dict):
+        async with self.lock:
+            self.tools[tool_data["name"]] = tool_data
+            self._cached_available_tools = (
+                None  # Invalidate the cache when a tool is added
+            )
+
+    async def get_next_id(self) -> int:
+        async with self.lock:
+            current_id = self.next_id
+            self.next_id += 1
+            return current_id
+
+    async def register_call_result_future(self, id: int, future: Future):
+        async with self.lock:
+            self.call_results[id] = future
+
+    async def resolve_call_result(self, id: int, result: any):
+        async with self.lock:
+            if id in self.call_results:
+                future = self.call_results.pop(id)
+                if not future.done():
+                    future.set_result(result)
+
+    async def reject_call_result(self, id: int, exception: Exception):
+        async with self.lock:
+            if id in self.call_results:
+                future = self.call_results.pop(id)
+                if not future.done():
+                    future.set_exception(exception)
+
+    async def cleanup_call_result(self, id: int):
+        async with self.lock:
+            if id in self.call_results:
+                self.call_results.pop(id)
+
+
+async def send_mcp_message(conn, payload: dict):
+    """Helper to send MCP messages, encapsulating common logic."""
+    if not conn.features.get("mcp"):
+        conn.logger.bind(tag=TAG).warning("客户端不支持MCP，无法发送MCP消息")
+        return
+
+    message = json.dumps({"type": "mcp", "payload": payload})
+
+    try:
+        await conn.websocket.send(message)
+        conn.logger.bind(tag=TAG).info(f"成功发送MCP消息: {message}")
+    except Exception as e:
+        conn.logger.bind(tag=TAG).error(f"发送MCP消息失败: {e}")
+
+
+async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
+    """处理MCP消息,包括初始化、工具列表和工具调用响应等"""
+    conn.logger.bind(tag=TAG).info(f"处理MCP消息: {payload}")
+
+    if not isinstance(payload, dict):
+        conn.logger.bind(tag=TAG).error("MCP消息缺少payload字段或格式错误")
+        return
+
+    # Handle result
+    if "result" in payload:
+        result = payload["result"]
+        msg_id = int(payload.get("id", 0))
+
+        # Check for tool call response first
+        if msg_id in mcp_client.call_results:
+            conn.logger.bind(tag=TAG).debug(
+                f"收到工具调用响应，ID: {msg_id}, 结果: {result}"
+            )
+            await mcp_client.resolve_call_result(msg_id, result)
+            return
+
+        if msg_id == 1:  # mcpInitializeID
+            conn.logger.bind(tag=TAG).debug("收到MCP初始化响应")
+            server_info = result.get("serverInfo")
+            if isinstance(server_info, dict):
+                name = server_info.get("name")
+                version = server_info.get("version")
+                conn.logger.bind(tag=TAG).info(
+                    f"客户端MCP服务器信息: name={name}, version={version}"
+                )
+            await send_mcp_tools_list_request(
+                conn
+            )  # After initialization, request tool list
+            return
+
+        elif msg_id == 2:  # mcpToolsListID
+            conn.logger.bind(tag=TAG).debug("收到MCP工具列表响应")
+            if isinstance(result, dict) and "tools" in result:
+                tools_data = result["tools"]
+                if not isinstance(tools_data, list):
+                    conn.logger.bind(tag=TAG).error("工具列表格式错误")
+                    return
+
+                conn.logger.bind(tag=TAG).info(
+                    f"客户端设备支持的工具数量: {len(tools_data)}"
+                )
+
+                for i, tool in enumerate(tools_data):
+                    if not isinstance(tool, dict):
+                        continue
+
+                    name = tool.get("name", "")
+                    description = tool.get("description", "")
+                    input_schema = {"type": "object", "properties": {}, "required": []}
+
+                    if "inputSchema" in tool and isinstance(tool["inputSchema"], dict):
+                        schema = tool["inputSchema"]
+                        input_schema["type"] = schema.get("type", "object")
+                        input_schema["properties"] = schema.get("properties", {})
+                        input_schema["required"] = [
+                            s for s in schema.get("required", []) if isinstance(s, str)
+                        ]
+
+                    new_tool = {
+                        "name": name,
+                        "description": description,
+                        "inputSchema": input_schema,
+                    }
+                    await mcp_client.add_tool(new_tool)
+                    conn.logger.bind(tag=TAG).debug(f"客户端工具 #{i+1}: {name}")
+
+                next_cursor = result.get("nextCursor", "")
+                if next_cursor:
+                    conn.logger.bind(tag=TAG).info(
+                        f"有更多工具，nextCursor: {next_cursor}"
+                    )
+                    await send_mcp_tools_list_continue_request(conn, next_cursor)
+                else:
+                    await mcp_client.set_ready(True)
+                    conn.logger.bind(tag=TAG).info("所有工具已获取，MCP客户端准备就绪")
+            return
+
+    # Handle method calls (requests from the client)
+    elif "method" in payload:
+        method = payload["method"]
+        conn.logger.bind(tag=TAG).info(f"收到MCP客户端请求: {method}")
+
+    elif "error" in payload:
+        error_data = payload["error"]
+        error_msg = error_data.get("message", "未知错误")
+        conn.logger.bind(tag=TAG).error(f"收到MCP错误响应: {error_msg}")
+
+        msg_id = int(payload.get("id", 0))
+        if msg_id in mcp_client.call_results:
+            await mcp_client.reject_call_result(
+                msg_id, Exception(f"MCP错误: {error_msg}")
+            )
+
+
+# --- Outgoing MCP Messages ---
+
+
+async def send_mcp_initialize_message(conn):
+    """发送MCP初始化消息"""
+
+    vision_url = get_vision_url(conn.config)
+
+    # 密钥生成token
+    auth = AuthToken(conn.config["server"]["auth_key"])
+    token = auth.generate_token(conn.headers.get("device-id"))
+
+    vision = {
+        "url": vision_url,
+        "token": token,
+    }
+
+    conn.logger.bind(tag=TAG).info(f"视觉服务信息: {vision}")
+
+    payload = {
+        "jsonrpc": "2.0",
+        "id": 1,  # mcpInitializeID
+        "method": "initialize",
+        "params": {
+            "protocolVersion": "2024-11-05",
+            "capabilities": {
+                "roots": {"listChanged": True},
+                "sampling": {},
+                "vision": vision,
+            },
+            "clientInfo": {
+                "name": "XiaozhiClient",
+                "version": "1.0.0",
+            },
+        },
+    }
+    conn.logger.bind(tag=TAG).info("发送MCP初始化消息")
+    await send_mcp_message(conn, payload)
+
+
+async def send_mcp_tools_list_request(conn):
+    """发送MCP工具列表请求"""
+    payload = {
+        "jsonrpc": "2.0",
+        "id": 2,  # mcpToolsListID
+        "method": "tools/list",
+    }
+    conn.logger.bind(tag=TAG).debug("发送MCP工具列表请求")
+    await send_mcp_message(conn, payload)
+
+
+async def send_mcp_tools_list_continue_request(conn, cursor: str):
+    """发送带有cursor的MCP工具列表请求"""
+    payload = {
+        "jsonrpc": "2.0",
+        "id": 2,  # mcpToolsListID (same ID for continuation)
+        "method": "tools/list",
+        "params": {"cursor": cursor},
+    }
+    conn.logger.bind(tag=TAG).info(f"发送带cursor的MCP工具列表请求: {cursor}")
+    await send_mcp_message(conn, payload)
+
+
+async def call_mcp_tool(
+    conn, mcp_client: MCPClient, tool_name: str, args: str = "{}", timeout: int = 30
+):
+    """
+    调用指定的工具，并等待响应
+    """
+    if not await mcp_client.is_ready():
+        raise RuntimeError("MCP客户端尚未准备就绪")
+
+    if not mcp_client.has_tool(tool_name):
+        raise ValueError(f"工具 {tool_name} 不存在")
+
+    tool_call_id = await mcp_client.get_next_id()
+    result_future = asyncio.Future()
+    await mcp_client.register_call_result_future(tool_call_id, result_future)
+
+    # 处理参数
+    try:
+        if isinstance(args, str):
+            # 确保字符串是有效的JSON
+            if not args.strip():
+                arguments = {}
+            else:
+                try:
+                    # 尝试直接解析
+                    arguments = json.loads(args)
+                except json.JSONDecodeError:
+                    # 如果解析失败，尝试合并多个JSON对象
+                    try:
+                        # 使用正则表达式匹配所有JSON对象
+                        import re
+
+                        json_objects = re.findall(r"\{[^{}]*\}", args)
+                        if len(json_objects) > 1:
+                            # 合并所有JSON对象
+                            merged_dict = {}
+                            for json_str in json_objects:
+                                try:
+                                    obj = json.loads(json_str)
+                                    if isinstance(obj, dict):
+                                        merged_dict.update(obj)
+                                except json.JSONDecodeError:
+                                    continue
+                            if merged_dict:
+                                arguments = merged_dict
+                            else:
+                                raise ValueError(f"无法解析任何有效的JSON对象: {args}")
+                        else:
+                            raise ValueError(f"参数JSON解析失败: {args}")
+                    except Exception as e:
+                        conn.logger.bind(tag=TAG).error(
+                            f"参数JSON解析失败: {str(e)}, 原始参数: {args}"
+                        )
+                        raise ValueError(f"参数JSON解析失败: {str(e)}")
+        elif isinstance(args, dict):
+            arguments = args
+        else:
+            raise ValueError(f"参数类型错误，期望字符串或字典，实际类型: {type(args)}")
+
+        # 确保参数是字典类型
+        if not isinstance(arguments, dict):
+            raise ValueError(f"参数必须是字典类型，实际类型: {type(arguments)}")
+
+    except Exception as e:
+        if not isinstance(e, ValueError):
+            raise ValueError(f"参数处理失败: {str(e)}")
+        raise e
+
+    payload = {
+        "jsonrpc": "2.0",
+        "id": tool_call_id,
+        "method": "tools/call",
+        "params": {"name": tool_name, "arguments": arguments},
+    }
+
+    conn.logger.bind(tag=TAG).info(
+        f"发送客户端mcp工具调用请求: {tool_name}，参数: {args}"
+    )
+    await send_mcp_message(conn, payload)
+
+    try:
+        # Wait for response or timeout
+        raw_result = await asyncio.wait_for(result_future, timeout=timeout)
+        conn.logger.bind(tag=TAG).info(
+            f"客户端mcp工具调用 {tool_name} 成功，原始结果: {raw_result}"
+        )
+
+        if isinstance(raw_result, dict):
+            if raw_result.get("isError") is True:
+                error_msg = raw_result.get(
+                    "error", "工具调用返回错误，但未提供具体错误信息"
+                )
+                raise RuntimeError(f"工具调用错误: {error_msg}")
+
+            content = raw_result.get("content")
+            if isinstance(content, list) and len(content) > 0:
+                if isinstance(content[0], dict) and "text" in content[0]:
+                    # 直接返回文本内容，不进行JSON解析
+                    return content[0]["text"]
+        # 如果结果不是预期的格式，将其转换为字符串
+        return str(raw_result)
+    except asyncio.TimeoutError:
+        await mcp_client.cleanup_call_result(tool_call_id)
+        raise TimeoutError("工具调用请求超时")
+    except Exception as e:
+        await mcp_client.cleanup_call_result(tool_call_id)
+        raise e

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -1,10 +1,8 @@
-import time
-import copy
-from core.utils.util import remove_punctuation_and_length
 from core.handle.sendAudioHandle import send_stt_message
 from core.handle.intentHandler import handle_user_intent
 from core.utils.output_counter import check_device_output_limit
-from core.handle.reportHandle import enqueue_asr_report
+from core.handle.abortHandle import handleAbortMessage
+import time
 from core.handle.sendAudioHandle import SentenceType
 from core.utils.util import audio_to_data
 
@@ -13,46 +11,20 @@ TAG = __name__
 
 async def handleAudioMessage(conn, audio):
     if conn.vad is None:
+        conn.logger.bind(tag=TAG).warning("VAD模块未初始化，继续等待")
         return
-    if not conn.asr_server_receive:
+    if conn.asr is None or not hasattr(conn.asr, "conn") or conn.asr.conn is None:
-        conn.logger.bind(tag=TAG).debug(f"前期数据处理中，暂停接收")
+        conn.logger.bind(tag=TAG).warning("ASR模块未初始化或通道未就绪，继续等待")
         return
-    if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
+    # 当前片段是否有人说话
-        have_voice = conn.vad.is_vad(conn, audio)
+    have_voice = conn.vad.is_vad(conn, audio)
-    else:
+    if have_voice:
-        have_voice = conn.client_have_voice
+        if conn.client_is_speaking:
-
+            await handleAbortMessage(conn)
-    # 如果本次没有声音，本段也没声音，就把声音丢弃了
+    # 设备长时间空闲检测，用于say goodbye
-    if have_voice == False and conn.client_have_voice == False:
+    await no_voice_close_connect(conn, have_voice)
-        await no_voice_close_connect(conn)
+    # 接收音频
-        conn.asr_audio.append(audio)
+    await conn.asr.receive_audio(audio, have_voice)
-        conn.asr_audio = conn.asr_audio[
-            -10:
-        ]  # 保留最新的10帧音频内容，解决ASR句首丢字问题
-        return
-    conn.client_no_voice_last_time = 0.0
-    conn.asr_audio.append(audio)
-    # 如果本段有声音，且已经停止了
-    if conn.client_voice_stop:
-        conn.client_abort = False
-        conn.asr_server_receive = False
-        # 音频太短了，无法识别
-        if len(conn.asr_audio) < 15:
-            conn.asr_server_receive = True
-        else:
-            raw_text, _ = await conn.asr.speech_to_text(
-                conn.asr_audio, conn.session_id
-            )  # 确保ASR模块返回原始文本
-            conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
-            text_len, _ = remove_punctuation_and_length(raw_text)
-            if text_len > 0:
-                # 使用自定义模块进行上报
-                await startToChat(conn, raw_text)
-                enqueue_asr_report(conn, raw_text, copy.deepcopy(conn.asr_audio))
-            else:
-                conn.asr_server_receive = True
-        conn.asr_audio.clear()
-        conn.reset_vad_states()
 
 
 async def startToChat(conn, text):
@@ -67,13 +39,14 @@ async def startToChat(conn, text):
         ):
             await max_out_size(conn)
             return
+    if conn.client_is_speaking:
+        await handleAbortMessage(conn)
 
     # 首先进行意图分析
     intent_handled = await handle_user_intent(conn, text)
 
     if intent_handled:
         # 如果意图已被处理，不再进行聊天
-        conn.asr_server_receive = True
         return
 
     # 意图未被处理，继续常规聊天流程
@@ -81,7 +54,10 @@ async def startToChat(conn, text):
     conn.executor.submit(conn.chat, text)
 
 
-async def no_voice_close_connect(conn):
+async def no_voice_close_connect(conn, have_voice):
+    if have_voice:
+        conn.client_no_voice_last_time = 0.0
+        return
     if conn.client_no_voice_last_time == 0.0:
         conn.client_no_voice_last_time = time.time() * 1000
     else:
@@ -95,7 +71,6 @@ async def no_voice_close_connect(conn):
         ):
             conn.close_after_chat = True
             conn.client_abort = False
-            conn.asr_server_receive = False
             end_prompt = conn.config.get("end_prompt", {})
             if end_prompt and end_prompt.get("enable", True) is False:
                 conn.logger.bind(tag=TAG).info("结束对话，无需发送结束提示语")
@@ -103,7 +78,7 @@ async def no_voice_close_connect(conn):
                 return
             prompt = end_prompt.get("prompt")
             if not prompt:
-                prompt = "请你以“时间过得真快”未来头，用富有感情、依依不舍的话来结束这场对话吧。！"
+                prompt = "请你以```时间过得真快```未来头，用富有感情、依依不舍的话来结束这场对话吧。！"
             await startToChat(conn, prompt)
 
 

main/xiaozhi-server/core/handle/reportHandle.py

@@ -8,6 +8,7 @@ TTS上报功能已集成到ConnectionHandler类中。
 
 具体实现请参考core/connection.py中的相关代码。
 """
+
 import time
 
 import opuslib_next
@@ -145,4 +146,4 @@ def enqueue_asr_report(conn, text, opus_data):
                 f"ASR数据已加入上报队列: {conn.device_id}, 不上报音频"
             )
     except Exception as e:
-        conn.logger.bind(tag=TAG).error(f"加入ASR上报队列失败: {text}, {e}")
+        conn.logger.bind(tag=TAG).debug(f"加入ASR上报队列失败: {text}, {e}")

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -62,6 +62,7 @@ async def sendAudioMessage(conn, sentenceType, audios, text):
     # 发送结束消息（如果是最后一个文本）
     if conn.llm_finish_task and sentenceType == SentenceType.LAST:
         await send_tts_message(conn, "stop", None)
+        conn.client_is_speaking = False
         if conn.close_after_chat:
             await conn.close()
 
@@ -88,6 +89,7 @@ async def sendAudio(conn, audios, pre_buffer=True):
     # 播放剩余音频帧
     for opus_packet in remaining_audios:
         if conn.client_abort:
+            conn.client_abort = False
             return
 
         # 每分钟重置一次计时器
@@ -131,9 +133,15 @@ async def send_tts_message(conn, state, text=None):
 
 
 async def send_stt_message(conn, text):
+    end_prompt_str = conn.config.get("end_prompt", {}).get("prompt")
+    if end_prompt_str and end_prompt_str == text:
+        await send_tts_message(conn, "start")
+        return
+
     """发送 STT 状态消息"""
     stt_text = get_string_no_punctuation_or_emoji(text)
     await conn.websocket.send(
         json.dumps({"type": "stt", "text": stt_text, "session_id": conn.session_id})
     )
+    conn.client_is_speaking = True
     await send_tts_message(conn, "start")

main/xiaozhi-server/core/handle/textHandle.py

@@ -1,6 +1,7 @@
 import json
 from core.handle.abortHandle import handleAbortMessage
 from core.handle.helloHandle import handleHelloMessage
+from core.handle.mcpHandle import handle_mcp_message
 from core.utils.util import remove_punctuation_and_length, filter_sensitive_info
 from core.handle.receiveAudioHandle import startToChat, handleAudioMessage
 from core.handle.sendAudioHandle import send_stt_message, send_tts_message
@@ -41,12 +42,13 @@ async def handleTextMessage(conn, message):
                 if len(conn.asr_audio) > 0:
                     await handleAudioMessage(conn, b"")
             elif msg_json["state"] == "detect":
-                conn.asr_server_receive = False
                 conn.client_have_voice = False
                 conn.asr_audio.clear()
                 if "text" in msg_json:
                     original_text = msg_json["text"]  # 保留原始文本
-                    filtered_len, filtered_text = remove_punctuation_and_length(original_text)
+                    filtered_len, filtered_text = remove_punctuation_and_length(
+                        original_text
+                    )
 
                     # 识别是否是唤醒词
                     is_wakeup_words = filtered_text in conn.config.get("wakeup_words")
@@ -57,6 +59,7 @@ async def handleTextMessage(conn, message):
                         # 如果是唤醒词，且关闭了唤醒词回复，就不用回答
                         await send_stt_message(conn, original_text)
                         await send_tts_message(conn, "stop", None)
+                        conn.client_is_speaking = False
                     elif is_wakeup_words:
                         # 上报纯文字数据（复用ASR上报功能，但不提供音频数据）
                         enqueue_asr_report(conn, "嘿，你好呀", [])
@@ -72,6 +75,10 @@ async def handleTextMessage(conn, message):
                 asyncio.create_task(handleIotDescriptors(conn, msg_json["descriptors"]))
             if "states" in msg_json:
                 asyncio.create_task(handleIotStatus(conn, msg_json["states"]))
+        elif msg_json["type"] == "mcp":
+            conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message}")
+            if "payload" in msg_json:
+                asyncio.create_task(handle_mcp_message(conn, conn.mcp_client, msg_json["payload"]))
         elif msg_json["type"] == "server":
             # 记录日志时过滤敏感信息
             conn.logger.bind(tag=TAG).info(
@@ -151,5 +158,7 @@ async def handleTextMessage(conn, message):
             # 重启服务器
             elif msg_json["action"] == "restart":
                 await conn.handle_restart(msg_json)
+        else:
+            conn.logger.bind(tag=TAG).error(f"收到未知类型消息：{message}")
     except json.JSONDecodeError:
         await conn.websocket.send(message)

main/xiaozhi-server/core/http_server.py

@@ -0,0 +1,71 @@
+import asyncio
+from aiohttp import web
+from config.logger import setup_logging
+from core.api.ota_handler import OTAHandler
+from core.api.vision_handler import VisionHandler
+
+TAG = __name__
+
+
+class SimpleHttpServer:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        self.ota_handler = OTAHandler(config)
+        self.vision_handler = VisionHandler(config)
+
+    def _get_websocket_url(self, local_ip: str, port: int) -> str:
+        """获取websocket地址
+
+        Args:
+            local_ip: 本地IP地址
+            port: 端口号
+
+        Returns:
+            str: websocket地址
+        """
+        server_config = self.config["server"]
+        websocket_config = server_config.get("websocket")
+
+        if websocket_config and "你" not in websocket_config:
+            return websocket_config
+        else:
+            return f"ws://{local_ip}:{port}/xiaozhi/v1/"
+
+    async def start(self):
+        server_config = self.config["server"]
+        host = server_config.get("ip", "0.0.0.0")
+        port = int(server_config.get("http_port", 8003))
+
+        if port:
+            app = web.Application()
+
+            read_config_from_api = server_config.get("read_config_from_api", False)
+
+            if not read_config_from_api:
+                # 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
+                app.add_routes(
+                    [
+                        web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
+                        web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
+                        web.options("/xiaozhi/ota/", self.ota_handler.handle_post),
+                    ]
+                )
+            # 添加路由
+            app.add_routes(
+                [
+                    web.get("/mcp/vision/explain", self.vision_handler.handle_get),
+                    web.post("/mcp/vision/explain", self.vision_handler.handle_post),
+                    web.options("/mcp/vision/explain", self.vision_handler.handle_post),
+                ]
+            )
+
+            # 运行服务
+            runner = web.AppRunner(app)
+            await runner.setup()
+            site = web.TCPSite(runner, host, port)
+            await site.start()
+
+            # 保持服务运行
+            while True:
+                await asyncio.sleep(3600)  # 每隔 1 小时检查一次

main/xiaozhi-server/core/mcp/MCPClient.py

@@ -92,11 +92,21 @@ class MCPClient:
                         args=self.config.get("args", []),
                         env=env,
                     )
-                    stdio_r, stdio_w = await stack.enter_async_context(stdio_client(params))
+                    stdio_r, stdio_w = await stack.enter_async_context(
+                        stdio_client(params)
+                    )
                     read_stream, write_stream = stdio_r, stdio_w
                 # 建立SSEClient
                 elif "url" in self.config:
-                    sse_r, sse_w = await stack.enter_async_context(sse_client(self.config["url"]))
+                    if "API_ACCESS_TOKEN" in self.config:
+                        headers = {
+                            "Authorization": f"Bearer {self.config['API_ACCESS_TOKEN']}"
+                        }
+                    else:
+                        headers = {}
+                    sse_r, sse_w = await stack.enter_async_context(
+                        sse_client(self.config["url"], headers=headers)
+                    )
                     read_stream, write_stream = sse_r, sse_w
 
                 else:

main/xiaozhi-server/core/providers/asr/aliyun.py

@@ -2,9 +2,6 @@ import http.client
 import json
 import asyncio
 from typing import Optional, Tuple, List
-import opuslib_next
-import wave
-import io
 import os
 import uuid
 import hmac
@@ -16,6 +13,7 @@ import time
 from datetime import datetime
 from config.logger import setup_logging
 from core.providers.asr.base import ASRProviderBase
+from core.providers.asr.dto.dto import InterfaceType
 
 TAG = __name__
 logger = setup_logging()
@@ -92,6 +90,7 @@ class AccessToken:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         """阿里云ASR初始化"""
         # 新增空值判断逻辑
         self.access_key_id = config.get("access_key_id")
@@ -165,21 +164,6 @@ class ASRProvider(ASRProviderBase):
         request += "&enable_voice_detection=false"
         return request
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)  # 单声道
-            wf.setsampwidth(2)  # 16-bit
-            wf.setframerate(self.sample_rate)
-            wf.writeframes(b"".join(pcm_data))
-
-        logger.bind(tag=TAG).debug(f"音频文件已保存至: {file_path}")
-        return file_path
-
     async def _send_request(self, pcm_data: bytes) -> Optional[str]:
         """发送请求到阿里云ASR服务"""
         try:

main/xiaozhi-server/core/providers/asr/baidu.py

@@ -1,18 +1,10 @@
-import base64
-import hashlib
-import hmac
-import json
 import time
-from datetime import datetime, timezone
 import os
-import uuid
 from typing import Optional, Tuple, List
-import wave
-import opuslib_next
-
 from aip import AipSpeech
 from core.providers.asr.base import ASRProviderBase
 from config.logger import setup_logging
+from core.providers.asr.dto.dto import InterfaceType
 
 TAG = __name__
 logger = setup_logging()
@@ -21,6 +13,7 @@ logger = setup_logging()
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool = True):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.app_id = config.get("app_id")
         self.api_key = config.get("api_key")
         self.secret_key = config.get("secret_key")
@@ -36,20 +29,6 @@ class ASRProvider(ASRProviderBase):
         # 确保输出目录存在
         os.makedirs(self.output_dir, exist_ok=True)
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def speech_to_text(
         self, opus_data: List[bytes], session_id: str
     ) -> Tuple[Optional[str], Optional[str]]:

main/xiaozhi-server/core/providers/asr/base.py

@@ -1,7 +1,15 @@
-from abc import ABC, abstractmethod
+import os
-from typing import Optional, Tuple, List
+import time
+import copy
+import uuid
+import wave
 import opuslib_next
+from abc import ABC, abstractmethod
 from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.utils.util import remove_punctuation_and_length
+from core.handle.reportHandle import enqueue_asr_report
+from core.handle.receiveAudioHandle import startToChat
 
 TAG = __name__
 logger = setup_logging()
@@ -10,11 +18,66 @@ logger = setup_logging()
 class ASRProviderBase(ABC):
     def __init__(self):
         self.audio_format = "opus"
+        self.conn = None
+
+    # 打开音频通道
+    # 这里默认是非流式的处理方式
+    # 流式处理方式请在子类中重写
+    async def open_audio_channels(self, conn):
+        self.conn = conn
+
+    # 接收音频
+    # 这里默认是非流式的处理方式
+    # 流式处理方式请在子类中重写
+    async def receive_audio(self, audio, audio_have_voice):
+        if (
+            self.conn.client_listen_mode == "auto"
+            or self.conn.client_listen_mode == "realtime"
+        ):
+            have_voice = audio_have_voice
+        else:
+            have_voice = self.conn.client_have_voice
+        # 如果本次没有声音，本段也没声音，就把声音丢弃了
+        self.conn.asr_audio.append(audio)
+        if have_voice == False and self.conn.client_have_voice == False:
+            self.conn.asr_audio = self.conn.asr_audio[-10:]
+            return
+
+        # 如果本段有声音，且已经停止了
+        if self.conn.client_voice_stop:
+            asr_audio_task = copy.deepcopy(self.conn.asr_audio)
+            self.conn.asr_audio.clear()
+
+            # 音频太短了，无法识别
+            self.conn.reset_vad_states()
+            if len(asr_audio_task) > 15:
+                await self.handle_voice_stop(asr_audio_task)
+
+    # 处理语音停止
+    async def handle_voice_stop(self, asr_audio_task):
+        raw_text, _ = await self.speech_to_text(
+            asr_audio_task, self.conn.session_id
+        )  # 确保ASR模块返回原始文本
+        self.conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+        text_len, _ = remove_punctuation_and_length(raw_text)
+        if text_len > 0:
+            # 使用自定义模块进行上报
+            await startToChat(self.conn, raw_text)
+            enqueue_asr_report(self.conn, raw_text, asr_audio_task)
 
-    @abstractmethod
     def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
         """PCM数据保存为WAV文件"""
-        pass
+        module_name = __name__.split(".")[-1]
+        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
+        file_path = os.path.join(self.output_dir, file_name)
+
+        with wave.open(file_path, "wb") as wf:
+            wf.setnchannels(1)
+            wf.setsampwidth(2)  # 2 bytes = 16-bit
+            wf.setframerate(16000)
+            wf.writeframes(b"".join(pcm_data))
+
+        return file_path
 
     @abstractmethod
     async def speech_to_text(

main/xiaozhi-server/core/providers/asr/doubao.py

@@ -1,283 +1,534 @@
-import time
-import io
-import wave
-import os
-from typing import Optional, Tuple, List
-import uuid
-import websockets
 import json
 import gzip
-
+import uuid
+import asyncio
+import websockets
 import opuslib_next
 from core.providers.asr.base import ASRProviderBase
-
 from config.logger import setup_logging
+from core.providers.asr.dto.dto import InterfaceType
+import threading
 
 TAG = __name__
 logger = setup_logging()
 
 CLIENT_FULL_REQUEST = 0b0001
 CLIENT_AUDIO_ONLY_REQUEST = 0b0010
-
-NO_SEQUENCE = 0b0000
-NEG_SEQUENCE = 0b0010
-
 SERVER_FULL_RESPONSE = 0b1001
 SERVER_ACK = 0b1011
 SERVER_ERROR_RESPONSE = 0b1111
-
+NO_SEQUENCE = 0b0000
-NO_SERIALIZATION = 0b0000
+NEG_SEQUENCE = 0b0010
-JSON = 0b0001
+JSON_SERIALIZATION = 0b0001
-THRIFT = 0b0011
+GZIP_COMPRESSION = 0b0001
-CUSTOM_TYPE = 0b1111
+PROTOCOL_VERSION = 0b0001
-NO_COMPRESSION = 0b0000
-GZIP = 0b0001
-CUSTOM_COMPRESSION = 0b1111
-
-
-def parse_response(res):
-    """
-    protocol_version(4 bits), header_size(4 bits),
-    message_type(4 bits), message_type_specific_flags(4 bits)
-    serialization_method(4 bits) message_compression(4 bits)
-    reserved （8bits) 保留字段
-    header_extensions 扩展头(大小等于 8 * 4 * (header_size - 1) )
-    payload 类似与http 请求体
-    """
-    protocol_version = res[0] >> 4
-    header_size = res[0] & 0x0F
-    message_type = res[1] >> 4
-    message_type_specific_flags = res[1] & 0x0F
-    serialization_method = res[2] >> 4
-    message_compression = res[2] & 0x0F
-    reserved = res[3]
-    header_extensions = res[4 : header_size * 4]
-    payload = res[header_size * 4 :]
-    result = {}
-    payload_msg = None
-    payload_size = 0
-    if message_type == SERVER_FULL_RESPONSE:
-        payload_size = int.from_bytes(payload[:4], "big", signed=True)
-        payload_msg = payload[4:]
-    elif message_type == SERVER_ACK:
-        seq = int.from_bytes(payload[:4], "big", signed=True)
-        result["seq"] = seq
-        if len(payload) >= 8:
-            payload_size = int.from_bytes(payload[4:8], "big", signed=False)
-            payload_msg = payload[8:]
-    elif message_type == SERVER_ERROR_RESPONSE:
-        code = int.from_bytes(payload[:4], "big", signed=False)
-        result["code"] = code
-        payload_size = int.from_bytes(payload[4:8], "big", signed=False)
-        payload_msg = payload[8:]
-    if payload_msg is None:
-        return result
-    if message_compression == GZIP:
-        payload_msg = gzip.decompress(payload_msg)
-    if serialization_method == JSON:
-        payload_msg = json.loads(str(payload_msg, "utf-8"))
-    elif serialization_method != NO_SERIALIZATION:
-        payload_msg = str(payload_msg, "utf-8")
-    result["payload_msg"] = payload_msg
-    result["payload_size"] = payload_size
-    return result
 
 
 class ASRProvider(ASRProviderBase):
-    def __init__(self, config: dict, delete_audio_file: bool):
+    def __init__(self, config, delete_audio_file):
         super().__init__()
-        self.appid = config.get("appid")
+        self.interface_type = InterfaceType.STREAM
+        self.config = config
+        self.text = ""
+        self.max_retries = 3
+        self.retry_delay = 2  # 重试延迟秒数
+        self.recv_lock = asyncio.Lock()  # 添加接收锁
+        self.reconnect_lock = asyncio.Lock()  # 添加重连锁
+        self.last_reconnect_time = 0  # 上次重连时间
+        self.reconnect_cooldown = 1  # 增加重连冷却时间到10秒
+        self.reconnect_count = 0  # 当前重连次数
+        self.max_reconnect_count = 3  # 减少最大重连次数到3次
+        self.asr_thread = None  # ASR监听线程
+        self.thread_lock = threading.Lock()  # 线程管理锁
+        self.is_reconnecting = False  # 添加重连状态标志
+
+        # 添加会话管理相关属性
+        self._session_lock = asyncio.Lock()  # 会话操作的并发锁
+        self._current_session_id = None  # 当前会话ID
+        self._session_started = False  # 会话是否已开始
+        self._session_finished = False  # 会话是否已结束
+        self._session_close_event = asyncio.Event()  # 添加会话关闭事件
+
+        self.appid = str(config.get("appid"))
         self.cluster = config.get("cluster")
         self.access_token = config.get("access_token")
         self.boosting_table_name = config.get("boosting_table_name", "")
         self.correct_table_name = config.get("correct_table_name", "")
-        self.output_dir = config.get("output_dir")
+        self.output_dir = config.get("output_dir", "temp/")
         self.delete_audio_file = delete_audio_file
 
-        self.host = "openspeech.bytedance.com"
+        self.ws_url = "wss://openspeech.bytedance.com/api/v2/asr"
-        self.ws_url = f"wss://{self.host}/api/v2/asr"
+        self.uid = config.get("uid", "streaming_asr_service")
-        self.success_code = 1000
+        self.workflow = config.get(
-        self.seg_duration = 15000
+            "workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
+        )
+        self.result_type = config.get("result_type", "single")
+        self.format = config.get("format", "raw")
+        self.codec = config.get("codec", "pcm")
+        self.rate = config.get("sample_rate", 16000)
+        self.language = config.get("language", "zh-CN")
+        self.bits = config.get("bits", 16)
+        self.channel = config.get("channel", 1)
+        self.auth_method = config.get("auth_method", "token")
+        self.secret = config.get("secret", "access_secret")
+        self.decoder = opuslib_next.Decoder(16000, 1)
+        self.asr_ws = None
+        self.forward_task = None
+        self.conn = None
 
-        # 确保输出目录存在
+    ###################################################################################
-        os.makedirs(self.output_dir, exist_ok=True)
+    # 豆包流式ASR重写父类的方法--开始
+    ###################################################################################
+    async def open_audio_channels(self, conn):
+        await super().open_audio_channels(conn)
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
+        async with self._session_lock:
-        """PCM数据保存为WAV文件"""
+            # 如果正在重连，等待重连完成
-        module_name = __name__.split(".")[-1]
+            if self.is_reconnecting:
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
+                logger.bind(tag=TAG).info("等待当前重连完成...")
-        file_path = os.path.join(self.output_dir, file_name)
+                await self._session_close_event.wait()
+                self._session_close_event.clear()
 
-        with wave.open(file_path, "wb") as wf:
+            # 如果已有会话未结束，先关闭它
-            wf.setnchannels(1)
+            if self._session_started and not self._session_finished:
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
+                logger.bind(tag=TAG).warning(
-            wf.setframerate(16000)
+                    f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
-            wf.writeframes(b"".join(pcm_data))
+                )
+                if self.asr_ws is not None:
+                    try:
+                        await self.asr_ws.close()
+                    except Exception as e:
+                        logger.bind(tag=TAG).warning(f"关闭旧连接时发生错误: {e}")
+                    finally:
+                        self.asr_ws = None
+                        self._session_finished = True
+                        self._session_close_event.set()
 
-        return file_path
+            # 重置会话状态
+            self._current_session_id = str(uuid.uuid4())
+            self._session_started = True
+            self._session_finished = False
+            self.is_reconnecting = True
 
-    @staticmethod
+            try:
-    def _generate_header(
+                retry_count = 0
-        message_type=CLIENT_FULL_REQUEST, message_type_specific_flags=NO_SEQUENCE
+                while retry_count < self.max_retries:
-    ) -> bytearray:
+                    try:
-        """Generate protocol header."""
+                        headers = (
-        header = bytearray()
+                            self.token_auth() if self.auth_method == "token" else None
-        header_size = 1
+                        )
-        header.append((0b0001 << 4) | header_size)  # Protocol version
+                        self.asr_ws = await websockets.connect(
-        header.append((message_type << 4) | message_type_specific_flags)
+                            self.ws_url,
-        header.append((0b0001 << 4) | 0b0001)  # JSON serialization & GZIP compression
+                            additional_headers=headers,
-        header.append(0x00)  # reserved
+                            max_size=1000000000,
-        return header
+                            ping_interval=None,
+                            ping_timeout=None,
+                            close_timeout=10,
+                        )
 
-    def _construct_request(self, reqid) -> dict:
+                        # 发送初始化请求
-        """Construct the request payload."""
+                        request_params = self.construct_request(
-        return {
+                            self._current_session_id
+                        )
+                        try:
+                            payload_bytes = str.encode(json.dumps(request_params))
+                            payload_bytes = gzip.compress(payload_bytes)
+                            full_client_request = self.generate_header()
+                            full_client_request.extend(
+                                (len(payload_bytes)).to_bytes(4, "big")
+                            )
+                            full_client_request.extend(payload_bytes)
+                            await self.asr_ws.send(full_client_request)
+                        except Exception as e:
+                            logger.bind(tag=TAG).error(f"发送初始化请求失败: {e}")
+                            raise e
+
+                        # 等待初始化响应
+                        try:
+                            init_res = await self.asr_ws.recv()
+                            self.parse_response(init_res)
+                        except Exception as e:
+                            logger.bind(tag=TAG).error(f"ASR服务初始化失败: {e}")
+                            raise e
+
+                        # 启动接收ASR结果的异步任务
+                        with self.thread_lock:
+                            if (
+                                self.asr_thread is None
+                                or not self.asr_thread.is_alive()
+                            ):
+                                logger.bind(tag=TAG).info("创建新的ASR监听线程...")
+                                self.asr_thread = threading.Thread(
+                                    target=self._start_monitor_asr_response_thread,
+                                    daemon=True,
+                                )
+                                self.asr_thread.start()
+                                # 等待一小段时间确保线程启动
+                                await asyncio.sleep(0.1)
+                                if not self.asr_thread.is_alive():
+                                    logger.bind(tag=TAG).error("ASR监听线程启动失败")
+                                    raise Exception("ASR监听线程启动失败")
+                                logger.bind(tag=TAG).info("ASR监听线程已启动")
+                        return
+
+                    except websockets.exceptions.WebSocketException as e:
+                        retry_count += 1
+                        if retry_count < self.max_retries:
+                            logger.bind(tag=TAG).warning(
+                                f"WebSocket连接失败，正在进行第{retry_count}次重试: {e}"
+                            )
+                            await asyncio.sleep(self.retry_delay)
+                        else:
+                            logger.bind(tag=TAG).warning(
+                                f"WebSocket连接失败，已达到最大重试次数: {e}"
+                            )
+                            raise
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"WebSocket连接发生未知错误: {e}")
+                        raise
+            finally:
+                self.is_reconnecting = False
+                self._session_close_event.set()
+
+    async def receive_audio(self, audio, _):
+        if not isinstance(audio, bytes):
+            return
+
+        try:
+            # 解码opus得到PCM数据
+            pcm_frame = self.decoder.decode(audio, 960)
+            payload = gzip.compress(pcm_frame)
+            audio_request = bytearray(self.generate_audio_default_header())
+            audio_request.extend(len(payload).to_bytes(4, "big"))
+            audio_request.extend(payload)
+            if self.asr_ws:
+                await self.asr_ws.send(audio_request)
+        except Exception as e:
+            logger.bind(tag=TAG).debug(f"发送音频数据时发生错误: {e}")
+
+    ###################################################################################
+    # 豆包流式ASR重写父类的方法--结束
+    ###################################################################################
+
+    def construct_request(self, reqid):
+        req = {
             "app": {
-                "appid": f"{self.appid}",
+                "appid": self.appid,
                 "cluster": self.cluster,
                 "token": self.access_token,
             },
-            "user": {
+            "user": {"uid": self.uid},
-                "uid": str(uuid.uuid4()),
-            },
             "request": {
                 "reqid": reqid,
-                "show_utterances": False,
+                "workflow": self.workflow,
+                "show_utterances": True,
+                "result_type": self.result_type,
                 "sequence": 1,
                 "boosting_table_name": self.boosting_table_name,
                 "correct_table_name": self.correct_table_name,
             },
             "audio": {
-                "format": "raw",
+                "format": self.format,
-                "rate": 16000,
+                "codec": self.codec,
-                "language": "zh-CN",
+                "rate": self.rate,
-                "bits": 16,
+                "language": self.language,
-                "channel": 1,
+                "bits": self.bits,
-                "codec": "raw",
+                "channel": self.channel,
             },
         }
+        return req
 
-    async def _send_request(
+    def token_auth(self):
-        self, audio_data: List[bytes], segment_size: int
+        return {"Authorization": f"Bearer; {self.access_token}"}
-    ) -> Optional[str]:
+
-        """Send request to Volcano ASR service."""
+    def generate_header(
+        self,
+        version=PROTOCOL_VERSION,
+        message_type=CLIENT_FULL_REQUEST,
+        message_type_specific_flags=NO_SEQUENCE,
+        serial_method=JSON_SERIALIZATION,
+        compression_type=GZIP_COMPRESSION,
+        reserved_data=0x00,
+        extension_header: bytes = b"",
+    ):
+        """
+        生成协议头：
+         - 第1字节：高4位：协议版本，低4位：头部大小（单位 4 字节）
+         - 第2字节：高4位：消息类型，低4位：消息类型特定标志
+         - 第3字节：高4位：序列化方式，低4位：压缩方式
+         - 第4字节：保留字段
+         - 后续：扩展头（如果有）
+        """
+        header = bytearray()
+        header_size = int(len(extension_header) / 4) + 1
+        header.append((version << 4) | header_size)
+        header.append((message_type << 4) | message_type_specific_flags)
+        header.append((serial_method << 4) | compression_type)
+        header.append(reserved_data)
+        header.extend(extension_header)
+        return header
+
+    def generate_full_default_header(self):
+        # full client request 默认头
+        return self.generate_header(
+            version=PROTOCOL_VERSION,
+            message_type=CLIENT_FULL_REQUEST,
+            message_type_specific_flags=NO_SEQUENCE,
+            serial_method=JSON_SERIALIZATION,
+            compression_type=GZIP_COMPRESSION,
+        )
+
+    def generate_audio_default_header(self):
+        # 普通音频片段请求
+        return self.generate_header(
+            version=PROTOCOL_VERSION,
+            message_type=CLIENT_AUDIO_ONLY_REQUEST,
+            message_type_specific_flags=NO_SEQUENCE,
+            serial_method=JSON_SERIALIZATION,
+            compression_type=GZIP_COMPRESSION,
+        )
+
+    def generate_last_audio_default_header(self):
+        # 最后一个音频片段标志
+        return self.generate_header(
+            version=PROTOCOL_VERSION,
+            message_type=CLIENT_AUDIO_ONLY_REQUEST,
+            message_type_specific_flags=NEG_SEQUENCE,  # 用 NEG_SEQUENCE 表示结束
+            serial_method=JSON_SERIALIZATION,
+            compression_type=GZIP_COMPRESSION,
+        )
+
+    def _start_monitor_asr_response_thread(self):
+        # 初始化链接
         try:
-            auth_header = {"Authorization": "Bearer; {}".format(self.access_token)}
+            with self.thread_lock:
-            async with websockets.connect(
+                if self.conn is None or self.conn.loop is None:
-                self.ws_url, additional_headers=auth_header
+                    logger.bind(tag=TAG).error(
-            ) as websocket:
+                        "无法启动ASR监听线程：conn或loop未初始化"
-                # Prepare request data
+                    )
-                request_params = self._construct_request(str(uuid.uuid4()))
+                    return
-                payload_bytes = str.encode(json.dumps(request_params))
-                payload_bytes = gzip.compress(payload_bytes)
-                full_client_request = self._generate_header()
-                full_client_request.extend(
-                    (len(payload_bytes)).to_bytes(4, "big")
-                )  # payload size(4 bytes)
-                full_client_request.extend(payload_bytes)  # payload
 
-                # Send header and metadata
+                try:
-                # full_client_request
+                    logger.bind(tag=TAG).info("开始启动ASR监听...")
-                await websocket.send(full_client_request)
+                    asyncio.run_coroutine_threadsafe(
-                res = await websocket.recv()
+                        self._forward_asr_results(), loop=self.conn.loop
-                result = parse_response(res)
+                    )
-                if (
+                    logger.bind(tag=TAG).info("ASR监听已启动")
-                    "payload_msg" in result
+                except Exception as e:
-                    and result["payload_msg"]["code"] != self.success_code
+                    logger.bind(tag=TAG).error(f"启动ASR监听线程失败: {e}")
-                ):
+        except Exception as e:
-                    logger.bind(tag=TAG).error(f"ASR error: {result}")
+            logger.bind(tag=TAG).error(f"ASR监听线程发生未预期的错误: {e}")
-                    return None
 
-                for seq, (chunk, last) in enumerate(
+    async def _forward_asr_results(self):
-                    self.slice_data(audio_data, segment_size), 1
+        try:
-                ):
+            while not self.conn.stop_event.is_set():
-                    if last:
+                try:
-                        audio_only_request = self._generate_header(
+                    if self.asr_ws is None:
-                            message_type=CLIENT_AUDIO_ONLY_REQUEST,
+                        # 检查是否需要重连
-                            message_type_specific_flags=NEG_SEQUENCE,
+                        async with self.reconnect_lock:
+                            current_time = asyncio.get_event_loop().time()
+                            if (
+                                current_time - self.last_reconnect_time
+                                < self.reconnect_cooldown
+                            ):
+                                await asyncio.sleep(1)
+                                continue
+
+                            if self.reconnect_count >= self.max_reconnect_count:
+                                logger.bind(tag=TAG).error(
+                                    "达到最大重连次数限制，停止重连"
+                                )
+                                await asyncio.sleep(self.reconnect_cooldown)
+                                self.reconnect_count = 0
+                                continue
+
+                            self.last_reconnect_time = current_time
+                            self.reconnect_count += 1
+                            logger.bind(tag=TAG).info(
+                                f"尝试重新连接ASR服务... (第{self.reconnect_count}次)"
+                            )
+                            await self.open_audio_channels(self.conn)
+                            continue
+
+                    # 使用锁来确保同一时间只有一个协程在接收数据
+                    async with self.recv_lock:
+                        response = await self.asr_ws.recv()
+                        result = self.parse_response(response)
+
+                    # 检查是否需要重连
+                    if result.get("need_reconnect", False):
+                        logger.bind(tag=TAG).info(
+                            "检测到需要重连的错误，准备重新连接..."
                         )
-                    else:
+                        if self.asr_ws is not None:
-                        audio_only_request = self._generate_header(
+                            try:
-                            message_type=CLIENT_AUDIO_ONLY_REQUEST
+                                await self.asr_ws.close()
-                        )
+                            except Exception as e:
-                    payload_bytes = gzip.compress(chunk)
+                                logger.bind(tag=TAG).warning(
-                    audio_only_request.extend(
+                                    f"关闭旧连接时发生错误: {e}"
-                        (len(payload_bytes)).to_bytes(4, "big")
+                                )
-                    )  # payload size(4 bytes)
+                            finally:
-                    audio_only_request.extend(payload_bytes)  # payload
+                                self.asr_ws = None
-                    # Send audio data
+                        continue
-                    await websocket.send(audio_only_request)
 
-                # Receive response
+                    if "payload_msg" in result:
-                response = await websocket.recv()
+                        if "result" in result["payload_msg"]:
-                result = parse_response(response)
+                            # 检查是否有utterances并且definite为True
+                            utterances = result["payload_msg"]["result"][0].get(
+                                "utterances", []
+                            )
+                            for utterance in utterances:
+                                if utterance.get("definite", False):
+                                    self.text = utterance["text"]
+                                    await self.handle_voice_stop(None)
+                                    break
 
-                if (
+                except websockets.ConnectionClosed:
-                    "payload_msg" in result
+                    logger.bind(tag=TAG).debug("ASR服务连接已关闭，准备重连...")
-                    and result["payload_msg"]["code"] == self.success_code
+                    # 确保关闭旧连接
-                ):
+                    if self.asr_ws is not None:
-                    if len(result["payload_msg"]["result"]) > 0:
+                        try:
-                        return result["payload_msg"]["result"][0]["text"]
+                            await self.asr_ws.close()
-                    return None
+                        except Exception as e:
-                else:
+                            logger.bind(tag=TAG).warning(f"关闭旧连接时发生错误: {e}")
-                    logger.bind(tag=TAG).error(f"ASR error: {result}")
+                        finally:
-                    return None
+                            self.asr_ws = None
+
+                    # 等待冷却时间
+                    await asyncio.sleep(self.reconnect_cooldown)
+                    continue
+
+                except Exception as e:
+                    if not self.conn.stop_event.is_set():
+                        logger.bind(tag=TAG).error(f"ASR监听发生错误: {e}")
+                        await asyncio.sleep(self.retry_delay)
+                        continue
 
         except Exception as e:
-            logger.bind(tag=TAG).error(f"ASR request failed: {e}", exc_info=True)
+            logger.bind(tag=TAG).error(f"ASR监听线程发生错误: {e}")
-            return None
+            # 确保在发生严重错误时也能继续尝试重连
+            if not self.conn.stop_event.is_set():
+                await asyncio.sleep(self.retry_delay)
+                await self._forward_asr_results()  # 递归重试
 
-    @staticmethod
+    async def speech_to_text(self, opus_data, session_id):
-    def slice_data(data: bytes, chunk_size: int) -> (list, bool):
+        result = self.text
+        self.text = ""  # 清空text
+        return result, None
+
+    def parse_response(self, res: bytes) -> dict:
         """
-        slice data
+        解析 ASR 服务返回的二进制响应。
-        :param data: wav data
+        根据协议格式解析头部和 payload，若采用 GZIP 压缩则先解压，再根据 JSON 反序列化。
-        :param chunk_size: the segment size in one request
-        :return: segment data, last flag
         """
-        data_len = len(data)
+        protocol_version = res[0] >> 4
-        offset = 0
+        header_size = res[0] & 0x0F
-        while offset + chunk_size < data_len:
+        message_type = res[1] >> 4
-            yield data[offset : offset + chunk_size], False
+        serialization_method = res[2] >> 4
-            offset += chunk_size
+        message_compression = res[2] & 0x0F
+        payload = res[header_size * 4 :]
+        result = {}
+        payload_msg = None
+        payload_size = 0
+
+        if message_type == SERVER_FULL_RESPONSE:
+            payload_size = int.from_bytes(payload[:4], "big", signed=True)
+            payload_msg = payload[4:]
+        elif message_type == SERVER_ACK:
+            seq = int.from_bytes(payload[:4], "big", signed=True)
+            result["seq"] = seq
+            if len(payload) >= 8:
+                payload_size = int.from_bytes(payload[4:8], "big", signed=False)
+                payload_msg = payload[8:]
+        elif message_type == SERVER_ERROR_RESPONSE:
+            code = int.from_bytes(payload[:4], "big", signed=False)
+            result["code"] = code
+            payload_size = int.from_bytes(payload[4:8], "big", signed=False)
+            payload_msg = payload[8:]
+
+        if payload_msg is None:
+            return result
+        if message_compression == GZIP_COMPRESSION:
+            payload_msg = gzip.decompress(payload_msg)
+        if serialization_method == JSON_SERIALIZATION:
+            payload_msg = json.loads(payload_msg.decode("utf-8"))
         else:
-            yield data[offset:data_len], True
+            payload_msg = payload_msg.decode("utf-8")
+        result["payload_msg"] = payload_msg
+        result["payload_size"] = payload_size
 
-    async def speech_to_text(
+        # 错误码处理
-        self, opus_data: List[bytes], session_id: str
+        if "code" in result:
-    ) -> Tuple[Optional[str], Optional[str]]:
+            error_code = result["code"]
-        """将语音数据转换为文本"""
+            error_message = ""
 
-        file_path = None
+            if error_code == 1000:
-        try:
+                error_message = "成功"
-            # 合并所有opus数据包
+            elif error_code == 1001:
-            if self.audio_format == "pcm":
+                error_message = "请求参数无效：请求参数缺失必需字段/字段值无效/重复请求"
-                pcm_data = opus_data
+            elif error_code == 1002:
+                error_message = "无访问权限：token无效/过期/无权访问指定服务"
+            elif error_code == 1003:
+                error_message = "访问超频：当前appid访问QPS超出设定阈值"
+            elif error_code == 1004:
+                error_message = "访问超额：当前appid访问次数超出限制"
+            elif error_code == 1005:
+                error_message = "服务器繁忙：服务过载，无法处理当前请求"
+            elif error_code == 1010:
+                error_message = "音频过长：音频数据时长超出阈值"
+            elif error_code == 1011:
+                error_message = "音频过大：音频数据大小超出阈值"
+            elif error_code == 1012:
+                error_message = "音频格式无效：音频header有误/无法进行音频解码"
+            elif error_code == 1013:
+                error_message = "音频静音：音频未识别出任何文本结果"
+            elif error_code >= 1020 and error_code <= 1022:
+                error_message = "识别相关错误：需要重连"
+                if error_code == 1020:
+                    error_message = "识别等待超时：等待下一包就绪超时"
+                elif error_code == 1021:
+                    error_message = "识别处理超时：识别处理过程超时"
+                elif error_code == 1022:
+                    error_message = "识别错误：识别过程中发生错误"
             else:
-                pcm_data = self.decode_opus(opus_data)
+                error_message = "未知错误：未归类错误"
-            combined_pcm_data = b"".join(pcm_data)
 
-            # 判断是否保存为WAV文件
+            logger.bind(tag=TAG).debug(
-            if self.delete_audio_file:
+                f"ASR错误: {error_message} (错误码: {error_code})"
-                pass
+            )
-            else:
-                file_path = self.save_audio_to_file(pcm_data, session_id)
 
-            # 直接使用PCM数据
+            # 如果是识别相关错误，标记需要重连
-            # 计算分段大小 (单声道, 16bit, 16kHz采样率)
+            if error_code >= 1020 or error_code == 1001:
-            size_per_sec = 1 * 2 * 16000  # nchannels * sampwidth * framerate
+                result["need_reconnect"] = True
-            segment_size = int(size_per_sec * self.seg_duration / 1000)
 
-            # 语音识别
+        return result
-            start_time = time.time()
+
-            text = await self._send_request(combined_pcm_data, segment_size)
+    async def close_session(self):
-            if text:
+        """关闭当前会话"""
-                logger.bind(tag=TAG).debug(
+        async with self._session_lock:
-                    f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {text}"
+            if not self._session_started:
+                logger.bind(tag=TAG).warning("尝试关闭未开始的会话")
+                return
+
+            if self._session_finished:
+                logger.bind(tag=TAG).warning(
+                    f"会话 {self._current_session_id} 已经关闭"
                 )
-                return text, file_path
+                return
-            return "", file_path
 
-        except Exception as e:
+            try:
-            logger.bind(tag=TAG).error(f"语音识别失败: {e}", exc_info=True)
+                if self.asr_ws is not None:
-            return "", file_path
+                    await self.asr_ws.close()
+            except Exception as e:
+                logger.bind(tag=TAG).warning(f"关闭WebSocket连接时发生错误: {e}")
+            finally:
+                self.asr_ws = None
+                self._session_finished = True
+                self._session_started = False
+                self._current_session_id = None
+                # 重置重连计数
+                self.reconnect_count = 0
+
+    async def close(self):
+        """资源清理方法"""
+        await self.close_session()

main/xiaozhi-server/core/providers/asr/dto/dto.py

@@ -0,0 +1,9 @@
+from enum import Enum
+from typing import Union, Optional
+
+
+class InterfaceType(Enum):
+    # 接口类型
+    STREAM = "STREAM"  # 流式接口
+    NON_STREAM = "NON_STREAM"  # 非流式接口
+    LOCAL = "LOCAL"  # 本地服务

main/xiaozhi-server/core/providers/asr/fun_local.py

@@ -1,15 +1,14 @@
 import time
-import wave
 import os
 import sys
 import io
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
-import uuid
 from core.providers.asr.base import ASRProviderBase
 from funasr import AutoModel
 from funasr.utils.postprocess_utils import rich_transcription_postprocess
 import shutil
+from core.providers.asr.dto.dto import InterfaceType
 
 TAG = __name__
 logger = setup_logging()
@@ -38,6 +37,7 @@ class CaptureOutput:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.LOCAL
         self.model_dir = config.get("model_dir")
         self.output_dir = config.get("output_dir")  # 修正配置键名
         self.delete_audio_file = delete_audio_file
@@ -53,20 +53,6 @@ class ASRProvider(ASRProviderBase):
                 # device="cuda:0",  # 启用GPU加速
             )
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def speech_to_text(
         self, opus_data: List[bytes], session_id: str
     ) -> Tuple[Optional[str], Optional[str]]:

main/xiaozhi-server/core/providers/asr/fun_server.py

@@ -1,11 +1,8 @@
 from typing import Optional, Tuple, List
-import opuslib_next
 from core.providers.asr.base import ASRProviderBase
-import os
+from core.providers.asr.dto.dto import InterfaceType
 import ssl
 import json
-import uuid
-import wave
 import websockets
 from config.logger import setup_logging
 import asyncio
@@ -23,6 +20,7 @@ class ASRProvider(ASRProviderBase):
         :param delete_audio_file: Boolean to indicate whether to delete audio files after processing.
         """
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.host = config.get("host", "localhost")
         self.port = config.get("port", 10095)
         self.api_key = config.get("api_key", "none")
@@ -43,20 +41,6 @@ class ASRProvider(ASRProviderBase):
             self.ssl_context.check_hostname = False
             self.ssl_context.verify_mode = ssl.CERT_NONE
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def _receive_responses(self, ws) -> None:
         """
         Asynchronous generator to receive messages from the WebSocket.

main/xiaozhi-server/core/providers/asr/sherpa_onnx_local.py

@@ -5,8 +5,7 @@ import sys
 import io
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
-import uuid
+from core.providers.asr.dto.dto import InterfaceType
-import opuslib_next
 from core.providers.asr.base import ASRProviderBase
 
 import numpy as np
@@ -38,6 +37,7 @@ class CaptureOutput:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        self.interface_type = InterfaceType.LOCAL
         self.model_dir = config.get("model_dir")
         self.output_dir = config.get("output_dir")
         self.delete_audio_file = delete_audio_file
@@ -84,20 +84,6 @@ class ASRProvider(ASRProviderBase):
                 use_itn=True,
             )
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     def read_wave(self, wave_filename: str) -> Tuple[np.ndarray, int]:
         """
         Args:

main/xiaozhi-server/core/providers/asr/tencent.py

@@ -5,11 +5,8 @@ import json
 import time
 from datetime import datetime, timezone
 import os
-import uuid
 from typing import Optional, Tuple, List
-import wave
+from core.providers.asr.dto.dto import InterfaceType
-import opuslib_next
-
 import requests
 from core.providers.asr.base import ASRProviderBase
 from config.logger import setup_logging
@@ -25,6 +22,7 @@ class ASRProvider(ASRProviderBase):
 
     def __init__(self, config: dict, delete_audio_file: bool = True):
         super().__init__()
+        self.interface_type = InterfaceType.NON_STREAM
         self.secret_id = config.get("secret_id")
         self.secret_key = config.get("secret_key")
         self.output_dir = config.get("output_dir")
@@ -33,20 +31,6 @@ class ASRProvider(ASRProviderBase):
         # 确保输出目录存在
         os.makedirs(self.output_dir, exist_ok=True)
 
-    def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
-        """PCM数据保存为WAV文件"""
-        module_name = __name__.split(".")[-1]
-        file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
-        file_path = os.path.join(self.output_dir, file_name)
-
-        with wave.open(file_path, "wb") as wf:
-            wf.setnchannels(1)
-            wf.setsampwidth(2)  # 2 bytes = 16-bit
-            wf.setframerate(16000)
-            wf.writeframes(b"".join(pcm_data))
-
-        return file_path
-
     async def speech_to_text(
         self, opus_data: List[bytes], session_id: str
     ) -> Tuple[Optional[str], Optional[str]]:

main/xiaozhi-server/core/providers/tts/base.py

@@ -216,6 +216,7 @@ class TTSProviderBase(ABC):
                 logger.bind(tag=TAG).error(
                     f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
                 )
+                continue
 
     def _audio_play_priority_thread(self):
         while not self.conn.stop_event.is_set():

main/xiaozhi-server/core/providers/tts/huoshan_double_stream.py

@@ -6,6 +6,7 @@ import asyncio
 import threading
 import traceback
 import websockets
+import time
 from config.logger import setup_logging
 from core.utils import opus_encoder_utils
 from core.utils.util import check_model_key
@@ -138,6 +139,7 @@ class Response:
 class TTSProvider(TTSProviderBase):
     def __init__(self, config, delete_audio_file):
         super().__init__(config, delete_audio_file)
+        self.ws = None  # 初始化ws属性
         self.interface_type = InterfaceType.DUAL_STREAM
         self.appId = config.get("appid")
         self.access_token = config.get("access_token")
@@ -166,59 +168,133 @@ class TTSProvider(TTSProviderBase):
         self._current_session_id = None  # 当前会话ID
         self._session_started = False  # 会话是否已开始
         self._session_finished = False  # 会话是否已结束
+        self._connection_ready = False  # 连接是否就绪
+        self._reconnect_attempts = 0  # 重连尝试次数
+        self._max_reconnect_attempts = 3  # 最大重连次数
 
     ###################################################################################
     # 火山双流式TTS重写父类的方法--开始
     ###################################################################################
 
     async def open_audio_channels(self, conn):
-        await super().open_audio_channels(conn)
+        try:
-        ws_header = {
+            await super().open_audio_channels(conn)
-            "X-Api-App-Key": self.appId,
+            await self._ensure_connection()
-            "X-Api-Access-Key": self.access_token,
+            tts_priority = threading.Thread(
-            "X-Api-Resource-Id": self.resource_id,
+                target=self._start_monitor_tts_response_thread, daemon=True
-            "X-Api-Connect-Id": uuid.uuid4(),
+            )
-        }
+            tts_priority.start()
-        self.ws = await websockets.connect(
+        except Exception as e:
-            self.ws_url, additional_headers=ws_header, max_size=1000000000
+            logger.bind(tag=TAG).error(f"Failed to open audio channels: {str(e)}")
-        )
+            self.ws = None
-        tts_priority = threading.Thread(
+            raise
-            target=self._start_monitor_tts_response_thread, daemon=True
+
-        )
+    async def _ensure_connection(self):
-        tts_priority.start()
+        """确保WebSocket连接可用"""
+        try:
+            if self.ws is None:
+                logger.bind(tag=TAG).info("WebSocket连接不存在，开始建立新连接...")
+                ws_header = {
+                    "X-Api-App-Key": self.appId,
+                    "X-Api-Access-Key": self.access_token,
+                    "X-Api-Resource-Id": self.resource_id,
+                    "X-Api-Connect-Id": uuid.uuid4(),
+                }
+                self.ws = await websockets.connect(
+                    self.ws_url, additional_headers=ws_header, max_size=1000000000
+                )
+                self._connection_ready = True
+                self._reconnect_attempts = 0
+                logger.bind(tag=TAG).info("WebSocket连接建立成功")
+            else:
+                # 尝试发送ping来检查连接是否还活着
+                try:
+                    logger.bind(tag=TAG).debug("检查WebSocket连接状态...")
+                    pong_waiter = await self.ws.ping()
+                    await asyncio.wait_for(pong_waiter, timeout=1.0)
+                    logger.bind(tag=TAG).debug("WebSocket连接状态正常")
+                except (asyncio.TimeoutError, websockets.ConnectionClosed):
+                    # 如果ping失败，重新建立连接
+                    logger.bind(tag=TAG).warning("WebSocket连接已断开，准备重新连接...")
+                    try:
+                        await self.ws.close()
+                    except:
+                        pass
+                    self.ws = None
+                    self._connection_ready = False
+                    # 重新建立连接
+                    await self._ensure_connection()
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"确保连接失败: {str(e)}")
+            self._connection_ready = False
+            self.ws = None
+            raise
 
     def tts_text_priority_thread(self):
+        logger.bind(tag=TAG).info("TTS文本处理线程启动")
         while not self.conn.stop_event.is_set():
             try:
+                logger.bind(tag=TAG).debug("等待TTS文本队列消息...")
                 message = self.tts_text_queue.get(timeout=1)
-                logger.bind(tag=TAG).debug(
+                logger.bind(tag=TAG).info(
-                    f"TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name}"
+                    f"收到TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
                 )
                 if message.sentence_type == SentenceType.FIRST:
                     # 初始化参数
-                    future = asyncio.run_coroutine_threadsafe(
+                    try:
-                        self.start_session(self.conn.sentence_id), loop=self.conn.loop
+                        logger.bind(tag=TAG).info("开始启动TTS会话...")
-                    )
-                    future.result()
-                    self.tts_audio_first_sentence = True
-                    self.before_stop_play_files.clear()
-                elif ContentType.TEXT == message.content_type:
-                    if message.content_detail:
                         future = asyncio.run_coroutine_threadsafe(
-                            self.text_to_speak(message.content_detail, None),
+                            self.start_session(self.conn.sentence_id),
                             loop=self.conn.loop,
                         )
                         future.result()
+                        self.tts_audio_first_sentence = True
+                        self.before_stop_play_files.clear()
+                        logger.bind(tag=TAG).info("TTS会话启动成功")
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"启动TTS会话失败: {str(e)}")
+                        # 直接跳过当前消息，不重新入队
+                        time.sleep(1)
+                        continue
+                elif ContentType.TEXT == message.content_type:
+                    if message.content_detail:
+                        try:
+                            logger.bind(tag=TAG).info(
+                                f"开始发送TTS文本: {message.content_detail}"
+                            )
+                            future = asyncio.run_coroutine_threadsafe(
+                                self.text_to_speak(message.content_detail, None),
+                                loop=self.conn.loop,
+                            )
+                            future.result()
+                            logger.bind(tag=TAG).info("TTS文本发送成功")
+                        except Exception as e:
+                            logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
+                            # 直接跳过当前消息，不重新入队
+                            time.sleep(1)
+                            continue
                 elif ContentType.FILE == message.content_type:
+                    logger.bind(tag=TAG).info(
+                        f"添加音频文件到待播放列表: {message.content_file}"
+                    )
                     self.before_stop_play_files.append(
                         (message.content_file, message.content_detail)
                     )
 
                 if message.sentence_type == SentenceType.LAST:
-                    future = asyncio.run_coroutine_threadsafe(
+                    try:
-                        self.finish_session(self.conn.sentence_id), loop=self.conn.loop
+                        logger.bind(tag=TAG).info("开始结束TTS会话...")
-                    )
+                        future = asyncio.run_coroutine_threadsafe(
-                    future.result()
+                            self.finish_session(self.conn.sentence_id),
+                            loop=self.conn.loop,
+                        )
+                        future.result()
+                        logger.bind(tag=TAG).info("TTS会话结束成功")
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"结束TTS会话失败: {str(e)}")
+                        # 直接跳过当前消息，不重新入队
+                        time.sleep(1)
+                        continue
 
             except queue.Empty:
                 continue
@@ -226,11 +302,30 @@ class TTSProvider(TTSProviderBase):
                 logger.bind(tag=TAG).error(
                     f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
                 )
+                # 如果是WebSocket连接关闭错误，等待一段时间后继续
+                if "non-exist session" in str(e):
+                    time.sleep(1)
+                continue
 
     async def text_to_speak(self, text, _):
-        # 发送文本
+        """发送文本到TTS服务"""
-        await self.send_text(self.speaker, text, self.conn.sentence_id)
+        try:
-        return
+            # 确保WebSocket连接可用
+            if not self._connection_ready or self.ws is None:
+                logger.bind(tag=TAG).warning("WebSocket连接不可用，尝试重新连接...")
+                await self._ensure_connection()
+
+            # 发送文本
+            await self.send_text(self.speaker, text, self.conn.sentence_id)
+            return
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
+            # 如果是连接问题，尝试重新连接
+            if isinstance(e, websockets.ConnectionClosed):
+                self._connection_ready = False
+                self.ws = None
+                await self._handle_connection_error()
+            raise
 
     ###################################################################################
     # 火山双流式TTS重写父类的方法--结束
@@ -308,14 +403,21 @@ class TTSProvider(TTSProviderBase):
     async def send_event(
         self, header: bytes, optional: bytes | None = None, payload: bytes = None
     ):
-        full_client_request = bytearray(header)
+        try:
-        if optional is not None:
+            full_client_request = bytearray(header)
-            full_client_request.extend(optional)
+            if optional is not None:
-        if payload is not None:
+                full_client_request.extend(optional)
-            payload_size = len(payload).to_bytes(4, "big", signed=True)
+            if payload is not None:
-            full_client_request.extend(payload_size)
+                payload_size = len(payload).to_bytes(4, "big", signed=True)
-            full_client_request.extend(payload)
+                full_client_request.extend(payload_size)
-        await self.ws.send(full_client_request)
+                full_client_request.extend(payload)
+            await self.ws.send(full_client_request)
+        except websockets.ConnectionClosed:
+            if await self._handle_connection_error():
+                # 重连成功后重试发送
+                await self.ws.send(full_client_request)
+            else:
+                raise
 
     async def send_text(self, speaker: str, text: str, session_id):
         header = Header(
@@ -450,63 +552,154 @@ class TTSProvider(TTSProviderBase):
         return
 
     async def start_session(self, session_id):
-        logger.bind(tag=TAG).debug(f"开始会话～～{session_id}")
+        logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
-        async with self._session_lock:
+        try:
-            # 如果已有会话未结束，先关闭它
+            async with self._session_lock:
-            if self._session_started and not self._session_finished:
+                try:
-                logger.bind(tag=TAG).warning(
+                    # 确保连接可用
-                    f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
+                    logger.bind(tag=TAG).info("检查WebSocket连接状态...")
-                )
+                    await asyncio.wait_for(self._ensure_connection(), timeout=5)
-                await self.finish_session(self._current_session_id)
 
-            # 重置会话状态
+                    # 如果已有会话未结束，先关闭它
-            self._current_session_id = session_id
+                    if self._session_started and not self._session_finished:
-            self._session_started = True
+                        logger.bind(tag=TAG).warning(
-            self._session_finished = False
+                            f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
+                        )
+                        try:
+                            await asyncio.wait_for(
+                                self.finish_session(self._current_session_id), timeout=5
+                            )
+                        except Exception as e:
+                            logger.bind(tag=TAG).error(f"关闭旧会话失败: {str(e)}")
+                            # 强制重置会话状态
+                            self._session_started = False
+                            self._session_finished = True
+                            self._current_session_id = None
 
-            header = Header(
+                    # 重置会话状态
-                message_type=FULL_CLIENT_REQUEST,
+                    self._current_session_id = session_id
-                message_type_specific_flags=MsgTypeFlagWithEvent,
+                    self._session_started = True
-                serial_method=JSON,
+                    self._session_finished = False
-            ).as_bytes()
+                    logger.bind(tag=TAG).info(
-            optional = Optional(
+                        f"会话状态已更新 - 开始: {self._session_started}, 结束: {self._session_finished}"
-                event=EVENT_StartSession, sessionId=session_id
+                    )
-            ).as_bytes()
+
-            payload = self.get_payload_bytes(
+                    header = Header(
-                event=EVENT_StartSession, speaker=self.speaker
+                        message_type=FULL_CLIENT_REQUEST,
-            )
+                        message_type_specific_flags=MsgTypeFlagWithEvent,
-            await self.send_event(header, optional, payload)
+                        serial_method=JSON,
+                    ).as_bytes()
+                    optional = Optional(
+                        event=EVENT_StartSession, sessionId=session_id
+                    ).as_bytes()
+                    payload = self.get_payload_bytes(
+                        event=EVENT_StartSession, speaker=self.speaker
+                    )
+                    await asyncio.wait_for(
+                        self.send_event(header, optional, payload), timeout=5
+                    )
+                    logger.bind(tag=TAG).info("会话启动请求已发送")
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
+                    self._session_started = False
+                    self._session_finished = True
+                    self._current_session_id = None
+                    raise
+        except asyncio.TimeoutError:
+            logger.bind(tag=TAG).error(f"启动会话超时: {session_id}")
+            # 超时后强制重置会话状态
+            self._session_started = False
+            self._session_finished = True
+            self._current_session_id = None
+            # 尝试关闭WebSocket连接
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"启动会话时发生未知错误: {str(e)}")
+            # 发生未知错误时也重置会话状态
+            self._session_started = False
+            self._session_finished = True
+            self._current_session_id = None
 
     async def finish_session(self, session_id):
-        logger.bind(tag=TAG).debug(f"关闭会话～～{session_id}")
+        logger.bind(tag=TAG).info(f"关闭会话～～{session_id}")
-        async with self._session_lock:
+        try:
-            # 检查会话状态
+            async with self._session_lock:
-            if not self._session_started:
+                try:
-                logger.bind(tag=TAG).warning(f"尝试关闭未开始的会话 {session_id}")
+                    # 检查会话状态
-                return
+                    if not self._session_started:
+                        logger.bind(tag=TAG).warning(
+                            f"尝试关闭未开始的会话 {session_id}"
+                        )
+                        return
 
-            if self._session_finished:
+                    if self._session_finished:
-                logger.bind(tag=TAG).warning(f"会话 {session_id} 已经关闭")
+                        logger.bind(tag=TAG).warning(f"会话 {session_id} 已经关闭")
-                return
+                        return
 
-            if self._current_session_id != session_id:
+                    if self._current_session_id != session_id:
-                logger.bind(tag=TAG).warning(
+                        logger.bind(tag=TAG).warning(
-                    f"尝试关闭错误的会话 {session_id}，当前会话为 {self._current_session_id}"
+                            f"尝试关闭错误的会话 {session_id}，当前会话为 {self._current_session_id}"
-                )
+                        )
-                return
+                        # 即使会话ID不匹配，也尝试关闭当前会话
+                        if self._current_session_id:
+                            session_id = self._current_session_id
 
-            header = Header(
+                    # 确保WebSocket连接可用
-                message_type=FULL_CLIENT_REQUEST,
+                    if self.ws is None:
-                message_type_specific_flags=MsgTypeFlagWithEvent,
+                        logger.bind(tag=TAG).warning(
-                serial_method=JSON,
+                            "WebSocket连接不存在，尝试重新连接..."
-            ).as_bytes()
+                        )
-            optional = Optional(
+                        await asyncio.wait_for(self._ensure_connection(), timeout=5)
-                event=EVENT_FinishSession, sessionId=session_id
-            ).as_bytes()
-            payload = str.encode("{}")
-            await self.send_event(header, optional, payload)
 
-            # 更新会话状态
+                    header = Header(
+                        message_type=FULL_CLIENT_REQUEST,
+                        message_type_specific_flags=MsgTypeFlagWithEvent,
+                        serial_method=JSON,
+                    ).as_bytes()
+                    optional = Optional(
+                        event=EVENT_FinishSession, sessionId=session_id
+                    ).as_bytes()
+                    payload = str.encode("{}")
+                    await asyncio.wait_for(
+                        self.send_event(header, optional, payload), timeout=5
+                    )
+                    logger.bind(tag=TAG).info("会话结束请求已发送")
+
+                    # 更新会话状态
+                    self._session_finished = True
+                    self._session_started = False
+                    self._current_session_id = None
+                    logger.bind(tag=TAG).info(
+                        "会话状态已更新 - 开始: False, 结束: True"
+                    )
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
+                    # 即使发生错误，也要重置会话状态
+                    self._session_finished = True
+                    self._session_started = False
+                    self._current_session_id = None
+                    raise
+        except asyncio.TimeoutError:
+            logger.bind(tag=TAG).error(f"关闭会话超时: {session_id}")
+            # 超时后强制重置会话状态
+            self._session_finished = True
+            self._session_started = False
+            self._current_session_id = None
+            # 尝试关闭WebSocket连接
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"关闭会话时发生未知错误: {str(e)}")
+            # 发生未知错误时也重置会话状态
             self._session_finished = True
             self._session_started = False
             self._current_session_id = None
@@ -527,3 +720,20 @@ class TTSProvider(TTSProviderBase):
     def wav_to_opus_data_audio_raw(self, raw_data_var, is_end=False):
         opus_datas = self.opus_encoder.encode_pcm_to_opus(raw_data_var, is_end)
         return opus_datas
+
+    async def _handle_connection_error(self):
+        """处理连接错误"""
+        if self._reconnect_attempts < self._max_reconnect_attempts:
+            self._reconnect_attempts += 1
+            logger.bind(tag=TAG).warning(
+                f"尝试重新连接 (第{self._reconnect_attempts}次)"
+            )
+            try:
+                await self._ensure_connection()
+                return True
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"重新连接失败: {str(e)}")
+                return False
+        else:
+            logger.bind(tag=TAG).error("达到最大重连次数，放弃重连")
+            return False

main/xiaozhi-server/core/providers/vllm/base.py

@@ -0,0 +1,12 @@
+from abc import ABC, abstractmethod
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class VLLMProviderBase(ABC):
+    @abstractmethod
+    def response(self, question, base64_image):
+        """VLLM response generator"""
+        pass

main/xiaozhi-server/core/providers/vllm/openai.py

@@ -0,0 +1,63 @@
+import openai
+import json
+from config.logger import setup_logging
+from core.utils.util import check_model_key
+from core.providers.vllm.base import VLLMProviderBase
+
+TAG = __name__
+logger = setup_logging()
+
+
+class VLLMProvider(VLLMProviderBase):
+    def __init__(self, config):
+        self.model_name = config.get("model_name")
+        self.api_key = config.get("api_key")
+        if "base_url" in config:
+            self.base_url = config.get("base_url")
+        else:
+            self.base_url = config.get("url")
+
+        param_defaults = {
+            "max_tokens": (500, int),
+            "temperature": (0.7, lambda x: round(float(x), 1)),
+            "top_p": (1.0, lambda x: round(float(x), 1)),
+        }
+
+        for param, (default, converter) in param_defaults.items():
+            value = config.get(param)
+            try:
+                setattr(
+                    self,
+                    param,
+                    converter(value) if value not in (None, "") else default,
+                )
+            except (ValueError, TypeError):
+                setattr(self, param, default)
+
+        check_model_key("VLLM", self.api_key)
+        self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
+
+    def response(self, question, base64_image):
+        try:
+            messages = [
+                {
+                    "role": "user",
+                    "content": [
+                        {"type": "text", "text": question},
+                        {
+                            "type": "image_url",
+                            "image_url": {"url": f"{base64_image}"},
+                        },
+                    ],
+                }
+            ]
+
+            response = self.client.chat.completions.create(
+                model=self.model_name, messages=messages, stream=False
+            )
+
+            return response.choices[0].message.content
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"Error in response generation: {e}")
+            raise

main/xiaozhi-server/core/utils/auth.py

@@ -0,0 +1,43 @@
+import jwt
+import time
+from datetime import datetime, timedelta, timezone
+from typing import Optional, Tuple
+
+
+class AuthToken:
+    def __init__(self, secret_key: str):
+        self.secret_key = secret_key
+
+    def generate_token(self, device_id: str) -> str:
+        """
+        生成JWT token
+        :param device_id: 设备ID
+        :return: JWT token字符串
+        """
+        # 设置过期时间为1小时后
+        expire_time = datetime.now(timezone.utc) + timedelta(hours=1)
+
+        # 创建payload
+        payload = {"device_id": device_id, "exp": expire_time.timestamp()}
+
+        # 使用JWT进行编码
+        token = jwt.encode(payload, self.secret_key, algorithm="HS256")
+        return token
+
+    def verify_token(self, token: str) -> Tuple[bool, Optional[str]]:
+        """
+        验证token
+        :param token: JWT token字符串
+        :return: (是否有效, 设备ID)
+        """
+        try:
+            # 解码token
+            payload = jwt.decode(token, self.secret_key, algorithms=["HS256"])
+
+            # 检查是否过期
+            if payload["exp"] < time.time():
+                return False, None
+
+            return True, payload["device_id"]
+        except jwt.InvalidTokenError:
+            return False, None

main/xiaozhi-server/core/utils/modules_initialize.py

@@ -0,0 +1,128 @@
+from typing import Dict, Any
+from config.logger import setup_logging
+from core.utils import tts, llm, intent, memory, vad, asr
+
+TAG = __name__
+logger = setup_logging()
+
+
+def initialize_modules(
+    logger,
+    config: Dict[str, Any],
+    init_vad=False,
+    init_asr=False,
+    init_llm=False,
+    init_tts=False,
+    init_memory=False,
+    init_intent=False,
+) -> Dict[str, Any]:
+    """
+    初始化所有模块组件
+
+    Args:
+        config: 配置字典
+
+    Returns:
+        Dict[str, Any]: 包含所有初始化后的模块的字典
+    """
+    modules = {}
+
+    # 初始化TTS模块
+    if init_tts:
+        select_tts_module = config["selected_module"]["TTS"]
+        modules["tts"] = initialize_tts(config)
+        logger.bind(tag=TAG).info(f"初始化组件: tts成功 {select_tts_module}")
+
+    # 初始化LLM模块
+    if init_llm:
+        select_llm_module = config["selected_module"]["LLM"]
+        llm_type = (
+            select_llm_module
+            if "type" not in config["LLM"][select_llm_module]
+            else config["LLM"][select_llm_module]["type"]
+        )
+        modules["llm"] = llm.create_instance(
+            llm_type,
+            config["LLM"][select_llm_module],
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: llm成功 {select_llm_module}")
+
+    # 初始化Intent模块
+    if init_intent:
+        select_intent_module = config["selected_module"]["Intent"]
+        intent_type = (
+            select_intent_module
+            if "type" not in config["Intent"][select_intent_module]
+            else config["Intent"][select_intent_module]["type"]
+        )
+        modules["intent"] = intent.create_instance(
+            intent_type,
+            config["Intent"][select_intent_module],
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: intent成功 {select_intent_module}")
+
+    # 初始化Memory模块
+    if init_memory:
+        select_memory_module = config["selected_module"]["Memory"]
+        memory_type = (
+            select_memory_module
+            if "type" not in config["Memory"][select_memory_module]
+            else config["Memory"][select_memory_module]["type"]
+        )
+        modules["memory"] = memory.create_instance(
+            memory_type,
+            config["Memory"][select_memory_module],
+            config.get("summaryMemory", None),
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: memory成功 {select_memory_module}")
+
+    # 初始化VAD模块
+    if init_vad:
+        select_vad_module = config["selected_module"]["VAD"]
+        vad_type = (
+            select_vad_module
+            if "type" not in config["VAD"][select_vad_module]
+            else config["VAD"][select_vad_module]["type"]
+        )
+        modules["vad"] = vad.create_instance(
+            vad_type,
+            config["VAD"][select_vad_module],
+        )
+        logger.bind(tag=TAG).info(f"初始化组件: vad成功 {select_vad_module}")
+
+    # 初始化ASR模块
+    if init_asr:
+        select_asr_module = config["selected_module"]["ASR"]
+        modules["asr"] = initialize_asr(config)
+        logger.bind(tag=TAG).info(f"初始化组件: asr成功 {select_asr_module}")
+    return modules
+
+
+def initialize_tts(config):
+    select_tts_module = config["selected_module"]["TTS"]
+    tts_type = (
+        select_tts_module
+        if "type" not in config["TTS"][select_tts_module]
+        else config["TTS"][select_tts_module]["type"]
+    )
+    new_tts = tts.create_instance(
+        tts_type,
+        config["TTS"][select_tts_module],
+        str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
+    )
+    return new_tts
+
+
+def initialize_asr(config):
+    select_asr_module = config["selected_module"]["ASR"]
+    asr_type = (
+        select_asr_module
+        if "type" not in config["ASR"][select_asr_module]
+        else config["ASR"][select_asr_module]["type"]
+    )
+    new_asr = asr.create_instance(
+        asr_type,
+        config["ASR"][select_asr_module],
+        str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
+    )
+    return new_asr

main/xiaozhi-server/core/utils/util.py

@@ -7,8 +7,6 @@ import numpy as np
 import requests
 import opuslib_next
 from pydub import AudioSegment
-from typing import Dict, Any
-from core.utils import tts, llm, intent, memory, vad, asr
 import copy
 
 TAG = __name__
@@ -245,122 +243,6 @@ def extract_json_from_string(input_string):
     return None
 
 
-def initialize_modules(
-    logger,
-    config: Dict[str, Any],
-    init_vad=False,
-    init_asr=False,
-    init_llm=False,
-    init_tts=False,
-    init_memory=False,
-    init_intent=False,
-) -> Dict[str, Any]:
-    """
-    初始化所有模块组件
-
-    Args:
-        config: 配置字典
-
-    Returns:
-        Dict[str, Any]: 包含所有初始化后的模块的字典
-    """
-    modules = {}
-
-    # 初始化TTS模块
-    if init_tts:
-        select_tts_module = config["selected_module"]["TTS"]
-        modules["tts"] = initialize_tts(config)
-        logger.bind(tag=TAG).info(f"初始化组件: tts成功 {select_tts_module}")
-
-    # 初始化LLM模块
-    if init_llm:
-        select_llm_module = config["selected_module"]["LLM"]
-        llm_type = (
-            select_llm_module
-            if "type" not in config["LLM"][select_llm_module]
-            else config["LLM"][select_llm_module]["type"]
-        )
-        modules["llm"] = llm.create_instance(
-            llm_type,
-            config["LLM"][select_llm_module],
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: llm成功 {select_llm_module}")
-
-    # 初始化Intent模块
-    if init_intent:
-        select_intent_module = config["selected_module"]["Intent"]
-        intent_type = (
-            select_intent_module
-            if "type" not in config["Intent"][select_intent_module]
-            else config["Intent"][select_intent_module]["type"]
-        )
-        modules["intent"] = intent.create_instance(
-            intent_type,
-            config["Intent"][select_intent_module],
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: intent成功 {select_intent_module}")
-
-    # 初始化Memory模块
-    if init_memory:
-        select_memory_module = config["selected_module"]["Memory"]
-        memory_type = (
-            select_memory_module
-            if "type" not in config["Memory"][select_memory_module]
-            else config["Memory"][select_memory_module]["type"]
-        )
-        modules["memory"] = memory.create_instance(
-            memory_type,
-            config["Memory"][select_memory_module],
-            config.get("summaryMemory", None),
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: memory成功 {select_memory_module}")
-
-    # 初始化VAD模块
-    if init_vad:
-        select_vad_module = config["selected_module"]["VAD"]
-        vad_type = (
-            select_vad_module
-            if "type" not in config["VAD"][select_vad_module]
-            else config["VAD"][select_vad_module]["type"]
-        )
-        modules["vad"] = vad.create_instance(
-            vad_type,
-            config["VAD"][select_vad_module],
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: vad成功 {select_vad_module}")
-
-    # 初始化ASR模块
-    if init_asr:
-        select_asr_module = config["selected_module"]["ASR"]
-        asr_type = (
-            select_asr_module
-            if "type" not in config["ASR"][select_asr_module]
-            else config["ASR"][select_asr_module]["type"]
-        )
-        modules["asr"] = asr.create_instance(
-            asr_type,
-            config["ASR"][select_asr_module],
-            str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
-        )
-        logger.bind(tag=TAG).info(f"初始化组件: asr成功 {select_asr_module}")
-    return modules
-
-
-def initialize_tts(config):
-    select_tts_module = config["selected_module"]["TTS"]
-    tts_type = (
-        select_tts_module
-        if "type" not in config["TTS"][select_tts_module]
-        else config["TTS"][select_tts_module]["type"]
-    )
-    new_tts = tts.create_instance(
-        tts_type,
-        config["TTS"][select_tts_module],
-        str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
-    )
-    return new_tts
-
-
 def analyze_emotion(text):
     """
     分析文本情感并返回对应的emoji名称（支持中英文）
@@ -1000,3 +882,51 @@ def filter_sensitive_info(config: dict) -> dict:
         return filtered
 
     return _filter_dict(copy.deepcopy(config))
+
+
+def get_vision_url(config: dict) -> str:
+    """获取 vision URL
+
+    Args:
+        config: 配置字典
+
+    Returns:
+        str: vision URL
+    """
+    server_config = config["server"]
+    vision_explain = server_config.get("vision_explain", "")
+    if "你的" in vision_explain:
+        local_ip = get_local_ip()
+        port = int(server_config.get("http_port", 8003))
+        vision_explain = f"http://{local_ip}:{port}/mcp/vision/explain"
+    return vision_explain
+
+
+def is_valid_image_file(file_data: bytes) -> bool:
+    """
+    检查文件数据是否为有效的图片格式
+
+    Args:
+        file_data: 文件的二进制数据
+
+    Returns:
+        bool: 如果是有效的图片格式返回True，否则返回False
+    """
+    # 常见图片格式的魔数（文件头）
+    image_signatures = {
+        b"\xff\xd8\xff": "JPEG",
+        b"\x89PNG\r\n\x1a\n": "PNG",
+        b"GIF87a": "GIF",
+        b"GIF89a": "GIF",
+        b"BM": "BMP",
+        b"II*\x00": "TIFF",
+        b"MM\x00*": "TIFF",
+        b"RIFF": "WEBP",
+    }
+
+    # 检查文件头是否匹配任何已知的图片格式
+    for signature in image_signatures:
+        if file_data.startswith(signature):
+            return True
+
+    return False

main/xiaozhi-server/core/utils/vllm.py

@@ -0,0 +1,23 @@
+import os
+import sys
+
+# 添加项目根目录到Python路径
+current_dir = os.path.dirname(os.path.abspath(__file__))
+project_root = os.path.abspath(os.path.join(current_dir, "..", ".."))
+sys.path.insert(0, project_root)
+
+from config.logger import setup_logging
+import importlib
+
+logger = setup_logging()
+
+
+def create_instance(class_name, *args, **kwargs):
+    # 创建LLM实例
+    if os.path.exists(os.path.join("core", "providers", "vllm", f"{class_name}.py")):
+        lib_name = f"core.providers.vllm.{class_name}"
+        if lib_name not in sys.modules:
+            sys.modules[lib_name] = importlib.import_module(f"{lib_name}")
+        return sys.modules[lib_name].VLLMProvider(*args, **kwargs)
+
+    raise ValueError(f"不支持的VLLM类型: {class_name}，请检查该配置的type是否设置正确")

main/xiaozhi-server/core/websocket_server.py

@@ -2,8 +2,9 @@ import asyncio
 import websockets
 from config.logger import setup_logging
 from core.connection import ConnectionHandler
-from core.utils.util import initialize_modules, check_vad_update, check_asr_update
 from config.config_loader import get_config_from_api
+from core.utils.modules_initialize import initialize_modules
+from core.utils.util import check_vad_update, check_asr_update
 
 TAG = __name__
 

main/xiaozhi-server/docker-compose.yml

@@ -13,8 +13,8 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
-      # ota服务端
+      # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-      - "8002:8002"
+      - "8003:8003"
     volumes:
       # 配置文件目录
       - ./data:/opt/xiaozhi-esp32-server/data

main/xiaozhi-server/docker-compose_all.yml

@@ -15,6 +15,8 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
+      # http服务的端口，用于视觉分析接口
+      - "8003:8003"
     security_opt:
       - seccomp:unconfined
     environment:

main/xiaozhi-server/requirements.txt

@@ -31,3 +31,4 @@ chardet==5.2.0
 aioconsole==0.8.1
 markitdown==0.1.1
 mcp-proxy==0.6.0
+PyJWT==2.8.0

main/xiaozhi-server/test/test_page.html

@@ -1484,6 +1484,23 @@
                                 if (message.text && message.text !== '😊') {
                                     addMessage(message.text);
                                 }
+                            }else if (message.type === 'mcp') {
+                                const payload = message.payload || {};
+                                log(`服务器下发: ${JSON.stringify(message)}`, 'info');
+                                if (payload) {
+                                    // 模拟小智客户端行为
+                                    if(payload.method === 'tools/list'){
+                                        const replay_message = JSON.stringify({"session_id":"","type":"mcp","payload":{"jsonrpc":"2.0","id":2,"result":{"tools":[{"name":"self.get_device_status","description":"Provides the real-time information of the device, including the current status of the audio speaker, screen, battery, network, etc.\nUse this tool for: \n1. Answering questions about current condition (e.g. what is the current volume of the audio speaker?)\n2. As the first step to control the device (e.g. turn up / down the volume of the audio speaker, etc.)","inputSchema":{"type":"object","properties":{}}},{"name":"self.audio_speaker.set_volume","description":"Set the volume of the audio speaker. If the current volume is unknown, you must call `self.get_device_status` tool first and then call this tool.","inputSchema":{"type":"object","properties":{"volume":{"type":"integer","minimum":0,"maximum":100}},"required":["volume"]}},{"name":"self.screen.set_brightness","description":"Set the brightness of the screen.","inputSchema":{"type":"object","properties":{"brightness":{"type":"integer","minimum":0,"maximum":100}},"required":["brightness"]}},{"name":"self.screen.set_theme","description":"Set the theme of the screen. The theme can be 'light' or 'dark'.","inputSchema":{"type":"object","properties":{"theme":{"type":"string"}},"required":["theme"]}}]}}})
+                                        websocket.send(replay_message);
+                                        log(`回复MCP消息: ${replay_message}`, 'info');
+                                    } else if(payload.method === 'tools/call'){
+                                        // 模拟回复
+                                        const replay_message = JSON.stringify({"session_id":"9f261599","type":"mcp","payload":{"jsonrpc":"2.0","id": payload.id,"result":{"content":[{"type":"text","text":"true"}],"isError":false}}})
+                                        websocket.send(replay_message);
+                                        log(`回复MCP消息: ${replay_message}`, 'info');
+                                    }
+                                }
+                                
                             } else {
                                 // 未知消息类型
                                 log(`未知消息类型: ${message.type}`, 'info');
@@ -1523,7 +1540,10 @@
                     device_id: config.deviceId,
                     device_name: config.deviceName,
                     device_mac: config.deviceMac,
-                    token: config.token
+                    token: config.token,
+                    features: {
+                        mcp: true
+                    }
                 };
 
                 log('发送hello握手消息', 'info');