Merge pull request #1602 from xinnan-tech/py_fix_bug

update:更新版本号

README.md

@@ -121,6 +121,33 @@
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="实时打断" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="拍照识物品" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="多指令任务" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -143,8 +170,8 @@
 #### 🚀 部署方式选择
 | 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | 
 |---------|------|---------|---------|---------|---------|
-| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
+| **最简化安装** | 智能对话、IOT、MCP、视觉感知，数据存储在配置文件 | 低配置环境，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
-| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
+| **全模块安装** | 智能对话、IOT、MCP、视觉感知、OTA、智控台，数据存储在数据库 | 完整功能体验 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
 
 > 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
@@ -159,34 +186,51 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 #### 🚩 配置说明和推荐
 > [!Note]
-> 本项目默认的配置是`入门全免费`设置，如果想效果更优，推荐使用`全流式配置`。
+> 本项目提供两种配置方案：
 > 
-> 本项目自`0.5.2`版本，已支持整个生命周期全流式，相比`0.5`版本以前，响应速度提升约`2.5秒`
+> 1. `入门全免费`配置：适合个人家庭使用，所有组件均采用免费方案，无需额外付费。
+> 
+> 2. `流式配置`：适合演示、培训、超过2个并发等场景，采用流式处理技术，响应速度更快，体验更佳。
+> 
+> 自`0.5.2`版本起，项目支持流式配置，相比早期版本，响应速度提升约`2.5秒`，显著改善用户体验。
 
+| 模块名称 | 入门全免费设置 | 流式配置 |
+|:---:|:---:|:---:|
+| ASR(语音识别) | FunASR(本地) | 👍FunASRServer 或 👍DoubaoStreamASR |
+| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
+| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
+| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) |
+| Intent(意图识别) | function_call(函数调用) | function_call(函数调用) |
+| Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） |
 
-| 模块名称 | 入门全免费设置 | 全流式配置 |
+#### 🔧 测试工具
-|---------|---------|------|
+本项目提供以下测试工具，帮助您验证系统和选择合适的模型：
-| ASR(语音识别) | FunASR(本地) | ✅DoubaoASR(火山流式语音识别) |
+
-| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | ✅DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
+| 工具名称 | 位置 | 使用方法 | 功能说明 |
-| TTS(语音合成) | EdgeTTS(微软语音) | ✅HuoshanDoubleStreamTTS(火山双流式语音合成) |
+|:---:|:---|:---:|:---:|
-| Intent(意图识别) | function_call(函数调用) | ✅function_call(函数调用) |
+| 音频交互测试工具 | main》xiaozhi-server》test》test_page.html | 使用谷歌浏览器直接打开 | 测试音频播放和接收功能，验证Python端音频处理是否正常 |
-| Memory(记忆功能) | mem_local_short(本地短期记忆） | ✅mem_local_short（本地短期记忆） |
+| 模型响应测试工具1 | main》xiaozhi-server》performance_tester.py | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、TTS(语音合成)三个核心模块的响应速度 |
+| 模型响应测试工具2 | main》xiaozhi-server》performance_tester_vllm.py | 执行 `python performance_tester_vllm.py` | 测试VLLM(视觉模型)的响应速度 |
+
+> 💡 提示：测试模型速度时，只会测试配置了密钥的模型。
 
 ---
 ## 功能清单 ✨
 ### 已实现 ✅
 
 | 功能模块 | 描述 |
-|---------|------|
+|:---:|:---|
-| 通信协议 | 基于 `xiaozhi-esp32` 协议，通过 WebSocket 实现数据交互 |
+| 核心服务架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
-| 对话交互 | 支持唤醒对话、手动对话及实时打断。长时间无对话时自动休眠 |
+| 语音交互系统 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
-| 意图识别 | 支持使用LLM意图识别、function call函数调用，减少硬编码意图判断 |
+| 智能对话系统 | 支持多种LLM(大语言模型)，实现智能对话 |
-| 多语言识别 | 支持国语、粤语、英语、日语、韩语（默认使用 FunASR） |
+| 视觉感知系统 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
-| LLM 模块 | 支持灵活切换 LLM 模块，默认使用 ChatGLMLLM，也可选用阿里百炼、DeepSeek、Ollama 等接口 |
+| 意图识别系统 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
-| TTS 模块 | 支持 EdgeTTS（默认）、火山引擎豆包 TTS 等多种 TTS 接口，满足语音合成需求 |
+| 记忆系统 | 支持本地短期记忆、mem0ai接口记忆，具备记忆总结功能 |
-| 记忆功能 | 支持超长记忆、本地总结记忆、无记忆三种模式，满足不同场景需求 |
+| IOT/MCP控制协议 | 支持设备注册管理、智能控制接口，同时支持IOT、MCP控制协议 |
-| IOT功能 | 支持管理注册设备IOT功能，支持基于对话上下文语境下的智能物联网控制 |
+| 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理 |
-| 智控台 | 提供Web管理界面，支持智能体管理、用户管理、系统配置等功能，方便管理员和用户进行管理 |
+| 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
+| 部署支持 | 支持Docker部署和本地部署，提供完整的配置文件管理 |
+| 插件系统 | 支持功能插件扩展、自定义插件开发和插件热加载 |
 
 ### 正在开发 🚧
 
@@ -223,11 +267,21 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 ---
 
+### VLLM 视觉模型
+
+| 使用方式 | 支持平台 | 免费平台 |
+|:---:|:---:|:---:|
+| openai 接口调用 | 阿里百炼、智谱ChatGLMVLLM | 智谱ChatGLMVLLM |
+
+实际上，任何支持 openai 接口调用的 VLLM 均可接入使用。
+
+---
+
 ### TTS 语音合成
 
 | 使用方式 | 支持平台 | 免费平台 |
 |:---:|:---:|:---:|
-| 接口调用 | EdgeTTS、火山引擎豆包TTS、腾讯云、阿里云TTS、CosyVoiceSiliconflow、TTS302AI、CozeCnTTS、GizwitsTTS、ACGNTTS、OpenAITTS | EdgeTTS、CosyVoiceSiliconflow(部分) |
+| 接口调用 | EdgeTTS、火山引擎豆包TTS、腾讯云、阿里云TTS、CosyVoiceSiliconflow、TTS302AI、CozeCnTTS、GizwitsTTS、ACGNTTS、OpenAITTS、灵犀流式TTS | 灵犀流式TTS、EdgeTTS、CosyVoiceSiliconflow(部分) |
 | 本地服务 | FishSpeech、GPT_SOVITS_V2、GPT_SOVITS_V3、MinimaxTTS | FishSpeech、GPT_SOVITS_V2、GPT_SOVITS_V3、MinimaxTTS |
 
 ---

README_en.md

@@ -6,14 +6,14 @@
 This project provides backend services for the open-source smart hardware project
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
 Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
-Helping you quickly set up your Xiaozhi server
+Helps you quickly set up your Xiaozhi server
 </p>
 
 <p align="center">
 <a href="./README.md">中文</a>
 · <a href="./docs/FAQ.md">FAQ</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
-· <a href="./README_ed.md#deployment-documentation">Deployment Guide</a>
+· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
 </p>
 <p align="center">
@@ -50,7 +50,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
          <picture>
-           <img alt="Xiaozhi esp32 connecting to custom backend model" src="docs/images/demo1.png" />
+           <img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
          </picture>
         </a>
     </td>
@@ -64,14 +64,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
          <picture>
-           <img alt="Cantonese communication" src="docs/images/demo3.png" />
+           <img alt="Using Cantonese" src="docs/images/demo3.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
          <picture>
-           <img alt="Home appliance control" src="docs/images/demo5.png" />
+           <img alt="Control home appliances" src="docs/images/demo5.png" />
          </picture>
         </a>
     </td>
@@ -94,7 +94,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
          <picture>
-           <img alt="Music playback" src="docs/images/demo7.png" />
+           <img alt="Play music" src="docs/images/demo7.png" />
          </picture>
         </a>
     </td>
@@ -108,7 +108,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
          <picture>
-           <img alt="IOT device control" src="docs/images/demo9.png" />
+           <img alt="IOT command control" src="docs/images/demo9.png" />
          </picture>
         </a>
     </td>
@@ -120,6 +120,33 @@ Want to see it in action? Check out these videos 🎥
         </a>
     </td>
   </tr>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
+         <picture>
+           <img alt="Real-time interruption" src="docs/images/demo10.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
+         <picture>
+           <img alt="Photo recognition" src="docs/images/demo12.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
+         <picture>
+           <img alt="Multi-command tasks" src="docs/images/demo11.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+    </td>
+    <td>
+    </td>
+  </tr>
 </table>
 
 ---
@@ -137,91 +164,102 @@ It is recommended that users prioritize service providers with relevant business
 
 ![Banners](docs/images/banner2.png)
 
-This project offers two deployment methods. Please choose based on your specific needs:
+This project provides two deployment methods. Please choose according to your specific needs:
 
 #### 🚀 Deployment Method Selection
+| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial | 
+|---------|------|---------|---------|---------|---------|
+| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - | 
+| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) | 
 
-| Deployment Method | Features | Use Case | Docker Deployment Guide | Source Code Deployment Guide |
+> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
-|---------|------|---------|---------|---------|
-| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database required | [Docker Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [Local Source Code Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)|
-| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience |[Docker Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [Local Source Code Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) |
-
-> 💡 Note: Below is the test platform deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data cleared daily
 
 ```
-Control Panel: https://2662r3426b.vicp.fun
+Control Panel Address: https://2662r3426b.vicp.fun
 
 Service Test Tool: https://2662r3426b.vicp.fun/test/
-OTA Interface: https://2662r3426b.vicp.fun/xiaozhi/ota/
+OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
-Websocket Interface: wss://2662r3426b.vicp.fun/xiaozhi/v1/
+Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ```
 
+#### 🚩 Configuration Description and Recommendations
+> [!Note]
+> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
+> 
+> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
+
+| Module Name | Entry Level Free Settings | Full Streaming Configuration |
+|---------|---------|------|
+| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
+| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
+| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
+| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
+| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
+| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
+
 ---
 ## Feature List ✨
-
 ### Implemented ✅
 
 | Feature Module | Description |
 |---------|------|
 | Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
-| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of inactivity |
+| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
 | Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
 | Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
-| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, also supports Ali Bailing, DeepSeek, Ollama, etc. |
+| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
-| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces for speech synthesis |
+| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
-| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes for different scenarios |
+| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
-| IOT Function | Supports managing registered device IOT functionality, intelligent IoT control based on dialogue context |
+| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
-| Control Panel | Provides web management interface, supports agent management, user management, system configuration, etc. |
+| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
 
 ### In Development 🚧
 
 To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
 
-If you're a software developer, here's an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
+If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
 
 ---
 
 ## Product Ecosystem 👬
-Xiaozhi is an ecosystem. When using this product, you might want to check out other excellent projects in this ecosystem:
+Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
 
-| Project Name | Project Link | Description |
+| Project Name | Project Address | Project Description |
 |:---------------------|:--------|:--------|
-| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue |
+| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
-| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | A Python-based AI client that allows you to experience Xiaozhi AI functionality through code without physical hardware |
+| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
-| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | A Java-based open-source project providing complete backend service solutions |
+| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
 
 ---
 
-## Supported Platforms/Components 📋
+## Supported Platforms/Components List 📋
 
 ### LLM Language Models
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| openai API | Ali Bailing, Volcano Engine Doubao, DeepSeek, ChatGLM, Gemini | ChatGLM, Gemini |
+| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
-| ollama API | Ollama | - |
+| ollama interface call | Ollama | - |
-| dify API | Dify | - |
+| dify interface call | Dify | - |
-| fastgpt API | Fastgpt | - |
+| fastgpt interface call | Fastgpt | - |
-| coze API | Coze | - |
+| coze interface call | Coze | - |
 
-Actually, any LLM supporting openai API calls can be integrated.
+In fact, any LLM that supports openai interface calls can be integrated and used.
-
----
 
 ### TTS Speech Synthesis
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| API Calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Aliyun TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
+| API Call | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
 | Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
 
 ---
 
 ### VAD Voice Activity Detection
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:---:|:---------:|:----:|:----:|:--:|
-| VAD | SileroVAD | Local Use | Free | |
+| VAD | SileroVAD | Local Usage | Free | |
 
 ---
 
@@ -229,26 +267,26 @@ Actually, any LLM supporting openai API calls can be integrated.
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| Local Use | FunASR, SherpaASR | FunASR, SherpaASR |
+| Local Usage | FunASR, SherpaASR | FunASR, SherpaASR |
-| API Calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+| API Call | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
 
 ---
 
 ### Memory Storage
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:---------------:|:----:|:---------:|:--:|
-| Memory | mem0ai | API Calls | 1000 calls/month quota | |
+| Memory | mem0ai | API Call | 1000 calls/month quota | |
 | Memory | mem_local_short | Local Summary | Free | |
 
 ---
 
 ### Intent Recognition
 
-| Type | Platform Name | Usage Method | Pricing | Notes |
+| Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:-------------:|:----:|:-------:|:---------------------:|
-| Intent | intent_llm | API Calls | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
+| Intent | intent_llm | API Call | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
-| Intent | function_call | API Calls | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
+| Intent | function_call | API Call | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
 
 ---
 
@@ -257,10 +295,10 @@ Actually, any LLM supporting openai API calls can be integrated.
 | Logo | Project/Company | Description |
 |:---:|:---:|:---|
 | <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
-| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for developing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing comprehensive technical documentation support for this project |
+| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for establishing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing full-chain technical documentation support for this project |
-| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
+| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing the function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
-| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering product user experience with their design experience serving over a thousand enterprises |
+| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering the product user experience with their design experience serving over a thousand enterprises |
-| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
+| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening the visual system of this project, ensuring consistency and extensibility of the overall design style in multi-scenario applications |
 
 
 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">

docs/Deployment.md

@@ -269,7 +269,7 @@ LLM:
 如果你能看到，类似以下日志,则是本项目服务启动成功的标志。
 
 ```
-250427 13:04:20[0.3.11_SiFuChTTnofu][__main__]-INFO-OTA接口是           http://192.168.4.123:8002/xiaozhi/ota/
+250427 13:04:20[0.3.11_SiFuChTTnofu][__main__]-INFO-OTA接口是           http://192.168.4.123:8003/xiaozhi/ota/
 250427 13:04:20[0.3.11_SiFuChTTnofu][__main__]-INFO-Websocket地址是     ws://192.168.4.123:8000/xiaozhi/v1/
 250427 13:04:20[0.3.11_SiFuChTTnofu][__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问=======
 250427 13:04:20[0.3.11_SiFuChTTnofu][__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
@@ -280,7 +280,7 @@ LLM:
 但是如果你用docker部署，那么你的日志里给出的接口地址信息就不是真实的接口地址。
 
 最正确的方法，是根据电脑的局域网IP来确定你的接口地址。
-如果你的电脑的局域网IP比如是`192.168.1.25`，那么你的接口地址就是：`ws://192.168.1.25:8000/xiaozhi/v1/`，对应的OTA地址就是：`http://192.168.1.25:8002/xiaozhi/ota/`。
+如果你的电脑的局域网IP比如是`192.168.1.25`，那么你的接口地址就是：`ws://192.168.1.25:8000/xiaozhi/v1/`，对应的OTA地址就是：`http://192.168.1.25:8003/xiaozhi/ota/`。
 
 这个信息很有用的，后面`编译esp32固件`需要用到。
 

docs/FAQ.md

@@ -108,7 +108,11 @@ VAD:
 
 参考教程[阿里云短信集成指南](./ali-sms-integration.md)
 
-### 9、更多问题，可联系我们反馈 💬
+### 9、如何开启视觉模型实现拍照识物 📷
+
+参考教程[视觉模型使用指南](./mcp-vision-integration.md)
+
+### 10、更多问题，可联系我们反馈 💬
 
 可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
 

docs/dev-ops-integration.md

@@ -0,0 +1,163 @@
+# 全模块源码部署自动升级方法
+
+本教程是方便全模块源码部署的爱好者，如何通过自动命令，自动拉取源码，自动编译，自动启动端口运行。实现最高效率的升级系统。
+
+本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。
+
+# 开始条件
+- 你的电脑/服务器是linux操作系统
+- 你已经跑通了整个流程
+- 你喜欢跟进最新功能，但是觉得每次手动部署有点麻烦，期待有一个自动更新的方法
+
+第二个条件必须满足，因为本教程所涉及的某些文件，JDK、Node.js环境、Conda环境等，是需要你跑通整个流程才有的，如果你没有跑通，当我讲到某个文件的时候，你可能就不知道什么意思。
+
+# 教程效果
+- 解决国内不能拉取最新项目源码问题
+- 自动拉取代码编译前端文件
+- 自动拉取代码编译java文件，自动杀掉8002端口，自动启动8002端口
+- 自动拉取python代码，自动杀掉8000端口，自动启动8000端口
+
+# 第一步 选好你的项目目录
+
+例如，我规划了我的项目目录是，这是一个新建的空白的目录，如果你不想出错，可以和我一样
+```
+/home/system/xiaozhi
+```
+
+# 第二步 克隆本项目
+此刻，先要执行第一句话，拉取源码，这句命令适用于国内网络的服务器和电脑，无需翻墙
+
+```
+cd /home/system/xiaozhi
+git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-server.git
+```
+
+执行完后，你的项目目录会多了一个文件夹`xiaozhi-esp32-server`，这个就是项目的源码
+
+# 第三步 复制基础的文件
+
+如果你之前已经跑通了整个流程，对funasr的模型文件`xiaozhi-server/models/SenseVoiceSmall/model.pt`和你的私有配置文件`xiaozhi-server/data/.config.yaml`这两个文件不会陌生。
+
+此刻你需要把`model.pt`文件复制到新的目录去，你可以这样
+```
+cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
+cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
+```
+
+# 第四步 建立三个自动编译文件
+
+## 4.1 自动编译mananger-web模块
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8001.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git fetch --all
+git reset --hard
+git pull origin main
+
+
+cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web
+npm install
+npm run build
+rm -rf /home/system/xiaozhi/manager-web
+mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web/dist /home/system/xiaozhi/manager-web
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8001.sh
+```
+执行完后，继续往下
+
+## 4.2 自动编译运行manager-api模块
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8002.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git pull origin main
+
+
+cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api
+rm -rf target
+mvn clean package -Dmaven.test.skip=true
+cd /home/system/xiaozhi/
+
+# 查找占用8002端口的进程号
+PID=$(sudo netstat -tulnp | grep 8002 | awk '{print $7}' | cut -d'/' -f1)
+
+rm -rf /home/system/xiaozhi/xiaozhi-esp32-api.jar
+mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api/target/xiaozhi-esp32-api.jar /home/system/xiaozhi/xiaozhi-esp32-api.jar
+
+# 检查是否找到进程号
+if [ -z "$PID" ]; then
+  echo "没有找到占用8002端口的进程"
+else
+  echo "找到占用8002端口的进程，进程号为: $PID"
+  # 杀掉进程
+  kill -9 $PID
+  kill -9 $PID
+  echo "已杀掉进程 $PID"
+fi
+
+nohup java -jar xiaozhi-esp32-api.jar --spring.profiles.active=dev &
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8002.sh
+```
+执行完后，继续往下
+
+## 4.3 自动编译运行Python项目
+在`/home/system/xiaozhi/`目录下，创建名字为`update_8000.sh`的文件，内容如下
+
+```
+cd /home/system/xiaozhi/xiaozhi-esp32-server
+git pull origin main
+
+# 查找占用8000端口的进程号
+PID=$(sudo netstat -tulnp | grep 8000 | awk '{print $7}' | cut -d'/' -f1)
+
+# 检查是否找到进程号
+if [ -z "$PID" ]; then
+  echo "没有找到占用8000端口的进程"
+else
+  echo "找到占用8000端口的进程，进程号为: $PID"
+  # 杀掉进程
+  kill -9 $PID
+  kill -9 $PID
+  echo "已杀掉进程 $PID"
+fi
+cd main/xiaozhi-server
+pip install -r requirements.txt
+nohup python app.py >/dev/null &
+```
+
+保存好后执行赋权命令
+```
+chmod 777 update_8000.sh
+```
+执行完后，继续往下
+
+# 日常更新
+
+以上的脚本都建立好后，日常更新，我们只要依次执行以下命令就可以做到自动更新和启动
+
+```
+# 进入pyhton环境
+conda activate xiaozhi-esp32-server
+cd /home/system/xiaozhi
+# 更新并启动Java程序
+./update_8001.sh
+# 更新web程序
+./update_8002.sh
+# 更新并启动python程序
+./update_8000.sh
+# 查看Java日志
+tail -f nohup.out
+# 查看Python日志
+tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server.log
+```
+
+# 注意事项
+测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)

docs/firmware-build.md

@@ -9,7 +9,7 @@
 ### 如果你用的是简单Server部署
 此刻，请你用浏览器打开你的ota地址，例如我的ota地址
 ```
-http://192.168.1.25:8002/xiaozhi/ota/
+http://192.168.1.25:8003/xiaozhi/ota/
 ```
 如果显示“OTA接口运行正常，向设备发送的websocket地址是：ws://xxx:8000/xiaozhi/v1/
 

docs/homeassistant-integration.md

@@ -89,55 +89,25 @@ http://homeassistant.local:8123
 公司,台灯,switch.iot_cn_831898993_socn1_on_p_2_1;
 ```
 
-这段字符，我们成为“设备清单字符”需要保存好，等一下有用。
+这段字符，我们称为“设备清单字符”需要保存好，等一下有用。
-
 
 #### 2. 登录`智控台`
 
-使用管理员账号，登录`智控台`。点击顶部菜单`参数管理`，搜索`plugins.home_assistant.`，会有三条结果出来
+![image-20250504051716417](images/image-ha-integration-06.png)
 
-编辑`plugins.home_assistant.devices`，把刚才整理的设备清单字符粘贴进去。
+使用管理员账号，登录`智控台`。在`智能体管理`，找到你的智能体，再点击`配置角色`。
 
+将意图识别设置成`函数调用`或`LLM意图识别`。这时你会看到右侧有一个`编辑功能`。点击`编辑功能`按钮，会弹出`功能管理`的框。
 
-编辑`plugins.home_assistant.base_url`，把你部署的`HomeAssistant`接口地址粘贴进去，我粘贴进去的地址是这样的
+在`功能管理`的框里，你需要勾选`HomeAssistant设备状态查询`和`HomeAssistant设备状态修改`。
 
-```
+勾选后，在`已选功能`点击`HomeAssistant设备状态查询`，然后在`参数配置`里配置你的`HomeAssistant`地址、密钥、设备清单字符。
-http://192.168.4.7:8123
-```
 
-编辑`plugins.home_assistant.api_key`，把你从`HomeAssistant`复制过来的密钥，粘贴进去
+编辑好后，点击`保存配置`，这时`功能管理`的框会隐藏，这时你再点击保存智能体配置。
 
+保存成功后，即可唤醒设备操作。
 
-#### 3. 设置`意图识别`函数
+#### 3. 唤醒设别进行控制
-
-在智控台，点击顶部菜单“模型配置”，在左侧栏，找到“意图识别”，找到id为`Intent_function_call`的意图，点击编辑
-
-然后在弹框中，在原来的基础上追加两个函数：“hass_get_state”和“hass_set_state”
-
-修改前
-```
-change_role;get_weather;get_news;play_music
-```
-
-修改后
-
-```
-change_role;get_weather;get_news;play_music;hass_get_state;hass_set_state
-```
-
-#### 4. 手动重启xiaozhi-server
-
-重启xiaozhi-server程序
-
-
-#### 5. 确认角色配置是否设置了函数意图识别
-
-在智控台，点击顶部菜单“智能体管理”，找到设备所在的智能体，点击“配置角色”
-
-确认意图识别(Intent)，是否选择“函数调用意图识别”
-
-
-#### 6. 唤醒设别进行控制
 
 尝试和esp32说，“打开XXX灯”
 
@@ -202,7 +172,15 @@ change_role;get_weather;get_news;play_music;hass_get_state;hass_set_state
 
 #### 2. 配置小智开源服务端MCP配置信息
 
-切换到小智开源服务端`xiaozhi-esp32-server`的`mcp_server_settings.json`文件内，在`"mcpServers"`的括号内添加以下内容：
+
+进入`data`目录，找到`.mcp_server_settings.json`文件。
+
+如果你的`data`目录下没有`.mcp_server_settings.json`文件，
+- 请把在`xiaozhi-server`文件夹根目录的`mcp_server_settings.json`文件复制到`data`目录下，并重命名为`.mcp_server_settings.json`
+- 或[下载这个文件](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/main/xiaozhi-server/mcp_server_settings.json)，下载到`data`目录下，并重命名为`.mcp_server_settings.json`
+
+
+修改`"mcpServers"`里的这部分的内容：
 
 ```json
 "Home Assistant": {

docs/mcp-vision-integration.md

@@ -0,0 +1,171 @@
+# 视觉模型使用指南
+本教程分为两部分：
+- 第一部分：单模块运行xiaozhi-server开启视觉模型
+- 第二部分：全模块运行时，如何开启视觉模型
+
+开启视觉模型前，你需要准备三件事：
+- 你需要准备一台带摄像头的设备，而且这台设备已经在虾哥仓库里，实现了调用摄像头功能。例如`立创·实战派ESP32-S3开发板`
+- 你设备固件的版本升级到1.6.6及以上
+- 你已经成功跑通基础对话模块
+
+## 单模块运行xiaozhi-server开启视觉模型
+
+### 第一步确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose.yml`是否放了`8003`端口，如果没有就更新最新的`docker-compose.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步选择你的视觉模型
+打开你的`data/.config.yaml`文件，设置你的`selected_module.VLLM`设置为某个视觉模型。目前我们已经支持`openai`类型接口的视觉模型。`ChatGLMVLLM`就是其中一款兼容`openai`的模型。
+
+```
+selected_module:
+  VAD: ..
+  ASR: ..
+  LLM: ..
+  VLLM: ChatGLMVLLM
+  TTS: ..
+  Memory: ..
+  Intent: ..
+```
+
+假设我们使用`ChatGLMVLLM`作为视觉模型，那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+在你的配置文件中，增加这个配置，如果已经有了这个配置，就设置好你的api_key。
+
+```
+VLLM:
+  ChatGLMVLLM:
+    api_key: 你的api_key
+```
+
+### 第三步启动xiaozhi-server服务
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - OTA接口是           http://192.168.4.7:8003/xiaozhi/ota/
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。
+
+
+## 全模块运行时，如何开启视觉模型
+
+### 第一步 确认网络
+由于视觉模型会默认启动8003端口。
+
+如果你是docker运行，请确认一下你的`docker-compose_all.yml`是否映射了`8003`端口，如果没有就更新最新的`docker-compose_all.yml`文件
+
+如果你是源码运行，确认防火墙是否放行`8003`端口
+
+### 第二步 确认你配置文件
+
+打开你的`data/.config.yaml`文件，确认一下你的配置文件的结构，是否和`data/config_from_api.yaml`一样。如果不一样，或缺少某项，请补齐。
+
+### 第三步 配置视觉模型密钥
+
+那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
+
+登录`智控台`，顶部菜单点击`模型配置`，在左侧栏点击`视觉打语言模型`，找到`VLLM_ChatGLMVLLM`，点击修改按钮，在弹框中，在`API密钥`输入你密钥，点击保存。
+
+保存成功后，去到你需要测试的智能体哪里，点击`配置角色`，在打开的内容里，查看`视觉大语言模型(VLLM)`是否选择了刚才的视觉模型。点击保存。
+
+### 第三步 启动xiaozhi-server模块
+如果你是源码，就输入命令启动
+```
+python app.py
+```
+如果你是docker运行，就重启容器
+```
+docker restart xiaozhi-esp32-server
+```
+
+启动后会输出以下内容的日志。
+
+```
+2025-06-01 **** - 视觉分析接口是        http://192.168.4.7:8003/mcp/vision/explain
+2025-06-01 **** - Websocket地址是       ws://192.168.4.7:8000/xiaozhi/v1/
+2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
+2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+2025-06-01 **** - =============================================================
+```
+
+启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
+```
+curl -i 你的视觉分析接口
+```
+
+正常来说会这样显示
+```
+MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
+```
+
+请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
+```
+server:
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
+```
+
+为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
+
+假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
+
+```
+server:
+  vision_explain: http://111.111.111.111:8003/mcp/vision/explain
+```
+
+如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
+
+### 第四步 设备唤醒开启
+
+对设备说“请打开摄像头，说你你看到了什么”
+
+留意xiaozhi-server的日志输出，看看有没有报错。

main/README.md

@@ -1,7 +1,70 @@
-本文档是开发类文档，如需部署小智服务端，[点击这里查看部署教程](../README.md#%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3)
+# 技术文档：`xiaozhi-esp32-server`
 
-# 项目目录介绍
+**目录：**
-如果你是一名软件开发者，这里有一份[《致开发者的公开信》](../docs/contributor_open_letter.md)，欢迎归队！
+
+1.  [引言](#1-引言)
+2.  [整体架构](#2-整体架构)
+3.  [核心组件深度剖析](#3-核心组件深度剖析)
+    *   [3.1. `xiaozhi-server` (核心AI引擎 - Python实现)](#31-xiaozhi-server-核心ai引擎---python实现)
+    *   [3.2. `manager-api` (管理后端 - Java Spring Boot实现)](#32-manager-api-管理后端---java-spring-boot实现)
+    *   [3.3. `manager-web` (Web管理前端 - Vue.js实现)](#33-manager-web-web管理前端---vuejs实现)
+4.  [数据流与交互机制](#4-数据流与交互机制)
+5.  [核心功能概要](#5-核心功能概要)
+6.  [部署与配置概述](#6-部署与配置概述)
+---
+
+## 1. 引言
+
+`xiaozhi-esp32-server` 项目是一个专为基于ESP32的智能硬件提供支持的**综合性后端系统**。其核心目标是使开发人员能够快速构建一个强大的服务器基础设施，该设施不仅能够理解自然语言指令，还能与多种AI服务（用于语音识别、自然语言理解及语音合成）进行高效交互、管理物联网（IoT）设备，并提供一个基于Web的用户界面以进行系统配置和管理。通过将多种尖端技术整合到一个高内聚且可扩展的平台中，本项目旨在简化和加速可定制化语音助手及智能控制系统的开发进程。它不仅仅是一个简单的服务器，更是一个连接硬件、AI能力与用户管理的桥梁。
+
+---
+
+## 2. 整体架构
+
+`xiaozhi-esp32-server` 系统采用了一种**分布式、多组件协作**的架构设计，确保了系统的模块化、可维护性和可扩展性。各个核心组件各司其职，协同工作。主要组件包括：
+
+1.  **ESP32 硬件 (客户端设备):**
+    这是终端用户直接与之交互的物理智能硬件设备。其主要职责包括：
+    *   捕捉用户的语音指令。
+    *   将捕捉到的原始音频数据安全地发送至 `xiaozhi-server` 进行处理。
+    *   接收来自 `xiaozhi-server` 合成的语音回复，并通过扬声器播放给用户。
+    *   根据从 `xiaozhi-server` 收到的指令，控制与之连接的其他外围设备或IoT设备（例如智能灯泡、传感器等）。
+
+2.  **`xiaozhi-server` (核心AI引擎 - Python实现):**
+    这个基于Python的服务器是整个系统的“大脑”，负责处理所有语音相关的逻辑和AI交互。其关键职责细化如下：
+    *   通过WebSocket协议与ESP32设备建立**稳定、低延迟的实时双向通信链路**。
+    *   接收来自ESP32的音频流，并利用语音活动检测（VAD）技术精确切分有效的语音片段。
+    *   集成并调用自动语音识别（ASR）服务（可配置本地或云端），将语音片段转换为文本。
+    *   通过与大型语言模型（LLM）的交互来解析用户意图、生成智能回复，并支持复杂的自然语言理解任务。
+    *   管理多轮对话中的上下文信息和用户记忆，以提供连贯的交互体验。
+    *   调用文本转语音（TTS）服务，将LLM生成的文本回复合成为自然流畅的语音。
+    *   通过一个灵活的**插件系统**执行自定义命令，包括对IoT设备的控制逻辑。
+    *   从 `manager-api` 服务获取其详细的运行时操作配置。
+
+3.  **`manager-api` (管理后端 - Java实现):**
+    这是一个基于Java Spring Boot框架构建的应用程序，它为整个系统的管理和配置提供了一套安全的RESTful API。它不仅是 `manager-web` 控制台的后端支撑，也是 `xiaozhi-server` 的配置数据来源。其核心功能包括：
+    *   为Web控制台提供用户认证（登录、权限验证）和用户账户管理功能。
+    *   ESP32设备的注册、信息管理以及设备特定配置的维护。
+    *   在**MySQL数据库**中持久化存储系统配置，例如用户选择的AI服务提供商、API密钥、设备参数、插件设置等。
+    *   提供特定的API端点，供 `xiaozhi-server` 拉取其所需的最新配置。
+    *   管理TTS音色选项、处理OTA（Over-The-Air）固件更新流程及相关元数据。
+    *   利用 **Redis** 作为高速缓存，存储热点数据（如会话信息、频繁访问的配置），以提升API响应速度和系统整体性能。
+
+4.  **`manager-web` (Web控制面板 - Vue.js实现):**
+    这是一个基于Vue.js构建的单页应用（SPA），为系统管理员提供了一个图形化、用户友好的操作界面。其主要能力包括：
+    *   便捷地配置 `xiaozhi-server` 所使用的各项AI服务（如ASR、LLM、TTS的提供商切换、参数调整）。
+    *   管理平台用户账户、角色分配及权限控制。
+    *   管理已注册的ESP32设备及其相关设置。
+    *   （潜在功能）监控系统运行状态、查看日志、进行故障排查等。
+    *   与 `manager-api` 提供的所有后端管理功能进行全面的交互。
+
+**高层交互流程概述:**
+
+*   **语音交互主线:** **ESP32设备**捕捉到用户语音后，通过**WebSocket**将音频数据实时传输给**`xiaozhi-server`**。`xiaozhi-server`完成一系列AI处理（VAD、ASR、LLM交互、TTS）后，再通过WebSocket将合成的语音回复发送回ESP32设备进行播放。所有与语音直接相关的实时交互均在此链路完成。
+*   **管理配置主线:** 管理员通过浏览器访问**`manager-web`**控制台。`manager-web`通过调用**`manager-api`**提供的**RESTful HTTP接口**来执行各种管理操作（如修改配置、管理用户或设备）。数据以JSON格式在两者间传递。
+*   **配置同步:** **`xiaozhi-server`**在启动或特定更新机制触发时，会主动通过HTTP请求从**`manager-api`**拉取其最新的操作配置。这确保了管理员在Web界面上所做的配置更改能够及时有效地应用到核心AI引擎的运行中。
+
+这种**前后端分离、核心服务与管理服务分离**的架构设计，使得 `xiaozhi-server`能够专注于高效的实时AI处理任务，而 `manager-api` 和 `manager-web` 则共同提供了一个功能强大且易于使用的管理和配置平台。各组件职责清晰，有利于独立开发、测试、部署和扩展。
 
 ```
 xiaozhi-esp32-server
@@ -10,10 +73,397 @@ xiaozhi-esp32-server
   ├─ manager-api 8002 端口 Java语言开发 负责提供控制台的api
 ```
 
-# xiaozhi-server 和ESP32通讯协议
+---
 
-https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh
+## 3. 核心组件深度剖析
 
-# manager-web 、manager-api接口协议
+### 3.1. `xiaozhi-server` (核心AI引擎 - Python实现)
 
-https://2662r3426b.vicp.fun/xiaozhi/doc.html
+`xiaozhi-server` 作为系统的智能核心，全权负责处理语音交互、对接各类AI服务以及管理与ESP32设备间的通信。其设计目标是实现高效、灵活且可扩展的语音AI处理能力。
+
+*   **核心目标:**
+    *   为ESP32设备提供实时的语音指令处理服务。
+    *   深度集成各类AI服务，包括：自动语音识别 (ASR)、大型语言模型 (LLM) 进行自然语言理解 (NLU)、文本转语音 (TTS)、语音活动检测 (VAD)、意图识别 (Intent Recognition) 及对话记忆 (Memory)。
+    *   精细管理用户与设备间的对话流程及上下文状态。
+    *   基于用户指令，通过插件化机制执行自定义函数及控制物联网 (IoT) 设备。
+    *   支持通过 `manager-api`进行动态配置加载与更新。
+
+*   **核心技术栈:**
+    *   **Python 3:** 作为主要编程语言，Python以其丰富的AI/ML生态库和快速开发特性被选用。
+    *   **Asyncio:** Python的异步编程框架，是`xiaozhi-server`高性能的关键。它被广泛用于高效处理来自大量ESP32设备的并发WebSocket连接，以及执行与外部AI服务API通信时的非阻塞I/O操作，确保服务器在高并发下的响应能力。
+    *   **`websockets` 库:** 提供WebSocket服务器的具体实现，支持与ESP32客户端进行全双工实时通信。
+    *   **HTTP客户端 (如 `aiohttp`, `httpx`):** 用于异步执行HTTP请求，主要目的是从`manager-api`获取配置信息，以及与云端AI服务的API进行交互。
+    *   **YAML (通常通过 PyYAML 库):** 用于解析本地的 `config.yaml` 配置文件。
+    *   **FFmpeg (外部依赖):** 在 `app.py` 启动时会进行检查 (`check_ffmpeg_installed()`)。FFmpeg通常用于音频处理和格式转换，例如，确保音频数据符合特定AI服务的要求或进行内部处理。
+
+*   **关键实现细节:**
+
+    1.  **AI服务提供者模式 (Provider Pattern - `core/providers/`):**
+        *   **设计思想:** 这是`xiaozhi-server`集成不同AI服务的核心设计模式，极大地增强了系统的灵活性和可扩展性。针对每一种AI服务类型（ASR, TTS, LLM, VAD, Intent, Memory, VLLM），都在其对应子目录下定义了一个抽象基类 (ABC, Abstract Base Class)，例如 `core/providers/asr/base.py`。这个基类规定了该类型服务必须实现的通用接口方法（如ASR的 `async def transcribe(self, audio_chunk: bytes) -> str: pass`）。
+        *   **具体实现:** 各种具体的AI服务提供商或本地模型的实现，则以独立的Python类形式存在（例如 `core/providers/asr/fun_local.py` 实现了本地FunASR的逻辑，`core/providers/llm/openai.py` 实现了与OpenAI GPT模型的对接）。这些具体类继承自相应的抽象基类，并实现其定义的接口。部分提供者还使用DTOs (Data Transfer Objects, 存在于各自的 `dto/` 目录) 来结构化与外部服务交换的数据。
+        *   **优势:** 使得核心业务逻辑能够以统一的方式调用不同的AI服务，而无需关心其底层具体实现。用户可以通过配置文件轻松切换AI服务后端。添加对新AI服务的支持也变得相对简单，只需实现对应的Provider接口。
+        *   **动态加载与初始化:** `core/utils/modules_initialize.py` 脚本扮演了工厂的角色。它在服务器启动时，或在接收到配置更新指令时，会根据配置文件中 `selected_module` 及各项服务的具体provider设置，动态地导入并实例化相应的Provider类。
+
+    2.  **WebSocket通信与连接处理 (`app.py`, `core/websocket_server.py`, `core/connection.py`):**
+        *   **服务器启动与入口 (`app.py`):**
+            *   `app.py` 作为主入口，负责初始化应用环境（如检查FFmpeg、加载配置、设置日志）。
+            *   它会生成或加载一个 `auth_key` (JWT密钥)，用于保护特定的HTTP接口（如视觉分析接口 `/mcp/vision/explain`）。若配置中 `manager-api.secret` 为空，则会生成一个UUID作为 `auth_key`。
+            *   使用 `asyncio.create_task()` 并发启动 `WebSocketServer` (监听如 `ws://0.0.0.0:8000/xiaozhi/v1/`) 和 `SimpleHttpServer` (监听如 `http://0.0.0.0:8003/xiaozhi/ota/`)。
+            *   包含一个 `monitor_stdin()` 协程，用于在某些环境下保持应用存活或处理终端输入。
+        *   **WebSocket服务器核心 (`core/websocket_server.py`):**
+            *   `WebSocketServer` 类使用 `websockets` 库监听来自ESP32设备的连接请求。
+            *   对于每一个成功的WebSocket连接，它都会创建一个**独立的 `ConnectionHandler` 实例** (推测定义于 `core/connection.py`)。这种每个连接一个处理程序实例的设计模式，是实现多设备状态隔离和并发处理的关键，确保每个设备的对话流程和上下文信息互不干扰。
+            *   该服务器还提供一个 `_http_response` 方法，允许在同一端口上对非WebSocket升级的HTTP GET请求做出简单响应（例如返回 "Server is running"），便于进行健康检查。
+        *   **动态配置更新:** `WebSocketServer` 包含一个 `update_config()` 异步方法。此方法使用 `config_lock` (一个 `asyncio.Lock`) 保证配置更新的原子性。它调用 `get_config_from_api()` (可能在 `config_loader.py` 中实现，通过 `manage_api_client.py` 与 `manager-api` 通信) 来获取新的配置。通过 `check_vad_update()` 和 `check_asr_update()` 等辅助函数判断是否需要重新初始化特定的AI模块，避免不必要的开销。更新后的配置会用于重新调用 `initialize_modules()`，从而实现AI服务提供者的热切换。
+
+    3.  **消息处理与对话流程控制 (`core/handle/` 和 `ConnectionHandler`):**
+        *   `ConnectionHandler` (推测) 作为每个连接的控制中心，负责接收来自ESP32的消息，并根据消息类型或当前对话状态，将其分发给 `core/handle/` 目录下的相应处理模块。这种模块化的处理器设计使得 `ConnectionHandler` 逻辑更清晰，易于扩展。
+        *   **主要处理模块及其职责:**
+            *   `helloHandle.py`: 处理与ESP32初次连接时的握手协议、设备认证或初始化信息交换。
+            *   `receiveAudioHandle.py`: 接收音频流数据，调用VAD Provider进行语音活动检测，并将有效的音频片段传递给ASR Provider进行识别。
+            *   `textHandle.py` / `intentHandler.py`: 获取ASR识别出的文本后，与Intent Provider (可能利用LLM进行意图识别) 和LLM Provider交互，以理解用户意图并生成初步回复或决策。
+            *   `functionHandler.py`: 当LLM的响应包含执行特定“函数调用”的指令时，此模块负责从插件注册表中查找并执行对应的插件函数。
+            *   `sendAudioHandle.py`: 将LLM最终生成的文本回复交给TTS Provider合成语音，并将音频流通过WebSocket发送回ESP32。
+            *   `abortHandle.py`: 处理来自ESP32的中断请求，例如停止当前的TTS播报。
+            *   `iotHandle.py`, `mcpHandle.py`: 处理与IoT设备控制相关的特定指令或更复杂的模块通信协议 (MCP)。
+
+    4.  **插件化功能扩展系统 (`plugins_func/`):**
+        *   **设计目的:** 提供一种标准化的方式来扩展语音助手的功能和“技能”，而无需修改核心代码。
+        *   **实现机制:**
+            *   各个具体功能以独立的Python脚本形式存在于 `plugins_func/functions/` 目录中（例如 `get_weather.py`, `hass_set_state.py` 用于Home Assistant集成）。
+            *   `loadplugins.py` 在服务器启动时负责扫描并加载这些插件模块。
+            *   `register.py` (或插件模块内部的特定装饰器/函数) 可能用于定义每个插件函数的元数据，包括：
+                *   **函数名称 (Function Name):** LLM调用时使用的标识符。
+                *   **功能描述 (Description):** 供LLM理解此函数的作用。
+                *   **参数模式 (Parameters Schema):** 通常是一个JSON Schema，详细定义了函数所需的参数、类型、是否必需以及描述。这是LLM能够正确生成函数调用参数的关键。
+        *   **执行流程:** 当LLM在其思考过程中决定需要调用某个外部工具或函数来获取信息或执行操作时，它会依据预先提供的函数模式生成一个结构化的“函数调用”请求。`xiaozhi-server`中的`functionHandler.py`捕获此请求，从插件注册表中找到对应的Python函数并执行，然后将执行结果返回给LLM，LLM再基于此结果生成最终给用户的自然语言回复。
+
+    5.  **配置管理 (`config/`):**
+        *   **加载机制:** `config_loader.py` (通过 `settings.py` 被调用) 负责从根目录的 `config.yaml` 文件加载基础配置。
+        *   **远程配置与合并:** 通过 `manage_api_client.py` (使用如`aiohttp`的库与`manager-api`通信) 可以从`manager-api`服务拉取配置。远程配置通常会覆盖本地 `config.yaml` 中的同名设置，从而实现通过Web界面动态调整服务器行为。
+        *   **日志系统:** `logger.py` 初始化应用日志系统（可能使用 `loguru` 或对标准 `logging` 模块进行封装，支持通过 `logger.bind(tag=TAG)` 添加标签，便于追踪和过滤）。
+        *   **静态资源:** `config/assets/` 目录下存放了用于系统提示音的静态音频文件（如设备绑定提示音 `bind_code.wav`、错误提示音等）。
+
+    6.  **辅助HTTP服务 (`core/http_server.py`):**
+        *   与WebSocket服务并行运行一个简单的HTTP服务器，用于处理特定的HTTP请求。最主要的功能是为ESP32设备提供OTA (Over-The-Air) 固件更新的下载服务 (通过 `/xiaozhi/ota/` 端点)。此外，也可能承载其他如 `/mcp/vision/explain` (视觉分析) 等工具性HTTP接口。
+
+综上所述，`xiaozhi-server` 是一个采用现代Python异步编程模型构建的、高度模块化、配置驱动的AI应用服务器。其精心设计的Provider模式和插件架构赋予了它强大的适应性和扩展性，能够灵活接入不同的AI能力并支持日益增长的功能需求。
+
+---
+
+### 3.2. `manager-api` (管理后端 - Java Spring Boot实现)
+
+`manager-api` 组件是使用Java和Spring Boot框架构建的强大后端服务，作为整个`xiaozhi-esp32-server`生态系统的中央行政管理和配置中枢。
+
+*   **核心目标:**
+    *   为`manager-web`（Vue.js前端）提供一套安全、稳定、符合RESTful规范的API接口，使得管理员能够便捷地管理用户、设备、系统配置及其他相关资源。
+    *   充当`xiaozhi-server`（Python核心AI引擎）的集中化配置数据提供者，允许`xiaozhi-server`实例在启动或运行时获取其最新的操作参数。
+    *   持久化存储关键数据，例如：用户账户信息、设备注册详情、AI服务提供商配置（包括API密钥、选定的服务模型等）、TTS音色参数，以及OTA固件版本信息等。
+
+*   **核心技术栈:**
+    *   **Java 21:** 项目采用的JDK版本，确保了对现代Java特性的支持。
+    *   **Spring Boot 3:** 作为核心开发框架，极大地简化了独立、生产级别的Spring应用的创建和部署。它提供了自动配置、内嵌Web服务器（默认为Tomcat）、依赖管理等关键功能。
+    *   **Spring MVC:** Spring框架中用于构建Web应用和RESTful API的模块。
+    *   **MyBatis-Plus:** 一个对MyBatis进行功能增强的ORM（对象关系映射）框架。它简化了数据库操作，提供了强大的CRUD（增删改查）功能、条件构造器、代码生成器等，并能很好地与Spring Boot集成。
+    *   **MySQL:** 作为主要的后端关系型数据库，用于存储所有需要持久化的管理数据和配置信息。
+    *   **Druid (Alibaba Druid):** 一个功能强大的JDBC连接池实现，提供了丰富的监控功能和优秀的性能，用于高效管理数据库连接。
+    *   **Redis (通过 Spring Data Redis):** 一个高性能的内存数据结构存储，常用于实现数据缓存（例如缓存热点配置数据、用户会话信息），以显著提升API的响应速度。
+    *   **Apache Shiro:** 一个成熟且易用的Java安全框架，负责处理应用的认证（用户身份验证）和授权（API访问权限控制）需求。
+    *   **Liquibase:** 一个用于跟踪、管理和应用数据库 schéma（模式）变更的开源工具。它允许开发者以数据库无关的方式定义和版本化数据库结构变更。
+    *   **Knife4j:** 一个集成了Swagger并增强了UI的API文档生成工具，专为Java MVC框架（尤其是Spring Boot）设计。它能生成美观且易于交互的API文档界面（通常通过 `/xiaozhi/doc.html` 访问）。
+    *   **Maven:** 用于项目的构建自动化和依赖项管理。
+    *   **Lombok:** 一个Java库，通过注解自动生成构造函数、getter/setter、equals/hashCode、toString等样板代码，减少冗余。
+    *   **HuTool / Google Guava:** 提供大量实用工具类，简化常见编程任务。
+    *   **Aliyun Dysmsapi:** 阿里云短信服务SDK，用于集成发送短信功能（如验证码、通知）。
+
+*   **关键实现细节:**
+
+    1.  **模块化项目结构 (`modules/` 包):**
+        *   `manager-api` 的核心业务逻辑被清晰地划分到 `src/main/java/xiaozhi/modules/` 目录下的不同模块中。这种按功能领域划分模块的方式（例如 `sys` 负责系统管理，`agent` 负责智能体配置，`device` 负责设备管理，`config` 负责为`xiaozhi-server`提供配置，`security` 负责安全，`timbre` 负责音色管理，`ota` 负责固件升级）极大地提高了代码的可维护性和可扩展性。
+        *   **各模块内部结构:** 每个业务模块通常遵循经典的三层架构或其变体：
+            *   **Controller (控制层):** 位于 `xiaozhi.modules.[模块名].controller`。
+            *   **Service (服务层):** 位于 `xiaozhi.modules.[模块名].service`。
+            *   **DAO/Mapper (数据访问层):** 位于 `xiaozhi.modules.[模块名].dao`。
+            *   **Entity (实体类):** 位于 `xiaozhi.modules.[模块名].entity`。
+            *   **DTO (数据传输对象):** 位于 `xiaozhi.modules.[模块名].dto`。
+
+    2.  **分层架构实现:**
+        *   **Controller层 (`@RestController`):** 这些类使用Spring MVC注解（如 `@GetMapping`, `@PostMapping` 等）来定义API的端点(endpoints)。它们负责接收HTTP请求，将请求体中的JSON数据反序列化为DTO对象，调用相应的Service层方法处理业务逻辑，最后将Service层的返回结果序列化为JSON并作为HTTP响应返回给客户端。
+        *   **Service层 (`@Service`):** 这些类（通常是接口及其实现类的组合）封装了核心的业务规则和操作流程。它们可能会调用一个或多个DAO/Mapper对象来与数据库交互，并常常使用 `@Transactional` 注解来管理数据库事务的原子性。
+        *   **Data Access (DAO/Mapper) 层 (MyBatis-Plus Mappers):** 这些是Java接口，继承自MyBatis-Plus提供的 `BaseMapper<Entity>` 接口。MyBatis-Plus会为这些接口自动提供标准的CRUD方法。对于更复杂的数据库查询，开发者可以通过在Mapper接口中定义方法并使用注解（如 `@Select`, `@Update`）或编写对应的XML映射文件来实现。例如，`UserMapper.selectById(userId)` 会被MyBatis-Plus自动实现。
+        *   **Entity层 (`@TableName`, `@TableId` 等MyBatis-Plus注解):** 这些POJO（Plain Old Java Objects）类直接映射到数据库中的表结构。Lombok的 `@Data` 注解常用于自动生成getter/setter等。
+        *   **DTO层:** 用于在各层之间，特别是Controller层与Service层之间，以及API的请求/响应体中传递数据。使用DTO有助于解耦API接口的数据结构与数据库实体的数据结构，使API更稳定。
+
+    3.  **通用功能与配置 (`common/` 包):**
+        *   `src/main/java/xiaozhi/common/` 包提供了一系列跨模块共享的通用组件和配置：
+            *   **基类:** 如 `BaseDao`, `BaseEntity`, `BaseService`, `CrudService`，为各模块的相应组件提供通用的属性或方法。
+            *   **全局配置:** 包括 `MybatisPlusConfig` (MyBatis-Plus的配置，如分页插件、数据权限插件等)、`RedisConfig` (Redis连接及序列化配置)、`SwaggerConfig` (Knife4j的配置)、`AsyncConfig` (异步任务执行器配置)。
+            *   **自定义注解:** 例如 `@LogOperation` 用于通过AOP记录操作日志，`@DataFilter` 可能用于实现数据范围过滤。
+            *   **AOP切面:** 如 `RedisAspect` 可能用于实现方法级别的缓存逻辑。
+            *   **全局异常处理:** `RenExceptionHandler` (使用 `@ControllerAdvice` 注解) 捕获应用中抛出的特定或所有异常 (如自定义的 `RenException`)，并返回统一格式的JSON错误响应给客户端。`ErrorCode` 定义了标准化的错误码。
+            *   **工具类:** 提供了日期转换、JSON处理(Jackson)、IP地址获取、HTTP上下文操作、统一结果封装 (`Result` 类)等多种实用工具。
+            *   **校验工具:** `ValidatorUtils` 和 `AssertUtils` 用于简化参数校验逻辑。
+            *   **XSS防护:** `XssFilter` 等组件用于防止跨站脚本攻击。
+            *   **MyBatis-Plus自动填充:** `FieldMetaObjectHandler` 用于在执行插入或更新数据库操作时，自动填充如 `createTime`, `updateTime` 等公共字段。
+
+    4.  **安全机制 (Apache Shiro):**
+        *   Shiro的配置（通常在 `modules/security/config/` 或 `common/config/` 下）定义了如何进行用户认证和授权。
+        *   **Realms (域):** 自定义的Shiro Realm类负责从数据库中查询用户信息（用户名、密码、盐值）进行身份验证，以及获取用户的角色和权限信息用于授权决策。
+        *   **Filters (过滤器):** Shiro过滤器链被应用于保护API端点，确保只有经过认证且拥有足够权限的用户才能访问特定资源。
+        *   **Session/Token Management:** Shiro管理用户会话。对于RESTful API，可能结合OAuth2或JWT等令牌机制实现无状态认证。
+
+    5.  **数据库版本控制 (Liquibase):**
+        *   数据库的表结构、索引、初始数据等变更，都通过Liquibase的 `changelog` 文件（通常是XML格式）进行定义和版本化管理。当应用启动时，Liquibase会自动检查并应用必要的数据库结构更新，确保开发、测试和生产环境数据库结构的一致性。
+
+    6.  **API文档:**
+        *   完整的API接口文档可通过以下地址访问: https://2662r3426b.vicp.fun/xiaozhi/doc.html
+        *   该文档使用Knife4j生成,提供了所有RESTful API端点的详细说明、请求/响应示例以及在线测试功能。
+
+`manager-api` 通过这些精心选择的技术和设计模式，构建了一个功能全面、结构清晰、安全可靠且易于维护和扩展的Java后端服务。其模块化的设计特别适合处理具有多种管理功能需求的复杂系统。
+
+---
+
+### 3.3. `manager-web` (Web管理前端 - Vue.js实现)
+
+`manager-web` 组件是一个采用 Vue.js 2 框架构建的单页应用 (SPA - Single Page Application)。它为系统管理员提供了一个功能丰富、交互友好的图形用户界面，用于全面管理和配置 `xiaozhi-esp32-server` 生态系统。
+
+*   **核心目标:**
+    *   提供一个基于Web的集中式控制面板，供管理员进行系统操作与监控。
+    *   实现对 `xiaozhi-server` 中AI服务提供商（ASR、LLM、TTS等）及其相关API密钥或许可配置的便捷管理。
+    *   支持用户账户、角色及权限的精细化管理。
+    *   提供ESP32设备的注册、配置及状态查看功能。
+    *   允许管理员自定义TTS音色、管理OTA固件更新流程、调整系统级参数及字典数据等。
+    *   作为 `manager-api` 所暴露各项功能的图形化交互前端。
+
+*   **核心技术栈:**
+    *   **Vue.js 2:** 一个渐进式的JavaScript框架，用于构建用户界面。其核心特性包括声明式渲染、组件化系统、数据绑定等，非常适合构建复杂的SPA。
+    *   **Vue CLI (`@vue/cli-service`):** Vue.js的官方命令行工具，用于项目的快速搭建、开发服务器的运行（支持热模块替换HMR）、以及生产环境构建打包（内部集成并配置了Webpack）。
+    *   **Vue Router (`vue-router`):** Vue.js官方的路由管理器。它负责在SPA内部实现不同“页面”或视图组件之间的导航切换，而无需重新加载整个HTML页面，提供了流畅的用户体验。
+    *   **Vuex (`vuex`):** Vue.js官方的状态管理模式和库。它充当了应用中所有组件的“中央数据存储”，用于管理全局共享状态（例如当前登录用户信息、设备列表、应用配置等），特别适用于大型复杂应用。
+    *   **Element UI (`element-ui`):** 一个广受欢迎的基于Vue 2.0的桌面端UI组件库。它提供了大量预先设计和实现的组件（如表单、表格、对话框、导航菜单、按钮、提示等），帮助开发者快速构建出专业且一致的用户界面。
+    *   **JavaScript (ES6+):** 前端逻辑实现的主要编程语言，利用其现代特性进行开发。
+    *   **SCSS (Sassy CSS):** 一种CSS预处理器，它为CSS增加了变量、嵌套规则、混合(Mixin)、继承等高级特性，使得CSS代码更易于组织、维护和复用。
+    *   **HTTP客户端 (Flyio 或 Axios 通过 `vue-axios`):** 用于在浏览器端向 `manager-api` 后端发起异步HTTP（AJAX）请求，以获取数据或提交操作。
+    *   **Webpack:** 一个强大的模块打包工具（由Vue CLI在底层管理和配置）。它将项目中的各种资源（JavaScript文件、CSS、图片、字体等）视为模块，并将它们打包成浏览器可识别的静态文件。
+    *   **Workbox (通过 `workbox-webpack-plugin`):** Google开发的一个库，用于简化Service Worker的编写和PWA（Progressive Web App - 渐进式Web应用）的实现。它可以帮助生成Service Worker脚本，实现资源缓存、离线访问等功能。
+    *   **Opus库 (`opus-decoder`, `opus-recorder`):** 这些音频处理库表明前端可能具备一些直接在浏览器中处理Opus格式音频的能力，例如：用于测试麦克风输入、允许管理员录制自定义音频片段（可能用于TTS音色样本或语音指令测试），或播放在管理界面中预览的Opus编码音频。
+
+*   **关键实现细节:**
+
+    1.  **单页应用 (SPA) 结构:**
+        *   整个前端应用加载一个主HTML文件 (`public/index.html`)。后续的所有页面切换和内容更新都在客户端由Vue Router动态完成，无需每次都从服务器请求新的HTML页面。这种模式能提供更快的页面加载速度和更流畅的交互体验。
+
+    2.  **组件化架构 (Component-Based Architecture):**
+        *   用户界面由一系列可复用的Vue组件 (`.vue` 单文件组件) 构成，形成一个组件树。这种方式提高了代码的模块化程度、可维护性和复用性。
+        *   **`src/main.js`:** 应用的入口JS文件。它负责创建和初始化根Vue实例，注册全局插件（如Vue Router, Vuex, Element UI），并把根Vue实例挂载到 `public/index.html` 中的某个DOM元素上（通常是 `#app`）。
+        *   **`src/App.vue`:** 应用的根组件。它通常定义了应用的基础布局结构（如包含导航栏、侧边栏、主内容区），并通过 `<router-view></router-view>` 标签来显示当前路由匹配到的视图组件。
+        *   **视图组件 (`src/views/`):** 这些组件代表了应用中的各个“页面”或主要功能区（例如 `Login.vue` 登录页, `DeviceManagement.vue` 设备管理页, `UserManagement.vue` 用户管理页, `ModelConfig.vue` 模型配置页）。它们通常由Vue Router直接映射。
+        *   **可复用UI组件 (`src/components/`):** 包含了在不同视图之间共享的、更小粒度的UI组件（例如 `HeaderBar.vue` 顶部导航栏, `AddDeviceDialog.vue` 添加设备对话框, `AudioPlayer.vue` 音频播放器组件）。
+
+    3.  **客户端路由 (`src/router/index.js`):**
+        *   Vue Router在此文件中进行配置，定义了应用的路由表。每个路由规则将一个特定的URL路径映射到一个视图组件。
+        *   常常包含**导航守卫 (Navigation Guards)**，例如 `beforeEach` 守卫，用于在路由跳转前执行逻辑，如检查用户是否已登录，如果未登录则重定向到登录页面，从而保护需要认证才能访问的页面。
+
+    4.  **状态管理 (`src/store/index.js`):**
+        *   Vuex被用来构建一个集中的状态管理中心（Store）。这个Store包含了：
+            *   **State:** 存储应用级别的共享数据（例如，当前登录用户的详细信息、从API获取的设备列表、系统配置等）。
+            *   **Getters:** 类似于Vue组件中的计算属性，用于从State派生出一些状态值，方便组件使用。
+            *   **Mutations:** **唯一**可以同步修改State中数据的方法。它们必须是同步函数。
+            *   **Actions:** 用于处理异步操作（如API调用）或封装多个Mutation提交。Actions会调用API，获取数据后，通过 `commit` 一个或多个Mutation来更新State。
+        *   例如，用户登录时，一个名为 `login` 的Action可能会被调用，它会向后端API发送登录请求，成功后获取到用户信息和token，然后 `commit` 一个名为 `SET_USER_INFO` 的Mutation来更新State中的用户信息和token。
+
+    5.  **API通信 (`src/apis/`):**
+        *   与 `manager-api` 后端的所有HTTP通信逻辑被封装在 `src/apis/` 目录下，通常会按照后端API的模块进行组织（例如 `src/apis/module/agent.js`, `src/apis/module/device.js`）。
+        *   每个模块导出一系列函数，每个函数对应一个具体的API请求。这些函数内部使用配置好的HTTP客户端实例 (例如，在 `src/apis/api.js` 或 `src/apis/httpRequest.js` 中统一配置Axios或Flyio实例，可能包含设置请求基地址、请求/响应拦截器等)。
+        *   **拦截器 (Interceptors):** HTTP客户端的请求拦截器常用于在每个请求发送前自动添加认证令牌（如JWT）；响应拦截器则可用于全局处理API错误（如权限不足、服务器错误）或对响应数据进行预处理。
+
+    6.  **样式与资源 (`src/styles/`, `src/assets/`):**
+        *   `Element UI` 提供了基础的组件样式。
+        *   `src/styles/global.scss` 文件用于定义全局共享的SCSS样式、变量、混合(Mixin)等。
+        *   Vue单文件组件内部的 `<style scoped>` 标签允许编写只作用于当前组件的局部样式。
+        *   `src/assets/` 目录存放图片、字体等静态资源。
+
+    7.  **构建与PWA特性:**
+        *   Vue CLI通过Webpack将所有代码和资源打包成优化的静态文件，用于生产部署。
+        *   `workbox-webpack-plugin` 的使用（体现在 `service-worker.js` 和 `registerServiceWorker.js` 文件）表明项目集成了Service Worker技术。Service Worker可以拦截网络请求，实现前端资源的智能缓存（从而加快后续访问速度），甚至在网络断开时提供一定的离线访问能力，是PWA的核心技术之一。
+
+    8.  **环境配置 (`.env`系列文件):**
+        *   项目根目录下的 `.env` (以及 `.env.development`, `.env.production` 等) 文件用于定义环境变量。这些变量（例如 `VUE_APP_API_BASE_URL` 来指定 `manager-api` 的基础URL）可以在应用代码中通过 `process.env.VUE_APP_XXX` 的形式访问，从而允许为不同构建环境（开发、测试、生产）配置不同的参数。
+
+`manager-web` 通过这些技术的综合运用，构建了一个功能强大、易于维护且用户体验良好的管理界面，为 `xiaozhi-esp32-server` 系统的配置和监控提供了坚实的前端支持。
+
+---
+
+## 4. 数据流与交互机制
+
+`xiaozhi-esp32-server` 系统通过各组件间定义清晰的数据流和交互协议来协同工作。主要的通信方式依赖于针对实时交互优化的WebSocket协议和适用于客户端-服务器请求的RESTful API。
+
+**4.1.核心语音交互流程 (ESP32设备 <-> `xiaozhi-server`)**
+
+此流程是实时的，主要通过WebSocket进行低延迟、双向的数据交换。
+
+*   **通讯协议文档:**
+    *   详细的通讯协议说明文档可通过以下地址访问: https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh
+    *   该文档详细描述了ESP32设备与`xiaozhi-server`之间的WebSocket通信协议,包括:
+        *   连接建立与握手流程
+        *   音频数据传输格式
+        *   控制命令格式
+        *   状态报告格式
+        *   错误处理机制
+
+*   **连接建立与握手:**
+    *   ESP32设备作为客户端，主动向`xiaozhi-server`的指定端点（例如 `ws://<服务器IP>:<WebSocket端口>/xiaozhi/v1/`）发起WebSocket连接请求。
+    *   `xiaozhi-server` (`core/websocket_server.py`) 接收连接，并为每个成功连接的ESP32设备实例化一个独立的`ConnectionHandler`对象来管理该会话的整个生命周期。
+    *   连接建立后，可能会执行一个初始握手流程（由`core/handle/helloHandle.py`处理），用于交换设备标识、认证信息、协议版本或基本状态。
+
+*   **音频上行传输 (ESP32 -> `xiaozhi-server`):**
+    *   用户对ESP32设备讲话后，设备上的麦克风捕捉原始音频数据（通常是PCM或经过压缩如Opus的格式）。
+    *   ESP32将这些音频数据块（chunks）作为WebSocket的**二进制消息 (binary messages)** 实时推送到`xiaozhi-server`对应的`ConnectionHandler`。
+    *   服务器端的`core/handle/receiveAudioHandle.py`模块负责接收、缓冲并处理这些音频数据。
+
+*   **AI核心处理 (在`xiaozhi-server`内部):**
+    *   **VAD (语音活动检测):** `receiveAudioHandle.py`利用配置的VAD提供者（如SileroVAD）分析音频流，以准确识别语音的起始和结束点，滤除静默或噪声片段。
+    *   **ASR (自动语音识别):** 检测到的有效语音片段被送往配置的ASR提供者（本地如FunASR，或云端服务）。ASR引擎将音频信号转换为文本字符串。
+    *   **NLU/LLM (自然语言理解/大型语言模型):** ASR输出的文本，连同从Memory提供者获取的当前对话上下文历史，以及从`plugins_func/`加载的可用函数（工具）的描述模式，一同被传递给配置的LLM提供者。
+    *   **函数调用执行 (若LLM决策需要):** 如果LLM分析后认为需要调用外部函数（例如查询天气、控制家电），它会生成一个结构化的函数调用请求。`core/handle/functionHandler.py`接收此请求，查找并执行在`plugins_func/`中定义的相应Python函数，并将函数的执行结果返回给LLM。LLM随后基于此结果生成最终的自然语言回复。
+    *   **回复生成:** LLM综合所有信息（用户输入、上下文、函数调用结果等）生成最终的文本回复。
+    *   **记忆更新:** 当前轮次的交互（用户问题、LLM回复、可能的功能调用）会被Memory提供者处理，以更新对话历史，供后续交互使用。
+    *   **TTS (文本转语音):** LLM生成的最终文本回复被送往配置的TTS提供者，后者将文本合成为语音数据流（例如MP3或WAV格式）。
+
+*   **音频下行响应 (`xiaozhi-server` -> ESP32):**
+    *   由TTS提供者合成的语音数据流，通过`core/handle/sendAudioHandle.py`模块，作为WebSocket的**二进制消息**实时发送回ESP32设备。
+    *   ESP32设备接收这些音频数据块并立即通过扬声器播放给用户。
+
+*   **控制与状态消息 (双向):**
+    *   除了音频流，ESP32与`xiaozhi-server`之间也通过WebSocket交换**文本消息 (text messages)**，这些消息通常采用JSON格式封装。
+    *   **ESP32 -> Server:** 设备可能发送状态报告（如网络状况、麦克风状态）、错误代码、或特定的控制命令（例如用户按键触发的“停止TTS播报”）。
+    *   **Server -> ESP32:** 服务器可能发送控制指令给设备（如“开始监听”、“停止监听”、调整灵敏度、下发特定配置参数）。
+    *   `core/handle/abortHandle.py`（处理中断请求）、`core/handle/reportHandle.py`（处理设备报告）等模块负责解析和响应这些控制/状态消息。
+
+**4.2.管理与配置流程 (`manager-web` <-> `manager-api` <-> `xiaozhi-server`)**
+
+此流程主要依赖于基于HTTP/HTTPS的RESTful API进行请求-响应式的交互。
+
+*   **管理员UI后端交互 (`manager-web` -> `manager-api`):**
+    *   当管理员在`manager-web`界面执行操作时（例如保存一项配置、添加一个新用户、注册一台ESP32设备）：
+        *   Vue.js前端应用 (`manager-web`) 会通过其API封装模块（位于`src/apis/module/`）向`manager-api`的对应REST API端点发起异步HTTP请求（通常是GET, POST, PUT, DELETE）。
+        *   请求体和响应体通常使用JSON格式。
+        *   `manager-api`中的`@RestController`类接收这些请求。**Apache Shiro**框架会首先对请求进行认证和授权检查。
+        *   通过验证后，Controller将请求分发给相应的Service层处理业务逻辑。Service层可能会与MySQL数据库（通过MyBatis-Plus）交互，并可能利用Redis进行数据缓存。
+        *   处理完成后，`manager-api`向`manager-web`返回一个JSON格式的HTTP响应。
+        *   `manager-web`根据响应结果更新其Vuex状态存储和用户界面显示。
+
+*   **配置同步 (`manager-api` -> `xiaozhi-server`):**
+    *   `xiaozhi-server`的运行依赖于从`manager-api`获取的动态配置（例如当前选用的AI服务提供商及其API密钥）。
+    *   **拉取机制 (Pull Mechanism):** `xiaozhi-server`内部的`config/manage_api_client.py`模块，在服务器启动时或通过特定更新触发器（例如`WebSocketServer.update_config()`被调用），会向`manager-api`的一个指定端点（例如由`modules/config/controller/`中的某个Controller提供）发起HTTP GET请求。
+    *   `manager-api`响应该请求，返回`xiaozhi-server`所需的配置数据（JSON格式）。
+    *   `xiaozhi-server`接收到配置后，会更新其内部状态，并可能重新初始化相关的AI服务模块，以使新配置生效。
+
+*   **OTA固件更新流程 (概念性描述):**
+    *   管理员通过`manager-web`界面上传新的ESP32固件包到`manager-api`的特定端点。
+    *   `manager-api`将固件文件存储起来，并记录相关元数据（版本号、适用设备型号等）。
+    *   当管理员触发对特定设备的OTA更新时：
+        *   `manager-api`可能会通知`xiaozhi-server`（具体通知机制可能是一个轮询检查点，或`xiaozhi-server`暴露一个接收更新通知的API，或者更松耦合的如消息队列）。
+        *   `xiaozhi-server`随后可以通过WebSocket向目标ESP32设备发送一条包含固件下载URL的指令消息。
+        *   ESP32设备收到指令后，通过HTTP GET请求从该URL下载固件。此URL可能指向`xiaozhi-server`自身运行的`SimpleHttpServer`所服务的路径（如`/xiaozhi/ota/`），或者在某些架构中，也可能直接指向`manager-api`或专用的文件服务器。
+
+**4.3. 主要协议总结:**
+
+*   **WebSocket:** 被选用于ESP32与`xiaozhi-server`之间的通信链路，因为它非常适合实时、低延迟、双向的数据流传输（尤其是音频），以及异步控制消息的传递。
+*   **RESTful APIs (基于HTTP/HTTPS，通常使用JSON作为数据交换格式):** 这是Web服务间通信的标准方式。用于`manager-web`（客户端）与`manager-api`（服务器）之间的请求-响应交互，也用于`xiaozhi-server`（作为客户端）从`manager-api`（作为服务器）拉取配置信息。其无状态特性、广泛的库支持和易于理解的语义使其成为此类交互的理想选择。
+
+这种多协议并用的通信策略，确保了系统内不同类型的交互需求都能得到高效和恰当的处理，兼顾了实时性和标准化的请求-响应模式。
+
+---
+
+## 5. 核心功能概要
+
+`xiaozhi-esp32-server` 系统提供了一系列丰富的功能，旨在支持开发者构建先进的语音控制应用：
+
+1.  **全面的语音交互后端:** 提供从语音捕获指导到响应生成和动作执行的端到端解决方案。
+2.  **模块化和可插拔的AI服务:**
+    *   支持广泛的ASR（自动语音识别）、LLM（大型语言模型）、TTS（文本转语音）、VAD（语音活动检测）、意图识别和记忆提供商。
+    *   允许动态选择和配置这些服务（包括基于云的API和本地模型），以平衡成本、性能、隐私和语言需求。
+3.  **高级对话管理:**
+    *   支持自然交互，具有唤醒词启动对话、手动（按键说话式）对话以及对系统响应的实时打断等功能。
+    *   包含上下文记忆，以在多轮对话中保持连贯性。
+    *   在一段时间不活动后具有自动休眠模式。
+4.  **多语言能力:**
+    *   支持多种语言的识别和合成，包括普通话、粤语、英语、日语和韩语（具体取决于所选的ASR/LLM/TTS提供商）。
+5.  **通过插件实现的可扩展功能:**
+    *   强大的插件系统允许开发人员添加自定义“技能”或函数（例如，获取天气、控制智能家居设备、访问新闻）。
+    *   这些函数可以由LLM使用其函数调用能力，根据提供的模式来触发。
+    *   内置对Home Assistant集成的支持。
+6.  **物联网设备控制:**
+    *   设计用于通过语音命令管理和控制智能家居设备及其他物联网硬件，并利用插件系统。
+7.  **基于Web的管理控制台 (`manager-web` & `manager-api`):**
+    *   提供全面的图形界面，用于：
+        *   系统配置（AI服务选择、API密钥、操作参数）。
+        *   基于角色的访问控制的用户管理。
+        *   ESP32设备注册和管理。
+        *   语音音色/TTS语音定制。
+        *   ESP32设备的OTA（空中下载）固件更新管理。
+        *   系统参数和字典的管理。
+8.  **灵活的部署选项:**
+    *   支持通过Docker容器（用于简化的仅服务器或全栈设置）和直接从源代码部署，以适应各种环境和用户专业知识。
+9.  **动态远程配置:**
+    *   `xiaozhi-server`可以从`manager-api`获取其配置，允许实时更新AI提供商和设置，而无需重新启动服务器。
+10. **开源和社区驱动:**
+    *   根据MIT许可证授权，鼓励透明、协作和社区贡献。
+11. **经济高效的解决方案:**
+    *   提供“入门全免费设置”路径，利用AI服务的免费套餐或本地模型，使其易于进行实验和个人项目。
+12. **渐进式Web应用 (PWA) 特性:**
+    *   `manager-web`控制面板包含Service Worker集成，以增强缓存和潜在的离线访问能力。
+13. **详细的API文档:**
+    *   `manager-api`通过Knife4j提供OpenAPI (Swagger) 文档，以便清晰理解和测试其RESTful端点。
+
+这些功能共同使`xiaozhi-esp32-server`成为一个强大、适应性强且用户友好的平台，用于构建复杂的语音交互应用程序。
+
+---
+
+## 6. 部署与配置概述
+
+`xiaozhi-esp32-server`系统在设计上充分考虑了灵活性，提供了多种部署方法和全面的配置选项，以适应不同的使用场景和需求。
+
+**部署选项:**
+
+项目可以通过多种方式部署，主要包括使用Docker简化安装过程，或直接从源代码部署以获得更大的控制权和进行开发。
+
+1.  **基于Docker的部署:**
+    *   **简化安装 (仅`xiaozhi-server`):** 此选项仅部署核心的基于Python的`xiaozhi-server`。它适用于主要需要语音AI处理能力和IoT控制，而不需要完整Web管理界面和数据库支持功能（如OTA）的用户。在此模式下，配置通常通过本地文件（`config.yaml`）管理，但如果需要，仍可将其指向一个已存在的`manager-api`实例。
+    *   **全模块安装 (所有组件):** 此方案部署所有核心组件：`xiaozhi-server`、基于Java的`manager-api`、以及基于Vue.js的`manager-web`，同时还包括所需的数据库服务（MySQL和Redis）。这提供了完整的系统体验，包括用于全面配置和管理的Web控制面板。
+    *   项目为每个服务提供了`Dockerfile`定义，并使用`docker-compose.yml`文件（例如`docker-compose.yml`用于基础版，`docker-compose_all.yml`用于全功能版）来编排和管理多容器的部署。此外，还可能提供一个`docker-setup.sh`脚本来辅助自动化部分Docker环境的搭建工作。
+
+2.  **源代码部署:**
+    *   这种方法需要为每个组件手动设置相应的开发环境：Python环境用于`xiaozhi-server`，Java/Maven环境用于`manager-api`，Node.js/Vue CLI环境用于`manager-web`。
+    *   对于全模块安装，还需要手动安装和配置MySQL及Redis数据库服务。
+    *   这种方式通常用于项目开发、深度定制、调试，或者在对环境有特殊要求的生产场景中。
+
+**配置管理:**
+
+配置是定制系统行为的关键，尤其是在选择AI服务提供商和管理API密钥方面。
+
+1.  **`xiaozhi-server` 配置:**
+    *   **本地`config.yaml`:** 位于`xiaozhi-server`根目录下的一个主要的YAML格式配置文件。它定义了服务器端口、选定的AI服务提供商（ASR、LLM、TTS、VAD、意图识别、记忆模块等）、它们各自的API密钥或模型路径、插件配置以及日志级别等。
+    *   **通过`manager-api`进行远程配置:** `xiaozhi-server`被设计为可以从`manager-api`获取其运行配置。从`manager-api`获取的设置通常会覆盖本地`config.yaml`中的同名设置。这带来了两大好处：
+        *   **集中管理:** 所有配置都可以通过`manager-web`界面进行统一管理。
+        *   **动态更新:** `xiaozhi-server`可以刷新其配置并重新初始化AI模块，而无需完全重启服务。
+    *   `xiaozhi-server`中的`config/config_loader.py`和`config/manage_api_client.py`负责处理配置的加载、合并及从`manager-api`拉取的逻辑。
+
+2.  **`manager-api` 配置:**
+    *   作为一个Spring Boot应用，其配置主要通过位于`src/main/resources`目录下的`application.properties`或`application.yml`文件进行管理。
+    *   关键配置项包括：数据库连接信息（MySQL的URL、用户名、密码）、Redis服务器地址和端口、应用服务端口（默认为8002）、Apache Shiro安全相关的设置，以及任何集成的第三方服务（如阿里云短信）的配置参数。
+
+3.  **`manager-web` 配置:**
+    *   Vue.js前端应用的环境特定设置通过项目根目录下的`.env`系列文件（例如`.env`, `.env.development`, `.env.production`）进行管理。
+    *   这里最关键的配置通常是`manager-api`后端的API基础URL地址 (例如 `VUE_APP_API_BASE_URL`)，前端应用将向此地址发送所有API请求。
+
+4.  **预定义的配置方案:**
+    *   项目文档（通常是README）中会推荐一些常见的配置组合，例如：
+        *   **“入门全免费设置”:** 该方案旨在利用云AI服务的免费套餐额度或完全免费的本地模型，以最大程度地降低用户的初始使用成本和运营费用。
+        *   **“全流式配置”:** 该方案优先考虑系统的响应速度和交互的流畅性，通常会选用支持流式处理的（可能付费的）AI服务。
+    *   这些预定义方案为用户在`xiaozhi-server`中配置AI服务提供商（通过`manager-web`界面或直接修改`config.yaml`）提供了指导。
+
+在全模块部署的情况下，推荐使用`manager-web`控制面板作为大多数配置任务的主要操作界面，因为它提供了一种用户友好的方式来管理由`manager-api`持久化并最终由`xiaozhi-server`使用的各项设置。
+
+---

main/README_en.md

@@ -0,0 +1,369 @@
+# Technical Documentation: `xiaozhi-esp32-server`
+
+**Table of Contents:**
+
+1.  [Introduction](#1-introduction)
+2.  [Overall Architecture](#2-overall-architecture)
+3.  [Component Deep Dive](#3-component-deep-dive)
+    *   [3.1. `xiaozhi-server` (Core AI Engine - Python Implementation)](#31-xiaozhi-server-core-ai-engine---python-implementation)
+    *   [3.2. `manager-api` (Management Backend - Java Spring Boot Implementation)](#32-manager-api-management-backend---java-spring-boot-implementation)
+    *   [3.3. `manager-web` (Web Management Frontend - Vue.js Implementation)](#33-manager-web-web-management-frontend---vuejs-implementation)
+4.  [Data Flow and Interaction Mechanisms](#4-data-flow-and-interaction-mechanisms)
+5.  [Key Features Summary](#5-key-features-summary)
+6.  [Deployment and Configuration Overview](#6-deployment-and-configuration-overview)
+
+---
+
+## 1. Introduction
+
+The `xiaozhi-esp32-server` project is a **comprehensive backend system** designed to support intelligent hardware based on ESP32. Its core goal is to enable developers to quickly build a robust server infrastructure that can understand natural language commands, interact efficiently with various AI services (for speech recognition, natural language understanding, and speech synthesis), manage IoT devices, and provide a web-based user interface for system configuration and management. By integrating multiple cutting-edge technologies into a cohesive and extensible platform, this project aims to simplify and accelerate the development process of customizable voice assistants and intelligent control systems. It is not just a simple server, but a bridge connecting hardware, AI capabilities, and user management.
+
+---
+
+## 2. Overall Architecture
+
+The `xiaozhi-esp32-server` system adopts a **distributed, multi-component collaborative** architectural design, ensuring modularity, maintainability, and scalability. Each core component has its specific role and works in coordination. The main components include:
+
+1.  **ESP32 Hardware (Client Device):**
+    This is the physical smart hardware device that end-users directly interact with. Its main responsibilities include:
+    *   Capturing user voice commands.
+    *   Securely sending captured raw audio data to `xiaozhi-server` for processing.
+    *   Receiving synthesized voice responses from `xiaozhi-server` and playing them through speakers.
+    *   Controlling other connected peripherals or IoT devices (such as smart bulbs, sensors, etc.) based on instructions received from `xiaozhi-server`.
+
+2.  **`xiaozhi-server` (Core AI Engine - Python Implementation):**
+    This Python-based server is the "brain" of the entire system, responsible for handling all voice-related logic and AI interactions. Its key responsibilities are detailed as follows:
+    *   Establishing **stable, low-latency real-time bidirectional communication links** with ESP32 devices through the WebSocket protocol.
+    *   Receiving audio streams from ESP32 and using Voice Activity Detection (VAD) technology to precisely segment valid speech segments.
+    *   Integrating and calling Automatic Speech Recognition (ASR) services (configurable for local or cloud), converting speech segments to text.
+    *   Interacting with Large Language Models (LLMs) to parse user intent, generate intelligent responses, and support complex natural language understanding tasks.
+    *   Managing context information and user memory in multi-turn dialogues to provide coherent interaction experiences.
+    *   Calling Text-to-Speech (TTS) services to synthesize natural and fluent speech from LLM-generated text responses.
+    *   Executing custom commands through a flexible **plugin system**, including IoT device control logic.
+    *   Obtaining its detailed runtime operation configuration from the `manager-api` service.
+
+3.  **`manager-api` (Management Backend - Java Spring Boot Implementation):**
+    This is an application built using the Java Spring Boot framework, providing a secure RESTful API for system management and configuration. It serves not only as the backend support for the `manager-web` console but also as the configuration data source for `xiaozhi-server`. Its core functions include:
+    *   Providing user authentication (login, permission verification) and user account management functions for the Web console.
+    *   Registration, information management of ESP32 devices, and maintenance of device-specific configurations.
+    *   Persistently storing system configurations in the **MySQL database**, such as user-selected AI service providers, API keys, device parameters, plugin settings, etc.
+    *   Providing specific API endpoints for `xiaozhi-server` to pull its required latest configuration.
+    *   Managing TTS voice options, handling OTA (Over-The-Air) firmware update processes, and related metadata.
+    *   Utilizing **Redis** as a high-speed cache to store hotspot data (such as session information, frequently accessed configurations) to improve API response speed and overall system performance.
+
+4.  **`manager-web` (Web Control Panel - Vue.js Implementation):**
+    This is a Single Page Application (SPA) built with Vue.js, providing system administrators with a graphical, user-friendly operation interface. Its main capabilities include:
+    *   Conveniently configuring various AI services used by `xiaozhi-server` (such as ASR, LLM, TTS provider switching, parameter adjustment).
+    *   Managing platform user accounts, role assignment, and permission control.
+    *   Managing registered ESP32 devices and their related settings.
+    *   (Potential functionality) Monitoring system operation status, viewing logs, troubleshooting, etc.
+    *   Comprehensive interaction with all backend management functions provided by `manager-api`.
+
+**High-Level Interaction Flow Overview:**
+
+*   **Voice Interaction Main Line:** After the **ESP32 device** captures user voice, it transmits audio data in real-time to **`xiaozhi-server`** through **WebSocket**. After `xiaozhi-server` completes a series of AI processing (VAD, ASR, LLM interaction, TTS), it sends the synthesized voice response back to the ESP32 device for playback through WebSocket. All real-time interactions directly related to voice are completed in this link.
+*   **Management Configuration Main Line:** Administrators access the **`manager-web`** console through a browser. `manager-web` executes various management operations (such as modifying configurations, managing users or devices) by calling **RESTful HTTP interfaces** provided by **`manager-api`**. Data is passed between them in JSON format.
+*   **Configuration Synchronization:** **`xiaozhi-server`** actively pulls its latest operation configuration from **`manager-api`** through HTTP requests when starting or when specific update mechanisms are triggered. This ensures that configuration changes made by administrators in the Web interface can be effectively applied to the operation of the core AI engine in a timely manner.
+
+This **frontend-backend separation, core service and management service separation** architectural design allows `xiaozhi-server` to focus on efficient real-time AI processing tasks, while `manager-api` and `manager-web` together provide a powerful and easy-to-use management and configuration platform. Each component has clear responsibilities, facilitating independent development, testing, deployment, and expansion.
+
+```
+xiaozhi-esp32-server
+  ├─ xiaozhi-server Port 8000 Python development Responsible for ESP32 communication
+  ├─ manager-web Port 8001 Node.js+Vue development Responsible for providing web interface for console
+  ├─ manager-api Port 8002 Java development Responsible for providing console API
+```
+
+---
+
+## 3. Component Deep Dive
+
+### 3.1. `xiaozhi-server` (Core AI Engine - Python Implementation)
+
+The `xiaozhi-server` is the intelligent core of the system, responsible for processing voice interactions, interfacing with AI services, and managing communication with ESP32 devices.
+
+*   **Purpose:**
+    *   To provide real-time processing of voice commands from ESP32 devices.
+    *   To integrate with various AI services for Speech-to-Text (ASR), Natural Language Understanding (via Large Language Models - LLMs), Text-to-Speech (TTS), Voice Activity Detection (VAD), Intent Recognition, and Memory.
+    *   To manage dialogue flow and context with users.
+    *   To execute custom functions and control IoT devices based on user commands.
+    *   To be dynamically configurable through the `manager-api`.
+
+*   **Core Technologies:**
+    *   **Python 3:** The primary programming language.
+    *   **Asyncio:** Python's asynchronous programming framework, crucial for handling concurrent WebSocket connections and non-blocking I/O for AI service API calls.
+    *   **`websockets` Library:** For WebSocket server implementation.
+    *   **HTTP Client (e.g., `aiohttp`, `httpx`):** For asynchronous HTTP requests to `manager-api` and external AI services.
+    *   **YAML (PyYAML):** For local configuration file parsing.
+
+*   **Key Implementation Aspects:**
+
+    1.  **AI Service Provider Pattern (`core/providers/`):**
+        *   **Concept:** A flexible design for integrating AI services. Each service type (ASR, TTS, LLM, etc.) has an abstract base class defining a common interface. Concrete classes implement this interface for specific vendors or local models.
+        *   **Benefit:** Allows easy switching of AI service backends via configuration and simplifies adding new service integrations.
+        *   **Initialization:** `core/utils/modules_initialize.py` acts as a factory to load and instantiate configured providers.
+
+    2.  **WebSocket Communication & Connection Handling (`core/websocket_server.py`, `core/connection.py`):**
+        *   **Server Setup:** Manages WebSocket connections from ESP32 devices.
+        *   **Connection Isolation:** Each ESP32 client gets a dedicated `ConnectionHandler` instance, isolating its session state and dialogue.
+        *   **Dynamic Configuration Updates:** Can fetch updated configurations from `manager-api` and re-initialize AI service modules live, without a full server restart.
+
+    3.  **Message Handling & Dialogue Flow (`core/handle/`):**
+        *   Employs a modular handler pattern. The `ConnectionHandler` dispatches message processing to specialized modules based on message type or dialogue phase (e.g., `receiveAudioHandle.py` for audio input, `intentHandler.py` for NLU, `functionHandler.py` for plugin execution, `sendAudioHandle.py` for TTS output).
+
+    4.  **Plugin System for Extensible Functions (`plugins_func/`):**
+        *   **Purpose:** Allows adding custom "skills" (e.g., weather, news, Home Assistant control).
+        *   **Mechanism:** Plugins define functions and schemas. The LLM can request execution of these functions (function calling). `loadplugins.py` and `register.py` manage plugin discovery and registration.
+
+    5.  **Configuration Management (`config/`):**
+        *   Loads settings from a local `config.yaml` and merges them with configurations fetched from `manager-api` (via `manage_api_client.py`), enabling remote dynamic configuration.
+        *   `logger.py` sets up structured application logging.
+        *   `config/assets/` stores predefined audio files for system notifications.
+
+    6.  **Auxiliary HTTP Server (`core/http_server.py`):**
+        *   Handles specific HTTP requests, notably for OTA firmware updates (`/xiaozhi/ota/`) and other utility endpoints.
+
+### 3.2. `manager-api` (Management Backend - Java Spring Boot Implementation)
+
+The `manager-api` component is a backend server built using Java and the Spring Boot framework, serving as the administrative hub.
+
+*   **Purpose:**
+    *   Provide a secure RESTful API for the `manager-web` frontend.
+    *   Act as a centralized configuration provider for `xiaozhi-server`.
+    *   Manage persistent data (users, devices, AI configurations, voice timbres, OTA firmware).
+
+*   **Core Technologies:**
+    *   **Java 21 & Spring Boot 3:** Core language and framework.
+    *   **Spring MVC:** For building REST controllers.
+    *   **MyBatis-Plus:** ORM for database interaction with MySQL.
+    *   **MySQL:** Relational database.
+    *   **Druid:** JDBC connection pool.
+    *   **Redis (Spring Data Redis):** For caching.
+    *   **Apache Shiro:** Security framework for authentication and authorization.
+    *   **Liquibase:** Database schema migration.
+    *   **Knife4j:** OpenAPI (Swagger) API documentation.
+    *   **Maven:** Build and dependency management.
+
+*   **Key Implementation Aspects:**
+
+    1.  **Modular Architecture (`modules/` package):**
+        *   Business logic is organized into distinct modules (e.g., `sys` for users/roles, `agent` for assistant configs, `device` for ESP32s, `config` for `xiaozhi-server` settings, `security`, `timbre`, `ota`).
+        *   Each module typically follows a layered pattern: Controller, Service, DAO (Mapper), Entity, DTO.
+
+    2.  **Layered Architecture:**
+        *   **Controller Layer (`@RestController`):** Defines API endpoints, handles HTTP request/response.
+        *   **Service Layer (`@Service`):** Contains business logic, transaction management.
+        *   **Data Access Layer (MyBatis-Plus Mappers):** Interacts with the MySQL database.
+
+    3.  **Common Functionalities (`common/` package):**
+        *   Provides shared code: base classes, global configurations (Spring, MyBatis, Redis, Knife4j), custom annotations (e.g., `@LogOperation`), AOP aspects, global exception handling, utility classes, and XSS protection.
+
+    4.  **Security (Apache Shiro):**
+        *   Manages user authentication and permissions for accessing API endpoints. Configured with Shiro Realms and security filters.
+
+    5.  **Database Schema Management (Liquibase):**
+        *   Ensures consistent database structure across environments through versioned schema changes.
+
+### 3.3. `manager-web` (Web Control Panel - Vue.js Implementation)
+
+The `manager-web` is a Single Page Application (SPA) providing the administrative user interface.
+
+*   **Purpose:**
+    *   Offer a web-based control panel for system configuration and management.
+    *   Enable administrators to configure `xiaozhi-server`'s AI services, manage users and devices, customize voice timbres, and handle OTA updates.
+
+*   **Core Technologies:**
+    *   **Vue.js 2 & Vue CLI:** Core JavaScript framework and build tools.
+    *   **Vue Router:** For client-side routing within the SPA.
+    *   **Vuex:** For centralized state management.
+    *   **Element UI:** UI component library for a consistent look and feel.
+    *   **SCSS:** CSS preprocessor.
+    *   **HTTP Client (Flyio or Axios):** For API calls to `manager-api`.
+    *   **Workbox:** For PWA features (caching, service worker).
+    *   **Opus Libraries:** For potential in-browser audio recording/playback.
+
+*   **Key Implementation Aspects:**
+
+    1.  **SPA Structure:** Single HTML page with dynamic view updates.
+    2.  **Component-Based Architecture:** UI built from reusable Vue components (`.vue` files in `src/views/` for pages and `src/components/` for smaller elements).
+    3.  **Client-Side Routing (`src/router/index.js`):** Maps browser URLs to view components, with route guards for authentication.
+    4.  **State Management (`src/store/index.js`):** Vuex manages global state (user info, device lists, etc.) via state, getters, mutations, and actions (often involving API calls).
+    5.  **API Communication (`src/apis/`):** Modularized API service files make asynchronous calls to `manager-api`.
+    6.  **Build Process & PWA Features:** Vue CLI (Webpack) bundles assets. Workbox enables PWA features like caching.
+    7.  **Environment Configuration (`.env` files):** Manages settings like the `manager-api` base URL for different environments.
+
+---
+
+## 4. Data Flow and Interaction Mechanisms
+
+The `xiaozhi-esp32-server` system coordinates work through well-defined data flows and interaction protocols between components. The main communication methods rely on WebSocket protocol optimized for real-time interaction and RESTful API suitable for client-server requests.
+
+**4.1. Core Voice Interaction Flow (ESP32 Device <-> `xiaozhi-server`)**
+
+This flow is real-time, primarily using WebSocket for low-latency, bidirectional data exchange.
+
+*   **Communication Protocol Documentation:**
+    *   Detailed communication protocol documentation can be accessed at: https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh
+    *   This document details the WebSocket communication protocol between ESP32 devices and `xiaozhi-server`, including:
+        *   Connection establishment and handshake process
+        *   Audio data transmission format
+        *   Control command format
+        *   Status report format
+        *   Error handling mechanism
+
+*   **Connection Establishment and Handshake:**
+    *   The ESP32 device, as a client, actively initiates a WebSocket connection request to the specified endpoint of `xiaozhi-server` (e.g., `ws://<server-IP>:<WebSocket-port>/xiaozhi/v1/`).
+    *   `xiaozhi-server` (`core/websocket_server.py`) receives the connection and instantiates an independent `ConnectionHandler` object for each successfully connected ESP32 device to manage the entire lifecycle of that session.
+    *   After the connection is established, an initial handshake process may be executed (handled by `core/handle/helloHandle.py`) to exchange device identification, authentication information, protocol version, or basic status.
+
+*   **Audio Uplink Transmission (ESP32 -> `xiaozhi-server`):**
+    *   After a user speaks to the ESP32 device, the device's microphone captures raw audio data (usually in PCM or compressed formats like Opus).
+    *   The ESP32 pushes these audio data chunks as WebSocket **binary messages** in real-time to the corresponding `ConnectionHandler` in `xiaozhi-server`.
+    *   The server-side `core/handle/receiveAudioHandle.py` module is responsible for receiving, buffering, and processing these audio data.
+
+*   **AI Core Processing (within `xiaozhi-server`):**
+    *   **VAD (Voice Activity Detection):** `receiveAudioHandle.py` uses the configured VAD provider (such as SileroVAD) to analyze the audio stream, accurately identifying the start and end points of speech, filtering out silent or noise segments.
+    *   **ASR (Automatic Speech Recognition):** Detected valid speech segments are sent to the configured ASR provider (local such as FunASR, or cloud services). The ASR engine converts audio signals into text strings.
+    *   **NLU/LLM (Natural Language Understanding/Large Language Model):** The ASR output text, along with the current dialogue context history obtained from the Memory provider, and the description schemas of available functions (tools) loaded from `plugins_func/`, are passed to the configured LLM provider.
+    *   **Function Call Execution (if LLM decides needed):** If the LLM analysis determines that an external function needs to be called (e.g., querying weather, controlling home appliances), it generates a structured function call request. `core/handle/functionHandler.py` receives this request, finds and executes the corresponding Python function defined in `plugins_func/`, and returns the function's execution result to the LLM. The LLM then generates the final natural language response based on this result.
+    *   **Response Generation:** The LLM synthesizes all information (user input, context, function call results, etc.) to generate the final text response.
+    *   **Memory Update:** The current round of interaction (user question, LLM response, possible function calls) is processed by the Memory provider to update the dialogue history for subsequent interactions.
+    *   **TTS (Text-to-Speech):** The final text response generated by the LLM is sent to the configured TTS provider, which synthesizes the text into a speech data stream (e.g., MP3 or WAV format).
+
+*   **Audio Downlink Response (`xiaozhi-server` -> ESP32):**
+    *   The speech data stream synthesized by the TTS provider is sent in real-time as WebSocket **binary messages** back to the ESP32 device through the `core/handle/sendAudioHandle.py` module.
+    *   The ESP32 device receives these audio data chunks and immediately plays them to the user through the speaker.
+
+*   **Control and Status Messages (Bidirectional):**
+    *   In addition to audio streams, ESP32 and `xiaozhi-server` also exchange **text messages** through WebSocket, these messages are usually encapsulated in JSON format.
+    *   **ESP32 -> Server:** The device may send status reports (such as network conditions, microphone status), error codes, or specific control commands (e.g., "stop TTS playback" triggered by user button press).
+    *   **Server -> ESP32:** The server may send control instructions to the device (such as "start listening", "stop listening", adjust sensitivity, send specific configuration parameters).
+    *   Modules like `core/handle/abortHandle.py` (handling interrupt requests), `core/handle/reportHandle.py` (handling device reports) are responsible for parsing and responding to these control/status messages.
+
+**4.2. Management and Configuration Flow (`manager-web` <-> `manager-api` <-> `xiaozhi-server`)**
+
+This flow primarily relies on HTTP/HTTPS-based RESTful API for request-response interactions.
+
+*   **Administrator UI Backend Interaction (`manager-web` -> `manager-api`):**
+    *   When administrators perform operations in the `manager-web` interface (e.g., saving a configuration, adding a new user, registering an ESP32 device):
+        *   The Vue.js frontend application (`manager-web`) will initiate asynchronous HTTP requests (usually GET, POST, PUT, DELETE) to the corresponding REST API endpoints of `manager-api` through its API encapsulation module (located in `src/apis/module/`).
+        *   Request and response bodies typically use JSON format.
+        *   The `@RestController` classes in `manager-api` receive these requests. The **Apache Shiro** framework will first perform authentication and authorization checks on the requests.
+        *   After verification, the Controller distributes the request to the corresponding Service layer to handle business logic. The Service layer may interact with the MySQL database (through MyBatis-Plus) and may utilize Redis for data caching.
+        *   After processing, `manager-api` returns an HTTP response in JSON format to `manager-web`.
+        *   `manager-web` updates its Vuex state store and user interface display based on the response results.
+
+*   **Configuration Synchronization (`manager-api` -> `xiaozhi-server`):**
+    *   The operation of `xiaozhi-server` depends on dynamic configurations obtained from `manager-api` (such as currently selected AI service providers and their API keys).
+    *   **Pull Mechanism:** The `config/manage_api_client.py` module within `xiaozhi-server`, when the server starts or through specific update triggers (e.g., when `WebSocketServer.update_config()` is called), will initiate an HTTP GET request to a specified endpoint of `manager-api` (e.g., provided by a Controller in `modules/config/controller/`).
+    *   `manager-api` responds to this request, returning the configuration data required by `xiaozhi-server` (in JSON format).
+    *   After receiving the configuration, `xiaozhi-server` will update its internal state and may reinitialize relevant AI service modules to make the new configuration effective.
+
+*   **OTA Firmware Update Flow (Conceptual Description):**
+    *   Administrators upload new ESP32 firmware packages to specific endpoints of `manager-api` through the `manager-web` interface.
+    *   `manager-api` stores the firmware files and records related metadata (version number, applicable device models, etc.).
+    *   When administrators trigger OTA updates for specific devices:
+        *   `manager-api` may notify `xiaozhi-server` (the specific notification mechanism may be a polling checkpoint, or `xiaozhi-server` exposes an API to receive update notifications, or more loosely coupled like message queues).
+        *   `xiaozhi-server` can then send an instruction message containing the firmware download URL to the target ESP32 device through WebSocket.
+        *   After receiving the instruction, the ESP32 device downloads the firmware through an HTTP GET request from that URL. This URL may point to a path served by the `SimpleHttpServer` running on `xiaozhi-server` itself (such as `/xiaozhi/ota/`), or in some architectures, it may directly point to `manager-api` or a dedicated file server.
+
+**4.3. Main Protocol Summary:**
+
+*   **WebSocket:** Selected for the communication link between ESP32 and `xiaozhi-server` because it is very suitable for real-time, low-latency, bidirectional data stream transmission (especially audio), as well as asynchronous control message delivery.
+*   **RESTful APIs (based on HTTP/HTTPS, usually using JSON as the data exchange format):** This is the standard way for web service communication. Used for request-response interactions between `manager-web` (client) and `manager-api` (server), and also for `xiaozhi-server` (as client) to pull configuration information from `manager-api` (as server). Its stateless nature, wide library support, and easy-to-understand semantics make it an ideal choice for such interactions.
+
+This multi-protocol communication strategy ensures that different types of interaction requirements within the system can be handled efficiently and appropriately, balancing real-time performance and standardized request-response patterns.
+
+---
+
+## 5. Key Features Summary
+
+The `xiaozhi-esp32-server` system provides a series of rich features aimed at supporting developers in building advanced voice control applications:
+
+1.  **Comprehensive Voice Interaction Backend:** Provides an end-to-end solution from voice capture guidance to response generation and action execution.
+2.  **Modular and Pluggable AI Services:**
+    *   Supports a wide range of ASR (Automatic Speech Recognition), LLM (Large Language Model), TTS (Text-to-Speech), VAD (Voice Activity Detection), Intent Recognition, and Memory providers.
+    *   Allows dynamic selection and configuration of these services (including cloud-based APIs and local models) to balance cost, performance, privacy, and language requirements.
+3.  **Advanced Dialogue Management:**
+    *   Supports natural interaction, with wake word to start dialogue, manual (push-to-talk) dialogue, and real-time interruption of system responses.
+    *   Includes contextual memory to maintain coherence in multi-turn dialogues.
+    *   Has automatic sleep mode after a period of inactivity.
+4.  **Multi-language Capabilities:**
+    *   Supports recognition and synthesis in multiple languages, including Mandarin, Cantonese, English, Japanese, and Korean (specific capabilities depend on the selected ASR/LLM/TTS providers).
+5.  **Extensible Functions through Plugins:**
+    *   Powerful plugin system allows developers to add custom "skills" or functions (e.g., getting weather, controlling smart home devices, accessing news).
+    *   These functions can be triggered by the LLM using its function calling capability, based on provided schemas.
+    *   Built-in support for Home Assistant integration.
+6.  **IoT Device Control:**
+    *   Designed to manage and control smart home devices and other IoT hardware through voice commands, utilizing the plugin system.
+7.  **Web-based Management Console (`manager-web` & `manager-api`):**
+    *   Provides a comprehensive graphical interface for:
+        *   System configuration (AI service selection, API keys, operation parameters).
+        *   Role-based access control user management.
+        *   ESP32 device registration and management.
+        *   Voice timbre/TTS voice customization.
+        *   OTA (Over-The-Air) firmware update management for ESP32 devices.
+        *   System parameter and dictionary management.
+8.  **Flexible Deployment Options:**
+    *   Supports deployment through Docker containers (for simplified server-only or full-stack setup) and directly from source code, adapting to various environments and user expertise.
+9.  **Dynamic Remote Configuration:**
+    *   `xiaozhi-server` can obtain its configuration from `manager-api`, allowing real-time updates of AI providers and settings without restarting the server.
+10. **Open Source and Community-Driven:**
+    *   Licensed under MIT License, encouraging transparency, collaboration, and community contribution.
+11. **Cost-Effective Solution:**
+    *   Provides an "Entry Level Free Settings" path, utilizing free tiers of AI services or local models, making it easy to experiment and for personal projects.
+12. **Progressive Web Application (PWA) Features:**
+    *   The `manager-web` control panel includes Service Worker integration to enhance caching and potential offline access capabilities.
+13. **Detailed API Documentation:**
+    *   `manager-api` provides OpenAPI (Swagger) documentation through Knife4j for clear understanding and testing of its RESTful endpoints.
+
+These features together make `xiaozhi-esp32-server` a powerful, adaptable, and user-friendly platform for building complex voice interaction applications.
+
+---
+
+## 6. Deployment and Configuration Overview
+
+The `xiaozhi-esp32-server` system is designed with flexibility in mind, providing multiple deployment methods and comprehensive configuration options to adapt to different usage scenarios and requirements.
+
+**Deployment Options:**
+
+The project can be deployed in multiple ways, mainly including using Docker to simplify the installation process, or deploying directly from source code for greater control and development.
+
+1.  **Docker-based Deployment:**
+    *   **Simplified Installation (Only `xiaozhi-server`):** This option only deploys the core Python-based `xiaozhi-server`. It is suitable for users who mainly need voice AI processing capabilities and IoT control, without requiring the complete Web management interface and database support functions (such as OTA). In this mode, configuration is typically managed through local files (`config.yaml`), but if needed, it can still point to an existing `manager-api` instance.
+    *   **Full Module Installation (All Components):** This scheme deploys all core components: `xiaozhi-server`, Java-based `manager-api`, and Vue.js-based `manager-web`, along with required database services (MySQL and Redis). This provides a complete system experience, including a Web control panel for comprehensive configuration and management.
+    *   The project provides `Dockerfile` definitions for each service and uses `docker-compose.yml` files (e.g., `docker-compose.yml` for basic version, `docker-compose_all.yml` for full-featured version) to orchestrate and manage multi-container deployment. Additionally, a `docker-setup.sh` script may be provided to assist in automating part of the Docker environment setup work.
+
+2.  **Source Code Deployment:**
+    *   This method requires manual setup of the corresponding development environment for each component: Python environment for `xiaozhi-server`, Java/Maven environment for `manager-api`, Node.js/Vue CLI environment for `manager-web`.
+    *   For full module installation, MySQL and Redis database services also need to be manually installed and configured.
+    *   This approach is typically used for project development, deep customization, debugging, or in production scenarios with special environmental requirements.
+
+**Configuration Management:**
+
+Configuration is key to customizing system behavior, especially in selecting AI service providers and managing API keys.
+
+1.  **`xiaozhi-server` Configuration:**
+    *   **Local `config.yaml`:** A main YAML format configuration file located in the `xiaozhi-server` root directory. It defines server ports, selected AI service providers (ASR, LLM, TTS, VAD, Intent Recognition, Memory modules, etc.), their respective API keys or model paths, plugin configurations, and log levels.
+    *   **Remote Configuration through `manager-api`:** `xiaozhi-server` is designed to obtain its operation configuration from `manager-api`. Settings obtained from `manager-api` typically override settings with the same name in the local `config.yaml`. This brings two major benefits:
+        *   **Centralized Management:** All configurations can be managed uniformly through the `manager-web` interface.
+        *   **Dynamic Updates:** `xiaozhi-server` can refresh its configuration and reinitialize AI modules without completely restarting the service.
+    *   `config/config_loader.py` and `config/manage_api_client.py` in `xiaozhi-server` are responsible for handling configuration loading, merging, and pulling logic from `manager-api`.
+
+2.  **`manager-api` Configuration:**
+    *   As a Spring Boot application, its configuration is mainly managed through the `application.properties` or `application.yml` file located in the `src/main/resources` directory.
+    *   Key configuration items include: database connection information (MySQL URL, username, password), Redis server address and port, application service port (default 8002), Apache Shiro security-related settings, and configuration parameters for any integrated third-party services (such as Aliyun SMS).
+
+3.  **`manager-web` Configuration:**
+    *   Environment-specific settings for the Vue.js frontend application are managed through `.env` series files (e.g., `.env`, `.env.development`, `.env.production`) in the project root directory.
+    *   The most critical configuration here is usually the API base URL address of the `manager-api` backend (e.g., `VUE_APP_API_BASE_URL`), to which the frontend application will send all API requests.
+
+4.  **Predefined Configuration Schemes:**
+    *   The project documentation (usually README) will recommend some common configuration combinations, for example:
+        *   **"Entry Level Free Settings":** This scheme aims to utilize free tier quotas of cloud AI services or completely free local models to minimize users' initial usage costs and operating expenses.
+        *   **"Full Streaming Configuration":** This scheme prioritizes system response speed and interaction fluency, typically choosing AI services that support streaming processing (possibly paid).
+    *   These predefined schemes provide guidance for users to configure AI service providers in `xiaozhi-server` (through the `manager-web` interface or directly modifying `config.yaml`).
+
+In the case of full module deployment, it is recommended to use the `manager-web` control panel as the main operation interface for most configuration tasks, as it provides a user-friendly way to manage various settings that are persisted by `manager-api` and ultimately used by `xiaozhi-server`.
+
+---

main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java

@@ -227,7 +227,7 @@ public interface Constant {
     /**
      * 版本号
      */
-    public static final String VERSION = "0.5.2";
+    public static final String VERSION = "0.5.7";
 
     /**
      * 无效固件URL

main/manager-api/src/main/java/xiaozhi/common/utils/ResultUtils.java

@@ -0,0 +1,31 @@
+package xiaozhi.common.utils;
+
+/**
+ * 返回响应体工具类
+ */
+public class ResultUtils
+{
+    public static <T> Result<T> success(T data) {
+        return new Result<T>().ok(data);
+    }
+
+    public static <T> Result<T> error() {
+        return new Result<T>().error();
+    }
+
+    public static <T> Result<T> error(String msg) {
+        return new Result<T>().error(msg);
+    }
+
+    public static <T> Result<T> error(int errorCode, String msg) {
+        return new Result<T>().error(errorCode, msg);
+    }
+
+    public static <T> Result<T> error(int errorCode) {
+        return new Result<T>().error(errorCode);
+    }
+
+    public static <T> Result<T> empty() {
+        return new Result<T>();
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java

@@ -1,6 +1,5 @@
 package xiaozhi.modules.agent.controller;
 
-import java.util.Date;
 import java.util.List;
 import java.util.Map;
 import java.util.UUID;
@@ -33,8 +32,8 @@ import xiaozhi.common.page.PageData;
 import xiaozhi.common.redis.RedisKeys;
 import xiaozhi.common.redis.RedisUtils;
 import xiaozhi.common.user.UserDetail;
-import xiaozhi.common.utils.ConvertUtils;
 import xiaozhi.common.utils.Result;
+import xiaozhi.common.utils.ResultUtils;
 import xiaozhi.modules.agent.dto.AgentChatHistoryDTO;
 import xiaozhi.modules.agent.dto.AgentChatSessionDTO;
 import xiaozhi.modules.agent.dto.AgentCreateDTO;
@@ -45,8 +44,10 @@ import xiaozhi.modules.agent.entity.AgentEntity;
 import xiaozhi.modules.agent.entity.AgentTemplateEntity;
 import xiaozhi.modules.agent.service.AgentChatAudioService;
 import xiaozhi.modules.agent.service.AgentChatHistoryService;
+import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.AgentTemplateService;
+import xiaozhi.modules.agent.vo.AgentInfoVO;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
 import xiaozhi.modules.security.user.SecurityUser;
@@ -61,6 +62,7 @@ public class AgentController {
     private final DeviceService deviceService;
     private final AgentChatHistoryService agentChatHistoryService;
     private final AgentChatAudioService agentChatAudioService;
+    private final AgentPluginMappingService agentPluginMappingService;
     private final RedisUtils redisUtils;
 
     @GetMapping("/list")
@@ -88,45 +90,17 @@ public class AgentController {
     @GetMapping("/{id}")
     @Operation(summary = "获取智能体详情")
     @RequiresPermissions("sys:role:normal")
-    public Result<AgentEntity> getAgentById(@PathVariable("id") String id) {
+    public Result<AgentInfoVO> getAgentById(@PathVariable("id") String id) {
-        AgentEntity agent = agentService.getAgentById(id);
+        AgentInfoVO agent = agentService.getAgentById(id);
-        return new Result<AgentEntity>().ok(agent);
+        return ResultUtils.success(agent);
     }
 
     @PostMapping
     @Operation(summary = "创建智能体")
     @RequiresPermissions("sys:role:normal")
     public Result<String> save(@RequestBody @Valid AgentCreateDTO dto) {
-        AgentEntity entity = ConvertUtils.sourceToTarget(dto, AgentEntity.class);
+        String agentId = agentService.createAgent(dto);
-
+        return new Result<String>().ok(agentId);
-        // 获取默认模板
-        AgentTemplateEntity template = agentTemplateService.getDefaultTemplate();
-        if (template != null) {
-            // 设置模板中的默认值
-            entity.setAsrModelId(template.getAsrModelId());
-            entity.setVadModelId(template.getVadModelId());
-            entity.setLlmModelId(template.getLlmModelId());
-            entity.setTtsModelId(template.getTtsModelId());
-            entity.setTtsVoiceId(template.getTtsVoiceId());
-            entity.setMemModelId(template.getMemModelId());
-            entity.setIntentModelId(template.getIntentModelId());
-            entity.setSystemPrompt(template.getSystemPrompt());
-            entity.setSummaryMemory(template.getSummaryMemory());
-            entity.setChatHistoryConf(template.getChatHistoryConf());
-            entity.setLangCode(template.getLangCode());
-            entity.setLanguage(template.getLanguage());
-        }
-
-        // 设置用户ID和创建者信息
-        UserDetail user = SecurityUser.getUser();
-        entity.setUserId(user.getId());
-        entity.setCreator(user.getId());
-        entity.setCreatedAt(new Date());
-
-        // ID、智能体编码和排序会在Service层自动生成
-        agentService.insert(entity);
-
-        return new Result<String>().ok(entity.getId());
     }
 
     @PutMapping("/saveMemory/{macAddress}")
@@ -138,85 +112,15 @@ public class AgentController {
         }
         AgentUpdateDTO agentUpdateDTO = new AgentUpdateDTO();
         agentUpdateDTO.setSummaryMemory(dto.getSummaryMemory());
-        return updateAgentById(device.getAgentId(), agentUpdateDTO);
+        agentService.updateAgentById(device.getAgentId(), agentUpdateDTO);
+        return new Result<>();
     }
 
     @PutMapping("/{id}")
     @Operation(summary = "更新智能体")
     @RequiresPermissions("sys:role:normal")
     public Result<Void> update(@PathVariable String id, @RequestBody @Valid AgentUpdateDTO dto) {
-        return updateAgentById(id, dto);
+        agentService.updateAgentById(id, dto);
-    }
-
-    private Result<Void> updateAgentById(String id, AgentUpdateDTO dto) {
-        // 先查询现有实体
-        AgentEntity existingEntity = agentService.getAgentById(id);
-        if (existingEntity == null) {
-            return new Result<Void>().error("智能体不存在");
-        }
-
-        // 只更新提供的非空字段
-        if (dto.getAgentName() != null) {
-            existingEntity.setAgentName(dto.getAgentName());
-        }
-        if (dto.getAgentCode() != null) {
-            existingEntity.setAgentCode(dto.getAgentCode());
-        }
-        if (dto.getAsrModelId() != null) {
-            existingEntity.setAsrModelId(dto.getAsrModelId());
-        }
-        if (dto.getVadModelId() != null) {
-            existingEntity.setVadModelId(dto.getVadModelId());
-        }
-        if (dto.getLlmModelId() != null) {
-            existingEntity.setLlmModelId(dto.getLlmModelId());
-        }
-        if (dto.getTtsModelId() != null) {
-            existingEntity.setTtsModelId(dto.getTtsModelId());
-        }
-        if (dto.getTtsVoiceId() != null) {
-            existingEntity.setTtsVoiceId(dto.getTtsVoiceId());
-        }
-        if (dto.getMemModelId() != null) {
-            existingEntity.setMemModelId(dto.getMemModelId());
-        }
-        if (dto.getIntentModelId() != null) {
-            existingEntity.setIntentModelId(dto.getIntentModelId());
-        }
-        if (dto.getSystemPrompt() != null) {
-            existingEntity.setSystemPrompt(dto.getSystemPrompt());
-        }
-        if (dto.getSummaryMemory() != null) {
-            existingEntity.setSummaryMemory(dto.getSummaryMemory());
-        }
-        if (dto.getChatHistoryConf() != null) {
-            existingEntity.setChatHistoryConf(dto.getChatHistoryConf());
-        }
-        if (dto.getLangCode() != null) {
-            existingEntity.setLangCode(dto.getLangCode());
-        }
-        if (dto.getLanguage() != null) {
-            existingEntity.setLanguage(dto.getLanguage());
-        }
-        if (dto.getSort() != null) {
-            existingEntity.setSort(dto.getSort());
-        }
-
-        // 设置更新者信息
-        UserDetail user = SecurityUser.getUser();
-        existingEntity.setUpdater(user.getId());
-        existingEntity.setUpdatedAt(new Date());
-
-        // 更新记忆策略
-        if (existingEntity.getMemModelId() == null || existingEntity.getMemModelId().equals(Constant.MEMORY_NO_MEM)) {
-            // 删除所有记录
-            agentChatHistoryService.deleteByAgentId(existingEntity.getId(), true, true);
-            existingEntity.setSummaryMemory("");
-        } else if (existingEntity.getChatHistoryConf() != null && existingEntity.getChatHistoryConf() == 1) {
-            // 删除音频数据
-            agentChatHistoryService.deleteByAgentId(existingEntity.getId(), true, false);
-        }
-        agentService.updateById(existingEntity);
         return new Result<>();
     }
 
@@ -228,6 +132,8 @@ public class AgentController {
         deviceService.deleteByAgentId(id);
         // 删除关联的聊天记录
         agentChatHistoryService.deleteByAgentId(id, true, true);
+        // 删除关联的插件
+        agentPluginMappingService.deleteByAgentId(id);
         // 再删除智能体
         agentService.deleteById(id);
         return new Result<>();

main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentDao.java

@@ -6,6 +6,7 @@ import org.apache.ibatis.annotations.Param;
 import org.apache.ibatis.annotations.Select;
 import xiaozhi.common.dao.BaseDao;
 import xiaozhi.modules.agent.entity.AgentEntity;
+import xiaozhi.modules.agent.vo.AgentInfoVO;
 
 @Mapper
 public interface AgentDao extends BaseDao<AgentEntity> {
@@ -28,4 +29,11 @@ public interface AgentDao extends BaseDao<AgentEntity> {
             " WHERE d.mac_address = #{macAddress} " +
             " ORDER BY d.id DESC LIMIT 1")
     AgentEntity getDefaultAgentByMacAddress(@Param("macAddress") String macAddress);
+
+    /**
+     * 根据id查询agent信息，包括插件信息
+     *
+     * @param agentId 智能体ID
+     */
+    AgentInfoVO selectAgentInfoById(@Param("agentId") String agentId);
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentPluginMappingMapper.java

@@ -0,0 +1,22 @@
+package xiaozhi.modules.agent.dao;
+
+import org.apache.ibatis.annotations.Mapper;
+import org.apache.ibatis.annotations.Param;
+import xiaozhi.modules.agent.entity.AgentPluginMapping;
+import com.baomidou.mybatisplus.core.mapper.BaseMapper;
+
+import java.util.List;
+
+/**
+* @description 针对表【ai_agent_plugin_mapping(Agent与插件的唯一映射表)】的数据库操作Mapper
+* @createDate 2025-05-25 22:33:17
+* @Entity xiaozhi.modules.agent.entity.AgentPluginMapping
+*/
+@Mapper
+public interface AgentPluginMappingMapper extends BaseMapper<AgentPluginMapping> {
+    List<AgentPluginMapping> selectPluginsByAgentId(@Param("agentId") String agentId);
+}
+
+
+
+

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentDTO.java

@@ -27,6 +27,9 @@ public class AgentDTO {
     @Schema(description = "大语言模型名称", example = "llm_model_01")
     private String llmModelName;
 
+    @Schema(description = "视觉模型名称", example = "vllm_model_01")
+    private String vllmModelName;
+
     @Schema(description = "记忆模型ID", example = "mem_model_01")
     private String memModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentUpdateDTO.java

@@ -1,6 +1,8 @@
 package xiaozhi.modules.agent.dto;
 
 import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
 
 import io.swagger.v3.oas.annotations.media.Schema;
 import lombok.Data;
@@ -15,49 +17,67 @@ import lombok.Data;
 public class AgentUpdateDTO implements Serializable {
     private static final long serialVersionUID = 1L;
 
-    @Schema(description = "智能体编码", example = "AGT_1234567890", required = false)
+    @Schema(description = "智能体编码", example = "AGT_1234567890", nullable = true)
     private String agentCode;
 
-    @Schema(description = "智能体名称", example = "客服助手", required = false)
+    @Schema(description = "智能体名称", example = "客服助手", nullable = true)
     private String agentName;
 
-    @Schema(description = "语音识别模型标识", example = "asr_model_02", required = false)
+    @Schema(description = "语音识别模型标识", example = "asr_model_02", nullable = true)
     private String asrModelId;
 
-    @Schema(description = "语音活动检测标识", example = "vad_model_02", required = false)
+    @Schema(description = "语音活动检测标识", example = "vad_model_02", nullable = true)
     private String vadModelId;
 
-    @Schema(description = "大语言模型标识", example = "llm_model_02", required = false)
+    @Schema(description = "大语言模型标识", example = "llm_model_02", nullable = true)
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识", example = "vllm_model_02", required = false)
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识", example = "tts_model_02", required = false)
     private String ttsModelId;
 
-    @Schema(description = "音色标识", example = "voice_02", required = false)
+    @Schema(description = "音色标识", example = "voice_02", nullable = true)
     private String ttsVoiceId;
 
-    @Schema(description = "记忆模型标识", example = "mem_model_02", required = false)
+    @Schema(description = "记忆模型标识", example = "mem_model_02", nullable = true)
     private String memModelId;
 
-    @Schema(description = "意图模型标识", example = "intent_model_02", required = false)
+    @Schema(description = "意图模型标识", example = "intent_model_02", nullable = true)
     private String intentModelId;
 
-    @Schema(description = "角色设定参数", example = "你是一个专业的客服助手，负责回答用户问题并提供帮助", required = false)
+    @Schema(description = "插件函数信息", nullable = true)
+    private List<FunctionInfo> functions;
+
+    @Schema(description = "角色设定参数", example = "你是一个专业的客服助手，负责回答用户问题并提供帮助", nullable = true)
     private String systemPrompt;
 
-    @Schema(description = "总结记忆", example = "构建可生长的动态记忆网络，在有限空间内保留关键信息的同时，智能维护信息演变轨迹\n" +
+    @Schema(description = "总结记忆", example = "构建可生长的动态记忆网络，在有限空间内保留关键信息的同时，智能维护信息演变轨迹\n"
-            "根据对话记录，总结user的重要信息，以便在未来的对话中提供更个性化的服务", required = false)
+            + "根据对话记录，总结user的重要信息，以便在未来的对话中提供更个性化的服务", nullable = true)
     private String summaryMemory;
 
-    @Schema(description = "聊天记录配置（0不记录 1仅记录文本 2记录文本和语音）", example = "3", required = false)
+    @Schema(description = "聊天记录配置（0不记录 1仅记录文本 2记录文本和语音）", example = "3", nullable = true)
     private Integer chatHistoryConf;
 
-    @Schema(description = "语言编码", example = "zh_CN", required = false)
+    @Schema(description = "语言编码", example = "zh_CN", nullable = true)
     private String langCode;
 
-    @Schema(description = "交互语种", example = "中文", required = false)
+    @Schema(description = "交互语种", example = "中文", nullable = true)
     private String language;
 
-    @Schema(description = "排序", example = "1", required = false)
+    @Schema(description = "排序", example = "1", nullable = true)
     private Integer sort;
+
+    @Data
+    @Schema(description = "插件函数信息")
+    public static class FunctionInfo implements Serializable {
+        @Schema(description = "插件ID", example = "plugin_01")
+        private String pluginId;
+
+        @Schema(description = "函数参数信息", nullable = true)
+        private HashMap<String, Object> paramInfo;
+
+        private static final long serialVersionUID = 1L;
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentEntity.java

@@ -36,6 +36,9 @@ public class AgentEntity {
     @Schema(description = "大语言模型标识")
     private String llmModelId;
 
+    @Schema(description = "VLLM模型标识")
+    private String vllmModelId;
+
     @Schema(description = "语音合成模型标识")
     private String ttsModelId;
 

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentPluginMapping.java

@@ -0,0 +1,54 @@
+package xiaozhi.modules.agent.entity;
+
+import java.io.Serializable;
+
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableField;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import lombok.Data;
+
+/**
+ * Agent与插件的唯一映射表
+ * 
+ * @TableName ai_agent_plugin_mapping
+ */
+@Data
+@TableName(value = "ai_agent_plugin_mapping")
+@Schema(description = "Agent与插件的唯一映射表")
+public class AgentPluginMapping implements Serializable {
+    /**
+     * 主键
+     */
+    @TableId(type = IdType.ASSIGN_ID)
+    @Schema(description = "映射信息主键ID")
+    private Long id;
+
+    /**
+     * 智能体ID
+     */
+    @Schema(description = "智能体ID")
+    private String agentId;
+
+    /**
+     * 插件ID
+     */
+    @Schema(description = "插件ID")
+    private String pluginId;
+
+    /**
+     * 插件参数(Json)格式
+     */
+    @Schema(description = "插件参数(Json)格式")
+    private String paramInfo;
+
+    // 冗余字段，用于方便在根据id查询插件时，对照查出插件的Provider_code,详见dao层xml文件
+    @TableField(exist = false)
+    @Schema(description = "插件provider_code, 对应表ai_model_provider")
+    private String providerCode;
+
+    @TableField(exist = false)
+    private static final long serialVersionUID = 1L;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentTemplateEntity.java

@@ -49,6 +49,11 @@ public class AgentTemplateEntity implements Serializable {
      */
     private String llmModelId;
 
+    /**
+     * VLLM模型标识
+     */
+    private String vllmModelId;
+
     /**
      * 语音合成模型标识
      */

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentPluginMappingService.java

@@ -0,0 +1,29 @@
+package xiaozhi.modules.agent.service;
+
+import java.util.List;
+
+import com.baomidou.mybatisplus.extension.service.IService;
+
+import xiaozhi.modules.agent.entity.AgentPluginMapping;
+
+/**
+ * @description 针对表【ai_agent_plugin_mapping(Agent与插件的唯一映射表)】的数据库操作Service
+ * @createDate 2025-05-25 22:33:17
+ */
+public interface AgentPluginMappingService extends IService<AgentPluginMapping> {
+
+    /**
+     * 根据智能体id获取插件参数
+     * 
+     * @param agentId
+     * @return
+     */
+    List<AgentPluginMapping> agentPluginParamsByAgentId(String agentId);
+
+    /**
+     * 根据智能体id删除插件参数
+     * 
+     * @param agentId
+     */
+    void deleteByAgentId(String agentId);
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentService.java

@@ -5,8 +5,11 @@ import java.util.Map;
 
 import xiaozhi.common.page.PageData;
 import xiaozhi.common.service.BaseService;
+import xiaozhi.modules.agent.dto.AgentCreateDTO;
 import xiaozhi.modules.agent.dto.AgentDTO;
+import xiaozhi.modules.agent.dto.AgentUpdateDTO;
 import xiaozhi.modules.agent.entity.AgentEntity;
+import xiaozhi.modules.agent.vo.AgentInfoVO;
 
 /**
  * 智能体表处理service
@@ -30,7 +33,7 @@ public interface AgentService extends BaseService<AgentEntity> {
      * @param id 智能体ID
      * @return 智能体实体
      */
-    AgentEntity getAgentById(String id);
+    AgentInfoVO getAgentById(String id);
 
     /**
      * 插入智能体
@@ -79,4 +82,20 @@ public interface AgentService extends BaseService<AgentEntity> {
      * @return 是否有权限
      */
     boolean checkAgentPermission(String agentId, Long userId);
+
+    /**
+     * 更新智能体
+     *
+     * @param agentId 智能体ID
+     * @param dto     更新智能体所需的信息
+     */
+    void updateAgentById(String agentId, AgentUpdateDTO dto);
+
+    /**
+     * 创建智能体
+     *
+     * @param dto 创建智能体所需的信息
+     * @return 创建的智能体ID
+     */
+    String createAgent(AgentCreateDTO dto);
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentPluginMappingServiceImpl.java

@@ -0,0 +1,37 @@
+package xiaozhi.modules.agent.service.impl;
+
+import java.util.List;
+
+import org.springframework.stereotype.Service;
+
+import com.baomidou.mybatisplus.core.conditions.update.UpdateWrapper;
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
+
+import lombok.RequiredArgsConstructor;
+import xiaozhi.modules.agent.dao.AgentPluginMappingMapper;
+import xiaozhi.modules.agent.entity.AgentPluginMapping;
+import xiaozhi.modules.agent.service.AgentPluginMappingService;
+
+/**
+ * @description 针对表【ai_agent_plugin_mapping(Agent与插件的唯一映射表)】的数据库操作Service实现
+ * @createDate 2025-05-25 22:33:17
+ */
+@Service
+@RequiredArgsConstructor
+public class AgentPluginMappingServiceImpl extends ServiceImpl<AgentPluginMappingMapper, AgentPluginMapping>
+        implements AgentPluginMappingService {
+    private final AgentPluginMappingMapper agentPluginMappingMapper;
+
+    @Override
+    public List<AgentPluginMapping> agentPluginParamsByAgentId(String agentId) {
+        return agentPluginMappingMapper.selectPluginsByAgentId(agentId);
+    }
+
+    @Override
+    public void deleteByAgentId(String agentId) {
+        UpdateWrapper<AgentPluginMapping> updateWrapper = new UpdateWrapper<>();
+        updateWrapper.eq("agent_id", agentId);
+        agentPluginMappingMapper.delete(updateWrapper);
+    }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentServiceImpl.java

@@ -1,12 +1,17 @@
 package xiaozhi.modules.agent.service.impl;
 
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.UUID;
+import java.util.function.Function;
 import java.util.stream.Collectors;
 
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.stereotype.Service;
+import org.springframework.transaction.annotation.Transactional;
 
 import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
 import com.baomidou.mybatisplus.core.conditions.update.UpdateWrapper;
@@ -14,16 +19,30 @@ import com.baomidou.mybatisplus.core.metadata.IPage;
 
 import lombok.AllArgsConstructor;
 import xiaozhi.common.constant.Constant;
+import xiaozhi.common.exception.RenException;
 import xiaozhi.common.page.PageData;
 import xiaozhi.common.redis.RedisKeys;
 import xiaozhi.common.redis.RedisUtils;
 import xiaozhi.common.service.impl.BaseServiceImpl;
+import xiaozhi.common.user.UserDetail;
+import xiaozhi.common.utils.ConvertUtils;
+import xiaozhi.common.utils.JsonUtils;
 import xiaozhi.modules.agent.dao.AgentDao;
+import xiaozhi.modules.agent.dto.AgentCreateDTO;
 import xiaozhi.modules.agent.dto.AgentDTO;
+import xiaozhi.modules.agent.dto.AgentUpdateDTO;
 import xiaozhi.modules.agent.entity.AgentEntity;
+import xiaozhi.modules.agent.entity.AgentPluginMapping;
+import xiaozhi.modules.agent.entity.AgentTemplateEntity;
+import xiaozhi.modules.agent.service.AgentChatHistoryService;
+import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
+import xiaozhi.modules.agent.service.AgentTemplateService;
+import xiaozhi.modules.agent.vo.AgentInfoVO;
 import xiaozhi.modules.device.service.DeviceService;
+import xiaozhi.modules.model.dto.ModelProviderDTO;
 import xiaozhi.modules.model.service.ModelConfigService;
+import xiaozhi.modules.model.service.ModelProviderService;
 import xiaozhi.modules.security.user.SecurityUser;
 import xiaozhi.modules.sys.enums.SuperAdminEnum;
 import xiaozhi.modules.timbre.service.TimbreService;
@@ -36,6 +55,10 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
     private final ModelConfigService modelConfigService;
     private final RedisUtils redisUtils;
     private final DeviceService deviceService;
+    private final AgentPluginMappingService agentPluginMappingService;
+    private final AgentChatHistoryService agentChatHistoryService;
+    private final AgentTemplateService agentTemplateService;
+    private final ModelProviderService modelProviderService;
 
     @Override
     public PageData<AgentEntity> adminAgentList(Map<String, Object> params) {
@@ -46,15 +69,20 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
     }
 
     @Override
-    public AgentEntity getAgentById(String id) {
+    public AgentInfoVO getAgentById(String id) {
-        AgentEntity agent = agentDao.selectById(id);
+        AgentInfoVO agent = agentDao.selectAgentInfoById(id);
-        if (agent != null && agent.getMemModelId() != null && agent.getMemModelId().equals(Constant.MEMORY_NO_MEM)) {
+
-            agent.setChatHistoryConf(Constant.ChatHistoryConfEnum.IGNORE.getCode());
+        if (agent == null) {
-        } else if (agent != null && agent.getMemModelId() != null
+            throw new RenException("智能体不存在");
-                && !agent.getMemModelId().equals(Constant.MEMORY_NO_MEM)
-                && agent.getChatHistoryConf() == null) {
-            agent.setChatHistoryConf(Constant.ChatHistoryConfEnum.RECORD_TEXT_AUDIO.getCode());
         }
+
+        if (agent.getMemModelId() != null && agent.getMemModelId().equals(Constant.MEMORY_NO_MEM)) {
+            agent.setChatHistoryConf(Constant.ChatHistoryConfEnum.IGNORE.getCode());
+            if (agent.getChatHistoryConf() == null) {
+                agent.setChatHistoryConf(Constant.ChatHistoryConfEnum.RECORD_TEXT_AUDIO.getCode());
+            }
+        }
+        // 无需额外查询插件列表，已通过SQL查询出来
         return agent;
     }
 
@@ -102,6 +130,9 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
             // 获取 LLM 模型名称
             dto.setLlmModelName(modelConfigService.getModelNameById(agent.getLlmModelId()));
 
+            // 获取 VLLM 模型名称
+            dto.setVllmModelName(modelConfigService.getModelNameById(agent.getVllmModelId()));
+
             // 获取记忆模型名称
             dto.setMemModelId(agent.getMemModelId());
 
@@ -164,4 +195,198 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
         // 检查是否是智能体的所有者
         return userId.equals(agent.getUserId());
     }
+
+    // 根据id更新智能体信息
+    @Override
+    @Transactional(rollbackFor = Exception.class)
+    public void updateAgentById(String agentId, AgentUpdateDTO dto) {
+        // 先查询现有实体
+        AgentEntity existingEntity = this.getAgentById(agentId);
+        if (existingEntity == null) {
+            throw new RuntimeException("智能体不存在");
+        }
+
+        // 只更新提供的非空字段
+        if (dto.getAgentName() != null) {
+            existingEntity.setAgentName(dto.getAgentName());
+        }
+        if (dto.getAgentCode() != null) {
+            existingEntity.setAgentCode(dto.getAgentCode());
+        }
+        if (dto.getAsrModelId() != null) {
+            existingEntity.setAsrModelId(dto.getAsrModelId());
+        }
+        if (dto.getVadModelId() != null) {
+            existingEntity.setVadModelId(dto.getVadModelId());
+        }
+        if (dto.getLlmModelId() != null) {
+            existingEntity.setLlmModelId(dto.getLlmModelId());
+        }
+        if (dto.getVllmModelId() != null) {
+            existingEntity.setVllmModelId(dto.getVllmModelId());
+        }
+        if (dto.getTtsModelId() != null) {
+            existingEntity.setTtsModelId(dto.getTtsModelId());
+        }
+        if (dto.getTtsVoiceId() != null) {
+            existingEntity.setTtsVoiceId(dto.getTtsVoiceId());
+        }
+        if (dto.getMemModelId() != null) {
+            existingEntity.setMemModelId(dto.getMemModelId());
+        }
+        if (dto.getIntentModelId() != null) {
+            existingEntity.setIntentModelId(dto.getIntentModelId());
+        }
+        if (dto.getSystemPrompt() != null) {
+            existingEntity.setSystemPrompt(dto.getSystemPrompt());
+        }
+        if (dto.getSummaryMemory() != null) {
+            existingEntity.setSummaryMemory(dto.getSummaryMemory());
+        }
+        if (dto.getChatHistoryConf() != null) {
+            existingEntity.setChatHistoryConf(dto.getChatHistoryConf());
+        }
+        if (dto.getLangCode() != null) {
+            existingEntity.setLangCode(dto.getLangCode());
+        }
+        if (dto.getLanguage() != null) {
+            existingEntity.setLanguage(dto.getLanguage());
+        }
+        if (dto.getSort() != null) {
+            existingEntity.setSort(dto.getSort());
+        }
+
+        // 更新函数插件信息
+        List<AgentUpdateDTO.FunctionInfo> functions = dto.getFunctions();
+        if (functions != null) {
+            // 1. 收集本次提交的 pluginId
+            List<String> newPluginIds = functions.stream()
+                    .map(AgentUpdateDTO.FunctionInfo::getPluginId)
+                    .toList();
+
+            // 2. 查询当前agent现有的所有映射
+            List<AgentPluginMapping> existing = agentPluginMappingService.list(
+                    new QueryWrapper<AgentPluginMapping>()
+                            .eq("agent_id", agentId));
+            Map<String, AgentPluginMapping> existMap = existing.stream()
+                    .collect(Collectors.toMap(AgentPluginMapping::getPluginId, Function.identity()));
+
+            // 3. 构造所有要 保存或更新 的实体
+            List<AgentPluginMapping> allToPersist = functions.stream().map(info -> {
+                AgentPluginMapping m = new AgentPluginMapping();
+                m.setAgentId(agentId);
+                m.setPluginId(info.getPluginId());
+                m.setParamInfo(JsonUtils.toJsonString(info.getParamInfo()));
+                AgentPluginMapping old = existMap.get(info.getPluginId());
+                if (old != null) {
+                    // 已存在，设置id表示更新
+                    m.setId(old.getId());
+                }
+                return m;
+            }).toList();
+
+            // 4. 拆分：已有ID的走更新，无ID的走插入
+            List<AgentPluginMapping> toUpdate = allToPersist.stream()
+                    .filter(m -> m.getId() != null)
+                    .toList();
+            List<AgentPluginMapping> toInsert = allToPersist.stream()
+                    .filter(m -> m.getId() == null)
+                    .toList();
+
+            if (!toUpdate.isEmpty()) {
+                agentPluginMappingService.updateBatchById(toUpdate);
+            }
+            if (!toInsert.isEmpty()) {
+                agentPluginMappingService.saveBatch(toInsert);
+            }
+
+            // 5. 删除本次不在提交列表里的插件映射
+            List<Long> toDelete = existing.stream()
+                    .filter(old -> !newPluginIds.contains(old.getPluginId()))
+                    .map(AgentPluginMapping::getId)
+                    .toList();
+            if (!toDelete.isEmpty()) {
+                agentPluginMappingService.removeBatchByIds(toDelete);
+            }
+        }
+
+        // 设置更新者信息
+        UserDetail user = SecurityUser.getUser();
+        existingEntity.setUpdater(user.getId());
+        existingEntity.setUpdatedAt(new Date());
+
+        // 更新记忆策略
+        if (existingEntity.getMemModelId() == null || existingEntity.getMemModelId().equals(Constant.MEMORY_NO_MEM)) {
+            // 删除所有记录
+            agentChatHistoryService.deleteByAgentId(existingEntity.getId(), true, true);
+            existingEntity.setSummaryMemory("");
+        } else if (existingEntity.getChatHistoryConf() != null && existingEntity.getChatHistoryConf() == 1) {
+            // 删除音频数据
+            agentChatHistoryService.deleteByAgentId(existingEntity.getId(), true, false);
+        }
+        this.updateById(existingEntity);
+    }
+
+    @Override
+    @Transactional(rollbackFor = Exception.class)
+    public String createAgent(AgentCreateDTO dto) {
+        // 转换为实体
+        AgentEntity entity = ConvertUtils.sourceToTarget(dto, AgentEntity.class);
+
+        // 获取默认模板
+        AgentTemplateEntity template = agentTemplateService.getDefaultTemplate();
+        if (template != null) {
+            // 设置模板中的默认值
+            entity.setAsrModelId(template.getAsrModelId());
+            entity.setVadModelId(template.getVadModelId());
+            entity.setLlmModelId(template.getLlmModelId());
+            entity.setVllmModelId(template.getVllmModelId());
+            entity.setTtsModelId(template.getTtsModelId());
+            entity.setTtsVoiceId(template.getTtsVoiceId());
+            entity.setMemModelId(template.getMemModelId());
+            entity.setIntentModelId(template.getIntentModelId());
+            entity.setSystemPrompt(template.getSystemPrompt());
+            entity.setSummaryMemory(template.getSummaryMemory());
+            entity.setChatHistoryConf(template.getChatHistoryConf());
+            entity.setLangCode(template.getLangCode());
+            entity.setLanguage(template.getLanguage());
+        }
+
+        // 设置用户ID和创建者信息
+        UserDetail user = SecurityUser.getUser();
+        entity.setUserId(user.getId());
+        entity.setCreator(user.getId());
+        entity.setCreatedAt(new Date());
+
+        // 保存智能体
+        insert(entity);
+
+        // 设置默认插件
+        List<AgentPluginMapping> toInsert = new ArrayList<>();
+        // 播放音乐、查天气、查新闻
+        String[] pluginIds = new String[] { "SYSTEM_PLUGIN_MUSIC", "SYSTEM_PLUGIN_WEATHER",
+                "SYSTEM_PLUGIN_NEWS_NEWSNOW" };
+        for (String pluginId : pluginIds) {
+            ModelProviderDTO provider = modelProviderService.getById(pluginId);
+            if (provider == null) {
+                continue;
+            }
+            AgentPluginMapping mapping = new AgentPluginMapping();
+            mapping.setPluginId(pluginId);
+
+            Map<String, Object> paramInfo = new HashMap<>();
+            List<Map<String, Object>> fields = JsonUtils.parseObject(provider.getFields(), List.class);
+            if (fields != null) {
+                for (Map<String, Object> field : fields) {
+                    paramInfo.put((String) field.get("key"), field.get("default"));
+                }
+            }
+            mapping.setParamInfo(JsonUtils.toJsonString(paramInfo));
+            mapping.setAgentId(entity.getId());
+            toInsert.add(mapping);
+        }
+        // 保存默认插件
+        agentPluginMappingService.saveBatch(toInsert);
+        return entity.getId();
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentInfoVO.java

@@ -0,0 +1,24 @@
+package xiaozhi.modules.agent.vo;
+
+import com.baomidou.mybatisplus.annotation.TableField;
+import com.baomidou.mybatisplus.extension.handlers.JacksonTypeHandler;
+import io.swagger.v3.oas.annotations.media.Schema;
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import xiaozhi.modules.agent.entity.AgentEntity;
+import xiaozhi.modules.agent.entity.AgentPluginMapping;
+
+import java.util.List;
+
+/**
+ * Agent信息返回体VO
+ * 这里直接extend了Agent实体类AgentEntity，后续需要规范返回字段可以copy字段出来
+ */
+@EqualsAndHashCode(callSuper = true)
+@Data
+public class AgentInfoVO extends AgentEntity
+{
+    @Schema(description = "插件列表Id")
+    @TableField(typeHandler = JacksonTypeHandler.class)
+    private List<AgentPluginMapping> functions;
+}

main/manager-api/src/main/java/xiaozhi/modules/config/controller/ConfigController.java

@@ -27,7 +27,7 @@ public class ConfigController {
     private final ConfigService configService;
 
     @PostMapping("server-base")
-    @Operation(summary = "获取配置")
+    @Operation(summary = "服务端获取配置接口")
     public Result<Object> getConfig() {
         Object config = configService.getConfig(true);
         return new Result<Object>().ok(config);

main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java

@@ -1,9 +1,6 @@
 package xiaozhi.modules.config.service.impl;
 
-import java.util.ArrayList;
+import java.util.*;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
 
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.stereotype.Service;
@@ -16,7 +13,9 @@ import xiaozhi.common.redis.RedisKeys;
 import xiaozhi.common.redis.RedisUtils;
 import xiaozhi.common.utils.JsonUtils;
 import xiaozhi.modules.agent.entity.AgentEntity;
+import xiaozhi.modules.agent.entity.AgentPluginMapping;
 import xiaozhi.modules.agent.entity.AgentTemplateEntity;
+import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.AgentTemplateService;
 import xiaozhi.modules.config.service.ConfigService;
@@ -39,6 +38,7 @@ public class ConfigServiceImpl implements ConfigService {
     private final AgentTemplateService agentTemplateService;
     private final RedisUtils redisUtils;
     private final TimbreService timbreService;
+    private final AgentPluginMappingService agentPluginMappingService;
 
     @Override
     public Object getConfig(Boolean isCache) {
@@ -72,6 +72,7 @@ public class ConfigServiceImpl implements ConfigService {
                 null,
                 null,
                 null,
+                null,
                 result,
                 isCache);
 
@@ -131,6 +132,19 @@ public class ConfigServiceImpl implements ConfigService {
             agent.setAsrModelId(null);
         }
 
+        // 添加函数调用参数信息
+        if (!Objects.equals(agent.getIntentModelId(), "Intent_nointent")) {
+            String agentId = agent.getId();
+            List<AgentPluginMapping> pluginMappings = agentPluginMappingService.agentPluginParamsByAgentId(agentId);
+            if (pluginMappings != null && !pluginMappings.isEmpty()) {
+                Map<String, Object> pluginParams = new HashMap<>();
+                for (AgentPluginMapping pluginMapping : pluginMappings) {
+                    pluginParams.put(pluginMapping.getProviderCode(), pluginMapping.getParamInfo());
+                }
+                result.put("plugins", pluginParams);
+            }
+        }
+
         // 构建模块配置
         buildModuleConfig(
                 agent.getAgentName(),
@@ -140,6 +154,7 @@ public class ConfigServiceImpl implements ConfigService {
                 agent.getVadModelId(),
                 agent.getAsrModelId(),
                 agent.getLlmModelId(),
+                agent.getVllmModelId(),
                 agent.getTtsModelId(),
                 agent.getMemModelId(),
                 agent.getIntentModelId(),
@@ -241,6 +256,7 @@ public class ConfigServiceImpl implements ConfigService {
             String vadModelId,
             String asrModelId,
             String llmModelId,
+            String vllmModelId,
             String ttsModelId,
             String memModelId,
             String intentModelId,
@@ -248,8 +264,8 @@ public class ConfigServiceImpl implements ConfigService {
             boolean isCache) {
         Map<String, String> selectedModule = new HashMap<>();
 
-        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM" };
+        String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM", "VLLM" };
-        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId };
+        String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId, vllmModelId };
         String intentLLMModelId = null;
         String memLocalShortLLMModelId = null;
 
@@ -281,6 +297,7 @@ public class ConfigServiceImpl implements ConfigService {
                             map.put("functions", functions);
                         }
                     }
+                    System.out.println("map: " + map);
                 }
                 if ("Memory".equals(modelTypes[i])) {
                     Map<String, Object> map = (Map<String, Object>) model.getConfigJson();

main/manager-api/src/main/java/xiaozhi/modules/device/controller/DeviceController.java

@@ -4,6 +4,7 @@ import java.util.List;
 
 import org.apache.commons.lang3.StringUtils;
 import org.apache.shiro.authz.annotation.RequiresPermissions;
+import org.springframework.beans.BeanUtils;
 import org.springframework.web.bind.annotation.GetMapping;
 import org.springframework.web.bind.annotation.PathVariable;
 import org.springframework.web.bind.annotation.PostMapping;
@@ -14,6 +15,7 @@ import org.springframework.web.bind.annotation.RestController;
 
 import io.swagger.v3.oas.annotations.Operation;
 import io.swagger.v3.oas.annotations.tags.Tag;
+import jakarta.validation.Valid;
 import lombok.AllArgsConstructor;
 import xiaozhi.common.exception.ErrorCode;
 import xiaozhi.common.redis.RedisKeys;
@@ -22,6 +24,7 @@ import xiaozhi.common.user.UserDetail;
 import xiaozhi.common.utils.Result;
 import xiaozhi.modules.device.dto.DeviceRegisterDTO;
 import xiaozhi.modules.device.dto.DeviceUnBindDTO;
+import xiaozhi.modules.device.dto.DeviceUpdateDTO;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
 import xiaozhi.modules.security.user.SecurityUser;
@@ -80,15 +83,19 @@ public class DeviceController {
         return new Result<Void>();
     }
 
-    @PutMapping("/enableOta/{id}/{status}")
+    @PutMapping("/update/{id}")
-    @Operation(summary = "启用/关闭OTA自动升级")
+    @Operation(summary = "更新设备信息")
     @RequiresPermissions("sys:role:normal")
-    public Result<Void> enableOtaUpgrade(@PathVariable String id, @PathVariable Integer status) {
+    public Result<Void> updateDeviceInfo(@PathVariable String id, @Valid @RequestBody DeviceUpdateDTO deviceUpdateDTO) {
         DeviceEntity entity = deviceService.selectById(id);
         if (entity == null) {
             return new Result<Void>().error("设备不存在");
         }
-        entity.setAutoUpdate(status);
+        UserDetail user = SecurityUser.getUser();
+        if (!entity.getUserId().equals(user.getId())) {
+            return new Result<Void>().error("设备不存在");
+        }
+        BeanUtils.copyProperties(deviceUpdateDTO, entity);
         deviceService.updateById(entity);
         return new Result<Void>();
     }

main/manager-api/src/main/java/xiaozhi/modules/device/dto/DeviceUpdateDTO.java

@@ -0,0 +1,29 @@
+package xiaozhi.modules.device.dto;
+
+import jakarta.validation.constraints.Max;
+import jakarta.validation.constraints.Min;
+import jakarta.validation.constraints.Size;
+import lombok.Data;
+
+import java.io.Serializable;
+
+/**
+ * 设备更新DTO
+ */
+@Data
+public class DeviceUpdateDTO implements Serializable {
+    /**
+    * 自动更新状态
+    */
+    @Max(1)
+    @Min(0)
+    private Integer autoUpdate;
+
+    /**
+    * 设备别名
+    */
+    @Size(max = 64)
+    private String alias;
+
+    private static final long serialVersionUID = 1L;
+}

main/manager-api/src/main/java/xiaozhi/modules/model/controller/ModelProviderController.java

@@ -18,6 +18,7 @@ import io.swagger.v3.oas.annotations.tags.Tag;
 import lombok.AllArgsConstructor;
 import xiaozhi.common.page.PageData;
 import xiaozhi.common.utils.Result;
+import xiaozhi.common.utils.ResultUtils;
 import xiaozhi.common.validator.group.UpdateGroup;
 import xiaozhi.modules.model.dto.ModelProviderDTO;
 import xiaozhi.modules.model.service.ModelProviderService;
@@ -65,4 +66,10 @@ public class ModelProviderController {
         return new Result<>();
     }
 
+    @GetMapping("/plugin/names")
+    @Tag(name = "获取插件名称列表")
+    public Result<List<ModelProviderDTO>> getPluginNameList() {
+        return ResultUtils.success(modelProviderService.getPluginList());
+    }
+
 }

main/manager-api/src/main/java/xiaozhi/modules/model/service/ModelProviderService.java

@@ -1,5 +1,6 @@
 package xiaozhi.modules.model.service;
 
+import java.util.Collection;
 import java.util.List;
 
 import xiaozhi.common.page.PageData;
@@ -7,7 +8,11 @@ import xiaozhi.modules.model.dto.ModelProviderDTO;
 
 public interface ModelProviderService {
 
-    // List<String> getModelNames(String modelType, String modelName);
+    List<ModelProviderDTO> getPluginList();
+
+    ModelProviderDTO getById(String id);
+
+    List<ModelProviderDTO> getPluginListByIds(Collection<String> ids);
 
     List<ModelProviderDTO> getListByModelType(String modelType);
 

main/manager-api/src/main/java/xiaozhi/modules/model/service/impl/ModelProviderServiceImpl.java

@@ -1,5 +1,6 @@
 package xiaozhi.modules.model.service.impl;
 
+import java.util.Collection;
 import java.util.Date;
 import java.util.HashMap;
 import java.util.List;
@@ -8,6 +9,7 @@ import java.util.Map;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.stereotype.Service;
 
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
 import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
 import com.baomidou.mybatisplus.core.metadata.IPage;
 
@@ -32,6 +34,29 @@ public class ModelProviderServiceImpl extends BaseServiceImpl<ModelProviderDao,
 
     private final ModelProviderDao modelProviderDao;
 
+    @Override
+    public List<ModelProviderDTO> getPluginList() {
+        LambdaQueryWrapper<ModelProviderEntity> queryWrapper = new LambdaQueryWrapper<>();
+        queryWrapper.eq(ModelProviderEntity::getModelType, "Plugin");
+        List<ModelProviderEntity> providerEntities = modelProviderDao.selectList(queryWrapper);
+        return ConvertUtils.sourceToTarget(providerEntities, ModelProviderDTO.class);
+    }
+
+    @Override
+    public ModelProviderDTO getById(String id) {
+        ModelProviderEntity entity = modelProviderDao.selectById(id);
+        return ConvertUtils.sourceToTarget(entity, ModelProviderDTO.class);
+    }
+
+    @Override
+    public List<ModelProviderDTO> getPluginListByIds(Collection<String> ids) {
+        LambdaQueryWrapper<ModelProviderEntity> queryWrapper = new LambdaQueryWrapper<>();
+        queryWrapper.in(ModelProviderEntity::getId, ids);
+        queryWrapper.eq(ModelProviderEntity::getModelType, "Plugin");
+        List<ModelProviderEntity> providerEntities = modelProviderDao.selectList(queryWrapper);
+        return ConvertUtils.sourceToTarget(providerEntities, ModelProviderDTO.class);
+    }
+
     @Override
     public List<ModelProviderDTO> getListByModelType(String modelType) {
 

main/manager-api/src/main/java/xiaozhi/modules/sys/enums/ServerActionEnum.java

@@ -2,36 +2,29 @@ package xiaozhi.modules.sys.enums;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonValue;
-import xiaozhi.common.exception.RenException;
 
 /**
  * 服务端动作枚举
  */
-public enum ServerActionEnum
+public enum ServerActionEnum {
-{
     RESTART("restart"),
     UPDATE_CONFIG("update_config");
 
     private final String value;
 
-    ServerActionEnum(String value)
+    ServerActionEnum(String value) {
-    {
         this.value = value;
     }
 
     @JsonValue
-    public String getValue()
+    public String getValue() {
-    {
         return value;
     }
 
     @JsonCreator
-    public static ServerActionEnum fromValue(String value)
+    public static ServerActionEnum fromValue(String value) {
-    {
+        for (ServerActionEnum action : ServerActionEnum.values()) {
-        for (ServerActionEnum action : ServerActionEnum.values())
+            if (action.value.equalsIgnoreCase(value)) {
-        {
-            if (action.value.equalsIgnoreCase(value))
-            {
                 return action;
             }
         }

main/manager-api/src/main/java/xiaozhi/modules/sys/enums/ServerActionResponseEnum.java

@@ -1,16 +1,16 @@
 package xiaozhi.modules.sys.enums;
 
+import org.apache.commons.lang3.StringUtils;
+
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonValue;
-import lombok.Getter;
-import org.apache.commons.lang3.StringUtils;
 
 /**
  * 服务端调用响应枚举
  */
-public enum ServerActionResponseEnum
+public enum ServerActionResponseEnum {
-{
     SUCCESS("success"), FAIL("fail");
+
     private final String value;
 
     ServerActionResponseEnum(String value) {
@@ -18,8 +18,7 @@ public enum ServerActionResponseEnum
     }
 
     @JsonValue
-    public String getValue()
+    public String getValue() {
-    {
         return value;
     }
 

main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java

@@ -1,13 +1,5 @@
 package xiaozhi.modules.sys.utils;
 
-import com.fasterxml.jackson.databind.ObjectMapper;
-import lombok.extern.slf4j.Slf4j;
-import org.springframework.util.StopWatch;
-import org.springframework.web.socket.*;
-import org.springframework.web.socket.client.standard.StandardWebSocketClient;
-import org.springframework.web.socket.handler.AbstractWebSocketHandler;
-import xiaozhi.common.utils.DateUtils;
-
 import java.io.Closeable;
 import java.io.IOException;
 import java.net.URI;
@@ -15,30 +7,51 @@ import java.nio.ByteBuffer;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Objects;
-import java.util.concurrent.*;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.function.Consumer;
 import java.util.function.Predicate;
 
+import org.springframework.util.StopWatch;
+import org.springframework.web.socket.BinaryMessage;
+import org.springframework.web.socket.CloseStatus;
+import org.springframework.web.socket.TextMessage;
+import org.springframework.web.socket.WebSocketHttpHeaders;
+import org.springframework.web.socket.WebSocketSession;
+import org.springframework.web.socket.client.standard.StandardWebSocketClient;
+import org.springframework.web.socket.handler.AbstractWebSocketHandler;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import lombok.extern.slf4j.Slf4j;
+import xiaozhi.common.utils.DateUtils;
+
 /**
  * WebSocketClientResource：支持 try-with-resources 模式
  */
 @Slf4j
-public class WebSocketClientManager implements Closeable
+public class WebSocketClientManager implements Closeable {
-{
     private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
 
     // 全局回调线程池
-    private static final ExecutorService CALLBACK_EXECUTOR = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors(), new ThreadFactory() {
+    private static final ExecutorService CALLBACK_EXECUTOR = Executors
-        private final AtomicInteger cnt = new AtomicInteger();
+            .newFixedThreadPool(Runtime.getRuntime().availableProcessors(), new ThreadFactory() {
+                private final AtomicInteger cnt = new AtomicInteger();
 
-        public Thread newThread(Runnable r)
+                public Thread newThread(Runnable r) {
-        {
+                    Thread t = new Thread(r, "ws-callback-" + cnt.getAndIncrement());
-            Thread t = new Thread(r, "ws-callback-" + cnt.getAndIncrement());
+                    t.setDaemon(true);
-            t.setDaemon(true);
+                    return t;
-            return t;
+                }
-        }
+            });
-    });
 
     private volatile WebSocketSession session;
     private final BlockingQueue<String> textMessageQueue;
@@ -51,18 +64,10 @@ public class WebSocketClientManager implements Closeable
     private volatile Consumer<byte[]> onBinary;
     private volatile Consumer<Throwable> onError;
 
-    private final String uri;
-    private final WebSocketHttpHeaders headers;
-    private final long connectTimeout;
-    private final TimeUnit connectUnit;
     private final int queueCapacity;
 
     // 私有构造，仅由 Builder 调用
     private WebSocketClientManager(Builder b) {
-        this.uri = b.uri;
-        this.headers = b.headers != null ? b.headers : new WebSocketHttpHeaders();
-        this.connectTimeout = b.connectTimeout;
-        this.connectUnit = b.connectUnit;
         this.maxSessionDuration = b.maxSessionDuration;
         this.maxSessionDurationUnit = b.maxSessionDurationUnit;
         this.queueCapacity = b.queueCapacity;
@@ -71,13 +76,14 @@ public class WebSocketClientManager implements Closeable
         this.errorFuture = new CompletableFuture<>();
     }
 
-    public static WebSocketClientManager build(Builder b) throws InterruptedException, ExecutionException, TimeoutException, IOException {
+    public static WebSocketClientManager build(Builder b)
+            throws InterruptedException, ExecutionException, TimeoutException, IOException {
         WebSocketClientManager ws = new WebSocketClientManager(b);
         StandardWebSocketClient client = new StandardWebSocketClient();
-        CompletableFuture<WebSocketSession> future = client.execute(ws.new InternalHandler(b.uri), b.headers, URI.create(b.uri));
+        CompletableFuture<WebSocketSession> future = client.execute(ws.new InternalHandler(b.uri), b.headers,
+                URI.create(b.uri));
         WebSocketSession sess = future.get(b.connectTimeout, b.connectUnit);
-        if (sess == null || !sess.isOpen())
+        if (sess == null || !sess.isOpen()) {
-        {
             throw new IOException("握手失败或会话未打开");
         }
         ws.session = sess;
@@ -100,13 +106,10 @@ public class WebSocketClientManager implements Closeable
         session.sendMessage(new TextMessage(json));
     }
 
-
-
     private <T> List<T> listenerCustom(
             BlockingQueue<T> queue,
             Predicate<T> predicate)
-            throws InterruptedException, TimeoutException, ExecutionException
+            throws InterruptedException, TimeoutException, ExecutionException {
-    {
         List<T> collected = new ArrayList<>();
         long deadline = System.currentTimeMillis() + maxSessionDurationUnit.toMillis(maxSessionDuration);
 
@@ -136,17 +139,16 @@ public class WebSocketClientManager implements Closeable
 
     /**
      * 同步接收多条消息，直到 predicate 为 true 或超时抛异常；
+     * 
      * @return 返回监听期间的所有消息列表
      */
     public List<String> listener(Predicate<String> predicate)
-            throws InterruptedException, TimeoutException, ExecutionException
+            throws InterruptedException, TimeoutException, ExecutionException {
-    {
         return listenerCustom(textMessageQueue, predicate);
     }
 
     public List<byte[]> listenerBinary(Predicate<byte[]> predicate)
-            throws InterruptedException, TimeoutException, ExecutionException
+            throws InterruptedException, TimeoutException, ExecutionException {
-    {
         return listenerCustom(binaryMessageQueue, predicate);
     }
 
@@ -183,8 +185,8 @@ public class WebSocketClientManager implements Closeable
             if (session != null && session.isOpen()) {
                 session.close(CloseStatus.NORMAL);
             }
+        } catch (IOException ignored) {
         }
-        catch (IOException ignored) {}
         textMessageQueue.clear();
         binaryMessageQueue.clear();
         errorFuture.completeExceptionally(new IOException("WebSocket 已关闭"));
@@ -207,7 +209,8 @@ public class WebSocketClientManager implements Closeable
             // 保存会话
             WebSocketClientManager.this.session = session;
             this.stopWatch.start();
-            log.info("ws连接成功, 目标URI: {}, 连接时间: {}", targetUri, DateUtils.getDateTimeNow(DateUtils.DATE_TIME_MILLIS_PATTERN));
+            log.info("ws连接成功, 目标URI: {}, 连接时间: {}", targetUri,
+                    DateUtils.getDateTimeNow(DateUtils.DATE_TIME_MILLIS_PATTERN));
         }
 
         /**
@@ -264,18 +267,19 @@ public class WebSocketClientManager implements Closeable
                 stopWatch.stop();
             }
             log.info("ws连接关闭, 目标URI: {}, 关闭时间: {}, 连接总时长: {}s",
-                    targetUri, DateUtils.getDateTimeNow(DateUtils.DATE_TIME_MILLIS_PATTERN), DateUtils.millsToSecond(stopWatch.getTotalTimeMillis()));
+                    targetUri, DateUtils.getDateTimeNow(DateUtils.DATE_TIME_MILLIS_PATTERN),
+                    DateUtils.millsToSecond(stopWatch.getTotalTimeMillis()));
         }
     }
 
     public static class Builder {
-        private String uri;                                         // 目标 WS URI
+        private String uri; // 目标 WS URI
-        private long connectTimeout = 3;                            // 请求连接等待时间
+        private long connectTimeout = 3; // 请求连接等待时间
-        private TimeUnit connectUnit = TimeUnit.SECONDS;            // 请求连接等待时间单位
+        private TimeUnit connectUnit = TimeUnit.SECONDS; // 请求连接等待时间单位
-        private long maxSessionDuration = 5;                        // 最大连线时间，默认5秒
+        private long maxSessionDuration = 5; // 最大连线时间，默认5秒
         private TimeUnit maxSessionDurationUnit = TimeUnit.SECONDS; // 最大连线时间单位
-        private int queueCapacity = 100;                            // 消息队列容量
+        private int queueCapacity = 100; // 消息队列容量
-        private WebSocketHttpHeaders headers;                       // 请求头
+        private WebSocketHttpHeaders headers; // 请求头
 
         /**
          * 目标 WS URI
@@ -307,7 +311,8 @@ public class WebSocketClientManager implements Closeable
             return this;
         }
 
-        public WebSocketClientManager build() throws InterruptedException, ExecutionException, TimeoutException, IOException {
+        public WebSocketClientManager build()
+                throws InterruptedException, ExecutionException, TimeoutException, IOException {
             return WebSocketClientManager.build(this);
         }
 

main/manager-api/src/main/resources/db/changelog/202505091409.sql

@@ -1,6 +0,0 @@
--- 更新intent_llmM供应器
-update `ai_model_provider` set fields =  '[{"key":"llm","label":"LLM模型","type":"string"},{"key":"functions","label":"函数列表","type":"dict","dict_name":"functions"}]' where  id = 'SYSTEM_Intent_intent_llm';
--- 更新ChatGLMLLM的意图识别配置
-update `ai_model_config` set config_json =  '{\"type\": \"intent_llm\", \"llm\": \"LLM_ChatGLMLLM\", \"functions\": \"get_weather;get_news_from_newsnow;play_music\"}' where  id = 'Intent_intent_llm';
--- 更新函数调用意图识别配置
-UPDATE `ai_model_config` SET config_json = REPLACE(config_json, ';get_news;', ';get_news_from_newsnow;') WHERE id = 'Intent_function_call';

main/manager-api/src/main/resources/db/changelog/202505292203.sql

@@ -0,0 +1,189 @@
+-- ===============================
+-- 一、在ai_model_provider中插入plugin 记录
+-- ===============================
+START TRANSACTION;
+
+
+-- intent_llm和function_call不设置函数列表
+update `ai_model_provider` set fields =  '[{"key":"llm","label":"LLM模型","type":"string"}]' where  id = 'SYSTEM_Intent_intent_llm';
+update `ai_model_provider` set fields =  '[]' where  id = 'SYSTEM_Intent_function_call';
+update `ai_model_config` set config_json =  '{\"type\": \"intent_llm\", \"llm\": \"LLM_ChatGLMLLM\"}' where  id = 'Intent_intent_llm';
+UPDATE `ai_model_config` SET config_json = '{\"type\": \"function_call\"}' WHERE id = 'Intent_function_call';
+
+
+delete from ai_model_provider where model_type = 'Plugin';
+-- 1. 天气查询
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_WEATHER',
+        'Plugin',
+        'get_weather',
+        '天气查询',
+        JSON_ARRAY(
+                JSON_OBJECT(
+                        'key', 'api_key',
+                        'type', 'string',
+                        'label', '天气插件 API 密钥',
+                        'default', (SELECT param_value FROM sys_params WHERE param_code = 'plugins.get_weather.api_key')
+                ),
+                JSON_OBJECT(
+                        'key', 'default_location',
+                        'type', 'string',
+                        'label', '默认查询城市',
+                        'default',
+                        (SELECT param_value FROM sys_params WHERE param_code = 'plugins.get_weather.default_location')
+                ),
+                JSON_OBJECT(
+                        'key', 'api_host',
+                        'type', 'string',
+                        'label', '开发者 API Host',
+                        'default',
+                        (SELECT param_value FROM sys_params WHERE param_code = 'plugins.get_weather.api_host')
+                )
+        ),
+        10, 0, NOW(), 0, NOW());
+
+-- 6. 本地播放音乐
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_MUSIC',
+        'Plugin',
+        'play_music',
+        '服务器音乐播放',
+        JSON_ARRAY(),
+        20, 0, NOW(), 0, NOW());
+
+-- 2. 新闻订阅
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_NEWS_CHINANEWS',
+        'Plugin',
+        'get_news_from_chinanews',
+        '中新网新闻',
+        JSON_ARRAY(
+                JSON_OBJECT(
+                        'key', 'default_rss_url',
+                        'type', 'string',
+                        'label', '默认 RSS 源',
+                        'default',
+                        (SELECT param_value FROM sys_params WHERE param_code = 'plugins.get_news.default_rss_url')
+                ),
+                JSON_OBJECT(
+                        'key', 'society_rss_url',
+                        'type', 'string',
+                        'label', '社会新闻 RSS 地址',
+                        'default',
+                        'https://www.chinanews.com.cn/rss/society.xml'
+                ),
+                JSON_OBJECT(
+                        'key', 'world_rss_url',
+                        'type', 'string',
+                        'label', '国际新闻 RSS 地址',
+                        'default',
+                        'https://www.chinanews.com.cn/rss/world.xml'
+                ),
+                JSON_OBJECT(
+                        'key', 'finance_rss_url',
+                        'type', 'string',
+                        'label', '财经新闻 RSS 地址',
+                        'default',
+                        'https://www.chinanews.com.cn/rss/finance.xml'
+                )
+        ),
+        30, 0, NOW(), 0, NOW());
+
+-- 3. 新闻订阅
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_NEWS_NEWSNOW',
+        'Plugin',
+        'get_news_from_newsnow',
+        'newsnow新闻聚合',
+        JSON_ARRAY(
+                JSON_OBJECT(
+                        'key', 'url',
+                        'type', 'string',
+                        'label', '接口地址',
+                        'default',
+                        'https://newsnow.busiyi.world/api/s?id='
+                )
+        ),
+        40, 0, NOW(), 0, NOW());
+
+
+-- 4. HomeAssistant 状态查询
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_HA_GET_STATE',
+        'Plugin',
+        'hass_get_state',
+        'HomeAssistant设备状态查询',
+        JSON_ARRAY(
+                JSON_OBJECT(
+                        'key', 'base_url',
+                        'type', 'string',
+                        'label', 'HA 服务器地址',
+                        'default',
+                        (SELECT param_value FROM sys_params WHERE param_code = 'plugins.home_assistant.base_url')
+                ),
+                JSON_OBJECT(
+                        'key', 'api_key',
+                        'type', 'string',
+                        'label', 'HA API 访问令牌',
+                        'default',
+                        (SELECT param_value FROM sys_params WHERE param_code = 'plugins.home_assistant.api_key')
+                ),
+                JSON_OBJECT(
+                        'key', 'devices',
+                        'type', 'array',
+                        'label', '设备列表（名称,实体ID;…）',
+                        'default',
+                        (SELECT param_value FROM sys_params WHERE param_code = 'plugins.home_assistant.devices')
+                )
+        ),
+        50, 0, NOW(), 0, NOW());
+
+-- 5. HomeAssistant 状态写入
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_HA_SET_STATE',
+        'Plugin',
+        'hass_set_state',
+        'HomeAssistant设备状态修改',
+        JSON_ARRAY(),
+        60, 0, NOW(), 0, NOW());
+
+-- 5. HomeAssistant 音乐播放
+INSERT INTO ai_model_provider (id, model_type, provider_code, name, fields,
+                               sort, creator, create_date, updater, update_date)
+VALUES ('SYSTEM_PLUGIN_HA_PLAY_MUSIC',
+        'Plugin',
+        'hass_play_music',
+        'HomeAssistant音乐播放',
+        JSON_ARRAY(),
+        70, 0, NOW(), 0, NOW());
+
+
+-- ===============================
+-- 二、删除sys_params中旧的plugins.*参数
+-- ===============================
+DELETE
+FROM sys_params
+WHERE param_code LIKE 'plugins.%';
+
+
+-- ===============================
+-- 三、添加智能体插件id字段
+-- ===============================
+CREATE TABLE IF NOT EXISTS ai_agent_plugin_mapping
+(
+    id         BIGINT AUTO_INCREMENT PRIMARY KEY COMMENT '主键',
+    agent_id   VARCHAR(32) NOT NULL COMMENT '智能体ID',
+    plugin_id  VARCHAR(32) NOT NULL COMMENT '插件ID',
+    param_info JSON        NOT NULL COMMENT '参数信息',
+    UNIQUE KEY uk_agent_provider (agent_id, plugin_id)
+) COMMENT 'Agent与插件的唯一映射表';
+
+
+COMMIT;
+

main/manager-api/src/main/resources/db/changelog/202506010920.sql

@@ -0,0 +1,29 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_VLLM_openai';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_VLLM_openai', 'VLLM', 'openai', 'OpenAI接口', '[{"key":"base_url","label":"基础URL","type":"string"},{"key":"model_name","label":"模型名称","type":"string"},{"key":"api_key","label":"API密钥","type":"string"}]', 9, 1, NOW(), 1, NOW());
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_ChatGLMVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_ChatGLMVLLM', 'VLLM', 'ChatGLMVLLM', '智谱视觉AI', 1, 1, '{\"type\": \"openai\", \"model_name\": \"glm-4v-flash\", \"base_url\": \"https://open.bigmodel.cn/api/paas/v4/\", \"api_key\": \"你的api_key\"}', NULL, NULL, 1, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bigmodel.cn/usercenter/proj-mgmt/apikeys',
+`remark` = '智谱视觉AI配置说明：
+1. 访问 https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_ChatGLMVLLM';
+
+
+-- 添加参数
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (113, 'server.http_port', '8003', 'number', 1, 'http服务的端口，用于启动视觉分析接口');
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (114, 'server.vision_explain', 'null', 'string', 1, '视觉分析接口地址，用于下发到设备，多个用;分隔');
+
+-- 智能体表增加VLLM模型配置
+ALTER TABLE `ai_agent` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;
+
+-- 智能体模版表增加VLLM模型配置
+ALTER TABLE `ai_agent_template` 
+ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;

main/manager-api/src/main/resources/db/changelog/202506031639.sql

@@ -0,0 +1,45 @@
+-- VLLM模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_DoubaoStreamASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_DoubaoStreamASR', 'ASR', 'doubao_stream', '火山引擎语音识别(流式)', '[{"key":"appid","label":"应用ID","type":"string"},{"key":"access_token","label":"访问令牌","type":"string"},{"key":"cluster","label":"集群","type":"string"},{"key":"boosting_table_name","label":"热词文件名称","type":"string"},{"key":"correct_table_name","label":"替换词文件名称","type":"string"},{"key":"output_dir","label":"输出目录","type":"string"}]', 3, 1, NOW(), 1, NOW());
+
+
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'ASR_DoubaoStreamASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_DoubaoStreamASR', 'ASR', 'DoubaoStreamASR', '豆包语音识别(流式)', 0, 1, '{\"type\": \"doubao_stream\", \"appid\": \"\", \"access_token\": \"\", \"cluster\": \"volcengine_input_common\", \"output_dir\": \"tmp/\"}', NULL, NULL, 3, NULL, NULL, NULL, NULL);
+
+
+-- 更新豆包ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.volcengine.com/speech/app',
+`remark` = '豆包ASR配置说明：
+1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
+2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
+3. 需要在火山引擎控制台创建应用并获取appid和access_token
+4. 支持中文语音识别
+5. 需要网络连接
+6. 输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://console.volcengine.com/speech/app
+2. 创建新应用
+3. 获取appid和access_token
+4. 填入配置文件中
+如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
+' WHERE `id` = 'ASR_DoubaoASR';
+
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.volcengine.com/speech/app',
+`remark` = '豆包ASR配置说明：
+1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
+2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
+3. 需要在火山引擎控制台创建应用并获取appid和access_token
+4. 支持中文语音识别
+5. 需要网络连接
+6. 输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://console.volcengine.com/speech/app
+2. 创建新应用
+3. 获取appid和access_token
+4. 填入配置文件中
+如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
+' WHERE `id` = 'ASR_DoubaoStreamASR';

main/manager-api/src/main/resources/db/changelog/202506032232.sql

@@ -0,0 +1,14 @@
+-- VLLM模型配置
+delete from `ai_model_config` where id = 'VLLM_QwenVLVLLM';
+INSERT INTO `ai_model_config` VALUES ('VLLM_QwenVLVLLM', 'VLLM', 'QwenVLVLLM', '千问视觉模型', 0, 1, '{\"type\": \"openai\", \"model_name\": \"qwen2.5-vl-3b-instruct\", \"base_url\": \"https://dashscope.aliyuncs.com/compatible-mode/v1\", \"api_key\": \"你的api_key\"}', NULL, NULL, 2, NULL, NULL, NULL, NULL);
+
+-- 更新文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://bailian.console.aliyun.com/?tab=api#/api/?type=model&url=https%3A%2F%2Fhelp.aliyun.com%2Fdocument_detail%2F2845564.html&renderType=iframe',
+`remark` = '千问视觉模型配置说明：
+1. 访问 https://bailian.console.aliyun.com/?tab=model#/api-key
+2. 注册并获取API密钥
+3. 填入配置文件中' WHERE `id` = 'VLLM_QwenVLVLLM';
+
+-- 删除参数，这两个参数已挪至python配置文件
+delete from `sys_params` where id  in (113,114);

main/manager-api/src/main/resources/db/changelog/202506051538.sql

@@ -0,0 +1,20 @@
+-- 增加LinkeraiTTS供应器和模型配置
+delete from `ai_model_provider` where id = 'SYSTEM_TTS_LinkeraiTTS';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_TTS_LinkeraiTTS', 'TTS', 'linkerai', 'Linkerai语音合成', '[{"key":"api_url","label":"API地址","type":"string"},{"key":"audio_format","label":"音频格式","type":"string"},{"key":"access_token","label":"访问令牌","type":"string"},{"key":"voice","label":"默认音色","type":"string"}]', 14, 1, NOW(), 1, NOW());
+
+delete from `ai_model_config` where id = 'TTS_LinkeraiTTS';
+INSERT INTO `ai_model_config` VALUES ('TTS_LinkeraiTTS', 'TTS', 'LinkeraiTTS', 'Linkerai语音合成', 0, 1, '{\"type\": \"linkerai\", \"api_url\": \"https://tts.linkerai.cn/tts\", \"audio_format\": \"pcm\", \"access_token\": \"U4YdYXVfpwWnk2t5Gp822zWPCuORyeJL\", \"voice\": \"OUeAo1mhq6IBExi\"}', NULL, NULL, 17, NULL, NULL, NULL, NULL);
+
+-- LinkeraiTTS模型配置说明文档
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://tts.linkerai.cn/docs',
+`remark` = 'Linkerai语音合成服务配置说明：
+1. 访问 https://linkerai.cn 注册并获取访问令牌
+2. 默认的access_token供测试使用，请勿用于商业用途
+3. 支持声音克隆功能，可自行上传音频，填入voice参数
+4. 如果voice参数为空，将使用默认声音' WHERE `id` = 'TTS_LinkeraiTTS';
+
+
+delete from `ai_tts_voice` where tts_model_id = 'TTS_LinkeraiTTS';
+INSERT INTO `ai_tts_voice` VALUES ('TTS_LinkeraiTTS_0001', 'TTS_LinkeraiTTS', '芷若', 'OUeAo1mhq6IBExi', '中文', NULL, NULL, 1, NULL, NULL, NULL, NULL);

main/manager-api/src/main/resources/db/changelog/202506080955.sql

@@ -0,0 +1,3 @@
+-- 智控台开启唤醒词加速
+update `sys_params` set param_value = '你好小智;你好小志;小爱同学;你好小鑫;你好小新;小美同学;小龙小龙;喵喵同学;小滨小滨;小冰小冰;嘿你好呀' where param_code = 'wakeup_words';
+update `sys_params` set param_value = 'true' where param_code = 'enable_wakeup_words_response_cache';

main/manager-api/src/main/resources/db/changelog/202506161101.sql

@@ -0,0 +1 @@
+ALTER TABLE ai_agent_plugin_mapping CONVERT TO CHARACTER SET utf8mb4;

main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml

@@ -107,13 +107,6 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202505081146.sql
-  - changeSet:
-      id: 202505091409
-      author: hrz
-      changes:
-        - sqlFile:
-            encoding: utf8
-            path: classpath:db/changelog/202505091409.sql
   - changeSet:
       id: 202505091555
       author: whosmyqueen
@@ -170,3 +163,52 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202505271414.sql
+  - changeSet:
+      id: 202505292203
+      author: CAIXYPROMISE
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202505292203.sql
+  - changeSet:
+      id: 202506010920
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506010920.sql
+  - changeSet:
+      id: 202506031639
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506031639.sql
+  - changeSet:
+      id: 202506032232
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506032232.sql
+  - changeSet:
+      id: 202506051538
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506051538.sql
+  - changeSet:
+      id: 202506080955
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506080955.sql
+  - changeSet:
+      id: 202506161101
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506161101.sql

main/manager-api/src/main/resources/mapper/agent/AgentDao.xml

@@ -5,4 +5,72 @@
     <select id="getDeviceCountByAgentId" resultType="java.lang.Integer">
         SELECT COUNT(*) FROM ai_device WHERE agent_id = #{agentId}
     </select>
+
+    <resultMap id="AgentInfoMap" type="xiaozhi.modules.agent.vo.AgentInfoVO">
+        <id column="id"           property="id"/>
+        <result column="userId"    property="userId"/>
+        <result column="agentCode" property="agentCode"/>
+        <result column="agentName" property="agentName"/>
+        <result column="asrModelId" property="asrModelId"/>
+        <result column="vadModelId" property="vadModelId"/>
+        <result column="llmModelId" property="llmModelId"/>
+        <result column="ttsModelId" property="ttsModelId"/>
+        <result column="ttsVoiceId" property="ttsVoiceId"/>
+        <result column="memModelId" property="memModelId"/>
+        <result column="intentModelId" property="intentModelId"/>
+
+        <result column="functions" property="functions"
+                typeHandler="com.baomidou.mybatisplus.extension.handlers.JacksonTypeHandler"/>
+
+        <result column="chatHistoryConf" property="chatHistoryConf"/>
+        <result column="systemPrompt" property="systemPrompt"/>
+        <result column="summaryMemory" property="summaryMemory"/>
+        <result column="langCode" property="langCode"/>
+        <result column="language" property="language"/>
+        <result column="sort" property="sort"/>
+        <result column="creator" property="creator"/>
+        <result column="createdAt" property="createdAt"/>
+        <result column="updater" property="updater"/>
+        <result column="updatedAt" property="updatedAt"/>
+    </resultMap>
+
+    <select id="selectAgentInfoById" resultMap="AgentInfoMap">
+        SELECT a.id,
+               a.user_id           AS userId,
+               a.agent_code        AS agentCode,
+               a.agent_name        AS agentName,
+               a.asr_model_id      AS asrModelId,
+               a.vad_model_id      AS vadModelId,
+               a.llm_model_id      AS llmModelId,
+               a.vllm_model_id     AS vllmModelId,
+               a.tts_model_id      AS ttsModelId,
+               a.tts_voice_id      AS ttsVoiceId,
+               a.mem_model_id      AS memModelId,
+               a.intent_model_id   AS intentModelId,
+               COALESCE(
+                       (SELECT JSON_ARRAYAGG(
+                                       JSON_OBJECT(
+                                               'id', m.id,
+                                               'agentId', m.agent_id,
+                                               'pluginId', m.plugin_id,
+                                               'paramInfo', m.param_info
+                                       )
+                               )
+                        FROM ai_agent_plugin_mapping m
+                        WHERE m.agent_id = a.id),
+                       JSON_ARRAY()
+               )                   AS functions,
+               a.chat_history_conf AS chatHistoryConf,
+               a.system_prompt     AS systemPrompt,
+               a.summary_memory    AS summaryMemory,
+               a.lang_code         AS langCode,
+               a.language          AS language,
+               a.sort,
+               a.creator,
+               a.created_at        AS createdAt,
+               a.updater,
+               a.updated_at        AS updatedAt
+        FROM ai_agent a
+        WHERE a.id = #{agentId}
+    </select>
 </mapper> 

main/manager-api/src/main/resources/mapper/agent/AgentPluginMappingMapper.xml

@@ -0,0 +1,48 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE mapper
+        PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
+        "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
+<mapper namespace="xiaozhi.modules.agent.dao.AgentPluginMappingMapper">
+
+    <resultMap id="BaseResultMap" type="xiaozhi.modules.agent.entity.AgentPluginMapping">
+        <id property="id" column="id" jdbcType="BIGINT"/>
+        <result property="agentId" column="agent_id" jdbcType="VARCHAR"/>
+        <result property="pluginId" column="plugin_id" jdbcType="VARCHAR"/>
+        <result property="paramInfo" column="param_info" jdbcType="VARCHAR"/>
+    </resultMap>
+
+    <!-- 用于映射根据agentId查询完整插件信息 -->
+    <resultMap id="AgentPluginWithCodeMap" type="xiaozhi.modules.agent.entity.AgentPluginMapping">
+        <id column="id" property="id" jdbcType="BIGINT"/>
+        <result column="agentId" property="agentId" jdbcType="VARCHAR"/>
+        <result column="pluginId" property="pluginId" jdbcType="VARCHAR"/>
+        <result column="paramInfo" property="paramInfo" jdbcType="VARCHAR"/>
+        <result column="providerCode" property="providerCode" jdbcType="VARCHAR"/>
+    </resultMap>
+
+    <sql id="Base_Column_List">
+        id,agent_id,plugin_id
+    </sql>
+
+
+    <select id="selectPluginsByAgentId" resultMap="AgentPluginWithCodeMap">
+        SELECT m.id         AS id,
+               m.agent_id   AS agentId,
+               m.plugin_id  AS pluginId,
+               m.param_info AS paramInfo,
+               (
+                   SELECT
+                       p.provider_code
+                   FROM
+                       ai_model_provider p
+                   WHERE
+                       p.id = m.plugin_id
+                   LIMIT
+                       1
+                )    AS providerCode
+        FROM ai_agent_plugin_mapping m
+        WHERE m.agent_id = #{agentId}
+    </select>
+
+
+</mapper>

main/manager-web/src/apis/module/device.js

@@ -51,10 +51,11 @@ export default {
                 });
             }).send();
     },
-    enableOtaUpgrade(id, status, callback) {
+    updateDeviceInfo(id, payload, callback) {
         RequestService.sendRequest()
-            .url(`${getServiceUrl()}/device/enableOta/${id}/${status}`)
+            .url(`${getServiceUrl()}/device/update/${id}`)
             .method('PUT')
+            .data(payload)
             .success((res) => {
                 RequestService.clearRequestTime()
                 callback(res)
@@ -63,7 +64,7 @@ export default {
                 console.error('更新OTA状态失败:', err)
                 this.$message.error(err.msg || '更新OTA状态失败')
                 RequestService.reAjaxFun(() => {
-                    this.enableOtaUpgrade(id, status, callback)
+                    this.updateDeviceInfo(id, payload, callback)
                 })
             }).send()
     },

main/manager-web/src/apis/module/model.js

@@ -305,4 +305,20 @@ export default {
         })
       }).send()
   },
+  // 获取插件列表
+  getPluginFunctionList(params, callback) {
+    RequestService.sendRequest()
+      .url(`${getServiceUrl()}/models/provider/plugin/names`)
+      .method('GET')
+      .success((res) => {
+        RequestService.clearRequestTime()
+        callback(res)
+      })
+      .networkFail((err) => {
+        this.$message.error(err.msg || '获取插件列表失败')
+        RequestService.reAjaxFun(() => {
+          this.getPluginFunctionList(params, callback)
+        })
+      }).send()
+  }
 }

main/manager-web/src/components/DeviceItem.vue

@@ -14,10 +14,10 @@
       </div>
     </div>
     <div class="device-name">
-      设备型号：{{ device.ttsModelName }}
+      语言模型：{{ device.llmModelName }}
     </div>
     <div class="device-name">
-      音色模型：{{ device.ttsVoiceName }}
+      音色模型：{{ device.ttsModelName }} ({{ device.ttsVoiceName }})
     </div>
     <div style="display: flex;gap: 10px;align-items: center;">
       <div class="settings-btn" @click="handleConfigure">

main/manager-web/src/components/FirmwareDialog.vue

@@ -1,5 +1,5 @@
 <template>
-  <el-dialog :title="title" :visible.sync="dialogVisible" @close="handleClose" @open="handleOpen">
+  <el-dialog :title="title" :visible.sync="dialogVisible" :close-on-click-modal="false" @close="handleClose" @open="handleOpen">
     <el-form ref="form" :model="form" :rules="rules" label-width="100px">
       <el-form-item label="固件名称" prop="firmwareName">
         <el-input v-model="form.firmwareName" placeholder="请输入固件名称(板子+版本号)"></el-input>

main/manager-web/src/components/FunctionDialog.vue

@@ -1,5 +1,5 @@
 <template>
-  <el-drawer :visible.sync="dialogVisible" direction="rtl" size="50%" :wrapperClosable="false" :withHeader="false">
+  <el-drawer :visible.sync="dialogVisible" direction="rtl" size="80%" :wrapperClosable="false" :withHeader="false">
     <!-- 自定义标题区域 -->
     <div class="custom-header">
       <div class="header-left">
@@ -16,15 +16,18 @@
           <el-button type="text" @click="selectAll" class="select-all-btn">全选</el-button>
         </div>
         <div class="function-list">
-          <div v-for="func in unselected" :key="func.name" class="function-item">
+          <div v-if="unselected.length">
-            <el-checkbox :label="func.name" v-model="selectedNames" @change="(val) => handleCheckboxChange(func, val)" @click.native.stop></el-checkbox>
+            <div v-for="func in unselected" :key="func.name" class="function-item">
-            <div class="func-tag" @click="handleFunctionClick(func)">
+              <el-checkbox :label="func.name" v-model="selectedNames" @change="(val) => handleCheckboxChange(func, val)"
-              <div class="color-dot" :style="{backgroundColor: getFunctionColor(func.name)}"></div>
+                @click.native.stop></el-checkbox>
-              <span>{{ func.name }}</span>
+              <div class="func-tag" @click="handleFunctionClick(func)">
+                <div class="color-dot" :style="{ backgroundColor: getFunctionColor(func.name) }"></div>
+                <span>{{ func.name }}</span>
+              </div>
             </div>
-            <el-tooltip class="item" effect="dark" :content="func.description || '暂无功能描述'" placement="top">
+          </div>
-              <img src="@/assets/home/info.png" alt="" class="info-icon">
+          <div v-else style="display: flex; justify-content: center; align-items: center;">
-            </el-tooltip>
+            <el-empty description="没有更多的插件了" />
           </div>
         </div>
       </div>
@@ -36,26 +39,62 @@
           <el-button type="text" @click="deselectAll" class="select-all-btn">全选</el-button>
         </div>
         <div class="function-list">
-          <div v-for="func in selectedList" :key="func.name" class="function-item">
+          <div v-if="selectedList.length > 0">
-            <el-checkbox :label="func.name" v-model="selectedNames" @change="(val) => handleCheckboxChange(func, val)" @click.native.stop></el-checkbox>
+            <div v-for="func in selectedList" :key="func.name" class="function-item">
-            <div class="func-tag" @click="handleFunctionClick(func)">
+              <el-checkbox :label="func.name" v-model="selectedNames" @change="(val) => handleCheckboxChange(func, val)"
-              <div class="color-dot" :style="{backgroundColor: getFunctionColor(func.name)}"></div>
+                @click.native.stop></el-checkbox>
-              <span>{{ func.name }}</span>
+              <div class="func-tag" @click="handleFunctionClick(func)">
+                <div class="color-dot" :style="{ backgroundColor: getFunctionColor(func.name) }"></div>
+                <span>{{ func.name }}</span>
+              </div>
             </div>
           </div>
+          <div v-else style="display: flex; justify-content: center; align-items: center;">
+            <el-empty description="请选择插件功能" />
+          </div>
         </div>
       </div>
 
       <!-- 右侧：参数配置 -->
       <div class="params-column">
         <h4 v-if="currentFunction" class="column-title">参数配置 - {{ currentFunction.name }}</h4>
-          <div v-if="currentFunction" class="params-container">
+        <div v-if="currentFunction" class="params-container">
-            <el-form :model="currentFunction" size="mini" class="param-form" v-loading="loading" element-loading-text="拼命加载中" element-loading-spinner="el-icon-loading" element-loading-background="rgba(255, 255, 255, 0.7)">
+          <el-form :model="currentFunction" class="param-form">
-              <el-form-item v-for="(value, key) in currentFunction.params" :key="key" :label="key" class="param-item">
+            <!-- 遍历 fieldsMeta，而不是 params 的 keys -->
-                <el-input v-model="currentFunction.params[key]" size="mini" class="param-input" @change="(val) => handleParamChange(currentFunction, key, val)"/>
+            <div v-if="currentFunction.fieldsMeta.length == 0">
-              </el-form-item>
+              <el-empty :description="currentFunction.name + ' 无需配置参数'" />
-            </el-form>
+            </div>
-          </div>
+            <el-form-item v-for="field in currentFunction.fieldsMeta" :key="field.key" :label="field.label"
+              class="param-item" :class="{ 'textarea-field': field.type === 'array' || field.type === 'json' }">
+              <template #label>
+                <span style="font-size: 16px; margin-right: 6px;">{{ field.label }}</span>
+                <el-tooltip effect="dark" :content="fieldRemark(field)" placement="top">
+                  <img src="@/assets/home/info.png" alt="" class="info-icon">
+                </el-tooltip>
+              </template>
+              <!-- ARRAY -->
+              <el-input v-if="field.type === 'array'" type="textarea" v-model="currentFunction.params[field.key]"
+                @change="val => handleParamChange(currentFunction, field.key, val)" />
+
+              <!-- JSON -->
+              <el-input v-else-if="field.type === 'json'" type="textarea" :rows="6" placeholder="请输入合法的 JSON"
+                v-model="textCache[field.key]" @blur="flushJson(field)" />
+
+              <!-- number -->
+              <el-input-number v-else-if="field.type === 'number'" :value="currentFunction.params[field.key]"
+                @change="val => handleParamChange(currentFunction, field.key, val)" />
+
+              <!-- boolean -->
+              <el-switch v-else-if="field.type === 'boolean' || field.type === 'bool'"
+                :value="currentFunction.params[field.key]"
+                @change="val => handleParamChange(currentFunction, field.key, val)" />
+
+              <!-- string or fallback -->
+              <el-input v-else v-model="currentFunction.params[field.key]"
+                @change="val => handleParamChange(currentFunction, field.key, val)" />
+            </el-form-item>
+          </el-form>
+        </div>
         <div v-else class="empty-tip">请选择已配置的功能进行参数设置</div>
       </div>
     </div>
@@ -74,24 +113,19 @@ export default {
     functions: {
       type: Array,
       default: () => []
+    },
+    allFunctions: {
+      type: Array,
+      default: () => []
     }
   },
   data() {
     return {
+      textCache: {},
       dialogVisible: this.value,
       selectedNames: [],
       currentFunction: null,
       modifiedFunctions: {},
-      allFunctions: [
-        {name: '天气', params: {city: '北京'}, description: '查看指定城市的天气情况'},
-        {name: '新闻', params: {type: '科技'}, description: '获取最新科技类新闻资讯'},
-        {name: '工具', params: {category: '常用'}, description: '提供常用工具集合'},
-        {name: '退出', params: {}, description: '退出当前系统'},
-        {name: '音乐', params: {genre: '流行'}, description: '播放流行音乐'},
-        {name: '翻译', params: {from: '中文', to: '英文'}, description: '提供中英文互译功能'},
-        {name: '计算', params: {precision: '2'}, description: '提供精确计算功能'},
-        {name: '日历', params: {view: '月'}, description: '查看月历视图'}
-      ],
       functionColorMap: [
         '#FF6B6B', '#4ECDC4', '#45B7D1',
         '#96CEB4', '#FFEEAD', '#D4A5A5', '#A2836E'
@@ -111,10 +145,37 @@ export default {
     }
   },
   watch: {
-    value(newVal) {
+    currentFunction(newFn) {
-      this.dialogVisible = newVal;
+      if (!newFn) return;
-      if (newVal) {
+      // 对每个字段，如果是 array 或 json，就在 textCache 里生成初始字符串
+      newFn.fieldsMeta.forEach(f => {
+        const v = newFn.params[f.key];
+        if (f.type === 'array') {
+          this.$set(this.textCache, f.key, Array.isArray(v) ? v.join('\n') : '');
+        }
+        else if (f.type === 'json') {
+          try {
+            this.$set(this.textCache, f.key, JSON.stringify(v ?? {}, null, 2));
+          } catch {
+            this.$set(this.textCache, f.key, '');
+          }
+        }
+      });
+    },
+    value(v) {
+      this.dialogVisible = v;
+      if (v) {
+        // 对话框打开时，初始化选中态
         this.selectedNames = this.functions.map(f => f.name);
+        // 把后端传来的 this.functions（带 params）merge 到 allFunctions 上
+        this.functions.forEach(saved => {
+          const idx = this.allFunctions.findIndex(f => f.name === saved.name);
+          if (idx >= 0) {
+            // 保留用户之前在 saved.params 上的改动
+            this.allFunctions[idx].params = { ...saved.params };
+          }
+        });
+        // 右侧默认指向第一个
         this.currentFunction = this.selectedList[0] || null;
       }
     },
@@ -123,14 +184,32 @@ export default {
     }
   },
   methods: {
+    flushArray(key) {
+      const text = this.textCache[key] || '';
+      const arr = text
+        .split('\n')
+        .map(s => s.trim())
+        .filter(Boolean);
+      this.handleParamChange(this.currentFunction, key, arr);
+    },
+
+    flushJson(field) {
+      const key = field.key;
+      if (!key) {
+        return;
+      }
+      const text = this.textCache[key] || '';
+      try {
+        const obj = JSON.parse(text);
+        this.handleParamChange(this.currentFunction, key, obj);
+      } catch {
+        this.$message.error(`${this.currentFunction.name}的${key}字段格式错误：JSON格式有误`);
+      }
+    },
     handleFunctionClick(func) {
       if (this.selectedNames.includes(func.name)) {
-        this.loading = true;
+        const tempFunc = this.tempFunctions[func.name];
-        setTimeout(() => {
+        this.currentFunction = tempFunc ? tempFunc : func;
-          const tempFunc = this.tempFunctions[func.name];
-          this.currentFunction = tempFunc ? tempFunc : JSON.parse(JSON.stringify(func));
-          this.loading = false;
-        }, 300);
       }
     },
     handleParamChange(func, key, value) {
@@ -185,23 +264,31 @@ export default {
 
       const selected = this.selectedList.map(f => {
         const modified = this.modifiedFunctions[f.name];
-        return modified || f;
+        return {
-      }).map(f => ({
+          id: f.id,
-        ...f,
+          name: f.name,
-        params: JSON.parse(JSON.stringify(f.params))
+          params: modified
-      }));
+            ? { ...modified.params }
+            : { ...f.params }
+        }
+      });
 
       this.$emit('update-functions', selected);
       this.dialogVisible = false;
-      this.$message.success('配置保存成功');
       // 通知父组件对话框已关闭且已保存
       this.$emit('dialog-closed', true);
     },
-
     getFunctionColor(name) {
       const hash = [...name].reduce((acc, char) => acc + char.charCodeAt(0), 0);
-      return this.functionColorMap[hash % 7];
+      return this.functionColorMap[hash % this.functionColorMap.length];
-    }
+    },
+    fieldRemark(field) {
+      let description = (field && field.label) ? field.label : '';
+      if (field.default) {
+        description += `（默认值：${field.default}）`;
+      }
+      return description;
+    },
   }
 }
 </script>
@@ -209,7 +296,7 @@ export default {
 <style lang="scss" scoped>
 .function-manager {
   display: grid;
-  grid-template-columns: minmax(120px, 0.5fr) minmax(120px, 0.5fr) minmax(200px, 2fr);
+  grid-template-columns: max-content max-content 1fr;
   gap: 12px;
   height: calc(70vh - 60px);
 }
@@ -248,6 +335,7 @@ export default {
   overflow-y: auto;
   border-right: 1px solid #EBEEF5;
   scrollbar-width: none;
+  overflow-x: hidden;
 }
 
 .function-column::-webkit-scrollbar {
@@ -257,7 +345,7 @@ export default {
 .function-list {
   display: flex;
   flex-direction: column;
-  gap: 4px;
+  gap: 8px;
 }
 
 .function-item {
@@ -317,9 +405,31 @@ export default {
 }
 
 .param-form {
+  .param-item {
+    font-size: 16px;
+
+    &.textarea-field {
+      ::v-deep .el-form-item__content {
+        margin-left: 0 !important;
+        display: block;
+        width: 100%;
+      }
+
+      ::v-deep .el-form-item__label {
+        display: block;
+        width: 100% !important;
+        margin-bottom: 8px;
+      }
+    }
+  }
+
+  .param-input {
+    width: 100%;
+  }
+
   ::v-deep .el-form-item {
     display: flex;
-    align-items: center;
+    flex-direction: column;
     margin-bottom: 12px;
 
     .el-form-item__label {
@@ -356,9 +466,6 @@ export default {
   text-align: center;
 }
 
-.param-input {
-  width: 100%;
-}
 
 .drawer-footer {
   position: absolute;

main/manager-web/src/components/ModelEditDialog.vue

@@ -1,5 +1,5 @@
 <template>
-  <el-dialog :visible.sync="dialogVisible" width="57%" center custom-class="custom-dialog" :show-close="false"
+  <el-dialog :visible.sync="dialogVisible" :close-on-click-modal="false" width="57%" center custom-class="custom-dialog" :show-close="false"
     class="center-dialog" >
     <div style="margin: 0 18px; text-align: left; padding: 10px; border-radius: 10px;">
       <div style="font-size: 30px; color: #3d4566; margin-top: -10px; margin-bottom: 10px; text-align: center;">

main/manager-web/src/components/ProviderDialog.vue

@@ -1,5 +1,5 @@
 <template>
-  <el-dialog :visible="visible" @update:visible="handleVisibleChange" width="57%" center custom-class="custom-dialog"
+  <el-dialog :visible="visible" :close-on-click-modal="false" @update:visible="handleVisibleChange" width="57%" center custom-class="custom-dialog"
     :show-close="false" class="center-dialog">
 
     <div style="margin: 0 18px; text-align: left; padding: 10px; border-radius: 10px;">
@@ -18,7 +18,7 @@
             </el-select>
           </el-form-item>
 
-          <el-form-item label="供应器编码" prop="providerCode" style="flex: 1;">
+          <el-form-item label="编码" prop="providerCode" style="flex: 1;">
             <el-input v-model="form.providerCode" placeholder="请输入供应器编码" class="custom-input-bg"></el-input>
           </el-form-item>
         </div>
@@ -87,6 +87,7 @@
                     <el-option label="数字" value="number"></el-option>
                     <el-option label="布尔值" value="boolean"></el-option>
                     <el-option label="字典" value="dict"></el-option>
+                    <el-option label="分号分割的列表" value="array"></el-option>
                   </el-select>
                 </template>
                 <template v-else>
@@ -97,10 +98,10 @@
             <el-table-column label="默认值">
               <template slot-scope="scope">
                 <template v-if="scope.row.editing">
-                  <el-input v-model="scope.row.default_value" placeholder="请输入默认值"></el-input>
+                  <el-input v-model="scope.row.default" placeholder="请输入默认值"></el-input>
                 </template>
                 <template v-else>
-                  {{ scope.row.default_value }}
+                  {{ scope.row.default }}
                 </template>
               </template>
             </el-table-column>
@@ -161,7 +162,8 @@ export default {
         'string': '字符串',
         'number': '数字',
         'boolean': '布尔值',
-        'dict': '字典'
+        'dict': '字典',
+        'array': '分号分割的列表'
       };
       return typeMap[type];
     },
@@ -220,7 +222,7 @@ export default {
         key: '',
         label: '',
         type: 'string',
-        default_value: '',
+        default: '',
         selected: false,
         editing: true
       });

main/manager-web/src/views/DeviceManagement.vue

@@ -1,12 +1,12 @@
 <template>
   <div class="welcome">
-    <HeaderBar />
+    <HeaderBar/>
 
     <div class="operation-bar">
       <h2 class="page-title">设备管理</h2>
       <div class="right-operations">
         <el-input placeholder="请输入设备型号或Mac地址查询" v-model="searchKeyword" class="search-input"
-          @keyup.enter.native="handleSearch" clearable />
+                  @keyup.enter.native="handleSearch" clearable/>
         <el-button class="btn-search" @click="handleSearch">搜索</el-button>
       </div>
     </div>
@@ -16,8 +16,9 @@
         <div class="content-area">
           <el-card class="device-card" shadow="never">
             <el-table ref="deviceTable" :data="paginatedDeviceList" class="transparent-table"
-              :header-cell-class-name="headerCellClassName" v-loading="loading" element-loading-text="拼命加载中"
+                      :header-cell-class-name="headerCellClassName" v-loading="loading"
-              element-loading-spinner="el-icon-loading" element-loading-background="rgba(255, 255, 255, 0.7)">
+                      element-loading-text="拼命加载中"
+                      element-loading-spinner="el-icon-loading" element-loading-background="rgba(255, 255, 255, 0.7)">
               <el-table-column label="选择" align="center" width="120">
                 <template slot-scope="scope">
                   <el-checkbox v-model="scope.row.selected"></el-checkbox>
@@ -33,22 +34,32 @@
               <el-table-column label="绑定时间" prop="bindTime" align="center"></el-table-column>
               <el-table-column label="最近对话" prop="lastConversation" align="center"></el-table-column>
               <el-table-column label="备注" align="center">
-                <template slot-scope="scope">
+                <template #default="{ row }">
-                  <el-input v-if="scope.row.isEdit" v-model="scope.row.remark" size="mini"
+                  <el-input
-                    @blur="stopEditRemark(scope.$index)"></el-input>
+                      v-show="row.isEdit"
-                  <span v-else>
+                      v-model="row.remark"
-                    <i v-if="!scope.row.remark" class="el-icon-edit"
+                      size="mini"
-                      @click="startEditRemark(scope.$index, scope.row)"></i>
+                      maxlength="64"
-                    <span v-else @click="startEditRemark(scope.$index, scope.row)">
+                      show-word-limit
-                      {{ scope.row.remark }}
+                      @blur="onRemarkBlur(row)"
-                    </span>
+                      @keyup.enter.native="onRemarkEnter(row)"
+                  />
+                  <span v-show="!row.isEdit" class="remark-view">
+                  <i
+                      class="el-icon-edit"
+                      @click="row.isEdit = true"
+                      style="cursor: pointer;"
+                  ></i>
+                  <span @click="row.isEdit = true">
+                    {{ row.remark || '—' }}
                   </span>
+                </span>
                 </template>
               </el-table-column>
               <el-table-column label="OTA升级" align="center">
                 <template slot-scope="scope">
                   <el-switch v-model="scope.row.otaSwitch" size="mini" active-color="#13ce66" inactive-color="#ff4949"
-                    @change="handleOtaSwitchChange(scope.row)"></el-switch>
+                             @change="handleOtaSwitchChange(scope.row)"></el-switch>
                 </template>
               </el-table-column>
               <el-table-column label="操作" align="center">
@@ -78,7 +89,7 @@
                 <button class="pagination-btn" :disabled="currentPage === 1" @click="goFirst">首页</button>
                 <button class="pagination-btn" :disabled="currentPage === 1" @click="goPrev">上一页</button>
                 <button v-for="page in visiblePages" :key="page" class="pagination-btn"
-                  :class="{ active: page === currentPage }" @click="goToPage(page)">
+                        :class="{ active: page === currentPage }" @click="goToPage(page)">
                   {{ page }}
                 </button>
                 <button class="pagination-btn" :disabled="currentPage === pageCount" @click="goNext">下一页</button>
@@ -91,7 +102,7 @@
     </div>
 
     <AddDeviceDialog :visible.sync="addDeviceDialogVisible" :agent-id="currentAgentId"
-      @refresh="fetchBindDevices(currentAgentId)" />
+                     @refresh="fetchBindDevices(currentAgentId)"/>
 
   </div>
 </template>
@@ -102,7 +113,7 @@ import AddDeviceDialog from "@/components/AddDeviceDialog.vue";
 import HeaderBar from "@/components/HeaderBar.vue";
 
 export default {
-  components: { HeaderBar, AddDeviceDialog },
+  components: {HeaderBar, AddDeviceDialog},
   data() {
     return {
       addDeviceDialogVisible: false,
@@ -125,18 +136,15 @@ export default {
       const keyword = this.activeSearchKeyword.toLowerCase();
       if (!keyword) return this.deviceList;
       return this.deviceList.filter(device =>
-        (device.model && device.model.toLowerCase().includes(keyword)) ||
+          (device.model && device.model.toLowerCase().includes(keyword)) ||
-        (device.macAddress && device.macAddress.toLowerCase().includes(keyword))
+          (device.macAddress && device.macAddress.toLowerCase().includes(keyword))
       );
     },
 
     paginatedDeviceList() {
       const start = (this.currentPage - 1) * this.pageSize;
       const end = start + this.pageSize;
-      return this.filteredDeviceList.slice(start, end).map(item => ({
+      return this.filteredDeviceList.slice(start, end);
-        ...item,
-        selected: false
-      }));
     },
     pageCount() {
       return Math.ceil(this.filteredDeviceList.length / this.pageSize);
@@ -212,11 +220,10 @@ export default {
         this.batchUnbindDevices(deviceIds);
       });
     },
-
     batchUnbindDevices(deviceIds) {
       const promises = deviceIds.map(id => {
         return new Promise((resolve, reject) => {
-          Api.device.unbindDevice(id, ({ data }) => {
+          Api.device.unbindDevice(id, ({data}) => {
             if (data.code === 0) {
               resolve();
             } else {
@@ -225,33 +232,61 @@ export default {
           });
         });
       });
-
       Promise.all(promises)
-        .then(() => {
+          .then(() => {
-          this.$message.success({
+            this.$message.success({
-            message: `成功解绑 ${deviceIds.length} 台设备`,
+              message: `成功解绑 ${deviceIds.length} 台设备`,
-            showClose: true
+              showClose: true
+            });
+            this.fetchBindDevices(this.currentAgentId);
+            this.selectedDevices = [];
+            this.isAllSelected = false;
+          })
+          .catch(error => {
+            this.$message.error({
+              message: error || '批量解绑过程中出现错误',
+              showClose: true
+            });
           });
-          this.fetchBindDevices(this.currentAgentId);
-          this.selectedDevices = [];
-          this.isAllSelected = false;
-        })
-        .catch(error => {
-          this.$message.error({
-            message: error || '批量解绑过程中出现错误',
-            showClose: true
-          });
-        });
     },
-
     handleAddDevice() {
       this.addDeviceDialogVisible = true;
     },
-    startEditRemark(index, row) {
+    submitRemark(row) {
-      this.deviceList[index].isEdit = true;
+      if (row._submitting) return;
+
+      const text = (row.remark || '').trim();
+      if (text.length > 64) {
+        this.$message.warning('备注不能超过 64 字符');
+        return;
+      }
+      if (text === row._originalRemark) {
+        return;
+      }
+
+      row._submitting = true;
+      this.updateDeviceInfo(row.device_id, { alias: text }, (ok, resp) => {
+        if (ok) {
+          row._originalRemark = text;
+          this.$message.success('备注已保存');
+        } else {
+          row.remark = row._originalRemark;
+          this.$message.error(resp.msg || '备注保存失败');
+        }
+        row._submitting = false;
+      });
     },
-    stopEditRemark(index) {
+    // 备注输入框：失焦时提交
-      this.deviceList[index].isEdit = false;
+    onRemarkBlur(row) {
+      row.isEdit = false;
+      setTimeout(() => {
+        this.submitRemark(row);
+      }, 100); // 延迟 100ms，避开 enter+blur 同时触发的窗口
+    },
+    // 备注输入框：按回车时提交
+    onRemarkEnter(row) {
+      row.isEdit = false;
+      this.submitRemark(row);
     },
     handleUnbind(device_id) {
       this.$confirm('确认要解绑该设备吗？', '警告', {
@@ -259,7 +294,7 @@ export default {
         cancelButtonText: '取消',
         type: 'warning'
       }).then(() => {
-        Api.device.unbindDevice(device_id, ({ data }) => {
+        Api.device.unbindDevice(device_id, ({data}) => {
           if (data.code === 0) {
             this.$message.success({
               message: '设备解绑成功',
@@ -290,7 +325,7 @@ export default {
 
     fetchBindDevices(agentId) {
       this.loading = true;
-      Api.device.getAgentBindDevices(agentId, ({ data }) => {
+      Api.device.getAgentBindDevices(agentId, ({data}) => {
         this.loading = false;
         if (data.code === 0) {
           this.deviceList = data.data.map(device => {
@@ -302,12 +337,14 @@ export default {
               bindTime: device.createDate,
               lastConversation: device.lastConnectedAt,
               remark: device.alias,
+              _originalRemark: device.alias,
               isEdit: false,
+              _submitting: false,
               otaSwitch: device.autoUpdate === 1,
               rawBindTime: new Date(device.createDate).getTime()
             };
           })
-            .sort((a, b) => a.rawBindTime - b.rawBindTime);
+              .sort((a, b) => a.rawBindTime - b.rawBindTime);
           this.activeSearchKeyword = "";
           this.searchKeyword = "";
         } else {
@@ -315,7 +352,7 @@ export default {
         }
       });
     },
-    headerCellClassName({ columnIndex }) {
+    headerCellClassName({columnIndex}) {
       if (columnIndex === 0) {
         return "custom-selection-header";
       }
@@ -325,14 +362,19 @@ export default {
       const firmwareType = this.firmwareTypes.find(item => item.key === type)
       return firmwareType ? firmwareType.name : type
     },
+    updateDeviceInfo(device_id, payload, callback) {
+      return Api.device.updateDeviceInfo(device_id, payload, ({data}) => {
+        callback(data.code === 0, data);
+      })
+    },
     handleOtaSwitchChange(row) {
-      Api.device.enableOtaUpgrade(row.device_id, row.otaSwitch ? 1 : 0, ({ data }) => {
+      this.updateDeviceInfo(row.device_id, {autoUpdate: row.otaSwitch ? 1 : 0}, (result, {msg}) => {
-        if (data.code === 0) {
+        if (result) {
-          this.$message.success(row.otaSwitch ? '已设置成自动升级' : '已关闭自动升级')
+          this.$message.success(row.otaSwitch ? '已设置成自动升级' : '已关闭自动升级');
-        } else {
+          return;
-          row.otaSwitch = !row.otaSwitch
-          this.$message.error(data.msg || '操作失败')
         }
+        row.otaSwitch = !row.otaSwitch
+        this.$message.error(msg || '操作失败')
       })
     },
   }
@@ -645,7 +687,6 @@ export default {
 }
 
 
-
 :deep(.el-table .el-button--text) {
   color: #7079aa;
 }

main/manager-web/src/views/DictManagement.vue

@@ -538,7 +538,7 @@ export default {
 
 .data-table {
     border-radius: 6px;
-    overflow-y: auto;
+    overflow-y: hidden;
     background-color: transparent !important;
     --table-max-height: calc(100vh - 40vh);
     max-height: var(--table-max-height);

main/manager-web/src/views/ModelConfig.vue

@@ -30,6 +30,9 @@
           <el-menu-item index="llm">
             <span class="menu-text">大语言模型</span>
           </el-menu-item>
+          <el-menu-item index="vllm">
+            <span class="menu-text">视觉大模型</span>
+          </el-menu-item>
           <el-menu-item index="intent">
             <span class="menu-text">意图识别</span>
           </el-menu-item>
@@ -173,6 +176,7 @@ export default {
         vad: '语言活动检测模型(VAD)',
         asr: '语音识别模型(ASR)',
         llm: '大语言模型（LLM）',
+        vllm: '视觉大模型（VLLM）',
         intent: '意图识别模型(Intent)',
         tts: '语音合成模型(TTS)',
         memory: '记忆模型(Memory)'

main/manager-web/src/views/ProviderManagement.vue

@@ -143,9 +143,11 @@ export default {
         { value: "ASR", label: "语音识别" },
         { value: "TTS", label: "语音合成" },
         { value: "LLM", label: "大语言模型" },
+        { value: "VLLM", label: "视觉大语言模型" },
         { value: "Intent", label: "意图识别" },
         { value: "Memory", label: "记忆模块" },
-        { value: "VAD", label: "语音活动检测" }
+        { value: "VAD", label: "语音活动检测" },
+        { value: "Plugin", label: "插件工具" }
       ],
       currentPage: 1,
       loading: false,

main/manager-web/src/views/roleConfig.vue

@@ -64,7 +64,27 @@
                     </el-form-item>
                   </div>
                   <div class="form-column">
-                    <el-form-item v-for="(model, index) in models" :key="`model-${index}`" :label="model.label"
+                    <div class="model-row">
+                      <el-form-item label="语音活动检测(VAD)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.vadModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('VAD', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['VAD']"
+                              :key="`option-vad-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                      <el-form-item label="语音识别(ASR)" class="model-item">
+                        <div class="model-select-wrapper">
+                          <el-select v-model="form.model.asrModelId" filterable placeholder="请选择" class="form-select"
+                            @change="handleModelChange('ASR', $event)">
+                            <el-option v-for="(item, optionIndex) in modelOptions['ASR']"
+                              :key="`option-asr-${optionIndex}`" :label="item.label" :value="item.value" />
+                          </el-select>
+                        </div>
+                      </el-form-item>
+                    </div>
+                    <el-form-item v-for="(model, index) in models.slice(2)" :key="`model-${index}`" :label="model.label"
                       class="model-item">
                       <div class="model-select-wrapper">
                         <el-select v-model="form.model[model.key]" filterable placeholder="请选择" class="form-select"
@@ -77,19 +97,12 @@
                             popper-class="custom-tooltip">
                             <div slot="content">
                               <div><strong>功能名称:</strong> {{ func.name }}</div>
-                              <div v-if="Object.keys(func.params).length > 0">
-                                <strong>参数配置:</strong>
-                                <div v-for="(value, key) in func.params" :key="key">
-                                  {{ key }}: {{ value }}
-                                </div>
-                              </div>
-                              <div v-else>无参数配置</div>
                             </div>
                             <div class="icon-dot" :style="{ backgroundColor: getFunctionColor(func.name) }">
                               {{ func.name.charAt(0) }}
                             </div>
                           </el-tooltip>
-                          <el-button class="edit-function-btn" @click="showFunctionDialog = true"
+                          <el-button class="edit-function-btn" @click="openFunctionDialog"
                             :class="{ 'active-btn': showFunctionDialog }">
                             编辑功能
                           </el-button>
@@ -118,7 +131,7 @@
       </div>
     </div>
 
-    <function-dialog v-model="showFunctionDialog" :functions="currentFunctions"
+    <function-dialog v-model="showFunctionDialog" :functions="currentFunctions" :all-functions="allFunctions"
       @update-functions="handleUpdateFunctions" @dialog-closed="handleDialogClosed" />
   </div>
 </template>
@@ -148,6 +161,7 @@ export default {
           vadModelId: "",
           asrModelId: "",
           llmModelId: "",
+          vllmModelId: "",
           memModelId: "",
           intentModelId: "",
         }
@@ -156,6 +170,7 @@ export default {
         { label: '语音活动检测(VAD)', key: 'vadModelId', type: 'VAD' },
         { label: '语音识别(ASR)', key: 'asrModelId', type: 'ASR' },
         { label: '大语言模型(LLM)', key: 'llmModelId', type: 'LLM' },
+        { label: '视觉大模型(VLLM)', key: 'vllmModelId', type: 'VLLM' },
         { label: '意图识别(Intent)', key: 'intentModelId', type: 'Intent' },
         { label: '记忆(Memory)', key: 'memModelId', type: 'Memory' },
         { label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' },
@@ -170,12 +185,8 @@ export default {
         '#FF6B6B', '#4ECDC4', '#45B7D1',
         '#96CEB4', '#FFEEAD', '#D4A5A5', '#A2836E'
       ],
-      allFunctions: [
+      allFunctions: [],
-        { name: '天气', params: {} },
+      originalFunctions: [],
-        { name: '新闻', params: {} },
-        { name: '工具', params: {} },
-        { name: '退出', params: {} }
-      ],
     }
   },
   methods: {
@@ -189,6 +200,7 @@ export default {
         asrModelId: this.form.model.asrModelId,
         vadModelId: this.form.model.vadModelId,
         llmModelId: this.form.model.llmModelId,
+        vllmModelId: this.form.model.vllmModelId,
         ttsModelId: this.form.model.ttsModelId,
         ttsVoiceId: this.form.ttsVoiceId,
         chatHistoryConf: this.form.chatHistoryConf,
@@ -199,7 +211,12 @@ export default {
         langCode: this.form.langCode,
         language: this.form.language,
         sort: this.form.sort,
-        functions: this.currentFunctions
+        functions: this.currentFunctions.map(item => {
+          return ({
+            pluginId: item.id,
+            paramInfo: item.params
+          })
+        })
       };
       Api.agent.updateAgentConfig(this.$route.query.agentId, configData, ({ data }) => {
         if (data.code === 0) {
@@ -236,6 +253,7 @@ export default {
             vadModelId: "",
             asrModelId: "",
             llmModelId: "",
+            vllmModelId: "",
             memModelId: "",
             intentModelId: "",
           }
@@ -245,7 +263,8 @@ export default {
           message: '配置已重置',
           showClose: true
         })
-      }).catch(() => { });
+      }).catch(() => {
+      });
     },
     fetchTemplates() {
       Api.agent.getAgentTemplate(({ data }) => {
@@ -289,6 +308,7 @@ export default {
           vadModelId: templateData.vadModelId || this.form.model.vadModelId,
           asrModelId: templateData.asrModelId || this.form.model.asrModelId,
           llmModelId: templateData.llmModelId || this.form.model.llmModelId,
+          vllmModelId: templateData.vllmModelId || this.form.model.vllmModelId,
           memModelId: templateData.memModelId || this.form.model.memModelId,
           intentModelId: templateData.intentModelId || this.form.model.intentModelId
         }
@@ -305,11 +325,38 @@ export default {
               vadModelId: data.data.vadModelId,
               asrModelId: data.data.asrModelId,
               llmModelId: data.data.llmModelId,
+              vllmModelId: data.data.vllmModelId,
               memModelId: data.data.memModelId,
               intentModelId: data.data.intentModelId
             }
           };
-          this.currentFunctions = data.data.functions || [];
+          // 后端只给了最小映射：[{ id, agentId, pluginId }, ...]
+          const savedMappings = data.data.functions || [];
+
+          // 先保证 allFunctions 已经加载（如果没有，则先 fetchAllFunctions）
+          const ensureFuncs = this.allFunctions.length
+            ? Promise.resolve()
+            : this.fetchAllFunctions();
+
+          ensureFuncs.then(() => {
+            // 合并：按照 pluginId（id 字段）把全量元数据信息补齐
+            this.currentFunctions = savedMappings.map(mapping => {
+              const meta = this.allFunctions.find(f => f.id === mapping.pluginId);
+              if (!meta) {
+                // 插件定义没找到，退化处理
+                return { id: mapping.pluginId, name: mapping.pluginId, params: {} };
+              }
+              return {
+                id: mapping.pluginId,
+                name: meta.name,
+                // 后端如果还有 paramInfo 字段就用 mapping.paramInfo，否则用 meta.params 默认值
+                params: mapping.paramInfo || { ...meta.params },
+                fieldsMeta: meta.fieldsMeta  // 保留以便对话框渲染 tooltip
+              };
+            });
+            // 备份原始，以备取消时恢复
+            this.originalFunctions = JSON.parse(JSON.stringify(this.currentFunctions));
+          });
         } else {
           this.$message.error(data.msg || '获取配置失败');
         }
@@ -347,17 +394,15 @@ export default {
     },
     getFunctionColor(name) {
       const hash = [...name].reduce((acc, char) => acc + char.charCodeAt(0), 0);
-      return this.functionColorMap[hash % 7];
+      return this.functionColorMap[hash % this.functionColorMap.length];
     },
     showFunctionIcons(type) {
-      // TODO 暂时不放出来
+      return type === 'Intent' &&
-      return false;
+        this.form.model.intentModelId !== 'Intent_nointent';
-      // return type === 'Intent' &&
-      //   this.form.model.intentModelId !== 'Intent_nointent';
     },
     handleModelChange(type, value) {
       if (type === 'Intent' && value !== 'Intent_nointent') {
-        this.fetchFunctionList();
+        this.fetchAllFunctions();
       }
       if (type === 'Memory' && value === 'Memory_nomem') {
         this.form.chatHistoryConf = 0;
@@ -366,28 +411,44 @@ export default {
         this.form.chatHistoryConf = 2;
       }
     },
-    fetchFunctionList() {
+    fetchAllFunctions() {
-      // 使用假数据代替API调用
+      return new Promise((resolve, reject) => {
-      return new Promise(resolve => {
+        Api.model.getPluginFunctionList(null, ({ data }) => {
-        setTimeout(() => {
+          if (data.code === 0) {
-          this.currentFunctions = [
+            this.allFunctions = data.data.map(item => {
-            { name: '天气', params: { city: '北京' } },
+              const meta = JSON.parse(item.fields || '[]');
-            { name: '新闻', params: { type: '科技' } }
+              const params = meta.reduce((m, f) => {
-          ];
+                m[f.key] = f.default;
-          resolve();
+                return m;
-        }, 500);
+              }, {});
+              return { ...item, fieldsMeta: meta, params };
+            });
+            resolve();
+          } else {
+            this.$message.error(data.msg || '获取插件列表失败');
+            reject();
+          }
+        });
       });
     },
+    openFunctionDialog() {
+      // 显示编辑对话框时，确保 allFunctions 已经加载
+      if (this.allFunctions.length === 0) {
+        this.fetchAllFunctions().then(() => this.showFunctionDialog = true);
+      } else {
+        this.showFunctionDialog = true;
+      }
+    },
     handleUpdateFunctions(selected) {
       this.currentFunctions = selected;
-      console.log('保存的功能列表:', selected);
-      this.$message.success('功能配置已保存');
     },
     handleDialogClosed(saved) {
       if (!saved) {
-        // 如果未保存，恢复原始功能列表
         this.currentFunctions = JSON.parse(JSON.stringify(this.originalFunctions));
+      } else {
+        this.originalFunctions = JSON.parse(JSON.stringify(this.currentFunctions));
       }
+      this.showFunctionDialog = false;
     },
     updateChatHistoryConf() {
       if (this.form.model.memModelId === 'Memory_nomem') {
@@ -420,9 +481,7 @@ export default {
     const agentId = this.$route.query.agentId;
     if (agentId) {
       this.fetchAgentConfig(agentId);
-      this.fetchFunctionList().then(() => {
+      this.fetchAllFunctions();
-        this.originalFunctions = JSON.parse(JSON.stringify(this.currentFunctions));
-      });
     }
     this.fetchModelOptions();
     this.fetchTemplates();
@@ -587,6 +646,25 @@ export default {
   width: 100%;
 }
 
+.model-row {
+  display: flex;
+  gap: 20px;
+  margin-bottom: 6px;
+}
+
+.model-row .model-item {
+  flex: 1;
+  margin-bottom: 0;
+}
+
+.model-row .el-form-item__label {
+  font-size: 12px !important;
+  color: #3d4566 !important;
+  font-weight: 400;
+  line-height: 22px;
+  padding-bottom: 2px;
+}
+
 .function-icons {
   display: flex;
   align-items: center;

main/xiaozhi-server/app.py

@@ -1,11 +1,12 @@
 import sys
+import uuid
 import signal
 import asyncio
 from aioconsole import ainput
 from config.settings import load_config
 from config.logger import setup_logging
 from core.utils.util import get_local_ip
-from core.ota_server import SimpleOtaServer
+from core.http_server import SimpleHttpServer
 from core.websocket_server import WebSocketServer
 from core.utils.util import check_ffmpeg_installed
 
@@ -45,25 +46,37 @@ async def main():
     check_ffmpeg_installed()
     config = load_config()
 
+    # 默认使用manager-api的secret作为auth_key
+    # 如果secret为空，则生成随机密钥
+    # auth_key用于jwt认证，比如视觉分析接口的jwt认证
+    auth_key = config.get("manager-api", {}).get("secret", "")
+    if not auth_key or len(auth_key) == 0 or "你" in auth_key:
+        auth_key = str(uuid.uuid4().hex)
+    config["server"]["auth_key"] = auth_key
+
     # 添加 stdin 监控任务
     stdin_task = asyncio.create_task(monitor_stdin())
 
     # 启动 WebSocket 服务器
     ws_server = WebSocketServer(config)
     ws_task = asyncio.create_task(ws_server.start())
-    ota_task = None
+    # 启动 Simple http 服务器
+    ota_server = SimpleHttpServer(config)
+    ota_task = asyncio.create_task(ota_server.start())
 
     read_config_from_api = config.get("read_config_from_api", False)
+    port = int(config["server"].get("http_port", 8003))
     if not read_config_from_api:
-        # 启动 Simple OTA 服务器
-        ota_server = SimpleOtaServer(config)
-        ota_task = asyncio.create_task(ota_server.start())
-
         logger.bind(tag=TAG).info(
             "OTA接口是\t\thttp://{}:{}/xiaozhi/ota/",
             get_local_ip(),
-            config["server"]["ota_port"],
+            port,
         )
+    logger.bind(tag=TAG).info(
+        "视觉分析接口是\thttp://{}:{}/mcp/vision/explain",
+        get_local_ip(),
+        port,
+    )
 
     # 获取WebSocket配置，使用安全的默认值
     websocket_port = 8000

main/xiaozhi-server/config.yaml

@@ -2,6 +2,7 @@
 # 然后你想修改覆盖修改什么配置，就修改【.config.yaml】文件，而不是修改【config.yaml】文件
 # 系统会优先读取【data/.config.yaml】文件的配置，如果【.config.yaml】文件里的配置不存在，系统会自动去读取【config.yaml】文件的配置。
 # 这样做，可以最简化配置，保护您的密钥安全。
+# 如果你使用了智控台，那么以下所有配置，都不会生效，请在智控台中修改配置
 
 # #####################################################################################
 # #############################以下是服务器基本运行配置####################################
@@ -9,13 +10,21 @@ server:
   # 服务器监听地址和端口(Server listening address and port)
   ip: 0.0.0.0
   port: 8000
-  # OTA接口的端口号
+  # http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
-  ota_port: 8002
+  http_port: 8003
   # 这个websocket配置是指ota接口向设备发送的websocket地址
-  # 如果按默认的写法，ota接口会自动生成websocket地址。这个地址你可以直接用浏览器访问ota接口确认一下
+  # 如果按默认的写法，ota接口会自动生成websocket地址，并输出在启动日志里，这个地址你可以直接用浏览器访问ota接口确认一下
   # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
-  # 所以如果你使用docker部署或使用公网部署时，请设置正确的websocket地址
+  # 所以如果你使用docker部署时，将websocket设置成局域网地址
+  # 如果你使用公网部署时，将vwebsocket设置成公网地址
   websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
   # OTA返回信息时区偏移量
   timezone_offset: +8
   # 认证配置
@@ -108,10 +117,9 @@ plugins:
   # 更多类型的新闻列表查看 https://www.chinanews.com.cn/rss/
   get_news_from_chinanews:
     default_rss_url: "https://www.chinanews.com.cn/rss/society.xml"
-    category_urls:
+    society_rss_url: "https://www.chinanews.com.cn/rss/society.xml"
-      society: "https://www.chinanews.com.cn/rss/society.xml"
+    world_rss_url: "https://www.chinanews.com.cn/rss/world.xml"
-      world: "https://www.chinanews.com.cn/rss/world.xml"
+    finance_rss_url: "https://www.chinanews.com.cn/rss/finance.xml"
-      finance: "https://www.chinanews.com.cn/rss/finance.xml"
   get_news_from_newsnow: {"url": "https://newsnow.busiyi.world/api/s?id="}
   home_assistant:
     devices:
@@ -160,6 +168,8 @@ selected_module:
   ASR: FunASR
   # 将根据配置名称对应的type调用实际的LLM适配器
   LLM: ChatGLMLLM
+  # 视觉语言大模型
+  VLLM: ChatGLMVLLM
   # TTS将根据配置名称对应的type调用实际的TTS适配器
   TTS: EdgeTTS
   # 记忆模块，默认不开启记忆；如果想使用超长记忆，推荐使用mem0ai；如果注重隐私，请使用本地的mem_local_short
@@ -253,6 +263,8 @@ ASR:
   DoubaoASR:
     # 可以在这里申请相关Key等信息
     # https://console.volcengine.com/speech/app
+    # DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
+    # 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
     type: doubao
     appid: 你的火山引擎语音合成服务appid
     access_token: 你的火山引擎语音合成服务access_token
@@ -261,6 +273,20 @@ ASR:
     boosting_table_name: （选填）你的热词文件名称
     correct_table_name: （选填）你的替换词文件名称
     output_dir: tmp/
+  DoubaoStreamASR:
+    # 可以在这里申请相关Key等信息
+    # https://console.volcengine.com/speech/app
+    # DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
+    # 开通地址https://console.volcengine.com/speech/service/10011
+    # 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
+    type: doubao_stream
+    appid: 你的火山引擎语音合成服务appid
+    access_token: 你的火山引擎语音合成服务access_token
+    cluster: volcengine_input_common
+    # 热词、替换词使用流程：https://www.volcengine.com/docs/6561/155738
+    boosting_table_name: （选填）你的热词文件名称
+    correct_table_name: （选填）你的替换词文件名称
+    output_dir: tmp/
   TencentASR:
     # token申请地址：https://console.cloud.tencent.com/cam/capi
     # 免费领取资源：https://console.cloud.tencent.com/asr/resourcebundle
@@ -297,7 +323,7 @@ VAD:
     type: silero
     threshold: 0.5
     model_dir: models/snakers4_silero-vad
-    min_silence_duration_ms: 700  # 如果说话停顿比较长，可以把这个值设置大一些
+    min_silence_duration_ms: 200  # 如果说话停顿比较长，可以把这个值设置大一些
 
 LLM:
   # 所有openai类型均可以修改超参，以AliLLM为例
@@ -432,6 +458,21 @@ LLM:
     # Xinference服务地址和模型名称
     model_name: qwen2.5:3b-AWQ  # 使用的小模型名称，用于意图识别
     base_url: http://localhost:9997  # Xinference服务地址
+# VLLM配置（视觉语言大模型）
+VLLM:
+  ChatGLMVLLM:
+    type: openai
+    # glm-4v-flash是智谱免费AI的视觉模型，需要先在智谱AI平台创建API密钥并获取api_key
+    # 可在这里找到你的api key https://bigmodel.cn/usercenter/proj-mgmt/apikeys
+    model_name: glm-4v-flash  # 智谱AI的视觉模型
+    url: https://open.bigmodel.cn/api/paas/v4/
+    api_key: 你的api_key
+  QwenVLVLLM:
+    type: openai
+    model_name: qwen2.5-vl-3b-instruct
+    url: https://dashscope.aliyuncs.com/compatible-mode/v1
+    # 可在这里找到你的api key https://bailian.console.aliyun.com/?apiKey=1#/api-key
+    api_key: 你的api_key
 TTS:
   # 当前支持的type为edge、doubao，可自行适配
   EdgeTTS:
@@ -714,3 +755,14 @@ TTS:
       # Authorization: Bearer xxxx
     format: mp3 # 接口返回的音频格式
     output_dir: tmp/
+  LinkeraiTTS:
+    type: linkerai
+    api_url: https://tts.linkerai.cn/tts
+    audio_format: "pcm"
+    # 默认的access_token供大家测试时免费使用的，此access_token请勿用于商业用途
+    # 如果效果不错，可自行申请token，申请地址：https://linkerai.cn
+    # 各参数意义见开发文档：https://tts.linkerai.cn/docs
+    # 支持声音克隆，可自行上传音频，填入voice参数，voice参数为空时，使用默认声音
+    access_token: "U4YdYXVfpwWnk2t5Gp822zWPCuORyeJL"
+    voice: "OUeAo1mhq6IBExi"
+    output_dir: tmp/

main/xiaozhi-server/config/config_loader.py

@@ -59,10 +59,14 @@ def get_config_from_api(config):
         "url": config["manager-api"].get("url", ""),
         "secret": config["manager-api"].get("secret", ""),
     }
+    # server的配置以本地为准
     if config.get("server"):
         config_data["server"] = {
             "ip": config["server"].get("ip", ""),
             "port": config["server"].get("port", ""),
+            "http_port": config["server"].get("http_port", ""),
+            "vision_explain": config["server"].get("vision_explain", ""),
+            "auth_key": config["server"].get("auth_key", ""),
         }
     return config_data
 

main/xiaozhi-server/config/logger.py

@@ -3,8 +3,9 @@ import sys
 from loguru import logger
 from config.config_loader import load_config
 from config.settings import check_config_file
+from datetime import datetime
 
-SERVER_VERSION = "0.5.2"
+SERVER_VERSION = "0.5.7"
 _logger_initialized = False
 
 
@@ -59,7 +60,7 @@ def setup_logging():
         )
         log_format_file = log_config.get(
             "log_format_file",
-            "{time:YYYY-MM-DD HH:mm:ss} - {version_{extra[selected_module]}} - {name} - {level} - {extra[tag]} - {message}",
+            "{time:YYYY-MM-DD HH:mm:ss} - {version}_{extra[selected_module]} - {name} - {level} - {extra[tag]} - {message}",
         )
         selected_module_str = logger._core.extra["selected_module"]
 
@@ -84,12 +85,23 @@ def setup_logging():
         # 输出到控制台
         logger.add(sys.stdout, format=log_format, level=log_level, filter=formatter)
 
-        # 输出到文件
+        # 输出到文件 - 统一目录，按大小轮转
+        # 日志文件完整路径
+        log_file_path = os.path.join(log_dir, log_file)
+        
+        # 添加日志处理器
         logger.add(
-            os.path.join(log_dir, log_file),
+            log_file_path,
             format=log_format_file,
             level=log_level,
             filter=formatter,
+            rotation="10 MB",  # 每个文件最大10MB
+            retention="30 days",  # 保留30天
+            compression=None,
+            encoding="utf-8",
+            enqueue=True,  # 异步安全
+            backtrace=True,
+            diagnose=True,
         )
         _logger_initialized = True  # 标记为已初始化
 
@@ -102,6 +114,7 @@ def update_module_string(selected_module_str):
     current_module = logger._core.extra["selected_module"]
 
     if current_module == selected_module_str:
+        logger.debug(f"组件未更改无需更新")
         return
 
     try:
@@ -116,7 +129,7 @@ def update_module_string(selected_module_str):
         )
         log_format_file = log_config.get(
             "log_format_file",
-            "{time:YYYY-MM-DD HH:mm:ss} - {version_{extra[selected_module]}} - {name} - {level} - {extra[tag]} - {message}",
+            "{time:YYYY-MM-DD HH:mm:ss} - {version}_{extra[selected_module]} - {name} - {level} - {extra[tag]} - {message}",
         )
 
         log_format = log_format.replace("{version}", SERVER_VERSION)
@@ -133,14 +146,26 @@ def update_module_string(selected_module_str):
             level=log_config.get("log_level", "INFO"),
             filter=formatter,
         )
+        
+        # 更新文件日志配置 - 统一目录，按大小轮转
+        log_dir = log_config.get("log_dir", "tmp")
+        log_file = log_config.get("log_file", "server.log")
+        
+        # 日志文件完整路径
+        log_file_path = os.path.join(log_dir, log_file)
+        
         logger.add(
-            os.path.join(
+            log_file_path,
-                log_config.get("log_dir", "tmp"),
-                log_config.get("log_file", "server.log"),
-            ),
             format=log_format_file,
             level=log_config.get("log_level", "INFO"),
             filter=formatter,
+            rotation="10 MB",  # 每个文件最大10MB
+            retention="30 days",  # 保留30天
+            compression=None,
+            encoding="utf-8",
+            enqueue=True,  # 异步安全
+            backtrace=True,
+            diagnose=True,
         )
 
     except Exception as e:

main/xiaozhi-server/config_from_api.yaml

@@ -8,6 +8,15 @@
 server:
   ip: 0.0.0.0
   port: 8000
+  # http服务的端口，用于视觉分析接口
+  http_port: 8003
+  # 视觉分析接口地址
+  # 向设备发送的视觉分析的接口地址
+  # 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
+  # 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
+  # 所以如果你使用docker部署时，将vision_explain设置成局域网地址
+  # 如果你使用公网部署时，将vision_explain设置成公网地址
+  vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
 manager-api:
   # 你的manager-api的地址，最好使用局域网ip
   url: http://127.0.0.1:8002/xiaozhi

main/xiaozhi-server/core/api/base_handler.py

@@ -0,0 +1,16 @@
+from aiohttp import web
+from config.logger import setup_logging
+
+
+class BaseHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/api/ota_handler.py

@@ -1,18 +1,15 @@
 import json
 import time
-import asyncio
 from aiohttp import web
-from config.logger import setup_logging
 from core.utils.util import get_local_ip
-from core.utils.modules_initialize import initialize_modules
+from core.api.base_handler import BaseHandler
 
 TAG = __name__
 
 
-class SimpleOtaServer:
+class OTAHandler(BaseHandler):
     def __init__(self, config: dict):
-        self.config = config
+        super().__init__(config)
-        self.logger = setup_logging()
 
     def _get_websocket_url(self, local_ip: str, port: int) -> str:
         """获取websocket地址
@@ -25,41 +22,15 @@ class SimpleOtaServer:
             str: websocket地址
         """
         server_config = self.config["server"]
-        websocket_config = server_config.get("websocket")
+        websocket_config = server_config.get("websocket", "")
 
-        if websocket_config and "你" not in websocket_config:
+        if "你的" not in websocket_config:
             return websocket_config
         else:
             return f"ws://{local_ip}:{port}/xiaozhi/v1/"
 
-    async def start(self):
+    async def handle_post(self, request):
-        server_config = self.config["server"]
+        """处理 OTA POST 请求"""
-        host = server_config.get("ip", "0.0.0.0")
-        port = int(server_config.get("ota_port"))
-
-        if port:
-            app = web.Application()
-            # 添加路由
-            app.add_routes(
-                [
-                    web.get("/xiaozhi/ota/", self._handle_ota_get_request),
-                    web.post("/xiaozhi/ota/", self._handle_ota_request),
-                    web.options("/xiaozhi/ota/", self._handle_ota_request),
-                ]
-            )
-
-            # 运行服务
-            runner = web.AppRunner(app)
-            await runner.setup()
-            site = web.TCPSite(runner, host, port)
-            await site.start()
-
-            # 保持服务运行
-            while True:
-                await asyncio.sleep(3600)  # 每隔 1 小时检查一次
-
-    async def _handle_ota_request(self, request):
-        """处理 /xiaozhi/ota/ 的 POST 请求"""
         try:
             data = await request.text()
             self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}")
@@ -75,11 +46,9 @@ class SimpleOtaServer:
             data_json = json.loads(data)
 
             server_config = self.config["server"]
-            host = server_config.get("ip", "0.0.0.0")
             port = int(server_config.get("port", 8000))
             local_ip = get_local_ip()
 
-            # OTA基础信息
             return_json = {
                 "server_time": {
                     "timestamp": int(round(time.time() * 1000)),
@@ -104,16 +73,11 @@ class SimpleOtaServer:
                 content_type="application/json",
             )
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response
 
-    async def _handle_ota_get_request(self, request):
+    async def handle_get(self, request):
-        """处理 /xiaozhi/ota/ 的 GET 请求"""
+        """处理 OTA GET 请求"""
         try:
             server_config = self.config["server"]
             local_ip = get_local_ip()
@@ -125,10 +89,5 @@ class SimpleOtaServer:
             self.logger.bind(tag=TAG).error(f"OTA GET请求异常: {e}")
             response = web.Response(text="OTA接口异常", content_type="text/plain")
         finally:
-            # 添加header，允许跨域访问
+            self._add_cors_headers(response)
-            response.headers["Access-Control-Allow-Headers"] = (
-                "client-id, content-type, device-id"
-            )
-            response.headers["Access-Control-Allow-Credentials"] = "true"
-            response.headers["Access-Control-Allow-Origin"] = "*"
             return response

main/xiaozhi-server/core/api/vision_handler.py

@@ -0,0 +1,180 @@
+import json
+import copy
+from aiohttp import web
+from config.logger import setup_logging
+from core.utils.util import get_vision_url, is_valid_image_file
+from core.utils.vllm import create_instance
+from config.config_loader import get_private_config_from_api
+from core.utils.auth import AuthToken
+import base64
+from typing import Tuple, Optional
+
+TAG = __name__
+
+# 设置最大文件大小为5MB
+MAX_FILE_SIZE = 5 * 1024 * 1024
+
+
+class VisionHandler:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        # 初始化认证工具
+        self.auth = AuthToken(config["server"]["auth_key"])
+
+    def _create_error_response(self, message: str) -> dict:
+        """创建统一的错误响应格式"""
+        return {"success": False, "message": message}
+
+    def _verify_auth_token(self, request) -> Tuple[bool, Optional[str]]:
+        """验证认证token"""
+        auth_header = request.headers.get("Authorization", "")
+        if not auth_header.startswith("Bearer "):
+            return False, None
+
+        token = auth_header[7:]  # 移除"Bearer "前缀
+        return self.auth.verify_token(token)
+
+    async def handle_post(self, request):
+        """处理 MCP Vision POST 请求"""
+        response = None  # 初始化response变量
+        try:
+            # 验证token
+            is_valid, token_device_id = self._verify_auth_token(request)
+            if not is_valid:
+                response = web.Response(
+                    text=json.dumps(
+                        self._create_error_response("无效的认证token或token已过期")
+                    ),
+                    content_type="application/json",
+                    status=401,
+                )
+                return response
+
+            # 获取请求头信息
+            device_id = request.headers.get("Device-Id", "")
+            client_id = request.headers.get("Client-Id", "")
+            if device_id != token_device_id:
+                raise ValueError("设备ID与token不匹配")
+            # 解析multipart/form-data请求
+            reader = await request.multipart()
+
+            # 读取question字段
+            question_field = await reader.next()
+            if question_field is None:
+                raise ValueError("缺少问题字段")
+            question = await question_field.text()
+            self.logger.bind(tag=TAG).debug(f"Question: {question}")
+
+            # 读取图片文件
+            image_field = await reader.next()
+            if image_field is None:
+                raise ValueError("缺少图片文件")
+
+            # 读取图片数据
+            image_data = await image_field.read()
+            if not image_data:
+                raise ValueError("图片数据为空")
+
+            # 检查文件大小
+            if len(image_data) > MAX_FILE_SIZE:
+                raise ValueError(
+                    f"图片大小超过限制，最大允许{MAX_FILE_SIZE/1024/1024}MB"
+                )
+
+            # 检查文件格式
+            if not is_valid_image_file(image_data):
+                raise ValueError(
+                    "不支持的文件格式，请上传有效的图片文件（支持JPEG、PNG、GIF、BMP、TIFF、WEBP格式）"
+                )
+
+            # 将图片转换为base64编码
+            image_base64 = base64.b64encode(image_data).decode("utf-8")
+
+            # 如果开启了智控台，则从智控台获取模型配置
+            current_config = copy.deepcopy(self.config)
+            read_config_from_api = current_config.get("read_config_from_api", False)
+            if read_config_from_api:
+                current_config = get_private_config_from_api(
+                    current_config,
+                    device_id,
+                    client_id,
+                )
+
+            select_vllm_module = current_config["selected_module"].get("VLLM")
+            if not select_vllm_module:
+                raise ValueError("您还未设置默认的视觉分析模块")
+
+            vllm_type = (
+                select_vllm_module
+                if "type" not in current_config["VLLM"][select_vllm_module]
+                else current_config["VLLM"][select_vllm_module]["type"]
+            )
+
+            if not vllm_type:
+                raise ValueError(f"无法找到VLLM模块对应的供应器{vllm_type}")
+
+            vllm = create_instance(
+                vllm_type, current_config["VLLM"][select_vllm_module]
+            )
+
+            result = vllm.response(question, image_base64)
+
+            return_json = {
+                "success": True,
+                "result": result,
+            }
+
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except ValueError as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response(str(e))
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
+            return_json = self._create_error_response("处理请求时发生错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            if response:
+                self._add_cors_headers(response)
+            return response
+
+    async def handle_get(self, request):
+        """处理 MCP Vision GET 请求"""
+        try:
+            vision_explain = get_vision_url(self.config)
+            if vision_explain and len(vision_explain) > 0 and "null" != vision_explain:
+                message = (
+                    f"MCP Vision 接口运行正常，视觉解释接口地址是：{vision_explain}"
+                )
+            else:
+                message = "MCP Vision 接口运行不正常，请打开data目录下的.config.yaml文件，找到【server.vision_explain】，设置好地址"
+
+            response = web.Response(text=message, content_type="text/plain")
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"MCP Vision GET请求异常: {e}")
+            return_json = self._create_error_response("服务器内部错误")
+            response = web.Response(
+                text=json.dumps(return_json, separators=(",", ":")),
+                content_type="application/json",
+            )
+        finally:
+            self._add_cors_headers(response)
+            return response
+
+    def _add_cors_headers(self, response):
+        """添加CORS头信息"""
+        response.headers["Access-Control-Allow-Headers"] = (
+            "client-id, content-type, device-id"
+        )
+        response.headers["Access-Control-Allow-Credentials"] = "true"
+        response.headers["Access-Control-Allow-Origin"] = "*"

main/xiaozhi-server/core/connection.py

@@ -35,7 +35,6 @@ from plugins_func.loadplugins import auto_import_modules
 from plugins_func.register import Action, ActionResponse
 from core.auth import AuthMiddleware, AuthenticationError
 from config.config_loader import get_private_config_from_api
-from core.handle.receiveAudioHandle import handleAudioMessage
 from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType
 from config.logger import setup_logging, build_module_string, update_module_string
 from config.manage_api_client import DeviceNotFoundException, DeviceBindException
@@ -81,6 +80,7 @@ class ConnectionHandler:
         self.welcome_msg = None
         self.max_output_size = 0
         self.chat_history_conf = 0
+        self.audio_format = "opus"
 
         # 客户端状态相关
         self.client_abort = False
@@ -117,7 +117,10 @@ class ConnectionHandler:
         self.client_voice_stop = False
 
         # asr相关变量
+        # 因为实际部署时可能会用到公共的本地ASR，不能把变量暴露给公共ASR
+        # 所以涉及到ASR的变量，需要在这里定义，属于connection的私有变量
         self.asr_audio = []
+        self.asr_audio_queue = queue.Queue()
 
         # llm相关变量
         self.llm_finish_task = True
@@ -146,7 +149,6 @@ class ConnectionHandler:
             int(self.config.get("close_connection_no_voice_time", 120)) + 60
         )  # 在原来第一道关闭的基础上加60秒，进行二道关闭
 
-        self.audio_format = "opus"
         # {"mcp":true} 表示启用MCP功能
         self.features = None
 
@@ -254,7 +256,11 @@ class ConnectionHandler:
         if isinstance(message, str):
             await handleTextMessage(self, message)
         elif isinstance(message, bytes):
-            await handleAudioMessage(self, message)
+            if self.vad is None:
+                return
+            if self.asr is None:
+                return
+            self.asr_audio_queue.put(message)
 
     async def handle_restart(self, message):
         """处理服务器重启请求"""
@@ -418,6 +424,16 @@ class ConnectionHandler:
         init_vad = check_vad_update(self.common_config, private_config)
         init_asr = check_asr_update(self.common_config, private_config)
 
+        if init_vad:
+            self.config["VAD"] = private_config["VAD"]
+            self.config["selected_module"]["VAD"] = private_config["selected_module"][
+                "VAD"
+            ]
+        if init_asr:
+            self.config["ASR"] = private_config["ASR"]
+            self.config["selected_module"]["ASR"] = private_config["selected_module"][
+                "ASR"
+            ]
         if private_config.get("TTS", None) is not None:
             init_tts = True
             self.config["TTS"] = private_config["TTS"]
@@ -439,9 +455,17 @@ class ConnectionHandler:
         if private_config.get("Intent", None) is not None:
             init_intent = True
             self.config["Intent"] = private_config["Intent"]
-            self.config["selected_module"]["Intent"] = private_config[
+            model_intent = private_config.get("selected_module", {}).get("Intent", {})
-                "selected_module"
+            self.config["selected_module"]["Intent"] = model_intent
-            ]["Intent"]
+            # 加载插件配置
+            if model_intent != "Intent_nointent":
+                plugin_from_server = private_config.get("plugins", {})
+                for plugin, config_str in plugin_from_server.items():
+                    plugin_from_server[plugin] = json.loads(config_str)
+                self.config["plugins"] = plugin_from_server
+                self.config["Intent"][self.config["selected_module"]["Intent"]][
+                    "functions"
+                ] = plugin_from_server.keys()
         if private_config.get("prompt", None) is not None:
             self.config["prompt"] = private_config["prompt"]
         if private_config.get("summaryMemory", None) is not None:
@@ -478,6 +502,8 @@ class ConnectionHandler:
             self.memory = modules["memory"]
 
     def _initialize_memory(self):
+        if self.memory is None:
+            return
         """初始化记忆模块"""
         self.memory.init_memory(
             role_id=self.device_id,
@@ -518,6 +544,8 @@ class ConnectionHandler:
                 self.logger.bind(tag=TAG).info("使用主LLM作为意图识别模型")
 
     def _initialize_intent(self):
+        if self.intent is None:
+            return
         self.intent_type = self.config["Intent"][
             self.config["selected_module"]["Intent"]
         ]["type"]
@@ -602,7 +630,7 @@ class ConnectionHandler:
             uuid_str = str(uuid.uuid4()).replace("-", "")
             self.sentence_id = uuid_str
 
-            if functions is not None:
+            if self.intent_type == "function_call" and functions is not None:
                 # 使用支持functions的streaming接口
                 llm_responses = self.llm.response_with_functions(
                     self.session_id,
@@ -629,7 +657,7 @@ class ConnectionHandler:
         for response in llm_responses:
             if self.client_abort:
                 break
-            if functions is not None:
+            if self.intent_type == "function_call" and functions is not None:
                 content, tools_call = response
                 if "content" in response:
                     content = response["content"]
@@ -641,7 +669,7 @@ class ConnectionHandler:
                     # print("content_arguments", content_arguments)
                     tool_call_flag = True
 
-                if tools_call is not None:
+                if tools_call is not None and len(tools_call) > 0:
                     tool_call_flag = True
                     if tools_call[0].id is not None:
                         function_id = tools_call[0].id
@@ -708,15 +736,24 @@ class ConnectionHandler:
                 # 处理Server端MCP工具调用
                 if self.mcp_manager.is_mcp_tool(function_name):
                     result = self._handle_mcp_tool_call(function_call_data)
-                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(function_name):
+                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(
+                    function_name
+                ):
                     # 如果是小智端MCP工具调用
                     self.logger.bind(tag=TAG).debug(
                         f"调用小智端MCP工具: {function_name}, 参数: {function_arguments}"
                     )
                     try:
-                        result = asyncio.run_coroutine_threadsafe(call_mcp_tool(self, self.mcp_client, function_name, function_arguments), self.loop).result()
+                        result = asyncio.run_coroutine_threadsafe(
+                            call_mcp_tool(
+                                self, self.mcp_client, function_name, function_arguments
+                            ),
+                            self.loop,
+                        ).result()
                         self.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
-                        result = ActionResponse(action=Action.REQLLM, result=result, response="")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result=result, response=""
+                        )
                     except Exception as e:
                         self.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
                         result = ActionResponse(

main/xiaozhi-server/core/handle/abortHandle.py

@@ -1,6 +1,4 @@
 import json
-import queue
-from config.logger import setup_logging
 
 TAG = __name__
 

main/xiaozhi-server/core/handle/functionHandler.py

@@ -21,26 +21,12 @@ class FunctionHandler:
         self.register_nessary_functions()
         self.register_config_functions()
         self.functions_desc = self.function_registry.get_all_function_desc()
-        func_names = self.current_support_functions()
-        self.modify_plugin_loader_des(func_names)
         self.finish_init = True
     
-    def modify_plugin_loader_des(self, func_names):
-        if "plugin_loader" not in func_names:
-            return
-        # 可编辑的列表中去掉plugin_loader
-        surport_plugins = [func for func in func_names if func != "plugin_loader"]
-        func_names = ",".join(surport_plugins)
-        for function_desc in self.functions_desc:
-            if function_desc["function"]["name"] == "plugin_loader":
-                function_desc["function"]["description"] = function_desc["function"][
-                    "description"
-                ].replace("[plugins]", func_names)
-                break
-
     def upload_functions_desc(self):
         self.functions_desc = self.function_registry.get_all_function_desc()
 
+
     def current_support_functions(self):
         func_names = []
         for func in self.functions_desc:
@@ -58,10 +44,8 @@ class FunctionHandler:
     def register_nessary_functions(self):
         """注册必要的函数"""
         self.function_registry.register_function("handle_exit_intent")
-        self.function_registry.register_function("plugin_loader")
         self.function_registry.register_function("get_time")
         self.function_registry.register_function("get_lunar")
-        # self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
 
     def register_config_functions(self):
         """注册配置中的函数,可以不同客户端使用不同的配置"""

main/xiaozhi-server/core/handle/helloHandle.py

@@ -1,26 +1,32 @@
-import os
 import time
 import json
 import random
-import shutil
 import asyncio
-from core.handle.mcpHandle import MCPClient, send_mcp_initialize_message, send_mcp_tools_list_request
+from core.utils.dialogue import Message
-from core.handle.sendAudioHandle import send_stt_message
+from core.utils.util import audio_to_data
-from core.utils.util import remove_punctuation_and_length
+from core.handle.sendAudioHandle import sendAudioMessage, send_stt_message
-from core.providers.tts.dto.dto import ContentType, InterfaceType
+from core.utils.util import remove_punctuation_and_length, opus_datas_to_wav_bytes
-
+from core.providers.tts.dto.dto import ContentType, SentenceType
+from core.handle.mcpHandle import (
+    MCPClient,
+    send_mcp_initialize_message,
+    send_mcp_tools_list_request,
+)
+from core.utils.wakeup_word import WakeupWordsConfig
 
 TAG = __name__
 
 WAKEUP_CONFIG = {
-    "dir": "config/assets/",
+    "refresh_time": 5,
-    "file_name": "wakeup_words",
+    "words": ["你好", "你好啊", "嘿，你好", "嗨"],
-    "create_time": time.time(),
-    "refresh_time": 10,
-    "words": ["你好小智", "你好啊小智", "小智你好", "小智"],
-    "text": "",
 }
 
+# 创建全局的唤醒词配置管理器
+wakeup_words_config = WakeupWordsConfig()
+
+# 用于防止并发调用wakeupWordsResponse的锁
+_wakeup_response_lock = asyncio.Lock()
+
 
 async def handleHelloMessage(conn, msg_json):
     """处理hello消息"""
@@ -29,8 +35,6 @@ async def handleHelloMessage(conn, msg_json):
         format = audio_params.get("format")
         conn.logger.bind(tag=TAG).info(f"客户端音频格式: {format}")
         conn.audio_format = format
-        if conn.asr is not None:
-            conn.asr.set_audio_format(format)
         conn.welcome_msg["audio_params"] = audio_params
     features = msg_json.get("features")
     if features:
@@ -51,81 +55,87 @@ async def checkWakeupWords(conn, text):
     enable_wakeup_words_response_cache = conn.config[
         "enable_wakeup_words_response_cache"
     ]
-    """是否用的是非流式tts"""
+
-    if conn.tts and conn.tts.interface_type != InterfaceType.NON_STREAM:
+    if not enable_wakeup_words_response_cache or not conn.tts:
         return False
 
-    """是否开启唤醒词加速"""
-    if not enable_wakeup_words_response_cache:
-        return False
-    """检查是否是唤醒词"""
     _, filtered_text = remove_punctuation_and_length(text)
-    if filtered_text in conn.config.get("wakeup_words"):
+    if filtered_text not in conn.config.get("wakeup_words"):
-        await send_stt_message(conn, text)
+        return False
 
-        file = getWakeupWordFile(WAKEUP_CONFIG["file_name"])
+    conn.just_woken_up = True
-        if file is None:
+    await send_stt_message(conn, text)
+
+    # 获取当前音色
+    voice = getattr(conn.tts, "voice", "default")
+    if not voice:
+        voice = "default"
+
+    # 获取唤醒词回复配置
+    response = wakeup_words_config.get_wakeup_response(voice)
+    if not response or not response.get("file_path"):
+        response = {
+            "voice": "default",
+            "file_path": "config/assets/wakeup_words.wav",
+            "time": 0,
+            "text": "哈啰啊，我是小智啦，声音好听的台湾女孩一枚，超开心认识你耶，最近在忙啥，别忘了给我来点有趣的料哦，我超爱听八卦的啦",
+        }
+
+    # 播放唤醒词回复
+    conn.client_abort = False
+    opus_packets, _ = audio_to_data(response.get("file_path"))
+
+    conn.logger.bind(tag=TAG).info(f"播放唤醒词回复: {response.get('text')}")
+    await sendAudioMessage(conn, SentenceType.FIRST, opus_packets, response.get("text"))
+    await sendAudioMessage(conn, SentenceType.LAST, [], None)
+
+    # 补充对话
+    conn.dialogue.put(Message(role="assistant", content=response.get("text")))
+
+    # 检查是否需要更新唤醒词回复
+    if time.time() - response.get("time", 0) > WAKEUP_CONFIG["refresh_time"]:
+        if not _wakeup_response_lock.locked():
             asyncio.create_task(wakeupWordsResponse(conn))
-            return False
+    return True
-        text_hello = WAKEUP_CONFIG["text"]
-        if not text_hello:
-            text_hello = text
-        conn.tts.tts_one_sentence(
-            conn, ContentType.FILE, content_file=file, content_detail=text_hello
-        )
-        if time.time() - WAKEUP_CONFIG["create_time"] > WAKEUP_CONFIG["refresh_time"]:
-            asyncio.create_task(wakeupWordsResponse(conn))
-        return True
-    return False
-
-
-def getWakeupWordFile(file_name):
-    for file in os.listdir(WAKEUP_CONFIG["dir"]):
-        if file.startswith("my_" + file_name):
-            """避免缓存文件是一个空文件"""
-            if os.stat(f"config/assets/{file}").st_size > (15 * 1024):
-                return f"config/assets/{file}"
-
-    """查找config/assets/目录下名称为wakeup_words的文件"""
-    for file in os.listdir(WAKEUP_CONFIG["dir"]):
-        if file.startswith(file_name):
-            return f"config/assets/{file}"
-    return None
 
 
 async def wakeupWordsResponse(conn):
-    wait_max_time = 5
+    if not conn.tts or not conn.llm or not conn.llm.response_no_stream:
-    while conn.llm is None or not conn.llm.response_no_stream:
+        return
-        await asyncio.sleep(1)
+
-        wait_max_time -= 1
+    try:
-        if wait_max_time <= 0:
+        # 尝试获取锁，如果获取不到就返回
-            conn.logger.bind(tag=TAG).error("连接对象没有llm")
+        if not await _wakeup_response_lock.acquire():
             return
 
-    """唤醒词响应"""
+        # 生成唤醒词回复
-    wakeup_word = random.choice(WAKEUP_CONFIG["words"])
+        wakeup_word = random.choice(WAKEUP_CONFIG["words"])
-    question = (
+        question = (
-        "此刻用户正在和你说```"
+            "此刻用户正在和你说```"
-        + wakeup_word
+            + wakeup_word
-        + "```。\n请你根据以上用户的内容，进行简短回复，文字内容控制在15个字以内。\n"
+            + "```。\n请你根据以上用户的内容进行20-30字回复。要符合系统设置的角色情感和态度，不要像机器人一样说话。\n"
-        + "请勿对这条内容本身进行任何解释和回应，仅返回对用户的内容的回复。"
+            + "请勿对这条内容本身进行任何解释和回应，请勿返回表情符号，仅返回对用户的内容的回复。"
-    )
-    result = conn.llm.response_no_stream(conn.config["prompt"], question)
-    if result is None or result == "":
-        return
-    tts_file = await asyncio.to_thread(conn.tts.to_tts, result)
-
-    if tts_file is not None and os.path.exists(tts_file):
-        file_type = os.path.splitext(tts_file)[1]
-        if file_type:
-            file_type = file_type.lstrip(".")
-        old_file = getWakeupWordFile("my_" + WAKEUP_CONFIG["file_name"])
-        if old_file is not None:
-            os.remove(old_file)
-        """将文件挪到"wakeup_words.mp3"""
-        shutil.move(
-            tts_file,
-            WAKEUP_CONFIG["dir"] + "my_" + WAKEUP_CONFIG["file_name"] + "." + file_type,
         )
-        WAKEUP_CONFIG["create_time"] = time.time()
+
-        WAKEUP_CONFIG["text"] = result
+        result = conn.llm.response_no_stream(conn.config["prompt"], question)
+        if not result or len(result) == 0:
+            return
+
+        # 生成TTS音频
+        tts_result = await asyncio.to_thread(conn.tts.to_tts, result)
+        if not tts_result:
+            return
+
+        # 获取当前音色
+        voice = getattr(conn.tts, "voice", "default")
+
+        wav_bytes = opus_datas_to_wav_bytes(tts_result, sample_rate=16000)
+        file_path = wakeup_words_config.generate_file_path(voice)
+        with open(file_path, "wb") as f:
+            f.write(wav_bytes)
+        # 更新配置
+        wakeup_words_config.update_wakeup_response(voice, file_path, result)
+    finally:
+        # 确保在任何情况下都释放锁
+        if _wakeup_response_lock.locked():
+            _wakeup_response_lock.release()

main/xiaozhi-server/core/handle/intentHandler.py

@@ -1,11 +1,13 @@
 import json
+import asyncio
 import uuid
 from core.handle.sendAudioHandle import send_stt_message
 from core.handle.helloHandle import checkWakeupWords
 from core.utils.util import remove_punctuation_and_length
 from core.providers.tts.dto.dto import ContentType
 from core.utils.dialogue import Message
-from plugins_func.register import Action
+from core.handle.mcpHandle import call_mcp_tool
+from plugins_func.register import Action, ActionResponse
 from loguru import logger
 
 TAG = __name__
@@ -82,9 +84,11 @@ async def process_intent_result(conn, intent_result, original_text):
                 if not funcItem:
                     conn.func_handler.function_registry.register_function("play_music")
 
-            function_args = None
+            function_args = {}
             if "arguments" in intent_data["function_call"]:
                 function_args = intent_data["function_call"]["arguments"]
+                if function_args is None:
+                    function_args = {}
             # 确保参数是字符串格式的JSON
             if isinstance(function_args, dict):
                 function_args = json.dumps(function_args)
@@ -96,14 +100,43 @@ async def process_intent_result(conn, intent_result, original_text):
             }
 
             await send_stt_message(conn, original_text)
+            conn.client_abort = False
 
             # 使用executor执行函数调用和结果处理
             def process_function_call():
                 conn.dialogue.put(Message(role="user", content=original_text))
-                result = conn.func_handler.handle_llm_function_call(
+
-                    conn, function_call_data
+                # 处理Server端MCP工具调用
-                )
+                if conn.mcp_manager.is_mcp_tool(function_name):
-                logger.bind(tag=TAG).debug(f"检测到Action : {result.action}")
+                    result = conn._handle_mcp_tool_call(function_call_data)
+                elif hasattr(conn, "mcp_client") and conn.mcp_client.has_tool(
+                    function_name
+                ):
+                    # 如果是小智端MCP工具调用
+                    conn.logger.bind(tag=TAG).debug(
+                        f"调用小智端MCP工具: {function_name}, 参数: {function_args}"
+                    )
+                    try:
+                        result = asyncio.run_coroutine_threadsafe(
+                            call_mcp_tool(
+                                conn, conn.mcp_client, function_name, function_args
+                            ),
+                            conn.loop,
+                        ).result()
+                        conn.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result=result, response=""
+                        )
+                    except Exception as e:
+                        conn.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
+                        result = ActionResponse(
+                            action=Action.REQLLM, result="MCP工具调用失败", response=""
+                        )
+                else:
+                    # 处理系统函数
+                    result = conn.func_handler.handle_llm_function_call(
+                        conn, function_call_data
+                    )
 
                 if result:
                     if result.action == Action.RESPONSE:  # 直接回复前端

main/xiaozhi-server/core/handle/iotHandle.py

@@ -339,6 +339,7 @@ async def handleIotDescriptors(conn, descriptors):
     # 如果注册了新函数，更新function描述列表
     if functions_changed and hasattr(conn, "func_handler"):
         conn.func_handler.upload_functions_desc()
+
         func_names = conn.func_handler.current_support_functions()
         conn.logger.bind(tag=TAG).info(f"设备类型: {type_id}")
         conn.logger.bind(tag=TAG).info(

main/xiaozhi-server/core/handle/mcpHandle.py

@@ -1,6 +1,8 @@
 import json
 import asyncio
 from concurrent.futures import Future
+from core.utils.util import get_vision_url, sanitize_tool_name
+from core.utils.auth import AuthToken
 
 TAG = __name__
 
@@ -9,7 +11,8 @@ class MCPClient:
     """MCPClient，用于管理MCP状态和工具"""
 
     def __init__(self):
-        self.tools = {}  # Dictionary for O(1) lookup
+        self.tools = {}  # sanitized_name -> tool_data
+        self.name_mapping = {}
         self.ready = False
         self.call_results = {}  # To store Futures for tool call responses
         self.next_id = 1
@@ -28,7 +31,7 @@ class MCPClient:
         result = []
         for tool_name, tool_data in self.tools.items():
             function_def = {
-                "name": tool_data["name"],
+                "name": tool_name,
                 "description": tool_data["description"],
                 "parameters": {
                     "type": tool_data["inputSchema"].get("type", "object"),
@@ -51,7 +54,9 @@ class MCPClient:
 
     async def add_tool(self, tool_data: dict):
         async with self.lock:
-            self.tools[tool_data["name"]] = tool_data
+            sanitized_name = sanitize_tool_name(tool_data["name"])
+            self.tools[sanitized_name] = tool_data
+            self.name_mapping[sanitized_name] = tool_data["name"]
             self._cached_available_tools = (
                 None  # Invalidate the cache when a tool is added
             )
@@ -131,9 +136,6 @@ async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
                 conn.logger.bind(tag=TAG).info(
                     f"客户端MCP服务器信息: name={name}, version={version}"
                 )
-            await send_mcp_tools_list_request(
-                conn
-            )  # After initialization, request tool list
             return
 
         elif msg_id == 2:  # mcpToolsListID
@@ -172,6 +174,20 @@ async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
                     await mcp_client.add_tool(new_tool)
                     conn.logger.bind(tag=TAG).debug(f"客户端工具 #{i+1}: {name}")
 
+                # 替换所有工具描述中的工具名称
+                for tool_data in mcp_client.tools.values():
+                    if "description" in tool_data:
+                        description = tool_data["description"]
+                        # 遍历所有工具名称进行替换
+                        for (
+                            sanitized_name,
+                            original_name,
+                        ) in mcp_client.name_mapping.items():
+                            description = description.replace(
+                                original_name, sanitized_name
+                            )
+                        tool_data["description"] = description
+
                 next_cursor = result.get("nextCursor", "")
                 if next_cursor:
                     conn.logger.bind(tag=TAG).info(
@@ -205,6 +221,18 @@ async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
 
 async def send_mcp_initialize_message(conn):
     """发送MCP初始化消息"""
+
+    vision_url = get_vision_url(conn.config)
+
+    # 密钥生成token
+    auth = AuthToken(conn.config["server"]["auth_key"])
+    token = auth.generate_token(conn.headers.get("device-id"))
+
+    vision = {
+        "url": vision_url,
+        "token": token,
+    }
+
     payload = {
         "jsonrpc": "2.0",
         "id": 1,  # mcpInitializeID
@@ -214,6 +242,7 @@ async def send_mcp_initialize_message(conn):
             "capabilities": {
                 "roots": {"listChanged": True},
                 "sampling": {},
+                "vision": vision,
             },
             "clientInfo": {
                 "name": "XiaozhiClient",
@@ -316,15 +345,16 @@ async def call_mcp_tool(
             raise ValueError(f"参数处理失败: {str(e)}")
         raise e
 
+    actual_name = mcp_client.name_mapping.get(tool_name, tool_name)
     payload = {
         "jsonrpc": "2.0",
         "id": tool_call_id,
         "method": "tools/call",
-        "params": {"name": tool_name, "arguments": arguments},
+        "params": {"name": actual_name, "arguments": arguments},
     }
 
     conn.logger.bind(tag=TAG).info(
-        f"发送客户端mcp工具调用请求: {tool_name}，参数: {args}"
+        f"发送客户端mcp工具调用请求: {actual_name}，参数: {args}"
     )
     await send_mcp_message(conn, payload)
 
@@ -332,7 +362,7 @@ async def call_mcp_tool(
         # Wait for response or timeout
         raw_result = await asyncio.wait_for(result_future, timeout=timeout)
         conn.logger.bind(tag=TAG).info(
-            f"客户端mcp工具调用 {tool_name} 成功，原始结果: {raw_result}"
+            f"客户端mcp工具调用 {actual_name} 成功，原始结果: {raw_result}"
         )
 
         if isinstance(raw_result, dict):

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -3,6 +3,7 @@ from core.handle.intentHandler import handle_user_intent
 from core.utils.output_counter import check_device_output_limit
 from core.handle.abortHandle import handleAbortMessage
 import time
+import asyncio
 from core.handle.sendAudioHandle import SentenceType
 from core.utils.util import audio_to_data
 
@@ -10,21 +11,30 @@ TAG = __name__
 
 
 async def handleAudioMessage(conn, audio):
-    if conn.vad is None:
-        conn.logger.bind(tag=TAG).warning("VAD模块未初始化，继续等待")
-        return
-    if conn.asr is None or not hasattr(conn.asr, "conn") or conn.asr.conn is None:
-        conn.logger.bind(tag=TAG).warning("ASR模块未初始化或通道未就绪，继续等待")
-        return
     # 当前片段是否有人说话
     have_voice = conn.vad.is_vad(conn, audio)
+    # 如果设备刚刚被唤醒，短暂忽略VAD检测
+    if have_voice and hasattr(conn, "just_woken_up") and conn.just_woken_up:
+        have_voice = False
+        # 设置一个短暂延迟后恢复VAD检测
+        conn.asr_audio.clear()
+        if not hasattr(conn, "vad_resume_task") or conn.vad_resume_task.done():
+            conn.vad_resume_task = asyncio.create_task(resume_vad_detection(conn))
+        return
+
     if have_voice:
         if conn.client_is_speaking:
             await handleAbortMessage(conn)
     # 设备长时间空闲检测，用于say goodbye
     await no_voice_close_connect(conn, have_voice)
     # 接收音频
-    await conn.asr.receive_audio(audio, have_voice)
+    await conn.asr.receive_audio(conn, audio, have_voice)
+
+
+async def resume_vad_detection(conn):
+    # 等待2秒后恢复VAD检测
+    await asyncio.sleep(1)
+    conn.just_woken_up = False
 
 
 async def startToChat(conn, text):

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -34,6 +34,7 @@ emoji_map = {
 
 async def sendAudioMessage(conn, sentenceType, audios, text):
     # 发送句子开始消息
+    conn.logger.bind(tag=TAG).info(f"发送音频消息: {sentenceType}, {text}")
     if text is not None:
         emotion = analyze_emotion(text)
         emoji = emoji_map.get(emotion, "🙂")  # 默认使用笑脸
@@ -89,8 +90,7 @@ async def sendAudio(conn, audios, pre_buffer=True):
     # 播放剩余音频帧
     for opus_packet in remaining_audios:
         if conn.client_abort:
-            conn.client_abort = False
+            break
-            return
 
         # 每分钟重置一次计时器
         if time.perf_counter() - last_reset_time > 60:

main/xiaozhi-server/core/handle/textHandle.py

@@ -61,6 +61,7 @@ async def handleTextMessage(conn, message):
                         await send_tts_message(conn, "stop", None)
                         conn.client_is_speaking = False
                     elif is_wakeup_words:
+                        conn.just_woken_up = True
                         # 上报纯文字数据（复用ASR上报功能，但不提供音频数据）
                         enqueue_asr_report(conn, "嘿，你好呀", [])
                         await startToChat(conn, "嘿，你好呀")
@@ -78,7 +79,9 @@ async def handleTextMessage(conn, message):
         elif msg_json["type"] == "mcp":
             conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message}")
             if "payload" in msg_json:
-                asyncio.create_task(handle_mcp_message(conn, conn.mcp_client, msg_json["payload"]))
+                asyncio.create_task(
+                    handle_mcp_message(conn, conn.mcp_client, msg_json["payload"])
+                )
         elif msg_json["type"] == "server":
             # 记录日志时过滤敏感信息
             conn.logger.bind(tag=TAG).info(

main/xiaozhi-server/core/http_server.py

@@ -0,0 +1,71 @@
+import asyncio
+from aiohttp import web
+from config.logger import setup_logging
+from core.api.ota_handler import OTAHandler
+from core.api.vision_handler import VisionHandler
+
+TAG = __name__
+
+
+class SimpleHttpServer:
+    def __init__(self, config: dict):
+        self.config = config
+        self.logger = setup_logging()
+        self.ota_handler = OTAHandler(config)
+        self.vision_handler = VisionHandler(config)
+
+    def _get_websocket_url(self, local_ip: str, port: int) -> str:
+        """获取websocket地址
+
+        Args:
+            local_ip: 本地IP地址
+            port: 端口号
+
+        Returns:
+            str: websocket地址
+        """
+        server_config = self.config["server"]
+        websocket_config = server_config.get("websocket")
+
+        if websocket_config and "你" not in websocket_config:
+            return websocket_config
+        else:
+            return f"ws://{local_ip}:{port}/xiaozhi/v1/"
+
+    async def start(self):
+        server_config = self.config["server"]
+        host = server_config.get("ip", "0.0.0.0")
+        port = int(server_config.get("http_port", 8003))
+
+        if port:
+            app = web.Application()
+
+            read_config_from_api = server_config.get("read_config_from_api", False)
+
+            if not read_config_from_api:
+                # 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
+                app.add_routes(
+                    [
+                        web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
+                        web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
+                        web.options("/xiaozhi/ota/", self.ota_handler.handle_post),
+                    ]
+                )
+            # 添加路由
+            app.add_routes(
+                [
+                    web.get("/mcp/vision/explain", self.vision_handler.handle_get),
+                    web.post("/mcp/vision/explain", self.vision_handler.handle_post),
+                    web.options("/mcp/vision/explain", self.vision_handler.handle_post),
+                ]
+            )
+
+            # 运行服务
+            runner = web.AppRunner(app)
+            await runner.setup()
+            site = web.TCPSite(runner, host, port)
+            await site.start()
+
+            # 保持服务运行
+            while True:
+                await asyncio.sleep(3600)  # 每隔 1 小时检查一次

main/xiaozhi-server/core/mcp/MCPClient.py

@@ -9,6 +9,7 @@ from mcp import ClientSession, StdioServerParameters
 from mcp.client.stdio import stdio_client
 from mcp.client.sse import sse_client
 from config.logger import setup_logging
+from core.utils.util import sanitize_tool_name
 
 TAG = __name__
 
@@ -23,7 +24,9 @@ class MCPClient:
         self._shutdown_evt = asyncio.Event()
 
         self.session: Optional[ClientSession] = None
-        self.tools: List = []
+        self.tools: List = []  # original tool objects
+        self.tools_dict: Dict[str, Any] = {}
+        self.name_mapping: Dict[str, str] = {}
 
     async def initialize(self):
         if self._worker_task:
@@ -32,7 +35,7 @@ class MCPClient:
         await self._ready_evt.wait()
 
         self.logger.bind(tag=TAG).info(
-            f"Connected, tools = {[t.name for t in self.tools]}"
+            f"Connected, tools = {[name for name in self.name_mapping.values()]}"
         )
 
     async def cleanup(self):
@@ -48,27 +51,28 @@ class MCPClient:
             self._worker_task = None
 
     def has_tool(self, name: str) -> bool:
-        return any(t.name == name for t in self.tools)
+        return name in self.tools_dict
 
     def get_available_tools(self):
         return [
             {
                 "type": "function",
                 "function": {
-                    "name": t.name,
+                    "name": name,
-                    "description": t.description,
+                    "description": tool.description,
-                    "parameters": t.inputSchema,
+                    "parameters": tool.inputSchema,
                 },
             }
-            for t in self.tools
+            for name, tool in self.tools_dict.items()
         ]
 
     async def call_tool(self, name: str, args: dict):
         if not self.session:
             raise RuntimeError("MCPClient not initialized")
 
+        real_name = self.name_mapping.get(name, name)
         loop = self._worker_task.get_loop()
-        coro = self.session.call_tool(name, args)
+        coro = self.session.call_tool(real_name, args)
 
         if loop is asyncio.get_running_loop():
             return await coro
@@ -92,11 +96,21 @@ class MCPClient:
                         args=self.config.get("args", []),
                         env=env,
                     )
-                    stdio_r, stdio_w = await stack.enter_async_context(stdio_client(params))
+                    stdio_r, stdio_w = await stack.enter_async_context(
+                        stdio_client(params)
+                    )
                     read_stream, write_stream = stdio_r, stdio_w
                 # 建立SSEClient
                 elif "url" in self.config:
-                    sse_r, sse_w = await stack.enter_async_context(sse_client(self.config["url"]))
+                    if "API_ACCESS_TOKEN" in self.config:
+                        headers = {
+                            "Authorization": f"Bearer {self.config['API_ACCESS_TOKEN']}"
+                        }
+                    else:
+                        headers = {}
+                    sse_r, sse_w = await stack.enter_async_context(
+                        sse_client(self.config["url"], headers=headers)
+                    )
                     read_stream, write_stream = sse_r, sse_w
 
                 else:
@@ -113,6 +127,10 @@ class MCPClient:
 
                 # 获取工具
                 self.tools = (await self.session.list_tools()).tools
+                for t in self.tools:
+                    sanitized = sanitize_tool_name(t.name)
+                    self.tools_dict[sanitized] = t
+                    self.name_mapping[sanitized] = t.name
 
                 self._ready_evt.set()
 

main/xiaozhi-server/core/providers/asr/aliyun.py

@@ -213,7 +213,7 @@ class ASRProvider(ASRProviderBase):
             return None
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if self._is_token_expired():
@@ -223,7 +223,7 @@ class ASRProvider(ASRProviderBase):
         file_path = None
         try:
             # 解码Opus为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/baidu.py

@@ -30,7 +30,7 @@ class ASRProvider(ASRProviderBase):
         os.makedirs(self.output_dir, exist_ok=True)
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if not opus_data:
@@ -45,7 +45,7 @@ class ASRProvider(ASRProviderBase):
                 return None, file_path
 
             # 将Opus音频数据解码为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/base.py

@@ -1,15 +1,19 @@
 import os
-import time
+import wave
 import copy
 import uuid
-import wave
+import queue
+import asyncio
+import traceback
+import threading
 import opuslib_next
 from abc import ABC, abstractmethod
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
-from core.utils.util import remove_punctuation_and_length
-from core.handle.reportHandle import enqueue_asr_report
 from core.handle.receiveAudioHandle import startToChat
+from core.handle.reportHandle import enqueue_asr_report
+from core.utils.util import remove_punctuation_and_length
+from core.handle.receiveAudioHandle import handleAudioMessage
 
 TAG = __name__
 logger = setup_logging()
@@ -17,53 +21,75 @@ logger = setup_logging()
 
 class ASRProviderBase(ABC):
     def __init__(self):
-        self.audio_format = "opus"
+        pass
-        self.conn = None
 
     # 打开音频通道
     # 这里默认是非流式的处理方式
     # 流式处理方式请在子类中重写
     async def open_audio_channels(self, conn):
-        self.conn = conn
+        # tts 消化线程
+        conn.asr_priority_thread = threading.Thread(
+            target=self.asr_text_priority_thread, args=(conn,), daemon=True
+        )
+        conn.asr_priority_thread.start()
+
+    # 有序处理ASR音频
+    def asr_text_priority_thread(self, conn):
+        while not conn.stop_event.is_set():
+            try:
+                message = conn.asr_audio_queue.get(timeout=1)
+                future = asyncio.run_coroutine_threadsafe(
+                    handleAudioMessage(conn, message),
+                    conn.loop,
+                )
+                future.result()
+            except queue.Empty:
+                continue
+            except Exception as e:
+                logger.bind(tag=TAG).error(
+                    f"处理ASR文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
+                )
+                continue
 
     # 接收音频
     # 这里默认是非流式的处理方式
     # 流式处理方式请在子类中重写
-    async def receive_audio(self, audio, audio_have_voice):
+    async def receive_audio(self, conn, audio, audio_have_voice):
-        if (
+        if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
-            self.conn.client_listen_mode == "auto"
-            or self.conn.client_listen_mode == "realtime"
-        ):
             have_voice = audio_have_voice
         else:
-            have_voice = self.conn.client_have_voice
+            have_voice = conn.client_have_voice
         # 如果本次没有声音，本段也没声音，就把声音丢弃了
-        self.conn.asr_audio.append(audio)
+        conn.asr_audio.append(audio)
-        if have_voice == False and self.conn.client_have_voice == False:
+        if have_voice == False and conn.client_have_voice == False:
-            self.conn.asr_audio = self.conn.asr_audio[-10:]
+            conn.asr_audio = conn.asr_audio[-10:]
             return
 
         # 如果本段有声音，且已经停止了
-        if self.conn.client_voice_stop:
+        if conn.client_voice_stop:
-            asr_audio_task = copy.deepcopy(self.conn.asr_audio)
+            asr_audio_task = copy.deepcopy(conn.asr_audio)
-            self.conn.asr_audio.clear()
+            conn.asr_audio.clear()
 
             # 音频太短了，无法识别
-            self.conn.reset_vad_states()
+            conn.reset_vad_states()
             if len(asr_audio_task) > 15:
-                await self.handle_voice_stop(asr_audio_task)
+                await self.handle_voice_stop(conn, asr_audio_task)
 
     # 处理语音停止
-    async def handle_voice_stop(self, asr_audio_task):
+    async def handle_voice_stop(self, conn, asr_audio_task):
         raw_text, _ = await self.speech_to_text(
-            asr_audio_task, self.conn.session_id
+            asr_audio_task, conn.session_id, conn.audio_format
         )  # 确保ASR模块返回原始文本
-        self.conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+        conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
         text_len, _ = remove_punctuation_and_length(raw_text)
+        self.stop_ws_connection()
         if text_len > 0:
             # 使用自定义模块进行上报
-            await startToChat(self.conn, raw_text)
+            await startToChat(conn, raw_text)
-            enqueue_asr_report(self.conn, raw_text, asr_audio_task)
+            enqueue_asr_report(conn, raw_text, asr_audio_task)
+
+    def stop_ws_connection(self):
+        pass
 
     def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
         """PCM数据保存为WAV文件"""
@@ -81,15 +107,11 @@ class ASRProviderBase(ABC):
 
     @abstractmethod
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         pass
 
-    def set_audio_format(self, format: str) -> None:
-        """设置音频格式"""
-        self.audio_format = format
-
     @staticmethod
     def decode_opus(opus_data: List[bytes]) -> bytes:
         """将Opus音频数据解码为PCM数据"""

main/xiaozhi-server/core/providers/asr/doubao.py

@@ -1,534 +1,271 @@
+import time
+import os
+import uuid
 import json
 import gzip
-import uuid
-import asyncio
 import websockets
-import opuslib_next
-from core.providers.asr.base import ASRProviderBase
 from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.providers.asr.base import ASRProviderBase
 from core.providers.asr.dto.dto import InterfaceType
-import threading
+
 
 TAG = __name__
 logger = setup_logging()
 
 CLIENT_FULL_REQUEST = 0b0001
 CLIENT_AUDIO_ONLY_REQUEST = 0b0010
+
+NO_SEQUENCE = 0b0000
+NEG_SEQUENCE = 0b0010
+
 SERVER_FULL_RESPONSE = 0b1001
 SERVER_ACK = 0b1011
 SERVER_ERROR_RESPONSE = 0b1111
-NO_SEQUENCE = 0b0000
+
-NEG_SEQUENCE = 0b0010
+NO_SERIALIZATION = 0b0000
-JSON_SERIALIZATION = 0b0001
+JSON = 0b0001
-GZIP_COMPRESSION = 0b0001
+THRIFT = 0b0011
-PROTOCOL_VERSION = 0b0001
+CUSTOM_TYPE = 0b1111
+NO_COMPRESSION = 0b0000
+GZIP = 0b0001
+CUSTOM_COMPRESSION = 0b1111
+
+
+def parse_response(res):
+    """
+    protocol_version(4 bits), header_size(4 bits),
+    message_type(4 bits), message_type_specific_flags(4 bits)
+    serialization_method(4 bits) message_compression(4 bits)
+    reserved （8bits) 保留字段
+    header_extensions 扩展头(大小等于 8 * 4 * (header_size - 1) )
+    payload 类似与http 请求体
+    """
+    protocol_version = res[0] >> 4
+    header_size = res[0] & 0x0F
+    message_type = res[1] >> 4
+    message_type_specific_flags = res[1] & 0x0F
+    serialization_method = res[2] >> 4
+    message_compression = res[2] & 0x0F
+    reserved = res[3]
+    header_extensions = res[4 : header_size * 4]
+    payload = res[header_size * 4 :]
+    result = {}
+    payload_msg = None
+    payload_size = 0
+    if message_type == SERVER_FULL_RESPONSE:
+        payload_size = int.from_bytes(payload[:4], "big", signed=True)
+        payload_msg = payload[4:]
+    elif message_type == SERVER_ACK:
+        seq = int.from_bytes(payload[:4], "big", signed=True)
+        result["seq"] = seq
+        if len(payload) >= 8:
+            payload_size = int.from_bytes(payload[4:8], "big", signed=False)
+            payload_msg = payload[8:]
+    elif message_type == SERVER_ERROR_RESPONSE:
+        code = int.from_bytes(payload[:4], "big", signed=False)
+        result["code"] = code
+        payload_size = int.from_bytes(payload[4:8], "big", signed=False)
+        payload_msg = payload[8:]
+    if payload_msg is None:
+        return result
+    if message_compression == GZIP:
+        payload_msg = gzip.decompress(payload_msg)
+    if serialization_method == JSON:
+        payload_msg = json.loads(str(payload_msg, "utf-8"))
+    elif serialization_method != NO_SERIALIZATION:
+        payload_msg = str(payload_msg, "utf-8")
+    result["payload_msg"] = payload_msg
+    result["payload_size"] = payload_size
+    return result
 
 
 class ASRProvider(ASRProviderBase):
-    def __init__(self, config, delete_audio_file):
+    def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
-        self.interface_type = InterfaceType.STREAM
+        self.interface_type = InterfaceType.NON_STREAM
-        self.config = config
+        self.appid = config.get("appid")
-        self.text = ""
-        self.max_retries = 3
-        self.retry_delay = 2  # 重试延迟秒数
-        self.recv_lock = asyncio.Lock()  # 添加接收锁
-        self.reconnect_lock = asyncio.Lock()  # 添加重连锁
-        self.last_reconnect_time = 0  # 上次重连时间
-        self.reconnect_cooldown = 1  # 增加重连冷却时间到10秒
-        self.reconnect_count = 0  # 当前重连次数
-        self.max_reconnect_count = 3  # 减少最大重连次数到3次
-        self.asr_thread = None  # ASR监听线程
-        self.thread_lock = threading.Lock()  # 线程管理锁
-        self.is_reconnecting = False  # 添加重连状态标志
-
-        # 添加会话管理相关属性
-        self._session_lock = asyncio.Lock()  # 会话操作的并发锁
-        self._current_session_id = None  # 当前会话ID
-        self._session_started = False  # 会话是否已开始
-        self._session_finished = False  # 会话是否已结束
-        self._session_close_event = asyncio.Event()  # 添加会话关闭事件
-
-        self.appid = str(config.get("appid"))
         self.cluster = config.get("cluster")
         self.access_token = config.get("access_token")
         self.boosting_table_name = config.get("boosting_table_name", "")
         self.correct_table_name = config.get("correct_table_name", "")
-        self.output_dir = config.get("output_dir", "temp/")
+        self.output_dir = config.get("output_dir")
         self.delete_audio_file = delete_audio_file
 
-        self.ws_url = "wss://openspeech.bytedance.com/api/v2/asr"
+        self.host = "openspeech.bytedance.com"
-        self.uid = config.get("uid", "streaming_asr_service")
+        self.ws_url = f"wss://{self.host}/api/v2/asr"
-        self.workflow = config.get(
+        self.success_code = 1000
-            "workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
+        self.seg_duration = 15000
-        )
-        self.result_type = config.get("result_type", "single")
-        self.format = config.get("format", "raw")
-        self.codec = config.get("codec", "pcm")
-        self.rate = config.get("sample_rate", 16000)
-        self.language = config.get("language", "zh-CN")
-        self.bits = config.get("bits", 16)
-        self.channel = config.get("channel", 1)
-        self.auth_method = config.get("auth_method", "token")
-        self.secret = config.get("secret", "access_secret")
-        self.decoder = opuslib_next.Decoder(16000, 1)
-        self.asr_ws = None
-        self.forward_task = None
-        self.conn = None
 
-    ###################################################################################
+        # 确保输出目录存在
-    # 豆包流式ASR重写父类的方法--开始
+        os.makedirs(self.output_dir, exist_ok=True)
-    ###################################################################################
-    async def open_audio_channels(self, conn):
-        await super().open_audio_channels(conn)
 
-        async with self._session_lock:
+    @staticmethod
-            # 如果正在重连，等待重连完成
+    def _generate_header(
-            if self.is_reconnecting:
+        message_type=CLIENT_FULL_REQUEST, message_type_specific_flags=NO_SEQUENCE
-                logger.bind(tag=TAG).info("等待当前重连完成...")
+    ) -> bytearray:
-                await self._session_close_event.wait()
+        """Generate protocol header."""
-                self._session_close_event.clear()
+        header = bytearray()
+        header_size = 1
+        header.append((0b0001 << 4) | header_size)  # Protocol version
+        header.append((message_type << 4) | message_type_specific_flags)
+        header.append((0b0001 << 4) | 0b0001)  # JSON serialization & GZIP compression
+        header.append(0x00)  # reserved
+        return header
 
-            # 如果已有会话未结束，先关闭它
+    def _construct_request(self, reqid) -> dict:
-            if self._session_started and not self._session_finished:
+        """Construct the request payload."""
-                logger.bind(tag=TAG).warning(
+        return {
-                    f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
-                )
-                if self.asr_ws is not None:
-                    try:
-                        await self.asr_ws.close()
-                    except Exception as e:
-                        logger.bind(tag=TAG).warning(f"关闭旧连接时发生错误: {e}")
-                    finally:
-                        self.asr_ws = None
-                        self._session_finished = True
-                        self._session_close_event.set()
-
-            # 重置会话状态
-            self._current_session_id = str(uuid.uuid4())
-            self._session_started = True
-            self._session_finished = False
-            self.is_reconnecting = True
-
-            try:
-                retry_count = 0
-                while retry_count < self.max_retries:
-                    try:
-                        headers = (
-                            self.token_auth() if self.auth_method == "token" else None
-                        )
-                        self.asr_ws = await websockets.connect(
-                            self.ws_url,
-                            additional_headers=headers,
-                            max_size=1000000000,
-                            ping_interval=None,
-                            ping_timeout=None,
-                            close_timeout=10,
-                        )
-
-                        # 发送初始化请求
-                        request_params = self.construct_request(
-                            self._current_session_id
-                        )
-                        try:
-                            payload_bytes = str.encode(json.dumps(request_params))
-                            payload_bytes = gzip.compress(payload_bytes)
-                            full_client_request = self.generate_header()
-                            full_client_request.extend(
-                                (len(payload_bytes)).to_bytes(4, "big")
-                            )
-                            full_client_request.extend(payload_bytes)
-                            await self.asr_ws.send(full_client_request)
-                        except Exception as e:
-                            logger.bind(tag=TAG).error(f"发送初始化请求失败: {e}")
-                            raise e
-
-                        # 等待初始化响应
-                        try:
-                            init_res = await self.asr_ws.recv()
-                            self.parse_response(init_res)
-                        except Exception as e:
-                            logger.bind(tag=TAG).error(f"ASR服务初始化失败: {e}")
-                            raise e
-
-                        # 启动接收ASR结果的异步任务
-                        with self.thread_lock:
-                            if (
-                                self.asr_thread is None
-                                or not self.asr_thread.is_alive()
-                            ):
-                                logger.bind(tag=TAG).info("创建新的ASR监听线程...")
-                                self.asr_thread = threading.Thread(
-                                    target=self._start_monitor_asr_response_thread,
-                                    daemon=True,
-                                )
-                                self.asr_thread.start()
-                                # 等待一小段时间确保线程启动
-                                await asyncio.sleep(0.1)
-                                if not self.asr_thread.is_alive():
-                                    logger.bind(tag=TAG).error("ASR监听线程启动失败")
-                                    raise Exception("ASR监听线程启动失败")
-                                logger.bind(tag=TAG).info("ASR监听线程已启动")
-                        return
-
-                    except websockets.exceptions.WebSocketException as e:
-                        retry_count += 1
-                        if retry_count < self.max_retries:
-                            logger.bind(tag=TAG).warning(
-                                f"WebSocket连接失败，正在进行第{retry_count}次重试: {e}"
-                            )
-                            await asyncio.sleep(self.retry_delay)
-                        else:
-                            logger.bind(tag=TAG).warning(
-                                f"WebSocket连接失败，已达到最大重试次数: {e}"
-                            )
-                            raise
-                    except Exception as e:
-                        logger.bind(tag=TAG).error(f"WebSocket连接发生未知错误: {e}")
-                        raise
-            finally:
-                self.is_reconnecting = False
-                self._session_close_event.set()
-
-    async def receive_audio(self, audio, _):
-        if not isinstance(audio, bytes):
-            return
-
-        try:
-            # 解码opus得到PCM数据
-            pcm_frame = self.decoder.decode(audio, 960)
-            payload = gzip.compress(pcm_frame)
-            audio_request = bytearray(self.generate_audio_default_header())
-            audio_request.extend(len(payload).to_bytes(4, "big"))
-            audio_request.extend(payload)
-            if self.asr_ws:
-                await self.asr_ws.send(audio_request)
-        except Exception as e:
-            logger.bind(tag=TAG).debug(f"发送音频数据时发生错误: {e}")
-
-    ###################################################################################
-    # 豆包流式ASR重写父类的方法--结束
-    ###################################################################################
-
-    def construct_request(self, reqid):
-        req = {
             "app": {
-                "appid": self.appid,
+                "appid": f"{self.appid}",
                 "cluster": self.cluster,
                 "token": self.access_token,
             },
-            "user": {"uid": self.uid},
+            "user": {
+                "uid": str(uuid.uuid4()),
+            },
             "request": {
                 "reqid": reqid,
-                "workflow": self.workflow,
+                "show_utterances": False,
-                "show_utterances": True,
-                "result_type": self.result_type,
                 "sequence": 1,
                 "boosting_table_name": self.boosting_table_name,
                 "correct_table_name": self.correct_table_name,
             },
             "audio": {
-                "format": self.format,
+                "format": "raw",
-                "codec": self.codec,
+                "rate": 16000,
-                "rate": self.rate,
+                "language": "zh-CN",
-                "language": self.language,
+                "bits": 16,
-                "bits": self.bits,
+                "channel": 1,
-                "channel": self.channel,
+                "codec": "raw",
             },
         }
-        return req
 
-    def token_auth(self):
+    async def _send_request(
-        return {"Authorization": f"Bearer; {self.access_token}"}
+        self, audio_data: List[bytes], segment_size: int
-
+    ) -> Optional[str]:
-    def generate_header(
+        """Send request to Volcano ASR service."""
-        self,
-        version=PROTOCOL_VERSION,
-        message_type=CLIENT_FULL_REQUEST,
-        message_type_specific_flags=NO_SEQUENCE,
-        serial_method=JSON_SERIALIZATION,
-        compression_type=GZIP_COMPRESSION,
-        reserved_data=0x00,
-        extension_header: bytes = b"",
-    ):
-        """
-        生成协议头：
-         - 第1字节：高4位：协议版本，低4位：头部大小（单位 4 字节）
-         - 第2字节：高4位：消息类型，低4位：消息类型特定标志
-         - 第3字节：高4位：序列化方式，低4位：压缩方式
-         - 第4字节：保留字段
-         - 后续：扩展头（如果有）
-        """
-        header = bytearray()
-        header_size = int(len(extension_header) / 4) + 1
-        header.append((version << 4) | header_size)
-        header.append((message_type << 4) | message_type_specific_flags)
-        header.append((serial_method << 4) | compression_type)
-        header.append(reserved_data)
-        header.extend(extension_header)
-        return header
-
-    def generate_full_default_header(self):
-        # full client request 默认头
-        return self.generate_header(
-            version=PROTOCOL_VERSION,
-            message_type=CLIENT_FULL_REQUEST,
-            message_type_specific_flags=NO_SEQUENCE,
-            serial_method=JSON_SERIALIZATION,
-            compression_type=GZIP_COMPRESSION,
-        )
-
-    def generate_audio_default_header(self):
-        # 普通音频片段请求
-        return self.generate_header(
-            version=PROTOCOL_VERSION,
-            message_type=CLIENT_AUDIO_ONLY_REQUEST,
-            message_type_specific_flags=NO_SEQUENCE,
-            serial_method=JSON_SERIALIZATION,
-            compression_type=GZIP_COMPRESSION,
-        )
-
-    def generate_last_audio_default_header(self):
-        # 最后一个音频片段标志
-        return self.generate_header(
-            version=PROTOCOL_VERSION,
-            message_type=CLIENT_AUDIO_ONLY_REQUEST,
-            message_type_specific_flags=NEG_SEQUENCE,  # 用 NEG_SEQUENCE 表示结束
-            serial_method=JSON_SERIALIZATION,
-            compression_type=GZIP_COMPRESSION,
-        )
-
-    def _start_monitor_asr_response_thread(self):
-        # 初始化链接
         try:
-            with self.thread_lock:
+            auth_header = {"Authorization": "Bearer; {}".format(self.access_token)}
-                if self.conn is None or self.conn.loop is None:
+            async with websockets.connect(
-                    logger.bind(tag=TAG).error(
+                self.ws_url, additional_headers=auth_header
-                        "无法启动ASR监听线程：conn或loop未初始化"
+            ) as websocket:
-                    )
+                # Prepare request data
-                    return
+                request_params = self._construct_request(str(uuid.uuid4()))
+                payload_bytes = str.encode(json.dumps(request_params))
+                payload_bytes = gzip.compress(payload_bytes)
+                full_client_request = self._generate_header()
+                full_client_request.extend(
+                    (len(payload_bytes)).to_bytes(4, "big")
+                )  # payload size(4 bytes)
+                full_client_request.extend(payload_bytes)  # payload
 
-                try:
+                # Send header and metadata
-                    logger.bind(tag=TAG).info("开始启动ASR监听...")
+                # full_client_request
-                    asyncio.run_coroutine_threadsafe(
+                await websocket.send(full_client_request)
-                        self._forward_asr_results(), loop=self.conn.loop
+                res = await websocket.recv()
-                    )
+                result = parse_response(res)
-                    logger.bind(tag=TAG).info("ASR监听已启动")
+                if (
-                except Exception as e:
+                    "payload_msg" in result
-                    logger.bind(tag=TAG).error(f"启动ASR监听线程失败: {e}")
+                    and result["payload_msg"]["code"] != self.success_code
-        except Exception as e:
+                    and result["payload_msg"]["code"] != 1013  # 忽略无有效语音的错误
-            logger.bind(tag=TAG).error(f"ASR监听线程发生未预期的错误: {e}")
+                ):
+                    logger.bind(tag=TAG).error(f"ASR error: {result}")
+                    return None
 
-    async def _forward_asr_results(self):
+                for seq, (chunk, last) in enumerate(
-        try:
+                    self.slice_data(audio_data, segment_size), 1
-            while not self.conn.stop_event.is_set():
+                ):
-                try:
+                    if last:
-                    if self.asr_ws is None:
+                        audio_only_request = self._generate_header(
-                        # 检查是否需要重连
+                            message_type=CLIENT_AUDIO_ONLY_REQUEST,
-                        async with self.reconnect_lock:
+                            message_type_specific_flags=NEG_SEQUENCE,
-                            current_time = asyncio.get_event_loop().time()
-                            if (
-                                current_time - self.last_reconnect_time
-                                < self.reconnect_cooldown
-                            ):
-                                await asyncio.sleep(1)
-                                continue
-
-                            if self.reconnect_count >= self.max_reconnect_count:
-                                logger.bind(tag=TAG).error(
-                                    "达到最大重连次数限制，停止重连"
-                                )
-                                await asyncio.sleep(self.reconnect_cooldown)
-                                self.reconnect_count = 0
-                                continue
-
-                            self.last_reconnect_time = current_time
-                            self.reconnect_count += 1
-                            logger.bind(tag=TAG).info(
-                                f"尝试重新连接ASR服务... (第{self.reconnect_count}次)"
-                            )
-                            await self.open_audio_channels(self.conn)
-                            continue
-
-                    # 使用锁来确保同一时间只有一个协程在接收数据
-                    async with self.recv_lock:
-                        response = await self.asr_ws.recv()
-                        result = self.parse_response(response)
-
-                    # 检查是否需要重连
-                    if result.get("need_reconnect", False):
-                        logger.bind(tag=TAG).info(
-                            "检测到需要重连的错误，准备重新连接..."
                         )
-                        if self.asr_ws is not None:
+                    else:
-                            try:
+                        audio_only_request = self._generate_header(
-                                await self.asr_ws.close()
+                            message_type=CLIENT_AUDIO_ONLY_REQUEST
-                            except Exception as e:
+                        )
-                                logger.bind(tag=TAG).warning(
+                    payload_bytes = gzip.compress(chunk)
-                                    f"关闭旧连接时发生错误: {e}"
+                    audio_only_request.extend(
-                                )
+                        (len(payload_bytes)).to_bytes(4, "big")
-                            finally:
+                    )  # payload size(4 bytes)
-                                self.asr_ws = None
+                    audio_only_request.extend(payload_bytes)  # payload
-                        continue
+                    # Send audio data
+                    await websocket.send(audio_only_request)
 
-                    if "payload_msg" in result:
+                # Receive response
-                        if "result" in result["payload_msg"]:
+                response = await websocket.recv()
-                            # 检查是否有utterances并且definite为True
+                result = parse_response(response)
-                            utterances = result["payload_msg"]["result"][0].get(
-                                "utterances", []
-                            )
-                            for utterance in utterances:
-                                if utterance.get("definite", False):
-                                    self.text = utterance["text"]
-                                    await self.handle_voice_stop(None)
-                                    break
 
-                except websockets.ConnectionClosed:
+                if (
-                    logger.bind(tag=TAG).debug("ASR服务连接已关闭，准备重连...")
+                    "payload_msg" in result
-                    # 确保关闭旧连接
+                    and result["payload_msg"]["code"] == self.success_code
-                    if self.asr_ws is not None:
+                ):
-                        try:
+                    if len(result["payload_msg"]["result"]) > 0:
-                            await self.asr_ws.close()
+                        return result["payload_msg"]["result"][0]["text"]
-                        except Exception as e:
+                    return None
-                            logger.bind(tag=TAG).warning(f"关闭旧连接时发生错误: {e}")
+                elif "payload_msg" in result and result["payload_msg"]["code"] == 1013:
-                        finally:
+                    # 无有效语音，返回空字符串
-                            self.asr_ws = None
+                    return ""
-
+                else:
-                    # 等待冷却时间
+                    logger.bind(tag=TAG).error(f"ASR error: {result}")
-                    await asyncio.sleep(self.reconnect_cooldown)
+                    return None
-                    continue
-
-                except Exception as e:
-                    if not self.conn.stop_event.is_set():
-                        logger.bind(tag=TAG).error(f"ASR监听发生错误: {e}")
-                        await asyncio.sleep(self.retry_delay)
-                        continue
 
         except Exception as e:
-            logger.bind(tag=TAG).error(f"ASR监听线程发生错误: {e}")
+            logger.bind(tag=TAG).error(f"ASR request failed: {e}", exc_info=True)
-            # 确保在发生严重错误时也能继续尝试重连
+            return None
-            if not self.conn.stop_event.is_set():
-                await asyncio.sleep(self.retry_delay)
-                await self._forward_asr_results()  # 递归重试
 
-    async def speech_to_text(self, opus_data, session_id):
+    @staticmethod
-        result = self.text
+    def slice_data(data: bytes, chunk_size: int) -> (list, bool):
-        self.text = ""  # 清空text
-        return result, None
-
-    def parse_response(self, res: bytes) -> dict:
         """
-        解析 ASR 服务返回的二进制响应。
+        slice data
-        根据协议格式解析头部和 payload，若采用 GZIP 压缩则先解压，再根据 JSON 反序列化。
+        :param data: wav data
+        :param chunk_size: the segment size in one request
+        :return: segment data, last flag
         """
-        protocol_version = res[0] >> 4
+        data_len = len(data)
-        header_size = res[0] & 0x0F
+        offset = 0
-        message_type = res[1] >> 4
+        while offset + chunk_size < data_len:
-        serialization_method = res[2] >> 4
+            yield data[offset : offset + chunk_size], False
-        message_compression = res[2] & 0x0F
+            offset += chunk_size
-        payload = res[header_size * 4 :]
-        result = {}
-        payload_msg = None
-        payload_size = 0
-
-        if message_type == SERVER_FULL_RESPONSE:
-            payload_size = int.from_bytes(payload[:4], "big", signed=True)
-            payload_msg = payload[4:]
-        elif message_type == SERVER_ACK:
-            seq = int.from_bytes(payload[:4], "big", signed=True)
-            result["seq"] = seq
-            if len(payload) >= 8:
-                payload_size = int.from_bytes(payload[4:8], "big", signed=False)
-                payload_msg = payload[8:]
-        elif message_type == SERVER_ERROR_RESPONSE:
-            code = int.from_bytes(payload[:4], "big", signed=False)
-            result["code"] = code
-            payload_size = int.from_bytes(payload[4:8], "big", signed=False)
-            payload_msg = payload[8:]
-
-        if payload_msg is None:
-            return result
-        if message_compression == GZIP_COMPRESSION:
-            payload_msg = gzip.decompress(payload_msg)
-        if serialization_method == JSON_SERIALIZATION:
-            payload_msg = json.loads(payload_msg.decode("utf-8"))
         else:
-            payload_msg = payload_msg.decode("utf-8")
+            yield data[offset:data_len], True
-        result["payload_msg"] = payload_msg
-        result["payload_size"] = payload_size
 
-        # 错误码处理
+    async def speech_to_text(
-        if "code" in result:
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
-            error_code = result["code"]
+    ) -> Tuple[Optional[str], Optional[str]]:
-            error_message = ""
+        """将语音数据转换为文本"""
 
-            if error_code == 1000:
+        file_path = None
-                error_message = "成功"
+        try:
-            elif error_code == 1001:
+            # 合并所有opus数据包
-                error_message = "请求参数无效：请求参数缺失必需字段/字段值无效/重复请求"
+            if audio_format == "pcm":
-            elif error_code == 1002:
+                pcm_data = opus_data
-                error_message = "无访问权限：token无效/过期/无权访问指定服务"
-            elif error_code == 1003:
-                error_message = "访问超频：当前appid访问QPS超出设定阈值"
-            elif error_code == 1004:
-                error_message = "访问超额：当前appid访问次数超出限制"
-            elif error_code == 1005:
-                error_message = "服务器繁忙：服务过载，无法处理当前请求"
-            elif error_code == 1010:
-                error_message = "音频过长：音频数据时长超出阈值"
-            elif error_code == 1011:
-                error_message = "音频过大：音频数据大小超出阈值"
-            elif error_code == 1012:
-                error_message = "音频格式无效：音频header有误/无法进行音频解码"
-            elif error_code == 1013:
-                error_message = "音频静音：音频未识别出任何文本结果"
-            elif error_code >= 1020 and error_code <= 1022:
-                error_message = "识别相关错误：需要重连"
-                if error_code == 1020:
-                    error_message = "识别等待超时：等待下一包就绪超时"
-                elif error_code == 1021:
-                    error_message = "识别处理超时：识别处理过程超时"
-                elif error_code == 1022:
-                    error_message = "识别错误：识别过程中发生错误"
             else:
-                error_message = "未知错误：未归类错误"
+                pcm_data = self.decode_opus(opus_data)
+            combined_pcm_data = b"".join(pcm_data)
 
-            logger.bind(tag=TAG).debug(
+            # 判断是否保存为WAV文件
-                f"ASR错误: {error_message} (错误码: {error_code})"
+            if self.delete_audio_file:
-            )
+                pass
+            else:
+                file_path = self.save_audio_to_file(pcm_data, session_id)
 
-            # 如果是识别相关错误，标记需要重连
+            # 直接使用PCM数据
-            if error_code >= 1020 or error_code == 1001:
+            # 计算分段大小 (单声道, 16bit, 16kHz采样率)
-                result["need_reconnect"] = True
+            size_per_sec = 1 * 2 * 16000  # nchannels * sampwidth * framerate
+            segment_size = int(size_per_sec * self.seg_duration / 1000)
 
-        return result
+            # 语音识别
-
+            start_time = time.time()
-    async def close_session(self):
+            text = await self._send_request(combined_pcm_data, segment_size)
-        """关闭当前会话"""
+            if text:
-        async with self._session_lock:
+                logger.bind(tag=TAG).debug(
-            if not self._session_started:
+                    f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {text}"
-                logger.bind(tag=TAG).warning("尝试关闭未开始的会话")
-                return
-
-            if self._session_finished:
-                logger.bind(tag=TAG).warning(
-                    f"会话 {self._current_session_id} 已经关闭"
                 )
-                return
+                return text, file_path
+            return "", file_path
 
-            try:
+        except Exception as e:
-                if self.asr_ws is not None:
+            logger.bind(tag=TAG).error(f"语音识别失败: {e}", exc_info=True)
-                    await self.asr_ws.close()
+            return "", file_path
-            except Exception as e:
-                logger.bind(tag=TAG).warning(f"关闭WebSocket连接时发生错误: {e}")
-            finally:
-                self.asr_ws = None
-                self._session_finished = True
-                self._session_started = False
-                self._current_session_id = None
-                # 重置重连计数
-                self.reconnect_count = 0
-
-    async def close(self):
-        """资源清理方法"""
-        await self.close_session()

main/xiaozhi-server/core/providers/asr/doubao_stream.py

@@ -0,0 +1,349 @@
+import json
+import gzip
+import uuid
+import asyncio
+import websockets
+import opuslib_next
+from core.providers.asr.base import ASRProviderBase
+from config.logger import setup_logging
+from core.providers.asr.dto.dto import InterfaceType
+
+TAG = __name__
+logger = setup_logging()
+
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__()
+        self.interface_type = InterfaceType.STREAM
+        self.config = config
+        self.text = ""
+        self.max_retries = 3
+        self.retry_delay = 2
+        self.decoder = opuslib_next.Decoder(16000, 1)
+        self.asr_ws = None
+        self.forward_task = None
+        self.is_processing = False  # 添加处理状态标志
+
+        # 配置参数
+        self.appid = str(config.get("appid"))
+        self.cluster = config.get("cluster")
+        self.access_token = config.get("access_token")
+        self.boosting_table_name = config.get("boosting_table_name", "")
+        self.correct_table_name = config.get("correct_table_name", "")
+        self.output_dir = config.get("output_dir", "tmp/")
+        self.delete_audio_file = delete_audio_file
+
+        # 火山引擎ASR配置
+        self.ws_url = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel"
+        self.uid = config.get("uid", "streaming_asr_service")
+        self.workflow = config.get(
+            "workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
+        )
+        self.result_type = config.get("result_type", "single")
+        self.format = config.get("format", "pcm")
+        self.codec = config.get("codec", "pcm")
+        self.rate = config.get("sample_rate", 16000)
+        self.language = config.get("language", "zh-CN")
+        self.bits = config.get("bits", 16)
+        self.channel = config.get("channel", 1)
+        self.auth_method = config.get("auth_method", "token")
+        self.secret = config.get("secret", "access_secret")
+
+    async def open_audio_channels(self, conn):
+        await super().open_audio_channels(conn)
+
+    async def receive_audio(self, conn, audio, audio_have_voice):
+        conn.asr_audio.append(audio)
+        conn.asr_audio = conn.asr_audio[-10:]
+
+        # 如果本次有声音，且之前没有建立连接
+        if audio_have_voice and self.asr_ws is None and not self.is_processing:
+            try:
+                self.is_processing = True
+                # 建立新的WebSocket连接
+                headers = self.token_auth() if self.auth_method == "token" else None
+                logger.bind(tag=TAG).info(f"正在连接ASR服务，headers: {headers}")
+
+                self.asr_ws = await websockets.connect(
+                    self.ws_url,
+                    additional_headers=headers,
+                    max_size=1000000000,
+                    ping_interval=None,
+                    ping_timeout=None,
+                    close_timeout=10,
+                )
+
+                # 发送初始化请求
+                request_params = self.construct_request(str(uuid.uuid4()))
+                try:
+                    payload_bytes = str.encode(json.dumps(request_params))
+                    payload_bytes = gzip.compress(payload_bytes)
+                    full_client_request = self.generate_header()
+                    full_client_request.extend((len(payload_bytes)).to_bytes(4, "big"))
+                    full_client_request.extend(payload_bytes)
+
+                    logger.bind(tag=TAG).info(f"发送初始化请求: {request_params}")
+                    await self.asr_ws.send(full_client_request)
+
+                    # 等待初始化响应
+                    init_res = await self.asr_ws.recv()
+                    result = self.parse_response(init_res)
+                    logger.bind(tag=TAG).info(f"收到初始化响应: {result}")
+
+                    # 检查初始化响应
+                    if "code" in result and result["code"] != 1000:
+                        error_msg = f"ASR服务初始化失败: {result.get('payload_msg', {}).get('message', '未知错误')}"
+                        if "payload_msg" in result:
+                            error_msg += f"\n详细错误信息: {json.dumps(result['payload_msg'], ensure_ascii=False)}"
+                        logger.bind(tag=TAG).error(error_msg)
+                        raise Exception(error_msg)
+
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"发送初始化请求失败: {str(e)}")
+                    if hasattr(e, "__cause__") and e.__cause__:
+                        logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                    raise e
+
+                # 启动接收ASR结果的异步任务
+                self.forward_task = asyncio.create_task(self._forward_asr_results(conn))
+
+                # 发送缓存的音频数据
+                if conn.asr_audio and len(conn.asr_audio) > 0:
+                    for cached_audio in conn.asr_audio[-10:]:
+                        try:
+                            pcm_frame = self.decoder.decode(cached_audio, 960)
+                            payload = gzip.compress(pcm_frame)
+                            audio_request = bytearray(
+                                self.generate_audio_default_header()
+                            )
+                            audio_request.extend(len(payload).to_bytes(4, "big"))
+                            audio_request.extend(payload)
+                            await self.asr_ws.send(audio_request)
+                        except Exception as e:
+                            logger.bind(tag=TAG).info(
+                                f"发送缓存音频数据时发生错误: {e}"
+                            )
+
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"建立ASR连接失败: {str(e)}")
+                if hasattr(e, "__cause__") and e.__cause__:
+                    logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                if self.asr_ws:
+                    await self.asr_ws.close()
+                    self.asr_ws = None
+                self.is_processing = False
+                return
+
+        # 发送当前音频数据
+        if self.asr_ws and self.is_processing:
+            try:
+                pcm_frame = self.decoder.decode(audio, 960)
+                payload = gzip.compress(pcm_frame)
+                audio_request = bytearray(self.generate_audio_default_header())
+                audio_request.extend(len(payload).to_bytes(4, "big"))
+                audio_request.extend(payload)
+                await self.asr_ws.send(audio_request)
+            except Exception as e:
+                logger.bind(tag=TAG).info(f"发送音频数据时发生错误: {e}")
+
+    async def _forward_asr_results(self, conn):
+        try:
+            while self.asr_ws and not conn.stop_event.is_set():
+                try:
+                    response = await self.asr_ws.recv()
+                    result = self.parse_response(response)
+                    logger.bind(tag=TAG).debug(f"收到ASR结果: {result}")
+
+                    if "payload_msg" in result:
+                        payload = result["payload_msg"]
+                        # 检查是否是错误码1013（无有效语音）
+                        if "code" in payload and payload["code"] == 1013:
+                            # 静默处理，不记录错误日志
+                            continue
+
+                        if "result" in payload:
+                            utterances = payload["result"].get("utterances", [])
+                            # 检查duration和空文本的情况
+                            if (
+                                payload.get("audio_info", {}).get("duration", 0) > 2000
+                                and not utterances
+                                and not payload["result"].get("text")
+                            ):
+                                logger.bind(tag=TAG).error(f"识别文本：空")
+                                self.text = ""
+                                conn.reset_vad_states()
+                                await self.handle_voice_stop(conn, None)
+                                break
+
+                            for utterance in utterances:
+                                if utterance.get("definite", False):
+                                    self.text = utterance["text"]
+                                    logger.bind(tag=TAG).info(
+                                        f"识别到文本: {self.text}"
+                                    )
+                                    conn.reset_vad_states()
+                                    await self.handle_voice_stop(conn, None)
+                                    break
+                        elif "error" in payload:
+                            error_msg = payload.get("error", "未知错误")
+                            logger.bind(tag=TAG).error(f"ASR服务返回错误: {error_msg}")
+                            break
+
+                except websockets.ConnectionClosed:
+                    logger.bind(tag=TAG).info("ASR服务连接已关闭")
+                    self.is_processing = False
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"处理ASR结果时发生错误: {str(e)}")
+                    if hasattr(e, "__cause__") and e.__cause__:
+                        logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+                    self.is_processing = False
+                    break
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"ASR结果转发任务发生错误: {str(e)}")
+            if hasattr(e, "__cause__") and e.__cause__:
+                logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
+        finally:
+            if self.asr_ws:
+                await self.asr_ws.close()
+                self.asr_ws = None
+            self.is_processing = False
+
+    def stop_ws_connection(self):
+        if self.asr_ws:
+            asyncio.create_task(self.asr_ws.close())
+            self.asr_ws = None
+        self.is_processing = False
+
+    def construct_request(self, reqid):
+        req = {
+            "app": {
+                "appid": self.appid,
+                "cluster": self.cluster,
+                "token": self.access_token,
+            },
+            "user": {"uid": self.uid},
+            "request": {
+                "reqid": reqid,
+                "workflow": self.workflow,
+                "show_utterances": True,
+                "result_type": self.result_type,
+                "sequence": 1,
+                "boosting_table_name": self.boosting_table_name,
+                "correct_table_name": self.correct_table_name,
+                "end_window_size": 200,
+            },
+            "audio": {
+                "format": self.format,
+                "codec": self.codec,
+                "rate": self.rate,
+                "language": self.language,
+                "bits": self.bits,
+                "channel": self.channel,
+                "sample_rate": self.rate,
+            },
+        }
+        logger.bind(tag=TAG).debug(
+            f"构造请求参数: {json.dumps(req, ensure_ascii=False)}"
+        )
+        return req
+
+    def token_auth(self):
+        return {
+            "X-Api-App-Key": self.appid,
+            "X-Api-Access-Key": self.access_token,
+            "X-Api-Resource-Id": "volc.bigasr.sauc.duration",
+            "X-Api-Connect-Id": str(uuid.uuid4()),
+            "Host": "openspeech.bytedance.com",
+        }
+
+    def generate_header(
+        self,
+        version=0x01,
+        message_type=0x01,
+        message_type_specific_flags=0x00,
+        serial_method=0x01,
+        compression_type=0x01,
+        reserved_data=0x00,
+        extension_header: bytes = b"",
+    ):
+        header = bytearray()
+        header_size = int(len(extension_header) / 4) + 1
+        header.append((version << 4) | header_size)
+        header.append((message_type << 4) | message_type_specific_flags)
+        header.append((serial_method << 4) | compression_type)
+        header.append(reserved_data)
+        header.extend(extension_header)
+        return header
+
+    def generate_audio_default_header(self):
+        return self.generate_header(
+            version=0x01,
+            message_type=0x02,
+            message_type_specific_flags=0x00,
+            serial_method=0x01,
+            compression_type=0x01,
+        )
+
+    def generate_last_audio_default_header(self):
+        return self.generate_header(
+            version=0x01,
+            message_type=0x02,
+            message_type_specific_flags=0x02,
+            serial_method=0x01,
+            compression_type=0x01,
+        )
+
+    def parse_response(self, res: bytes) -> dict:
+        try:
+            # 检查响应长度
+            if len(res) < 4:
+                logger.bind(tag=TAG).error(f"响应数据长度不足: {len(res)}")
+                return {"error": "响应数据长度不足"}
+
+            # 获取消息头
+            header = res[:4]
+            message_type = header[1] >> 4
+
+            # 如果是错误响应
+            if message_type == 0x0F:  # SERVER_ERROR_RESPONSE
+                code = int.from_bytes(header[4:8], "big", signed=False)
+                error_msg = res[8:].decode("utf-8")
+                return {"code": code, "error": error_msg}
+
+            # 获取JSON数据（跳过12字节头部）
+            try:
+                json_data = res[12:].decode("utf-8")
+                result = json.loads(json_data)
+                logger.bind(tag=TAG).debug(f"成功解析JSON响应: {result}")
+                return {"payload_msg": result}
+            except (UnicodeDecodeError, json.JSONDecodeError) as e:
+                logger.bind(tag=TAG).error(f"JSON解析失败: {str(e)}")
+                logger.bind(tag=TAG).error(f"原始数据: {res}")
+                raise
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"解析响应失败: {str(e)}")
+            logger.bind(tag=TAG).error(f"原始响应数据: {res.hex()}")
+            raise
+
+    async def speech_to_text(self, opus_data, session_id, audio_format):
+        result = self.text
+        self.text = ""  # 清空text
+        return result, None
+
+    async def close(self):
+        """资源清理方法"""
+        if self.asr_ws:
+            await self.asr_ws.close()
+            self.asr_ws = None
+        if self.forward_task:
+            self.forward_task.cancel()
+            try:
+                await self.forward_task
+            except asyncio.CancelledError:
+                pass
+            self.forward_task = None
+        self.is_processing = False

main/xiaozhi-server/core/providers/asr/fun_local.py

@@ -2,6 +2,7 @@ import time
 import os
 import sys
 import io
+import psutil
 from config.logger import setup_logging
 from typing import Optional, Tuple, List
 from core.providers.asr.base import ASRProviderBase
@@ -37,6 +38,13 @@ class CaptureOutput:
 class ASRProvider(ASRProviderBase):
     def __init__(self, config: dict, delete_audio_file: bool):
         super().__init__()
+        
+        # 内存检测，要求大于2G
+        min_mem_bytes = 2 * 1024 * 1024 * 1024
+        total_mem = psutil.virtual_memory().total
+        if total_mem < min_mem_bytes:
+            logger.bind(tag=TAG).error(f"可用内存不足2G，当前仅有 {total_mem / (1024*1024):.2f} MB，可能无法启动FunASR")
+        
         self.interface_type = InterfaceType.LOCAL
         self.model_dir = config.get("model_dir")
         self.output_dir = config.get("output_dir")  # 修正配置键名
@@ -54,7 +62,7 @@ class ASRProvider(ASRProviderBase):
             )
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """语音转文本主处理逻辑"""
         file_path = None
@@ -63,7 +71,7 @@ class ASRProvider(ASRProviderBase):
         while retry_count < MAX_RETRIES:
             try:
                 # 合并所有opus数据包
-                if self.audio_format == "pcm":
+                if audio_format == "pcm":
                     pcm_data = opus_data
                 else:
                     pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/fun_server.py

@@ -100,7 +100,7 @@ class ASRProvider(ASRProviderBase):
         logger.bind(tag=TAG).debug(f"Sent end message: {end_message}")
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """
         Convert speech data to text using FunASR.
@@ -109,7 +109,7 @@ class ASRProvider(ASRProviderBase):
         :return: Tuple containing recognized text and optional timestamp.
         """
         file_path = None
-        if self.audio_format == "pcm":
+        if audio_format == "pcm":
             pcm_data = opus_data
         else:
             pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/sherpa_onnx_local.py

@@ -109,14 +109,14 @@ class ASRProvider(ASRProviderBase):
             return samples_float32, f.getframerate()
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """语音转文本主处理逻辑"""
         file_path = None
         try:
             # 保存音频文件
             start_time = time.time()
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/asr/tencent.py

@@ -32,7 +32,7 @@ class ASRProvider(ASRProviderBase):
         os.makedirs(self.output_dir, exist_ok=True)
 
     async def speech_to_text(
-        self, opus_data: List[bytes], session_id: str
+        self, opus_data: List[bytes], session_id: str, audio_format="opus"
     ) -> Tuple[Optional[str], Optional[str]]:
         """将语音数据转换为文本"""
         if not opus_data:
@@ -47,7 +47,7 @@ class ASRProvider(ASRProviderBase):
                 return None, file_path
 
             # 将Opus音频数据解码为PCM
-            if self.audio_format == "pcm":
+            if audio_format == "pcm":
                 pcm_data = opus_data
             else:
                 pcm_data = self.decode_opus(opus_data)

main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py

@@ -76,6 +76,14 @@ class IntentProvider(IntentProviderBase):
             '返回: {"function_call": {"name": "get_battery_level", "arguments": {"response_success": "当前电池电量为{value}%", "response_failure": "无法获取Battery的当前电量百分比"}}}\n'
             "```\n"
             "```\n"
+            "用户: 当前屏幕亮度是多少？\n"
+            '返回: {"function_call": {"name": "self_screen_get_brightness"}}\n'
+            "```\n"
+            "```\n"
+            "用户: 设置屏幕亮度为50%\n"
+            '返回: {"function_call": {"name": "self_screen_set_brightness", "arguments": {"brightness": 50}}}\n'
+            "```\n"
+            "```\n"
             "用户: 我想结束对话\n"
             '返回: {"function_call": {"name": "handle_exit_intent", "arguments": {"say_goodbye": "goodbye"}}}\n'
             "```\n"
@@ -122,6 +130,8 @@ class IntentProvider(IntentProviderBase):
     async def detect_intent(self, conn, dialogue_history: List[Dict], text: str) -> str:
         if not self.llm:
             raise ValueError("LLM provider not set")
+        if conn.func_handler is None:
+            return '{"function_call": {"name": "continue_chat"}}'
 
         # 记录整体开始时间
         total_start_time = time.time()
@@ -148,15 +158,25 @@ class IntentProvider(IntentProviderBase):
         self.clean_cache()
 
         if self.promot == "":
-            if hasattr(conn, "func_handler"):
+            functions = conn.func_handler.get_functions()
-                functions = conn.func_handler.get_functions()
+            if hasattr(conn, "mcp_client"):
-                self.promot = self.get_intent_system_prompt(functions)
+                mcp_tools = conn.mcp_client.get_available_tools()
+                if mcp_tools is not None and len(mcp_tools) > 0:
+                    if functions is None:
+                        functions = []
+                    functions.extend(mcp_tools)
+
+            self.promot = self.get_intent_system_prompt(functions)
 
         music_config = initialize_music_handler(conn)
         music_file_names = music_config["music_file_names"]
         prompt_music = f"{self.promot}\n<musicNames>{music_file_names}\n</musicNames>"
 
-        devices = conn.config["plugins"]["home_assistant"].get("devices", [])
+        home_assistant_cfg = conn.config["plugins"].get("home_assistant")
+        if home_assistant_cfg:
+            devices = home_assistant_cfg.get("devices", [])
+        else:
+            devices = []
         if len(devices) > 0:
             hass_prompt = "\n下面是我家智能设备列表（位置，设备名，entity_id），可以通过homeassistant控制\n"
             for device in devices:

main/xiaozhi-server/core/providers/llm/coze/coze.py

@@ -23,7 +23,9 @@ class LLMProvider(LLMProviderBase):
         self.bot_id = str(config.get("bot_id"))
         self.user_id = str(config.get("user_id"))
         self.session_conversation_map = {}  # 存储session_id和conversation_id的映射
-        check_model_key("CozeLLM", self.personal_access_token)
+        model_key_msg = check_model_key("CozeLLM", self.personal_access_token)
+        if model_key_msg:
+            logger.bind(tag=TAG).error(model_key_msg)
 
     def response(self, session_id, dialogue, **kwargs):
         coze_api_token = self.personal_access_token

main/xiaozhi-server/core/providers/llm/dify/dify.py

@@ -15,7 +15,9 @@ class LLMProvider(LLMProviderBase):
         self.mode = config.get("mode", "chat-messages")
         self.base_url = config.get("base_url", "https://api.dify.ai/v1").rstrip("/")
         self.session_conversation_map = {}  # 存储session_id和conversation_id的映射
-        check_model_key("DifyLLM", self.api_key)
+        model_key_msg = check_model_key("DifyLLM", self.api_key)
+        if model_key_msg:
+            logger.bind(tag=TAG).error(model_key_msg)
 
     def response(self, session_id, dialogue, **kwargs):
         try:

main/xiaozhi-server/core/providers/llm/fastgpt/fastgpt.py

@@ -14,7 +14,9 @@ class LLMProvider(LLMProviderBase):
         self.base_url = config.get("base_url")
         self.detail = config.get("detail", False)
         self.variables = config.get("variables", {})
-        check_model_key("FastGPTLLM", self.api_key)
+        model_key_msg = check_model_key("FastGPTLLM", self.api_key)
+        if model_key_msg:
+            logger.bind(tag=TAG).error(model_key_msg)
 
     def response(self, session_id, dialogue, **kwargs):
         try:

main/xiaozhi-server/core/providers/llm/gemini/gemini.py

@@ -73,8 +73,9 @@ class LLMProvider(LLMProviderBase):
         http_proxy = cfg.get("http_proxy")
         https_proxy = cfg.get("https_proxy")
 
-        if not check_model_key("LLM", self.api_key):
+        model_key_msg = check_model_key("LLM", self.api_key)
-            raise ValueError("无效的Gemini API Key，请检查是否配置正确")
+        if model_key_msg:
+            log.bind(tag=TAG).error(model_key_msg)
 
         if http_proxy or https_proxy:
             log.bind(tag=TAG).info(

main/xiaozhi-server/core/providers/llm/openai/openai.py

@@ -21,20 +21,27 @@ class LLMProvider(LLMProviderBase):
             "max_tokens": (500, int),
             "temperature": (0.7, lambda x: round(float(x), 1)),
             "top_p": (1.0, lambda x: round(float(x), 1)),
-            "frequency_penalty": (0, lambda x: round(float(x), 1))
+            "frequency_penalty": (0, lambda x: round(float(x), 1)),
         }
 
         for param, (default, converter) in param_defaults.items():
             value = config.get(param)
             try:
-                setattr(self, param, converter(value) if value not in (None, "") else default)
+                setattr(
+                    self,
+                    param,
+                    converter(value) if value not in (None, "") else default,
+                )
             except (ValueError, TypeError):
                 setattr(self, param, default)
 
         logger.debug(
-            f"意图识别参数初始化: {self.temperature}, {self.max_tokens}, {self.top_p}, {self.frequency_penalty}")
+            f"意图识别参数初始化: {self.temperature}, {self.max_tokens}, {self.top_p}, {self.frequency_penalty}"
+        )
 
-        check_model_key("LLM", self.api_key)
+        model_key_msg = check_model_key("LLM", self.api_key)
+        if model_key_msg:
+            logger.bind(tag=TAG).error(model_key_msg)
         self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
 
     def response(self, session_id, dialogue, **kwargs):
@@ -46,7 +53,9 @@ class LLMProvider(LLMProviderBase):
                 max_tokens=kwargs.get("max_tokens", self.max_tokens),
                 temperature=kwargs.get("temperature", self.temperature),
                 top_p=kwargs.get("top_p", self.top_p),
-                frequency_penalty=kwargs.get("frequency_penalty", self.frequency_penalty),
+                frequency_penalty=kwargs.get(
+                    "frequency_penalty", self.frequency_penalty
+                ),
             )
 
             is_active = True
@@ -84,10 +93,12 @@ class LLMProvider(LLMProviderBase):
             for chunk in stream:
                 # 检查是否存在有效的choice且content不为空
                 if getattr(chunk, "choices", None):
-                    yield chunk.choices[0].delta.content, chunk.choices[0].delta.tool_calls
+                    yield chunk.choices[0].delta.content, chunk.choices[
+                        0
+                    ].delta.tool_calls
                 # 存在 CompletionUsage 消息时，生成 Token 消耗 log
-                elif isinstance(getattr(chunk, 'usage', None), CompletionUsage):
+                elif isinstance(getattr(chunk, "usage", None), CompletionUsage):
-                    usage_info = getattr(chunk, 'usage', None)
+                    usage_info = getattr(chunk, "usage", None)
                     logger.bind(tag=TAG).info(
                         f"Token 消耗：输入 {getattr(usage_info, 'prompt_tokens', '未知')}，"
                         f"输出 {getattr(usage_info, 'completion_tokens', '未知')}，"

main/xiaozhi-server/core/providers/memory/base.py

@@ -12,10 +12,6 @@ class MemoryProviderBase(ABC):
 
     def set_llm(self, llm):
         self.llm = llm
-        # 获取模型名称和类型信息
-        model_name = getattr(llm, "model_name", str(llm.__class__.__name__))
-        # 记录更详细的日志
-        logger.bind(tag=TAG).info(f"记忆总结设置LLM: {model_name}")
 
     @abstractmethod
     async def save_memory(self, msgs):

main/xiaozhi-server/core/providers/memory/mem0ai/mem0ai.py

@@ -12,12 +12,14 @@ class MemoryProvider(MemoryProviderBase):
         super().__init__(config)
         self.api_key = config.get("api_key", "")
         self.api_version = config.get("api_version", "v1.1")
-        have_key = check_model_key("Mem0ai", self.api_key)
+        model_key_msg = check_model_key("Mem0ai", self.api_key)
-        if not have_key:
+        if model_key_msg:
+            logger.bind(tag=TAG).error(model_key_msg)
             self.use_mem0 = False
             return
         else:
             self.use_mem0 = True
+
         try:
             self.client = MemoryClient(api_key=self.api_key)
             logger.bind(tag=TAG).info("成功连接到 Mem0ai 服务")

main/xiaozhi-server/core/providers/tts/aliyun.py

@@ -91,7 +91,7 @@ class TTSProvider(TTSProviderBase):
 
         self.appkey = config.get("appkey")
         self.format = config.get("format", "wav")
-
+        self.audio_file_type = config.get("format", "wav")
         sample_rate = config.get("sample_rate", "16000")
         self.sample_rate = int(sample_rate) if sample_rate else 16000
 
@@ -188,9 +188,12 @@ class TTSProvider(TTSProviderBase):
                 )
             # 检查返回请求数据的mime类型是否是audio/***，是则保存到指定路径下；返回的是binary格式的
             if resp.headers["Content-Type"].startswith("audio/"):
-                with open(output_file, "wb") as f:
+                if output_file:
-                    f.write(resp.content)
+                    with open(output_file, "wb") as f:
-                return output_file
+                        f.write(resp.content)
+                    return output_file
+                else:
+                    return resp.content
             else:
                 raise Exception(
                     f"{__name__} status_code: {resp.status_code} response: {resp.content}"