Merge pull request #1894 from xinnan-tech/update-doc

update:添加阿里云流式推荐

.gitignore

@@ -174,3 +174,4 @@ main/xiaozhi-server/mysql
 uploadfile
 *.json
 .vscode
+.cursor

Dockerfile-server

@@ -21,6 +21,7 @@ RUN apt-get update && \
 
 # 从构建阶段复制Python包和前端构建产物
 COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
+COPY --from=builder /usr/local/bin/mcp-proxy /usr/local/bin/mcp-proxy
 
 # 复制应用代码
 COPY main/xiaozhi-server .

README.md

@@ -3,10 +3,10 @@
 <h1 align="center">小智后端服务xiaozhi-esp32-server</h1>
 
 <p align="center">
-本项目为开源智能硬件项目
+本项目基于人机共生智能理论和技术研发智能终端软硬件体系<br/>为开源智能硬件项目
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a>提供后端服务<br/>
 根据<a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">小智通信协议</a>使用Python、Java、Vue实现<br/>
-帮助您快速搭建小智服务器
+支持MCP接入点和声纹识别
 </p>
 
 <p align="center">
@@ -37,6 +37,14 @@
   </a>
 </p>
 
+<p align="center">
+By Professor Siyuan Liu Research and Development Group ( South China University of Technology)
+</br>
+刘思源教授团队研发（华南理工大学）
+</br>
+<img src="./docs/images/hnlg.jpg" alt="华南理工大学" width="50%">
+</p>
+
 ---
 
 ## 适用人群 👥
@@ -144,8 +152,18 @@
         </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
+         <picture>
+           <img alt="MCP接入点" src="docs/images/demo13.png" />
+         </picture>
+        </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
+         <picture>
+           <img alt="声纹识别" src="docs/images/demo14.png" />
+         </picture>
+        </a>
     </td>
   </tr>
 </table>
@@ -170,8 +188,8 @@
 #### 🚀 部署方式选择
 | 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | 
 |---------|------|---------|---------|---------|---------|
-| **最简化安装** | 智能对话、IOT、MCP、视觉感知，数据存储在配置文件 | 低配置环境，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
+| **最简化安装** | 智能对话、IOT、MCP、视觉感知 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
-| **全模块安装** | 智能对话、IOT、MCP、视觉感知、OTA、智控台，数据存储在数据库 | 完整功能体验 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
+| **全模块安装** | 智能对话、IOT、MCP接入点、声纹识别、视觉感知、OTA、智控台 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
 
 > 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
@@ -199,7 +217,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 | ASR(语音识别) | FunASR(本地) | 👍FunASRServer 或 👍DoubaoStreamASR |
 | LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
 | VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
-| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) |
+| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) 或 👍AliyunStreamTTS(阿里云流式语音合成) |
 | Intent(意图识别) | function_call(函数调用) | function_call(函数调用) |
 | Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） |
 
@@ -220,13 +238,14 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 | 功能模块 | 描述 |
 |:---:|:---|
-| 核心服务架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
+| 核心架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
-| 语音交互系统 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
+| 语音交互 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
-| 智能对话系统 | 支持多种LLM(大语言模型)，实现智能对话 |
+| 声纹识别 | 支持多用户声纹注册、管理和识别，与ASR并行处理，实时识别说话人身份并传递给LLM进行个性化回应 |
-| 视觉感知系统 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
+| 智能对话 | 支持多种LLM(大语言模型)，实现智能对话 |
-| 意图识别系统 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
+| 视觉感知 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
+| 意图识别 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
 | 记忆系统 | 支持本地短期记忆、mem0ai接口记忆，具备记忆总结功能 |
-| IOT/MCP控制协议 | 支持设备注册管理、智能控制接口，同时支持IOT、MCP控制协议 |
+| 工具调用 | 支持客户端IOT协议、客户MCP协议、服务端MCP协议、MCP接入点协议、自定义工具函数 |
 | 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理 |
 | 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
 | 部署支持 | 支持Docker部署和本地部署，提供完整的配置文件管理 |
@@ -303,6 +322,14 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 ---
 
+### Voiceprint 声纹识别
+
+| 使用方式 | 支持平台 | 免费平台 |
+|:---:|:---:|:---:|
+| 本地使用 | 3D-Speaker | 3D-Speaker |
+
+---
+
 ### Memory 记忆存储
 
 |   类型   |      平台名称       | 使用方式 |   收费模式    | 备注 |

README_en.md

@@ -3,17 +3,17 @@
 <h1 align="center">Xiaozhi Backend Service xiaozhi-esp32-server</h1>
 
 <p align="center">
-This project provides backend services for the open-source smart hardware project
+This project is based on human-machine symbiotic intelligence theory and technology to develop intelligent terminal hardware and software systems<br/>providing backend services for the open-source intelligent hardware project
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
 Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
-Helps you quickly set up your Xiaozhi server
+Supports MCP endpoints and voiceprint recognition
 </p>
 
 <p align="center">
 <a href="./README.md">中文</a>
 · <a href="./docs/FAQ.md">FAQ</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
-· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
+· <a href="./README.md#%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3">Deployment Docs</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
 </p>
 <p align="center">
@@ -37,41 +37,49 @@ Helps you quickly set up your Xiaozhi server
   </a>
 </p>
 
+<p align="center">
+By Professor Siyuan Liu Research and Development Group (South China University of Technology)
+</br>
+刘思源教授团队研发（华南理工大学）
+</br>
+<img src="./docs/images/hnlg.jpg" alt="South China University of Technology" width="50%">
+</p>
+
 ---
 
 ## Target Users 👥
 
-This project requires ESP32 hardware devices. If you have purchased ESP32-related hardware, successfully connected to Brother Xia's backend service, and want to set up your own `xiaozhi-esp32` backend service, then this project is perfect for you.
+This project requires ESP32 hardware devices to work. If you have purchased ESP32-related hardware, successfully connected to Brother Xia's deployed backend service, and want to build your own `xiaozhi-esp32` backend service independently, then this project is perfect for you.
 
-Want to see it in action? Check out these videos 🎥
+Want to see the usage effects? Click the videos below 🎥
 
 <table>
   <tr>
     <td>
         <a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
          <picture>
-           <img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
+           <img alt="Xiaozhi ESP32 connecting to own backend model" src="docs/images/demo1.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
          <picture>
-           <img alt="Custom voice" src="docs/images/demo2.png" />
+           <img alt="Custom voice timbre" src="docs/images/demo2.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
          <picture>
-           <img alt="Using Cantonese" src="docs/images/demo3.png" />
+           <img alt="Using Cantonese for communication" src="docs/images/demo3.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
          <picture>
-           <img alt="Control home appliances" src="docs/images/demo5.png" />
+           <img alt="Controlling home appliances" src="docs/images/demo5.png" />
          </picture>
         </a>
     </td>
@@ -87,14 +95,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1Vy96YCE3R" target="_blank">
          <picture>
-           <img alt="Custom voice" src="docs/images/demo6.png" />
+           <img alt="Custom voice timbre" src="docs/images/demo6.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
          <picture>
-           <img alt="Play music" src="docs/images/demo7.png" />
+           <img alt="Playing music" src="docs/images/demo7.png" />
          </picture>
         </a>
     </td>
@@ -108,14 +116,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
          <picture>
-           <img alt="IOT command control" src="docs/images/demo9.png" />
+           <img alt="IOT command control devices" src="docs/images/demo9.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV17LXWYvENb" target="_blank">
          <picture>
-           <img alt="News broadcast" src="docs/images/demo0.png" />
+           <img alt="News broadcasting" src="docs/images/demo0.png" />
          </picture>
         </a>
     </td>
@@ -131,7 +139,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
       <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
          <picture>
-           <img alt="Photo recognition" src="docs/images/demo12.png" />
+           <img alt="Photo recognition of objects" src="docs/images/demo12.png" />
          </picture>
         </a>
     </td>
@@ -143,20 +151,29 @@ Want to see it in action? Check out these videos 🎥
         </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
+         <picture>
+           <img alt="MCP endpoint" src="docs/images/demo13.png" />
+         </picture>
+        </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
+         <picture>
+           <img alt="Voiceprint recognition" src="docs/images/demo14.png" />
+         </picture>
+        </a>
     </td>
   </tr>
 </table>
 
 ---
 
-## Warning ⚠️
+## Warnings ⚠️
 
-1. This project is open-source software. This software has no commercial relationship with any third-party API service providers (including but not limited to speech recognition, large models, speech synthesis, and other platforms) and does not provide any form of guarantee for their service quality or financial security.
+1. This project is open-source software. This software has no commercial partnership with any third-party API service providers (including but not limited to speech recognition, large models, speech synthesis, and other platforms) that it interfaces with, and does not provide any form of guarantee for their service quality or financial security. It is recommended that users prioritize service providers with relevant business licenses and carefully read their service agreements and privacy policies. This software does not host any account keys, does not participate in fund flows, and does not bear the risk of recharge fund losses.
-It is recommended that users prioritize service providers with relevant business licenses and carefully read their service agreements and privacy policies. This software does not host any account keys, does not participate in fund transfers, and does not bear the risk of recharge fund losses.
 
-2. This project's functionality is not complete and has not passed network security testing. Please do not use it in production environments. If you deploy this project for learning in a public network environment, please ensure necessary protection measures are in place.
+2. The functionality of this project is not complete and has not passed network security assessment. Please do not use it in production environments. If you deploy this project for learning purposes in a public network environment, please ensure necessary protection measures are in place.
 
 ---
 
@@ -164,18 +181,19 @@ It is recommended that users prioritize service providers with relevant business
 
 ![Banners](docs/images/banner2.png)
 
-This project provides two deployment methods. Please choose according to your specific needs:
+This project provides two deployment methods. Please choose based on your specific needs:
 
 #### 🚀 Deployment Method Selection
-| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial | 
+| Deployment Method | Features | Applicable Scenarios | Deployment Docs | Configuration Requirements | Video Tutorials | 
 |---------|------|---------|---------|---------|---------|
-| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - | 
+| **Simplified Installation** | Intelligent dialogue, IOT, MCP, visual perception | Low-configuration environments, data stored in config files, no database required | [①Docker Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - | 
-| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) | 
+| **Full Module Installation** | Intelligent dialogue, IOT, MCP endpoints, voiceprint recognition, visual perception, OTA, intelligent control console | Complete functionality experience, data stored in database |[①Docker Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](./docs/dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
-> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
+
+> 💡 Note: Below is a test platform deployed with the latest code. You can burn and test if needed. Concurrent users: 6, data will be cleared daily.
 
 ```
-Control Panel Address: https://2662r3426b.vicp.fun
+Intelligent Control Console Address: https://2662r3426b.vicp.fun
 
 Service Test Tool: https://2662r3426b.vicp.fun/test/
 OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
@@ -184,51 +202,69 @@ Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 #### 🚩 Configuration Description and Recommendations
 > [!Note]
-> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
+> This project provides two configuration schemes:
 > 
-> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
+> 1. `Entry Level Free Settings`: Suitable for personal and home use, all components use free solutions, no additional payment required.
+> 
+> 2. `Streaming Configuration`: Suitable for demonstrations, training, scenarios with more than 2 concurrent users, etc. Uses streaming processing technology for faster response speed and better experience.
+> 
+> Starting from version `0.5.2`, the project supports streaming configuration. Compared to earlier versions, response speed is improved by approximately `2.5 seconds`, significantly improving user experience.
 
-| Module Name | Entry Level Free Settings | Full Streaming Configuration |
+| Module Name | Entry Level Free Settings | Streaming Configuration |
-|---------|---------|------|
+|:---:|:---:|:---:|
-| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
+| ASR(Speech Recognition) | FunASR(Local) | 👍FunASRServer or 👍DoubaoStreamASR |
-| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
+| LLM(Large Model) | ChatGLMLLM(Zhipu glm-4-flash) | 👍DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
-| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
+| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | 👍QwenVLVLLM(Qwen qwen2.5-vl-3b-instructh) |
-| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
+| TTS(Speech Synthesis) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano dual-stream speech synthesis) |
-| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
+| Intent(Intent Recognition) | function_call(Function calling) | function_call(Function calling) |
-| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
+| Memory(Memory function) | mem_local_short(Local short-term memory) | mem_local_short(Local short-term memory) |
+
+#### 🔧 Testing Tools
+This project provides the following testing tools to help you verify the system and choose suitable models:
+
+| Tool Name | Location | Usage Method | Function Description |
+|:---:|:---|:---:|:---:|
+| Audio Interaction Test Tool | main》xiaozhi-server》test》test_page.html | Open directly with Google Chrome | Tests audio playback and reception functions, verifies if Python-side audio processing is normal |
+| Model Response Test Tool 1 | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Tests response speed of three core modules: ASR(speech recognition), LLM(large model), TTS(speech synthesis) |
+| Model Response Test Tool 2 | main》xiaozhi-server》performance_tester_vllm.py | Execute `python performance_tester_vllm.py` | Tests VLLM(vision model) response speed |
+
+> 💡 Note: When testing model speed, only models with configured keys will be tested.
 
 ---
 ## Feature List ✨
 ### Implemented ✅
 
 | Feature Module | Description |
-|---------|------|
+|:---:|:---|
-| Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
+| Core Architecture | Based on WebSocket and HTTP servers, provides complete console management and authentication system |
-| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
+| Voice Interaction | Supports streaming ASR(speech recognition), streaming TTS(speech synthesis), VAD(voice activity detection), supports multi-language recognition and voice processing |
-| Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
+| Voiceprint Recognition | Supports multi-user voiceprint registration, management, and recognition, processes in parallel with ASR, real-time speaker identity recognition and passes to LLM for personalized responses |
-| Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
+| Intelligent Dialogue | Supports multiple LLM(large language models), implements intelligent dialogue |
-| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
+| Visual Perception | Supports multiple VLLM(vision large models), implements multimodal interaction |
-| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
+| Intent Recognition | Supports LLM intent recognition, Function Call function calling, provides plugin-based intent processing mechanism |
-| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
+| Memory System | Supports local short-term memory, mem0ai interface memory, with memory summarization functionality |
-| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
+| Tool Calling | Supports client IOT protocol, client MCP protocol, server MCP protocol, MCP endpoint protocol, custom tool functions |
-| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
+| Management Backend | Provides Web management interface, supports user management, system configuration, and device management |
+| Testing Tools | Provides performance testing tools, vision model testing tools, and audio interaction testing tools |
+| Deployment Support | Supports Docker deployment and local deployment, provides complete configuration file management |
+| Plugin System | Supports functional plugin extensions, custom plugin development, and plugin hot-loading |
 
-### In Development 🚧
+### Under Development 🚧
 
-To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
+To learn about specific development plan progress, [click here](https://github.com/users/xinnan-tech/projects/3)
 
 If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
 
 ---
 
 ## Product Ecosystem 👬
-Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
+Xiaozhi is an ecosystem. When using this product, you can also check out other excellent projects in this ecosystem
 
 | Project Name | Project Address | Project Description |
 |:---------------------|:--------|:--------|
-| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
+| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | An Android and iOS voice dialogue application based on xiaozhi-server, supporting real-time voice interaction and text dialogue.<br/>Currently a Flutter version, connecting iOS and Android platforms. |
-| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
+| Xiaozhi Desktop Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based AI client for beginners, allowing users to experience Xiaozhi AI functionality through code even without physical hardware conditions. |
-| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
+| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | Xiaozhi open-source backend service Java version is a Java-based open-source project.<br/>It includes frontend and backend services, aiming to provide users with a complete backend service solution. |
 
 ---
 
@@ -238,20 +274,32 @@ Xiaozhi is an ecosystem. When using this product, you might also want to check o
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
+| OpenAI interface calls | Alibaba Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
-| ollama interface call | Ollama | - |
+| Ollama interface calls | Ollama | - |
-| dify interface call | Dify | - |
+| Dify interface calls | Dify | - |
-| fastgpt interface call | Fastgpt | - |
+| FastGPT interface calls | FastGPT | - |
-| coze interface call | Coze | - |
+| Coze interface calls | Coze | - |
 
-In fact, any LLM that supports openai interface calls can be integrated and used.
+In fact, any LLM that supports OpenAI interface calls can be integrated and used.
+
+---
+
+### VLLM Vision Models
+
+| Usage Method | Supported Platforms | Free Platforms |
+|:---:|:---:|:---:|
+| OpenAI interface calls | Alibaba Bailian, Zhipu ChatGLMVLLM | Zhipu ChatGLMVLLM |
+
+In fact, any VLLM that supports OpenAI interface calls can be integrated and used.
+
+---
 
 ### TTS Speech Synthesis
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| API Call | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
+| Interface calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow(partial) |
-| Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
+| Local services | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
 
 ---
 
@@ -259,7 +307,7 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:---:|:---------:|:----:|:----:|:--:|
-| VAD | SileroVAD | Local Usage | Free | |
+| VAD | SileroVAD | Local use | Free | |
 
 ---
 
@@ -267,8 +315,16 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| Local Usage | FunASR, SherpaASR | FunASR, SherpaASR |
+| Local use | FunASR, SherpaASR | FunASR, SherpaASR |
-| API Call | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+| Interface calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+
+---
+
+### Voiceprint Recognition
+
+| Usage Method | Supported Platforms | Free Platforms |
+|:---:|:---:|:---:|
+| Local use | 3D-Speaker | 3D-Speaker |
 
 ---
 
@@ -276,8 +332,8 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:---------------:|:----:|:---------:|:--:|
-| Memory | mem0ai | API Call | 1000 calls/month quota | |
+| Memory | mem0ai | Interface calls | 1000 times/month quota | |
-| Memory | mem_local_short | Local Summary | Free | |
+| Memory | mem_local_short | Local summarization | Free | |
 
 ---
 
@@ -285,8 +341,8 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:-------------:|:----:|:-------:|:---------------------:|
-| Intent | intent_llm | API Call | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
+| Intent | intent_llm | Interface calls | Based on LLM pricing | Recognizes intent through large models, strong generalization |
-| Intent | function_call | API Call | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
+| Intent | function_call | Interface calls | Based on LLM pricing | Completes intent through large model function calling, fast speed, good effect |
 
 ---
 
@@ -294,11 +350,11 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Logo | Project/Company | Description |
 |:---:|:---:|:---|
-| <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
+| <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project is inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented on its basis |
-| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for establishing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing full-chain technical documentation support for this project |
+| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for formulating standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practice demonstrations for the Xiaozhi ecosystem; providing full-link technical documentation support for this project |
-| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing the function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
+| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function calling framework, MCP communication protocol, and plugin-based calling mechanism implementation code. Through standardized instruction scheduling system and dynamic expansion capabilities, it significantly improves the interaction efficiency and functional extensibility of frontend devices (IoT) |
-| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering the product user experience with their design experience serving over a thousand enterprises |
+| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, using their design practical experience serving over a thousand enterprises to empower this project's product user experience |
-| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening the visual system of this project, ensuring consistency and extensibility of the overall design style in multi-scenario applications |
+| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
 
 
 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">

docs/FAQ.md

@@ -112,7 +112,17 @@ VAD:
 
 参考教程[视觉模型使用指南](./mcp-vision-integration.md)
 
-### 10、更多问题，可联系我们反馈 💬
+### 10、如何开启MCP接入点 🔧
+
+1、先参考教程[MCP 接入点部署使用指南](./mcp-endpoint-enable.md)
+
+2、再参考教程[MCP 接入点使用指南](./mcp-endpoint-integration.md)
+
+### 12、如何开启声纹识别 🔊
+
+参考教程[声纹识别启用指南](./voiceprint-integration.md)
+
+### 13、更多问题，可联系我们反馈 💬
 
 可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
 

docs/dev-ops-integration.md

@@ -4,6 +4,8 @@
 
 本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。
 
+教程可参考B站博主`毕乐labs`发布的视频教程：[《开源小智服务器xiaozhi-server自动更新以及最新版本MCP接入点配置保姆教程》](https://www.bilibili.com/video/BV15H37zHE7Q)
+
 # 开始条件
 - 你的电脑/服务器是linux操作系统
 - 你已经跑通了整个流程
@@ -40,6 +42,9 @@ git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-serve
 
 此刻你需要把`model.pt`文件复制到新的目录去，你可以这样
 ```
+# 创建需要的目录
+mkdir -p /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/
+
 cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
 cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
 ```
@@ -161,3 +166,11 @@ tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server
 
 # 注意事项
 测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)
+
+## 常见问题
+
+### 1、为什么没有见到8001端口？
+回答：8001是开发环境使用的，用于运行前端的端口。如果你是服务器部署，不建议使用`npm run serve`启动8001端口运行前端，而是像本教程一样编译成html文件，然后使用nginx来管理访问。
+
+### 2、每次更新需要更新手动SQL语句吗？
+回答：不需要，因为项目使用**Liquibase**管理数据库版本，会自动执行新的sql脚本。

docs/docker/nginx.conf

@@ -42,9 +42,9 @@ http {
             proxy_set_header   Referer $http_referer;
             proxy_set_header   Cookie $http_cookie;
 
-            proxy_connect_timeout 10;
+            proxy_connect_timeout 15;
-            proxy_send_timeout 10;
+            proxy_send_timeout 15;
-            proxy_read_timeout 10;
+            proxy_read_timeout 15;
 
             proxy_set_header   X-Real-IP  $remote_addr;
             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;

docs/docker/start.sh

@@ -6,6 +6,7 @@ java -jar /app/xiaozhi-esp32-api.jar \
   --spring.datasource.druid.username=${SPRING_DATASOURCE_DRUID_USERNAME} \
   --spring.datasource.druid.password=${SPRING_DATASOURCE_DRUID_PASSWORD} \
   --spring.data.redis.host=${SPRING_DATA_REDIS_HOST} \
+  --spring.data.redis.password=${SPRING_DATA_REDIS_PASSWORD} \
   --spring.data.redis.port=${SPRING_DATA_REDIS_PORT} &
 
 # 启动Nginx（前台运行保持容器存活）

docs/mcp-endpoint-enable.md

@@ -0,0 +1,125 @@
+# MCP 接入点部署使用指南
+
+本教程包含3个部分
+- 1、如何部署MCP接入点这个服务
+- 2、全模块部署时，怎么配置MCP接入点
+- 3、单模块部署时，怎么配置MCP接入点
+
+# 1、如何部署MCP接入点这个服务
+
+## 第一步，下载mcp接入点项目源码
+
+浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server)
+
+打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。
+
+点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`mcp-endpoint-server-main`
+你需要把它重命名成`mcp-endpoint-server`。
+
+## 第二步，启动程序
+这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法
+
+```
+# 进入本项目源码根目录
+cd mcp-endpoint-server
+
+# 清除缓存
+docker compose -f docker-compose.yml down
+docker stop mcp-endpoint-server
+docker rm mcp-endpoint-server
+docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest
+
+# 启动docker容器
+docker compose -f docker-compose.yml up -d
+# 查看日志
+docker logs -f mcp-endpoint-server
+```
+
+此时，日志里会输出类似以下的日志
+```
+250705 INFO-=====下面的地址分别是智控台/单模块MCP接入点地址====
+250705 INFO-智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc
+250705 INFO-单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def
+250705 INFO-=====请根据具体部署选择使用，请勿泄露给任何人======
+```
+
+请你把两个接口地址复制出来：
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么
+原来我的接口地址
+```
+智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc
+单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def
+```
+就要改成
+```
+智控台MCP参数配置: http://192.168.1.25:8004/mcp_endpoint/health?key=abc
+单模块部署MCP接入点: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def
+```
+
+改好后，请使用浏览器直接访问`智控台MCP参数配置`。当浏览器出现类似这样的代码，说明是成功了。
+```
+{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"}
+```
+
+请你保留好上面两个`接口地址`，下一步要用到。
+
+# 2、全模块部署时，怎么配置MCP接入点
+
+如果你是全模块部署，使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。
+
+然后搜索参数`server.mcp_endpoint`，此时，它的值应该是`null`值。
+点击修改按钮，把上一步得来的`智控台MCP参数配置`粘贴到`参数值`里。然后保存。
+
+如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问mcp接入点，很大概率是网络防火墙，或者没有填写正确的局域网ip。
+
+# 3、单模块部署时，怎么配置MCP接入点
+
+如果你是单模块部署，找到你的配置文件`data/.config.yaml`。
+在配置文件搜索`mcp_endpoint`，如果没有找到，你就增加`mcp_endpoint`配置。类似我是就是这样
+```
+server:
+  websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  http_port: 8002
+log:
+  log_level: INFO
+
+# 此处可能还更多配置..
+
+mcp_endpoint: 你的接入点websocket地址
+```
+这时，请你把`如何部署MCP接入点这个服务`中得到的`单模块部署MCP接入点` 粘贴到 `mcp_endpoint`中。类似这样
+
+```
+server:
+  websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  http_port: 8002
+log:
+  log_level: INFO
+
+# 此处可能还更多配置
+
+mcp_endpoint: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def
+```
+
+配置好后，启动单模块会输出如下的日志。
+```
+250705[__main__]-INFO-初始化组件: vad成功 SileroVAD
+250705[__main__]-INFO-初始化组件: asr成功 FunASRServer
+250705[__main__]-INFO-OTA接口是          http://192.168.1.25:8002/xiaozhi/ota/
+250705[__main__]-INFO-视觉分析接口是     http://192.168.1.25:8002/mcp/vision/explain
+250705[__main__]-INFO-mcp接入点是        ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
+250705[__main__]-INFO-Websocket地址是    ws://192.168.1.25:8000/xiaozhi/v1/
+250705[__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问=======
+250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+250705[__main__]-INFO-=============================================================
+```
+
+如上，如果能输出类似的`mcp接入点是`中`ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`说明配置成功了。
+

docs/mcp-endpoint-integration.md

@@ -0,0 +1,94 @@
+# MCP 接入点使用指南
+
+本教程以虾哥开源的mcp计算器功能为示例，介绍如何将自己自定义的mcp服务接入到自己的接入点里。
+
+本教程的前提是，你的`xiaozhi-server`已经启用了mcp接入点功能，如果你还没启用，可以先根据[这个教程](./mcp-endpoint-enable.md)启用。
+
+# 如何为智能体接入一个简单的mcp功能，如计算器功能
+
+### 如果你是全模块部署
+如果你是全模块部署，你可以进入智控台，智能体管理，点击`配置角色`，在`意图识别`的右边，有一个`编辑功能`的按钮。
+
+点击这个按钮。在弹出的页面里，位于底部，会有`MCP接入点`，正常来说，会显示这个智能体的`MCP接入点地址`，接下来，我们来给这个智能体扩展一个基于MCP技术的计算器的功能。
+
+这个`MCP接入点地址`很重要，你等一下会用到。
+
+### 如果你是单模块部署
+如果你是单模块部署，且你已经在配置文件里配置了MCP接入点地址，那么正常来说，单模块部署启动的时候，会输出如下的日志。
+```
+250705[__main__]-INFO-初始化组件: vad成功 SileroVAD
+250705[__main__]-INFO-初始化组件: asr成功 FunASRServer
+250705[__main__]-INFO-OTA接口是          http://192.168.1.25:8002/xiaozhi/ota/
+250705[__main__]-INFO-视觉分析接口是     http://192.168.1.25:8002/mcp/vision/explain
+250705[__main__]-INFO-mcp接入点是        ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
+250705[__main__]-INFO-Websocket地址是    ws://192.168.1.25:8000/xiaozhi/v1/
+250705[__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问=======
+250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+250705[__main__]-INFO-=============================================================
+```
+
+如上，输出`mcp接入点是`中`ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`就是你的`MCP接入点地址`。
+
+这个`MCP接入点地址`很重要，你等一下会用到。
+
+## 第一步 下载虾哥MCP计算器项目代码
+
+浏览器打开虾哥写的[计算器项目](https://github.com/78/mcp-calculator)，
+
+打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。
+
+点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`mcp-calculatorr-main`
+你需要把它重命名成`mcp-calculator`。接下来，我们用命令行进入项目目录即安装依赖
+
+
+```bash
+# 进入项目目录
+cd mcp-calculator
+
+conda remove -n mcp-calculator --all -y
+conda create -n mcp-calculator python=3.10 -y
+conda activate mcp-calculator
+
+pip install -r requirements.txt
+```
+
+## 第二步 启动
+
+启动前，先从你的智控台的智能体里，复制到了MCP接入点的地址。
+
+例如我的智能体的mcp地址是
+```
+ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
+```
+
+开始输入命令
+
+```bash
+export MCP_ENDPOINT=ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
+```
+
+输入完后，启动程序
+
+```bash
+python mcp_pipe.py calculator.py
+```
+
+### 如果你是智控台部署
+如果你是智控台部署，启动完后，你再进入智控台，点击刷新MCP的接入状态，就会看到你扩展的功能列表了。
+
+### 如果你是单模块部署
+如果你是单模块部署，当设备连接后，会输出类似的日志，说明成功了
+
+```
+250705 -INFO-正在初始化MCP接入点: wss://2662r3426b.vicp.fun/mcp_e 
+250705 -INFO-发送MCP接入点初始化消息
+250705 -INFO-MCP接入点连接成功
+250705 -INFO-MCP接入点初始化成功
+250705 -INFO-统一工具处理器初始化完成
+250705 -INFO-MCP接入点服务器信息: name=Calculator, version=1.9.4
+250705 -INFO-MCP接入点支持的工具数量: 1
+250705 -INFO-所有MCP接入点工具已获取，客户端准备就绪
+250705 -INFO-工具缓存已刷新
+250705 -INFO-当前支持的函数列表: [ 'get_time', 'get_lunar', 'play_music', 'get_weather', 'handle_exit_intent', 'calculator']
+```
+如果包含了 `'calculator'`，说明设备将可以根据意图识别，调用计算器这个工具。

docs/newsnow_plugin_config.md

@@ -0,0 +1,105 @@
+# get_news_from_newsnow 插件新闻源配置指南
+
+## 概述
+
+`get_news_from_newsnow` 插件现在支持通过Web管理界面动态配置新闻源，不再需要修改代码。用户可以在智控台中为每个智能体配置不同的新闻源。
+
+## 配置方式
+
+### 1. 通过Web管理界面配置（推荐）
+
+1. 登录智控台
+2. 进入"角色配置"页面
+3. 选择要配置的智能体
+4. 点击"编辑功能"按钮
+5. 在右侧参数配置区域找到"newsnow新闻聚合"插件
+6. 在"新闻源配置"字段中输入分号分隔的中文名称
+
+### 2. 配置文件方式
+
+在 `config.yaml` 中配置：
+
+```yaml
+plugins:
+  get_news_from_newsnow:
+    url: "https://newsnow.busiyi.world/api/s?id="
+    news_sources: "澎湃新闻;百度热搜;财联社;微博;抖音"
+```
+
+## 新闻源配置格式
+
+新闻源配置使用分号分隔的中文名称，格式为：
+
+```
+中文名称1;中文名称2;中文名称3
+```
+
+### 配置示例
+
+```
+澎湃新闻;百度热搜;财联社;微博;抖音;知乎;36氪
+```
+
+## 支持的新闻源
+
+插件支持以下新闻源的中文名称：
+
+- 澎湃新闻
+- 百度热搜
+- 财联社
+- 微博
+- 抖音
+- 知乎
+- 36氪
+- 华尔街见闻
+- IT之家
+- 今日头条
+- 虎扑
+- 哔哩哔哩
+- 快手
+- 雪球
+- 格隆汇
+- 法布财经
+- 金十数据
+- 牛客
+- 少数派
+- 稀土掘金
+- 凤凰网
+- 虫部落
+- 联合早报
+- 酷安
+- 远景论坛
+- 参考消息
+- 卫星通讯社
+- 百度贴吧
+- 靠谱新闻
+- 以及更多...
+
+## 默认配置
+
+如果未配置新闻源，插件将使用以下默认配置：
+
+```
+澎湃新闻;百度热搜;财联社
+```
+
+## 使用说明
+
+1. **配置新闻源**：在Web界面或配置文件中设置新闻源的中文名称，用分号分隔
+2. **调用插件**：用户可以说"播报新闻"或"获取新闻"
+3. **指定新闻源**：用户可以说"播报澎湃新闻"或"获取百度热搜"
+4. **获取详情**：用户可以说"详细介绍这条新闻"
+
+## 工作原理
+
+1. 插件接受中文名称作为参数（如"澎湃新闻"）
+2. 根据配置的新闻源列表，将中文名称转换为对应的英文ID（如"thepaper"）
+3. 使用英文ID调用API获取新闻数据
+4. 返回新闻内容给用户
+
+## 注意事项
+
+1. 配置的中文名称必须与 CHANNEL_MAP 中定义的名称完全一致
+2. 配置更改后需要重启服务或重新加载配置
+3. 如果配置的新闻源无效，插件会自动使用默认新闻源
+4. 多个新闻源之间使用英文分号(;)分隔，不要使用中文分号(；)

docs/voiceprint-integration.md

@@ -0,0 +1,233 @@
+# 声纹识别启用指南
+
+本教程包含3个部分
+- 1、如何部署声纹识别这个服务
+- 2、全模块部署时，怎么配置声纹识别接口
+- 3、最简化部署时，怎么配置声纹识别
+
+# 1、如何部署声纹识别这个服务
+
+## 第一步，下载声纹识别项目源码
+
+浏览器打开[声纹识别项目地址](https://github.com/xinnan-tech/voiceprint-api)
+
+打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。
+
+点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`voiceprint-api-main`
+你需要把它重命名成`voiceprint-api`。
+
+## 第二步， 创建数据库和表
+
+声纹识别需要依赖`mysql`数据库。如果你之前已经部署`智控台`，说明你已经安装了`mysql`。你可以共用它。
+
+你可以你试一下在宿主机使用`telnet`命令，看看能不能正常访问`mysql`的`3306`端口。
+```
+telnet 127.0.0.1 3306
+```
+如果能访问到3306端口，请忽略以下的内容，直接进入第三步。
+
+如果不能访问，你需要回忆一下，你的`mysql`是怎么安装的。
+
+如果你的mysql是通过自己使用安装包安装的，说明你的`mysql`做了网络隔离。你可能先解访问`mysql`的`3306`端口这个问题。
+
+如果你`mysql`是通过本项目的`docker-compose_all.yml`安装的。你需要找一下你当时创建数据库的`docker-compose_all.yml`文件，修改以下的内容
+
+修改前
+```
+  xiaozhi-esp32-server-db:
+    ...
+    networks:
+      - default
+    expose:
+      - "3306:3306"
+```
+
+修改后
+```
+  xiaozhi-esp32-server-db:
+    ...
+    networks:
+      - default
+    ports:
+      - "3306:3306"
+```
+
+注意是将`xiaozhi-esp32-server-db`下面的`expose`改成`ports`。改完后，需要重新启动。以下是重启mysql的命令：
+
+```
+# 进入你docker-compose_all.yml所在的文件夹，例如我的是xiaozhi-server
+cd xiaozhi-server
+docker compose -f docker-compose_all.yml down
+docker compose -f docker-compose.yml up -d
+```
+
+启动完后，在宿主机再使用`telnet`命令，看看能不能正常访问`mysql`的`3306`端口。
+```
+telnet 127.0.0.1 3306
+```
+正常来说这样就可以访问的了。
+
+## 第三步， 创建数据库和表
+如果你的宿主机，能正常访问mysql数据库，那就在mysql上创建一个名字为`voiceprint_db`的数据库和`voiceprints`表。
+
+```
+CREATE DATABASE voiceprint_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+USE voiceprint_db;
+
+CREATE TABLE voiceprints (
+    id INT AUTO_INCREMENT PRIMARY KEY,
+    speaker_id VARCHAR(255) NOT NULL UNIQUE,
+    feature_vector LONGBLOB NOT NULL,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    INDEX idx_speaker_id (speaker_id)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
+```
+
+## 第四步， 配置数据库连接
+
+进入`voiceprint-api`文件夹，创建名字为`data`的文件夹。
+
+把`voiceprint-api`根目录里的`voiceprint.yaml`，复制到`data`的文件夹，将它重命名为`.voiceprint.yaml`
+
+接下来，你需要重点配置一下`.voiceprint.yaml`里的数据库连接。
+
+```
+mysql:
+  host: "127.0.0.1"
+  port: 3306
+  user: "root"
+  password: "your_password"
+  database: "voiceprint_db"
+```
+
+注意！由于你的声纹识别服务是使用docker部署，`host`需要填写成你`mysql所在机器的局域网ip`。
+
+注意！由于你的声纹识别服务是使用docker部署，`host`需要填写成你`mysql所在机器的局域网ip`。
+
+注意！由于你的声纹识别服务是使用docker部署，`host`需要填写成你`mysql所在机器的局域网ip`。
+
+## 第五步，启动程序
+这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/voiceprint-api/blob/main/README.md)使用源码运行。以下是docker运行的方法
+
+```
+# 进入本项目源码根目录
+cd voiceprint-api
+
+# 清除缓存
+docker compose -f docker-compose.yml down
+docker stop voiceprint-api
+docker rm voiceprint-api
+docker rmi ghcr.nju.edu.cn/xinnan-tech/voiceprint-api:latest
+
+# 启动docker容器
+docker compose -f docker-compose.yml up -d
+# 查看日志
+docker logs -f voiceprint-api
+```
+
+此时，日志里会输出类似以下的日志
+```
+250711 INFO-🚀 开始: 生产环境服务启动（Uvicorn），监听地址: 0.0.0.0:8005
+250711 INFO-============================================================
+250711 INFO-声纹接口地址: http://127.0.0.1:8005/voiceprint/health?key=abcd
+250711 INFO-============================================================
+```
+
+请你把声纹接口地址复制出来：
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么
+原来我的接口地址
+```
+http://127.0.0.1:8005/voiceprint/health?key=abcd
+
+```
+就要改成
+```
+http://192.168.1.25:8005/voiceprint/health?key=abcd
+```
+
+改好后，请使用浏览器直接访问`声纹接口地址`。当浏览器出现类似这样的代码，说明是成功了。
+```
+{"total_voiceprints":0,"status":"healthy"}
+```
+
+请你保留好修改后的`声纹接口地址`，下一步要用到。
+
+# 2、全模块部署时，怎么配置声纹识别
+
+## 第一步 配置接口
+如果你是全模块部署，使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。
+
+然后搜索参数`server.voice_print`，此时，它的值应该是`null`值。
+点击修改按钮，把上一步得来的`声纹接口地址`粘贴到`参数值`里。然后保存。
+
+如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问声纹识别，很大概率是网络防火墙，或者没有填写正确的局域网ip。
+
+## 第二步 设置智能体记忆模式
+
+进入你的智能体的角色配置里，将记忆设置成`本地短期记忆`，一定要开启`上报文字+语音`。
+
+## 第三步 和你的智能体聊天
+
+将你的设备通电，然后和他用正常的语速和音调聊天。
+
+## 第四步 设置声纹
+
+在智控台，`智能体管理`页面，在智能体的面板里，有一个`声纹识别`按钮，点击它。在底部有一个`新增按钮`。就可以对某个人说的话进行声纹注册。
+在弹出的框里，`描述`这个属性建议填写上，可以是这个人的职业、性格、爱好。方便智能体对说话人进行分析和了解。
+
+## 第三步 和你的智能体聊天
+
+将你的设备通电，问它，你知道我是谁吗？如果他能回答得出，说明声纹识别功能正常。
+
+# 3、最简化部署时，怎么配置声纹识别
+
+## 第一步 配置接口
+打开 `xiaozhi-server/data/.config.yaml` 文件（如果没有需要创建），然后添加/修改以下内容：
+
+```
+# 声纹识别配置
+voiceprint:
+  # 声纹接口地址
+  url: 你的声纹接口地址
+  # 说话人配置：speaker_id,名称,描述
+  speakers:
+    - "test1,张三,张三是一个程序员"
+    - "test2,李四,李四是一个产品经理"
+    - "test3,王五,王五是一个设计师"
+```
+
+把上一步得来的 `声纹接口地址` 粘贴到 `url` 里。然后保存。
+
+`speakers` 参数依据需求添加。这里需要注意这个 `speaker_id` 参数，后面注册声纹会用到。
+
+## 第二步 注册声纹
+如果你已经启动了声纹服务，本地浏览器里访问 `http://localhost:8005/voiceprint/docs` 即可查看 API 文档，这里只说明注册声纹的 API 如何使用。
+
+注册声纹的 API 地址为 `http://localhost:8005/voiceprint/register`，请求方式为 POST。
+
+请求头需要包含 Bearer Token 认证，token 为 `声纹接口地址` 中 `?key=` 后的部分，比如如果我的声纹注册地址为 `http://127.0.0.1:8005/voiceprint/health?key=abcd`，那么我的 token 就是`abcd`。
+
+请求体包含说话人 ID（speaker_id），和 WAV 音频文件（file），请求示例如下：
+
+```
+curl -X POST \
+  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
+  -F "speaker_id=your_speaker_id_here" \
+  -F "file=@/path/to/your/file" \
+  http://localhost:8005/voiceprint/register
+```
+
+ 这里的 `file` 是要注册的说话人说话的音频文件， `speaker_id` 需要和第一步配置接口的 `speaker_id` 保持一致。比如说我需要注册张三的声纹，在 `.config.yaml` 中填的张三的 `speaker_id` 为 `test1`，那么我注册张三声纹的时候，请求体里填的 `speaker_id` 就是 `test1`， `file` 填的就是张三说一段话的音频文件。
+
+ ## 第三步 启动服务
+
+启动小智服务器和声纹服务，即可正常使用。

main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java

@@ -111,6 +111,16 @@ public interface Constant {
      */
     String FILE_EXTENSION_SEG = ".";
 
+    /**
+     * mcp接入点路径
+     */
+    String SERVER_MCP_ENDPOINT = "server.mcp_endpoint";
+
+    /**
+     * mcp接入点路径
+     */
+    String SERVER_VOICE_PRINT = "server.voice_print";
+
     /**
      * 无记忆
      */
@@ -227,7 +237,7 @@ public interface Constant {
     /**
      * 版本号
      */
-    public static final String VERSION = "0.5.8";
+    public static final String VERSION = "0.7.2";
 
     /**
      * 无效固件URL

main/manager-api/src/main/java/xiaozhi/common/utils/AESUtils.java

@@ -0,0 +1,78 @@
+package xiaozhi.common.utils;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+
+import javax.crypto.Cipher;
+import javax.crypto.spec.SecretKeySpec;
+
+public class AESUtils {
+
+    private static final String ALGORITHM = "AES";
+    private static final String TRANSFORMATION = "AES/ECB/PKCS5Padding";
+
+    /**
+     * AES加密
+     * 
+     * @param key       密钥（16位、24位或32位）
+     * @param plainText 待加密字符串
+     * @return 加密后的Base64字符串
+     */
+    public static String encrypt(String key, String plainText) {
+        try {
+            // 确保密钥长度为16、24或32位
+            byte[] keyBytes = padKey(key.getBytes(StandardCharsets.UTF_8));
+            SecretKeySpec secretKey = new SecretKeySpec(keyBytes, ALGORITHM);
+
+            Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+            cipher.init(Cipher.ENCRYPT_MODE, secretKey);
+
+            byte[] encryptedBytes = cipher.doFinal(plainText.getBytes(StandardCharsets.UTF_8));
+            return Base64.getEncoder().encodeToString(encryptedBytes);
+        } catch (Exception e) {
+            throw new RuntimeException("AES加密失败", e);
+        }
+    }
+
+    /**
+     * AES解密
+     * 
+     * @param key           密钥（16位、24位或32位）
+     * @param encryptedText 待解密的Base64字符串
+     * @return 解密后的字符串
+     */
+    public static String decrypt(String key, String encryptedText) {
+        try {
+            // 确保密钥长度为16、24或32位
+            byte[] keyBytes = padKey(key.getBytes(StandardCharsets.UTF_8));
+            SecretKeySpec secretKey = new SecretKeySpec(keyBytes, ALGORITHM);
+
+            Cipher cipher = Cipher.getInstance(TRANSFORMATION);
+            cipher.init(Cipher.DECRYPT_MODE, secretKey);
+
+            byte[] encryptedBytes = Base64.getDecoder().decode(encryptedText);
+            byte[] decryptedBytes = cipher.doFinal(encryptedBytes);
+            return new String(decryptedBytes, StandardCharsets.UTF_8);
+        } catch (Exception e) {
+            throw new RuntimeException("AES解密失败", e);
+        }
+    }
+
+    /**
+     * 填充密钥到指定长度（16、24或32位）
+     * 
+     * @param keyBytes 原始密钥字节数组
+     * @return 填充后的密钥字节数组
+     */
+    private static byte[] padKey(byte[] keyBytes) {
+        int keyLength = keyBytes.length;
+        if (keyLength == 16 || keyLength == 24 || keyLength == 32) {
+            return keyBytes;
+        }
+
+        // 如果密钥长度不足，用0填充；如果超过，截取前32位
+        byte[] paddedKey = new byte[32];
+        System.arraycopy(keyBytes, 0, paddedKey, 0, Math.min(keyLength, 32));
+        return paddedKey;
+    }
+}

main/manager-api/src/main/java/xiaozhi/common/utils/HashEncryptionUtil.java

@@ -0,0 +1,52 @@
+package xiaozhi.common.utils;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+
+/**
+ * 哈希加密算法的工具类
+ * @author zjy
+ */
+@Slf4j
+public class HashEncryptionUtil {
+    /**
+     * 使用md5进行加密
+     * @param context 被加密的内容
+     * @return 哈希值
+     */
+    public static String Md5hexDigest(String context){
+        return hexDigest(context,"MD5");
+    }
+
+    /**
+     * 指定哈希算法进行加密
+     * @param context 被加密的内容
+     * @param algorithm 哈希算法
+     * @return 哈希值
+     */
+   public static String hexDigest(String context,String algorithm ){
+       // 获取MD5算法实例
+       MessageDigest md = null;
+       try {
+           md = MessageDigest.getInstance(algorithm);
+       } catch (NoSuchAlgorithmException e) {
+           log.error("加密失败的算法：{}",algorithm);
+           throw new RuntimeException("加密失败，"+ algorithm +"哈希算法系统不支持");
+       }
+       // 计算智能体id的MD5值
+       byte[] messageDigest = md.digest(context.getBytes());
+       // 将字节数组转换为十六进制字符串
+       StringBuilder hexString = new StringBuilder();
+       for (byte b : messageDigest) {
+           String hex = Integer.toHexString(0xFF & b);
+           if (hex.length() == 1) {
+               hexString.append('0');
+           }
+           hexString.append(hex);
+       }
+       return hexString.toString();
+   }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/Enums/AgentChatHistoryType.java

@@ -0,0 +1,21 @@
+package xiaozhi.modules.agent.Enums;
+
+
+import lombok.Getter;
+
+/**
+ * 智能体聊天记录类型
+ */
+@Getter
+public enum AgentChatHistoryType {
+
+    USER((byte) 1),
+    AGENT((byte) 2);
+
+    private final byte value;
+
+    AgentChatHistoryType(byte i) {
+        this.value = i;
+    }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java

@@ -47,6 +47,7 @@ import xiaozhi.modules.agent.service.AgentChatHistoryService;
 import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.AgentTemplateService;
+import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO;
 import xiaozhi.modules.agent.vo.AgentInfoVO;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
@@ -181,6 +182,33 @@ public class AgentController {
         List<AgentChatHistoryDTO> result = agentChatHistoryService.getChatHistoryBySessionId(id, sessionId);
         return new Result<List<AgentChatHistoryDTO>>().ok(result);
     }
+    @GetMapping("/{id}/chat-history/user")
+    @Operation(summary = "获取智能体聊天记录（用户）")
+    @RequiresPermissions("sys:role:normal")
+    public Result<List<AgentChatHistoryUserVO>> getRecentlyFiftyByAgentId(
+            @PathVariable("id") String id) {
+        // 获取当前用户
+        UserDetail user = SecurityUser.getUser();
+
+        // 检查权限
+        if (!agentService.checkAgentPermission(id, user.getId())) {
+            return new Result<List<AgentChatHistoryUserVO>>().error("没有权限查看该智能体的聊天记录");
+        }
+
+        // 查询聊天记录
+        List<AgentChatHistoryUserVO> data = agentChatHistoryService.getRecentlyFiftyByAgentId(id);
+        return new Result<List<AgentChatHistoryUserVO>>().ok(data);
+    }
+
+    @GetMapping("/{id}/chat-history/audio")
+    @Operation(summary = "获取音频内容")
+    @RequiresPermissions("sys:role:normal")
+    public Result<String> getContentByAudioId(
+            @PathVariable("id") String id) {
+        // 查询聊天记录
+        String data = agentChatHistoryService.getContentByAudioId(id);
+        return new Result<String>().ok(data);
+    }
 
     @PostMapping("/audio/{audioId}")
     @Operation(summary = "获取音频下载ID")

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentMcpAccessPointController.java

@@ -0,0 +1,66 @@
+package xiaozhi.modules.agent.controller;
+
+import java.util.List;
+
+import org.apache.shiro.authz.annotation.RequiresPermissions;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import lombok.RequiredArgsConstructor;
+import xiaozhi.common.user.UserDetail;
+import xiaozhi.common.utils.Result;
+import xiaozhi.modules.agent.service.AgentMcpAccessPointService;
+import xiaozhi.modules.agent.service.AgentService;
+import xiaozhi.modules.security.user.SecurityUser;
+
+@Tag(name = "智能体Mcp接入点管理")
+@RequiredArgsConstructor
+@RestController
+@RequestMapping("/agent/mcp")
+public class AgentMcpAccessPointController {
+    private final AgentMcpAccessPointService agentMcpAccessPointService;
+    private final AgentService agentService;
+
+    /**
+     * 获取智能体的Mcp接入点地址
+     * 
+     * @param audioId 智能体id
+     * @return 返回错误提醒或者Mcp接入点地址
+     */
+    @Operation(summary = "获取智能体的Mcp接入点地址")
+    @GetMapping("/address/{agentId}")
+    @RequiresPermissions("sys:role:normal")
+    public Result<String> getAgentMcpAccessAddress(@PathVariable("agentId") String agentId) {
+        // 获取当前用户
+        UserDetail user = SecurityUser.getUser();
+
+        // 检查权限
+        if (!agentService.checkAgentPermission(agentId, user.getId())) {
+            return new Result<String>().error("没有权限查看该智能体的MCP接入点地址");
+        }
+        String agentMcpAccessAddress = agentMcpAccessPointService.getAgentMcpAccessAddress(agentId);
+        if (agentMcpAccessAddress == null) {
+            return new Result<String>().ok("请联系管理员进入参数管理配置mcp接入点地址");
+        }
+        return new Result<String>().ok(agentMcpAccessAddress);
+    }
+
+    @Operation(summary = "获取智能体的Mcp工具列表")
+    @GetMapping("/tools/{agentId}")
+    @RequiresPermissions("sys:role:normal")
+    public Result<List<String>> getAgentMcpToolsList(@PathVariable("agentId") String agentId) {
+        // 获取当前用户
+        UserDetail user = SecurityUser.getUser();
+
+        // 检查权限
+        if (!agentService.checkAgentPermission(agentId, user.getId())) {
+            return new Result<List<String>>().error("没有权限查看该智能体的MCP工具列表");
+        }
+        List<String> agentMcpToolsList = agentMcpAccessPointService.getAgentMcpToolsList(agentId);
+        return new Result<List<String>>().ok(agentMcpToolsList);
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentVoicePrintController.java

@@ -0,0 +1,86 @@
+package xiaozhi.modules.agent.controller;
+
+import java.util.List;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.shiro.authz.annotation.RequiresPermissions;
+import org.springframework.web.bind.annotation.DeleteMapping;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import jakarta.validation.Valid;
+import lombok.AllArgsConstructor;
+import xiaozhi.common.exception.RenException;
+import xiaozhi.common.utils.Result;
+import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO;
+import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO;
+import xiaozhi.modules.agent.service.AgentVoicePrintService;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
+import xiaozhi.modules.security.user.SecurityUser;
+import xiaozhi.modules.sys.service.SysParamsService;
+
+@Tag(name = "智能体声纹管理")
+@AllArgsConstructor
+@RestController
+@RequestMapping("/agent/voice-print")
+public class AgentVoicePrintController {
+    private final AgentVoicePrintService agentVoicePrintService;
+    private final SysParamsService sysParamsService;
+
+    @PostMapping
+    @Operation(summary = "创建智能体的声纹")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> save(@RequestBody @Valid AgentVoicePrintSaveDTO dto) {
+        boolean b = agentVoicePrintService.insert(dto);
+        if (b) {
+            return new Result<>();
+        }
+        return new Result<Void>().error("智能体的声纹创建失败");
+    }
+
+    @PutMapping
+    @Operation(summary = "更新智能体的对应声纹")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> update(@RequestBody @Valid AgentVoicePrintUpdateDTO dto) {
+        Long userId = SecurityUser.getUserId();
+        boolean b = agentVoicePrintService.update(userId, dto);
+        if (b) {
+            return new Result<>();
+        }
+        return new Result<Void>().error("智能体的对应声纹更新失败");
+    }
+
+    @DeleteMapping("/{id}")
+    @Operation(summary = "删除智能体对应声纹")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> delete(@PathVariable String id) {
+        Long userId = SecurityUser.getUserId();
+        // 先删除关联的设备
+        boolean delete = agentVoicePrintService.delete(userId, id);
+        if (delete) {
+            return new Result<>();
+        }
+        return new Result<Void>().error("智能体的对应声纹删除失败");
+    }
+
+    @GetMapping("/list/{id}")
+    @Operation(summary = "获取用户指定智能体声纹列表")
+    @RequiresPermissions("sys:role:normal")
+    public Result<List<AgentVoicePrintVO>> list(@PathVariable String id) {
+        String voiceprintUrl = sysParamsService.getValue("server.voice_print", true);
+        if (StringUtils.isBlank(voiceprintUrl) || "null".equals(voiceprintUrl)) {
+            throw new RenException("声纹接口未配置，请先在参数配置中配置声纹接口地址(server.voice_print)");
+        }
+        Long userId = SecurityUser.getUserId();
+        List<AgentVoicePrintVO> list = agentVoicePrintService.list(userId, id);
+        return new Result<List<AgentVoicePrintVO>>().ok(list);
+    }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentVoicePrintDao.java

@@ -0,0 +1,20 @@
+package xiaozhi.modules.agent.dao;
+
+import org.apache.ibatis.annotations.Mapper;
+
+import com.baomidou.mybatisplus.core.mapper.BaseMapper;
+
+import xiaozhi.modules.agent.entity.AgentChatHistoryEntity;
+import xiaozhi.modules.agent.entity.AgentVoicePrintEntity;
+
+/**
+ * {@link AgentChatHistoryEntity} 智能体聊天历史记录Dao对象
+ *
+ * @author Goody
+ * @version 1.0, 2025/4/30
+ * @since 1.0.0
+ */
+@Mapper
+public interface AgentVoicePrintDao extends BaseMapper<AgentVoicePrintEntity> {
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintSaveDTO.java

@@ -0,0 +1,28 @@
+package xiaozhi.modules.agent.dto;
+
+import lombok.Data;
+
+/**
+ * 保存智能体声纹的dto
+ *
+ * @author zjy
+ */
+@Data
+public class AgentVoicePrintSaveDTO {
+    /**
+     * 关联的智能体id
+     */
+    private String agentId;
+    /**
+     * 音频文件id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintUpdateDTO.java

@@ -0,0 +1,28 @@
+package xiaozhi.modules.agent.dto;
+
+import lombok.Data;
+
+/**
+ * 修改智能体声纹的dto
+ *
+ * @author zjy
+ */
+@Data
+public class AgentVoicePrintUpdateDTO {
+    /**
+     * 智能体声纹id
+     */
+    private String id;
+    /**
+     * 音频文件id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/IdentifyVoicePrintResponse.java

@@ -0,0 +1,21 @@
+package xiaozhi.modules.agent.dto;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import lombok.Data;
+
+/**
+ * 声纹识别接口返回的对象
+ */
+@Data
+public class IdentifyVoicePrintResponse {
+    /**
+     * 最匹配的声纹id
+     */
+    @JsonProperty("speaker_id")
+    private String speakerId;
+    /**
+     * 声纹的分数
+     */
+    private Double score;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/McpJsonRpcRequest.java

@@ -0,0 +1,32 @@
+package xiaozhi.modules.agent.dto;
+
+import lombok.Data;
+
+/**
+ * MCP JSON-RPC 请求 DTO
+ */
+@Data
+public class McpJsonRpcRequest {
+    private String jsonrpc = "2.0";
+    private String method;
+    private Object params;
+    private Integer id;
+
+    public McpJsonRpcRequest() {
+    }
+
+    public McpJsonRpcRequest(String method) {
+        this.method = method;
+    }
+
+    public McpJsonRpcRequest(String method, Object params, Integer id) {
+        this.method = method;
+        this.params = params;
+        this.id = id;
+    }
+
+    public McpJsonRpcRequest(String method, Object params) {
+        this.method = method;
+        this.params = params;
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/McpJsonRpcResponse.java

@@ -0,0 +1,48 @@
+package xiaozhi.modules.agent.dto;
+
+import lombok.Data;
+
+/**
+ * MCP JSON-RPC 响应 DTO
+ */
+@Data
+public class McpJsonRpcResponse {
+    private String jsonrpc = "2.0";
+    private Integer id;
+    private McpResult result;
+    private McpError error;
+
+    public McpJsonRpcResponse() {
+    }
+
+    @Data
+    public static class McpResult {
+        private String type;
+        private String message;
+        private String agent_id;
+        private McpTool[] tools;
+
+        public McpResult() {
+        }
+    }
+
+    @Data
+    public static class McpTool {
+        private String name;
+        private String description;
+        private Object inputSchema;
+
+        public McpTool() {
+        }
+    }
+
+    @Data
+    public static class McpError {
+        private Integer code;
+        private String message;
+        private Object data;
+
+        public McpError() {
+        }
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentVoicePrintEntity.java

@@ -0,0 +1,64 @@
+package xiaozhi.modules.agent.entity;
+
+import java.util.Date;
+
+import com.baomidou.mybatisplus.annotation.FieldFill;
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableField;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+
+import lombok.Data;
+
+/**
+ * 智能体声纹表
+ *
+ * @author zjy
+ */
+@TableName(value = "ai_agent_voice_print")
+@Data
+public class AgentVoicePrintEntity {
+    /**
+     * 主键id
+     */
+    @TableId(type = IdType.ASSIGN_UUID)
+    private String id;
+    /**
+     * 关联的智能体id
+     */
+    private String agentId;
+    /**
+     * 关联的音频id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+
+    /**
+     * 创建者
+     */
+    @TableField(fill = FieldFill.INSERT)
+    private Long creator;
+    /**
+     * 创建时间
+     */
+    @TableField(fill = FieldFill.INSERT)
+    private Date createDate;
+
+    /**
+     * 更新者
+     */
+    @TableField(fill = FieldFill.INSERT_UPDATE)
+    private Long updater;
+    /**
+     * 更新时间
+     */
+    @TableField(fill = FieldFill.INSERT_UPDATE)
+    private Date updateDate;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentChatHistoryService.java

@@ -9,6 +9,7 @@ import xiaozhi.common.page.PageData;
 import xiaozhi.modules.agent.dto.AgentChatHistoryDTO;
 import xiaozhi.modules.agent.dto.AgentChatSessionDTO;
 import xiaozhi.modules.agent.entity.AgentChatHistoryEntity;
+import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO;
 
 /**
  * 智能体聊天记录表处理service
@@ -44,4 +45,30 @@ public interface AgentChatHistoryService extends IService<AgentChatHistoryEntity
      * @param deleteText  是否删除文本
      */
     void deleteByAgentId(String agentId, Boolean deleteAudio, Boolean deleteText);
+
+    /**
+     * 根据智能体ID获取最近50条用户的聊天记录数据（带音频数据）
+     *
+     * @param agentId 智能体id
+     * @return 聊天记录列表（只有用户）
+     */
+    List<AgentChatHistoryUserVO> getRecentlyFiftyByAgentId(String agentId);
+
+    /**
+     * 根据音频数据ID获取聊天内容
+     *
+     * @param audioId 音频id
+     * @return 聊天内容
+     */
+    String getContentByAudioId(String audioId);
+
+
+    /**
+     * 查询此音频id是否属于此智能体
+     *
+     * @param audioId 音频id
+     * @param agentId 音频id
+     * @return T：属于 F：不属于
+     */
+    boolean isAudioOwnedByAgent(String audioId,String agentId);
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentMcpAccessPointService.java

@@ -0,0 +1,25 @@
+package xiaozhi.modules.agent.service;
+
+
+import java.util.List;
+
+/**
+ * 智能体Mcp接入点处理service
+ *
+ * @author zjy
+ */
+public interface AgentMcpAccessPointService {
+    /**
+     * 获取智能体的mcp接入点地址
+     * @param id 智能体id
+     * @return mcp接入点地址
+     */
+   String getAgentMcpAccessAddress(String id);
+
+    /**
+     * 获取智能体的mcp接入点已有的工具列表
+     * @param id 智能体id
+     * @return 工具列表
+     */
+   List<String> getAgentMcpToolsList(String id);
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentVoicePrintService.java

@@ -0,0 +1,50 @@
+package xiaozhi.modules.agent.service;
+
+import java.util.List;
+
+import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO;
+import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
+
+/**
+ * 智能体声纹处理service
+ *
+ * @author zjy
+ */
+public interface AgentVoicePrintService {
+    /**
+     * 添加智能体新的声纹
+     *
+     * @param dto 保存智能体声纹的数据
+     * @return T:成功 F：失败
+     */
+    boolean insert(AgentVoicePrintSaveDTO dto);
+
+    /**
+     * 删除智能体的指的声纹
+     *
+     * @param userId       当前登录的用户id
+     * @param voicePrintId 声纹id
+     * @return 是否成功 T:成功 F：失败
+     */
+    boolean delete(Long userId, String voicePrintId);
+
+    /**
+     * 获取指定智能体的所有声纹数据
+     *
+     * @param userId  当前登录的用户id
+     * @param agentId 智能体id
+     * @return 声纹数据集合
+     */
+    List<AgentVoicePrintVO> list(Long userId, String agentId);
+
+    /**
+     * 更新智能体的指的声纹数据
+     *
+     * @param userId 当前登录的用户id
+     * @param dto    修改的声纹的数据
+     * @return 是否成功 T:成功 F：失败
+     */
+    boolean update(Long userId, AgentVoicePrintUpdateDTO dto);
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/biz/impl/AgentChatHistoryBizServiceImpl.java

@@ -19,6 +19,8 @@ import xiaozhi.modules.agent.service.AgentChatAudioService;
 import xiaozhi.modules.agent.service.AgentChatHistoryService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.biz.AgentChatHistoryBizService;
+import xiaozhi.modules.device.entity.DeviceEntity;
+import xiaozhi.modules.device.service.DeviceService;
 
 /**
  * {@link AgentChatHistoryBizService} impl
@@ -35,6 +37,7 @@ public class AgentChatHistoryBizServiceImpl implements AgentChatHistoryBizServic
     private final AgentChatHistoryService agentChatHistoryService;
     private final AgentChatAudioService agentChatAudioService;
     private final RedisUtils redisUtils;
+    private final DeviceService deviceService;
 
     /**
      * 处理聊天记录上报，包括文件上传和相关信息记录
@@ -68,6 +71,15 @@ public class AgentChatHistoryBizServiceImpl implements AgentChatHistoryBizServic
 
         // 更新设备最后对话时间
         redisUtils.set(RedisKeys.getAgentDeviceLastConnectedAtById(agentId), new Date());
+
+        // 更新设备最后连接时间
+        DeviceEntity device = deviceService.getDeviceByMacAddress(macAddress);
+        if (device != null) {
+            deviceService.updateDeviceConnectionInfo(agentId, device.getId(), null);
+        } else {
+            log.warn("聊天记录上报时，未找到mac地址为 {} 的设备", macAddress);
+        }
+
         return Boolean.TRUE;
     }
 

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentChatHistoryServiceImpl.java

@@ -8,6 +8,7 @@ import java.util.stream.Collectors;
 import org.springframework.stereotype.Service;
 import org.springframework.transaction.annotation.Transactional;
 
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
 import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
 import com.baomidou.mybatisplus.core.metadata.IPage;
 import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
@@ -16,11 +17,14 @@ import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
 import xiaozhi.common.constant.Constant;
 import xiaozhi.common.page.PageData;
 import xiaozhi.common.utils.ConvertUtils;
+import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.Enums.AgentChatHistoryType;
 import xiaozhi.modules.agent.dao.AiAgentChatHistoryDao;
 import xiaozhi.modules.agent.dto.AgentChatHistoryDTO;
 import xiaozhi.modules.agent.dto.AgentChatSessionDTO;
 import xiaozhi.modules.agent.entity.AgentChatHistoryEntity;
 import xiaozhi.modules.agent.service.AgentChatHistoryService;
+import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO;
 
 /**
  * 智能体聊天记录表处理service {@link AgentChatHistoryService} impl
@@ -90,4 +94,74 @@ public class AgentChatHistoryServiceImpl extends ServiceImpl<AiAgentChatHistoryD
         }
 
     }
+
+    @Override
+    public List<AgentChatHistoryUserVO> getRecentlyFiftyByAgentId(String agentId) {
+        // 构建查询条件(不添加按照创建时间排序，数据本来就是主键越大创建时间越大
+        // 不添加这样可以减少排序全部数据在分页的全盘扫描消耗)
+        LambdaQueryWrapper<AgentChatHistoryEntity> wrapper = new LambdaQueryWrapper<>();
+        wrapper.select(AgentChatHistoryEntity::getContent, AgentChatHistoryEntity::getAudioId)
+                .eq(AgentChatHistoryEntity::getAgentId, agentId)
+                .eq(AgentChatHistoryEntity::getChatType, AgentChatHistoryType.USER.getValue())
+                .isNotNull(AgentChatHistoryEntity::getAudioId);
+
+        // 构建分页查询，查询前50页数据
+        Page<AgentChatHistoryEntity> pageParam = new Page<>(0, 50);
+        IPage<AgentChatHistoryEntity> result = this.baseMapper.selectPage(pageParam, wrapper);
+        return result.getRecords().stream().map(item -> {
+            AgentChatHistoryUserVO vo = ConvertUtils.sourceToTarget(item, AgentChatHistoryUserVO.class);
+            // 处理 content 字段，确保只返回聊天内容
+            if (vo != null && vo.getContent() != null) {
+                vo.setContent(extractContentFromString(vo.getContent()));
+            }
+            return vo;
+        }).toList();
+    }
+
+    /**
+     * 从 content 字段中提取聊天内容
+     * 如果 content 是 JSON 格式（如 {"speaker": "未知说话人", "content": "现在几点了。"}），则提取 content
+     * 字段
+     * 如果 content 是普通字符串，则直接返回
+     * 
+     * @param content 原始内容
+     * @return 提取的聊天内容
+     */
+    private String extractContentFromString(String content) {
+        if (content == null || content.trim().isEmpty()) {
+            return content;
+        }
+
+        // 尝试解析为 JSON
+        try {
+            Map<String, Object> jsonMap = JsonUtils.parseObject(content, Map.class);
+            if (jsonMap != null && jsonMap.containsKey("content")) {
+                Object contentObj = jsonMap.get("content");
+                return contentObj != null ? contentObj.toString() : content;
+            }
+        } catch (Exception e) {
+            // 如果不是有效的 JSON，直接返回原内容
+        }
+
+        // 如果不是 JSON 格式或没有 content 字段，直接返回原内容
+        return content;
+    }
+
+    @Override
+    public String getContentByAudioId(String audioId) {
+        AgentChatHistoryEntity agentChatHistoryEntity = baseMapper
+                .selectOne(new LambdaQueryWrapper<AgentChatHistoryEntity>()
+                        .select(AgentChatHistoryEntity::getContent)
+                        .eq(AgentChatHistoryEntity::getAudioId, audioId));
+        return agentChatHistoryEntity == null ? null : agentChatHistoryEntity.getContent();
+    }
+
+    @Override
+    public boolean isAudioOwnedByAgent(String audioId, String agentId) {
+        // 查询是否有指定音频id和智能体id的数据，如果有且只有一条说明此数据属性此智能体
+        Long row = baseMapper.selectCount(new LambdaQueryWrapper<AgentChatHistoryEntity>()
+                .eq(AgentChatHistoryEntity::getAudioId, audioId)
+                .eq(AgentChatHistoryEntity::getAgentId, agentId));
+        return row == 1;
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentMcpAccessPointServiceImpl.java

@@ -0,0 +1,247 @@
+package xiaozhi.modules.agent.service.impl;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.stereotype.Service;
+
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import xiaozhi.common.constant.Constant;
+import xiaozhi.common.utils.AESUtils;
+import xiaozhi.common.utils.HashEncryptionUtil;
+import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.dto.McpJsonRpcRequest;
+import xiaozhi.modules.agent.service.AgentMcpAccessPointService;
+import xiaozhi.modules.sys.service.SysParamsService;
+import xiaozhi.modules.sys.utils.WebSocketClientManager;
+
+@AllArgsConstructor
+@Service
+@Slf4j
+public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointService {
+    private SysParamsService sysParamsService;
+
+    @Override
+    public String getAgentMcpAccessAddress(String id) {
+        // 获取到mcp的地址
+        String url = sysParamsService.getValue(Constant.SERVER_MCP_ENDPOINT, true);
+        if (StringUtils.isBlank(url) || "null".equals(url)) {
+            return null;
+        }
+        URI uri = getURI(url);
+        // 获取智能体mcp的url前缀
+        String agentMcpUrl = getAgentMcpUrl(uri);
+        // 获取密钥
+        String key = getSecretKey(uri);
+        // 获取加密的token
+        String encryptToken = encryptToken(id, key);
+        // 对token进行URL编码
+        String encodedToken = URLEncoder.encode(encryptToken, StandardCharsets.UTF_8);
+        // 返回智能体Mcp路径的格式
+        agentMcpUrl = "%s/mcp/?token=%s".formatted(agentMcpUrl, encodedToken);
+        return agentMcpUrl;
+    }
+
+    @Override
+    public List<String> getAgentMcpToolsList(String id) {
+        String wsUrl = getAgentMcpAccessAddress(id);
+        if (StringUtils.isBlank(wsUrl)) {
+            return List.of();
+        }
+
+        // 将 /mcp 替换为 /call
+        wsUrl = wsUrl.replace("/mcp/", "/call/");
+
+        try {
+            // 创建 WebSocket 连接，增加超时时间到15秒
+            try (WebSocketClientManager client = WebSocketClientManager.build(
+                    new WebSocketClientManager.Builder()
+                            .uri(wsUrl)
+                            .bufferSize(1024 * 1024)
+                            .connectTimeout(8, TimeUnit.SECONDS)
+                            .maxSessionDuration(10, TimeUnit.SECONDS))) {
+
+                // 步骤1: 发送初始化消息并等待响应
+                log.info("发送MCP初始化消息，智能体ID: {}", id);
+                McpJsonRpcRequest initializeRequest = new McpJsonRpcRequest("initialize",
+                        Map.of(
+                                "protocolVersion", "2024-11-05",
+                                "capabilities", Map.of(
+                                        "roots", Map.of("listChanged", false),
+                                        "sampling", Map.of()),
+                                "clientInfo", Map.of(
+                                        "name", "xz-mcp-broker",
+                                        "version", "0.0.1")),
+                        1);
+                client.sendJson(initializeRequest);
+
+                // 等待初始化响应 (id=1) - 移除固定延迟，改为响应驱动
+                List<String> initResponses = client.listenerWithoutClose(response -> {
+                    try {
+                        Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
+                        if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) {
+                            // 检查是否有result字段，表示初始化成功
+                            return jsonMap.containsKey("result") && !jsonMap.containsKey("error");
+                        }
+                        return false;
+                    } catch (Exception e) {
+                        log.warn("解析初始化响应失败: {}", response, e);
+                        return false;
+                    }
+                });
+
+                // 验证初始化响应
+                boolean initSucceeded = false;
+                for (String response : initResponses) {
+                    try {
+                        Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
+                        if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) {
+                            if (jsonMap.containsKey("result")) {
+                                log.info("MCP初始化成功，智能体ID: {}", id);
+                                initSucceeded = true;
+                                break;
+                            } else if (jsonMap.containsKey("error")) {
+                                log.error("MCP初始化失败，智能体ID: {}, 错误: {}", id, jsonMap.get("error"));
+                                return List.of();
+                            }
+                        }
+                    } catch (Exception e) {
+                        log.warn("处理初始化响应失败: {}", response, e);
+                    }
+                }
+
+                if (!initSucceeded) {
+                    log.error("未收到有效的MCP初始化响应，智能体ID: {}", id);
+                    return List.of();
+                }
+
+                // 步骤2: 发送初始化完成通知 - 只有在收到initialize响应后才发送
+                log.info("发送MCP初始化完成通知，智能体ID: {}", id);
+                String notificationJson = "{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\"}";
+                client.sendText(notificationJson);
+
+                // 步骤3: 发送工具列表请求 - 立即发送，无需额外延迟
+                log.info("发送MCP工具列表请求，智能体ID: {}", id);
+                McpJsonRpcRequest toolsRequest = new McpJsonRpcRequest("tools/list", null, 2);
+                client.sendJson(toolsRequest);
+
+                // 等待工具列表响应 (id=2)
+                List<String> toolsResponses = client.listener(response -> {
+                    try {
+                        Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
+                        return jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"));
+                    } catch (Exception e) {
+                        log.warn("解析工具列表响应失败: {}", response, e);
+                        return false;
+                    }
+                });
+
+                // 处理工具列表响应
+                for (String response : toolsResponses) {
+                    try {
+                        Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
+                        if (jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"))) {
+                            // 检查是否有result字段
+                            Object resultObj = jsonMap.get("result");
+                            if (resultObj instanceof Map) {
+                                Map<String, Object> resultMap = (Map<String, Object>) resultObj;
+                                Object toolsObj = resultMap.get("tools");
+                                if (toolsObj instanceof List) {
+                                    List<Map<String, Object>> toolsList = (List<Map<String, Object>>) toolsObj;
+                                    // 提取工具名称列表
+                                    List<String> result = toolsList.stream()
+                                            .map(tool -> (String) tool.get("name"))
+                                            .filter(name -> name != null)
+                                            .collect(Collectors.toList());
+                                    log.info("成功获取MCP工具列表，智能体ID: {}, 工具数量: {}", id, result.size());
+                                    return result;
+                                }
+                            } else if (jsonMap.containsKey("error")) {
+                                log.error("获取工具列表失败，智能体ID: {}, 错误: {}", id, jsonMap.get("error"));
+                                return List.of();
+                            }
+                        }
+                    } catch (Exception e) {
+                        log.warn("处理工具列表响应失败: {}", response, e);
+                    }
+                }
+
+                log.warn("未找到有效的工具列表响应，智能体ID: {}", id);
+                return List.of();
+
+            }
+        } catch (Exception e) {
+            log.error("获取智能体 MCP 工具列表失败，智能体ID: {},错误原因：{}", id, e.getMessage());
+            return List.of();
+        }
+    }
+
+    /**
+     * 获取URI对象
+     * 
+     * @param url 路径
+     * @return URI对象
+     */
+    private static URI getURI(String url) {
+        try {
+            return new URI(url);
+        } catch (URISyntaxException e) {
+            log.error("路径格式不正确路径：{}，\n错误信息:{}", url, e.getMessage());
+            throw new RuntimeException("mcp的地址存在错误，请进入参数管理修改mcp接入点地址");
+        }
+    }
+
+    /**
+     * 获取密钥
+     *
+     * @param uri mcp地址
+     * @return 密钥
+     */
+    private static String getSecretKey(URI uri) {
+        // 获取参数
+        String query = uri.getQuery();
+        // 获取aes加密密钥
+        String str = "key=";
+        return query.substring(query.indexOf(str) + str.length());
+    }
+
+    /**
+     * 获取智能体mcp接入点url
+     *
+     * @param uri mcp地址
+     * @return 智能体mcp接入点url
+     */
+    private String getAgentMcpUrl(URI uri) {
+        // 获取协议
+        String wsScheme = (uri.getScheme().equals("https")) ? "wss" : "ws";
+        // 获取主机，端口，路径
+        String path = uri.getSchemeSpecificPart();
+        // 获取到最后一个/前的path
+        path = path.substring(0, path.lastIndexOf("/"));
+        return wsScheme + ":" + path;
+    }
+
+    /**
+     * 获取对智能体id加密的token
+     *
+     * @param agentId 智能体id
+     * @param key     加密密钥
+     * @return 加密后token
+     */
+    private static String encryptToken(String agentId, String key) {
+        // 使用md5对智能体id进行加密
+        String md5 = HashEncryptionUtil.Md5hexDigest(agentId);
+        // aes需要加密文本
+        String json = "{\"agentId\": \"%s\"}".formatted(md5);
+        // 加密后成token值
+        return AESUtils.encrypt(key, json);
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentVoicePrintServiceImpl.java

@@ -0,0 +1,410 @@
+package xiaozhi.modules.agent.service.impl;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.List;
+import java.util.concurrent.Executor;
+import java.util.stream.Collectors;
+
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.core.io.ByteArrayResource;
+import org.springframework.http.HttpEntity;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.HttpMethod;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.MediaType;
+import org.springframework.http.ResponseEntity;
+import org.springframework.stereotype.Service;
+import org.springframework.transaction.support.TransactionTemplate;
+import org.springframework.util.LinkedMultiValueMap;
+import org.springframework.util.MultiValueMap;
+import org.springframework.web.client.RestTemplate;
+
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
+
+import lombok.extern.slf4j.Slf4j;
+import xiaozhi.common.constant.Constant;
+import xiaozhi.common.exception.RenException;
+import xiaozhi.common.utils.ConvertUtils;
+import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.dao.AgentVoicePrintDao;
+import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO;
+import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO;
+import xiaozhi.modules.agent.dto.IdentifyVoicePrintResponse;
+import xiaozhi.modules.agent.entity.AgentVoicePrintEntity;
+import xiaozhi.modules.agent.service.AgentChatAudioService;
+import xiaozhi.modules.agent.service.AgentChatHistoryService;
+import xiaozhi.modules.agent.service.AgentVoicePrintService;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
+import xiaozhi.modules.sys.service.SysParamsService;
+
+/**
+ * @author zjy
+ */
+@Service
+@Slf4j
+public class AgentVoicePrintServiceImpl extends ServiceImpl<AgentVoicePrintDao, AgentVoicePrintEntity>
+        implements AgentVoicePrintService {
+    private final AgentChatAudioService agentChatAudioService;
+    private final RestTemplate restTemplate;
+    private final SysParamsService sysParamsService;
+    private final AgentChatHistoryService agentChatHistoryService;
+    // Springboot提供的编程事务类
+    private final TransactionTemplate transactionTemplate;
+    // 识别度
+    private final Double RECOGNITION = 0.5;
+    private final Executor taskExecutor;
+
+    public AgentVoicePrintServiceImpl(AgentChatAudioService agentChatAudioService, RestTemplate restTemplate,
+                                      SysParamsService sysParamsService, AgentChatHistoryService agentChatHistoryService,
+                                      TransactionTemplate transactionTemplate, @Qualifier("taskExecutor") Executor taskExecutor) {
+        this.agentChatAudioService = agentChatAudioService;
+        this.restTemplate = restTemplate;
+        this.sysParamsService = sysParamsService;
+        this.agentChatHistoryService = agentChatHistoryService;
+        this.transactionTemplate = transactionTemplate;
+        this.taskExecutor = taskExecutor;
+    }
+
+    @Override
+    public boolean insert(AgentVoicePrintSaveDTO dto) {
+        // 获取音频数据
+        ByteArrayResource resource = getVoicePrintAudioWAV(dto.getAgentId(), dto.getAudioId());
+        // 识别一下此声音是否注册过
+        IdentifyVoicePrintResponse response = identifyVoicePrint(dto.getAgentId(), resource);
+        if (response != null && response.getScore() > RECOGNITION) {
+            // 根据识别出的声纹ID查询对应的用户信息
+            AgentVoicePrintEntity existingVoicePrint = baseMapper.selectById(response.getSpeakerId());
+            String existingUserName = existingVoicePrint != null ? existingVoicePrint.getSourceName() : "未知用户";
+            throw new RenException("此声音声纹对应的人（" + existingUserName + "）已经注册，请选择其他声音注册");
+        }
+        AgentVoicePrintEntity entity = ConvertUtils.sourceToTarget(dto, AgentVoicePrintEntity.class);
+        // 开启事务
+        return Boolean.TRUE.equals(transactionTemplate.execute(status -> {
+            try {
+                // 保存声纹信息
+                int row = baseMapper.insert(entity);
+                // 插入一条数据，影响的数据不等于1说明出现了，保存问题回滚
+                if (row != 1) {
+                    status.setRollbackOnly(); // 标记事务回滚
+                    return false;
+                }
+                // 发送注册声纹请求
+                registerVoicePrint(entity.getId(), resource);
+                return true;
+            } catch (RenException e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                throw e;
+            } catch (Exception e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                log.error("保存声纹错误原因：{}", e.getMessage());
+                throw new RenException("保存声纹错误，请联系管理员");
+            }
+        }));
+    }
+
+    @Override
+    public boolean delete(Long userId, String voicePrintId) {
+        // 开启事务
+        boolean b = Boolean.TRUE.equals(transactionTemplate.execute(status -> {
+            try {
+                // 删除声纹,按照指定当前登录用户和智能体
+                int row = baseMapper.delete(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                        .eq(AgentVoicePrintEntity::getId, voicePrintId)
+                        .eq(AgentVoicePrintEntity::getCreator, userId));
+                if (row != 1) {
+                    status.setRollbackOnly(); // 标记事务回滚
+                    return false;
+                }
+
+                return true;
+            } catch (Exception e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                log.error("删除声纹存在错误原因：{}", e.getMessage());
+                throw new RenException("删除声纹出现了错误");
+            }
+        }));
+        // 数据库声纹数据删除成功才继续执行删除声纹服务的数据
+        if(b){
+            taskExecutor.execute(()-> {
+                try {
+                    cancelVoicePrint(voicePrintId);
+                }catch (RuntimeException e) {
+                    log.error("删除声纹存在运行时错误原因：{}，id：{}", e.getMessage(),voicePrintId);
+                }
+            });
+        }
+        return b;
+    }
+
+    @Override
+    public List<AgentVoicePrintVO> list(Long userId, String agentId) {
+        // 按照指定当前登录用户和智能体查找数据
+        List<AgentVoicePrintEntity> list = baseMapper.selectList(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                .eq(AgentVoicePrintEntity::getAgentId, agentId)
+                .eq(AgentVoicePrintEntity::getCreator, userId));
+        return list.stream().map(entity -> {
+            // 遍历转换成AgentVoicePrintVO类型
+            return ConvertUtils.sourceToTarget(entity, AgentVoicePrintVO.class);
+        }).toList();
+
+    }
+
+    @Override
+    public boolean update(Long userId, AgentVoicePrintUpdateDTO dto) {
+        AgentVoicePrintEntity agentVoicePrintEntity = baseMapper
+                .selectOne(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                        .eq(AgentVoicePrintEntity::getId, dto.getId())
+                        .eq(AgentVoicePrintEntity::getCreator, userId));
+        if (agentVoicePrintEntity == null) {
+            return false;
+        }
+        // 获取音频Id
+        String audioId = dto.getAudioId();
+        // 获取智能体id
+        String agentId = agentVoicePrintEntity.getAgentId();
+        ByteArrayResource resource;
+        // audioId不等于空，且audioId和之前的保存的音频id不一样，则需要重新获取音频数据生成声纹
+        if (!StringUtils.isEmpty(audioId) && !audioId.equals(agentVoicePrintEntity.getAudioId())) {
+            resource = getVoicePrintAudioWAV(agentId, audioId);
+
+            // 识别一下此声音是否注册过
+            IdentifyVoicePrintResponse response = identifyVoicePrint(agentId, resource);
+            // 返回分数高于RECOGNITION说明这个声纹已经有了
+            if (response != null && response.getScore() > RECOGNITION) {
+                // 判断返回的id如果不是要修改的声纹id，说明这个声纹id，现在要注册的声音已经存在且不是原来的声纹，不允许修改
+                if (!response.getSpeakerId().equals(dto.getId())) {
+                    // 根据识别出的声纹ID查询对应的用户信息
+                    AgentVoicePrintEntity existingVoicePrint = baseMapper.selectById(response.getSpeakerId());
+                    String existingUserName = existingVoicePrint != null ? existingVoicePrint.getSourceName() : "未知用户";
+                    throw new RenException("此次修改不允许，此声音已经注册为声纹了（" + existingUserName + "）");
+                }
+            }
+        } else {
+            resource = null;
+        }
+        // 开启事务
+        return Boolean.TRUE.equals(transactionTemplate.execute(status -> {
+            try {
+                AgentVoicePrintEntity entity = ConvertUtils.sourceToTarget(dto, AgentVoicePrintEntity.class);
+                int row = baseMapper.updateById(entity);
+                if (row != 1) {
+                    status.setRollbackOnly(); // 标记事务回滚
+                    return false;
+                }
+                if (resource != null) {
+                    String id = entity.getId();
+                    // 先注销之前这个声纹id上的声纹向量
+                    cancelVoicePrint(id);
+                    // 发送注册声纹请求
+                    registerVoicePrint(id, resource);
+                }
+                return true;
+            } catch (RenException e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                throw e;
+            } catch (Exception e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                log.error("修改声纹错误原因：{}", e.getMessage());
+                throw new RenException("修改声纹错误，请联系管理员");
+            }
+        }));
+    }
+
+    /**
+     * 获取生纹接口URI对象
+     *
+     * @return URI对象
+     */
+    private URI getVoicePrintURI() {
+        // 获取声纹接口地址
+        String voicePrint = sysParamsService.getValue(Constant.SERVER_VOICE_PRINT, true);
+        try {
+            return new URI(voicePrint);
+        } catch (URISyntaxException e) {
+            log.error("路径格式不正确路径：{}，\n错误信息:{}", voicePrint, e.getMessage());
+            throw new RuntimeException("声纹接口的地址存在错误，请进入参数管理修改声纹接口地址");
+        }
+    }
+
+    /**
+     * 获取声纹地址基础路径
+     * 
+     * @param uri 声纹地址uri
+     * @return 基础路径
+     */
+    private String getBaseUrl(URI uri) {
+        String protocol = uri.getScheme();
+        String host = uri.getHost();
+        int port = uri.getPort();
+        if (port == -1) {
+            return "%s://%s".formatted(protocol, host);
+        } else {
+            return "%s://%s:%s".formatted(protocol, host, port);
+        }
+    }
+
+    /**
+     * 获取验证Authorization
+     *
+     * @param uri 声纹地址uri
+     * @return Authorization值
+     */
+    private String getAuthorization(URI uri) {
+        // 获取参数
+        String query = uri.getQuery();
+        // 获取aes加密密钥
+        String str = "key=";
+        return "Bearer " + query.substring(query.indexOf(str) + str.length());
+    }
+
+    /**
+     * 获取声纹音频资源数据
+     *
+     * @param audioId 音频Id
+     * @return 声纹音频资源数据
+     */
+    private ByteArrayResource getVoicePrintAudioWAV(String agentId, String audioId) {
+        // 判断这个音频是否属于当前智能体
+        boolean b = agentChatHistoryService.isAudioOwnedByAgent(audioId, agentId);
+        if (!b) {
+            throw new RenException("音频数据不属于这个智能体");
+        }
+        // 获取到音频数据
+        byte[] audio = agentChatAudioService.getAudio(audioId);
+        // 如果音频数据为空的直接报错不进行下去
+        if (audio == null || audio.length == 0) {
+            throw new RenException("音频数据是空的请检查上传数据");
+        }
+        // 将字节数组包装为资源，返回
+        return new ByteArrayResource(audio) {
+            @Override
+            public String getFilename() {
+                return "VoicePrint.WAV"; // 设置文件名
+            }
+        };
+    }
+
+    /**
+     * 发送注册声纹http请求
+     * 
+     * @param id       声纹id
+     * @param resource 声纹音频资源
+     */
+    private void registerVoicePrint(String id, ByteArrayResource resource) {
+        // 处理声纹接口地址，获取前缀
+        URI uri = getVoicePrintURI();
+        String baseUrl = getBaseUrl(uri);
+        String requestUrl = baseUrl + "/voiceprint/register";
+        // 创建请求体
+        MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
+        body.add("speaker_id", id);
+        body.add("file", resource);
+
+        // 创建请求头
+        HttpHeaders headers = new HttpHeaders();
+        headers.set("Authorization", getAuthorization(uri));
+        headers.setContentType(MediaType.MULTIPART_FORM_DATA);
+        // 创建请求体
+        HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
+        // 发送 POST 请求
+        ResponseEntity<String> response = restTemplate.postForEntity(requestUrl, requestEntity, String.class);
+
+        if (response.getStatusCode() != HttpStatus.OK) {
+            log.error("声纹注册失败,请求路径：{}", requestUrl);
+            throw new RenException("声纹保存失败,请求不成功");
+        }
+        // 检查响应内容
+        String responseBody = response.getBody();
+        if (responseBody == null || !responseBody.contains("true")) {
+            log.error("声纹注册失败,请求处理失败内容：{}", responseBody == null ? "空内容" : responseBody);
+            throw new RenException("声纹保存失败,请求处理失败");
+        }
+    }
+
+    /**
+     * 发送注销声纹的请求
+     * 
+     * @param voicePrintId 声纹id
+     */
+    private void cancelVoicePrint(String voicePrintId) {
+        URI uri = getVoicePrintURI();
+        String baseUrl = getBaseUrl(uri);
+        String requestUrl = baseUrl + "/voiceprint/" + voicePrintId;
+        // 创建请求头
+        HttpHeaders headers = new HttpHeaders();
+        headers.set("Authorization", getAuthorization(uri));
+        // 创建请求体
+        HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(headers);
+
+        // 发送 POST 请求
+        ResponseEntity<String> response = restTemplate.exchange(requestUrl, HttpMethod.DELETE, requestEntity,
+                String.class);
+        if (response.getStatusCode() != HttpStatus.OK) {
+            log.error("声纹注销失败,请求路径：{}", requestUrl);
+            throw new RenException("声纹注销失败,请求不成功");
+        }
+        // 检查响应内容
+        String responseBody = response.getBody();
+        if (responseBody == null || !responseBody.contains("true")) {
+            log.error("声纹注销失败,请求处理失败内容：{}", responseBody == null ? "空内容" : responseBody);
+            throw new RenException("声纹注销失败,请求处理失败");
+        }
+    }
+
+    /**
+     * 发送识别声纹http请求
+     * 
+     * @param agentId  智能体id
+     * @param resource 声纹音频资源
+     * @return 返回识别数据
+     */
+    private IdentifyVoicePrintResponse identifyVoicePrint(String agentId, ByteArrayResource resource) {
+
+        // 获取该智能体所有注册的声纹
+        List<AgentVoicePrintEntity> agentVoicePrintList = baseMapper
+                .selectList(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                        .select(AgentVoicePrintEntity::getId)
+                        .eq(AgentVoicePrintEntity::getAgentId, agentId));
+
+        // 声纹数量为0，说明还没注册过声纹不需要发生识别请求
+        if (agentVoicePrintList.isEmpty()) {
+            return null;
+        }
+        // 处理声纹接口地址，获取前缀
+        URI uri = getVoicePrintURI();
+        String baseUrl = getBaseUrl(uri);
+        String requestUrl = baseUrl + "/voiceprint/identify";
+        // 创建请求体
+        MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
+
+        // 创建speaker_id参数
+        String speakerIds = agentVoicePrintList.stream()
+                .map(AgentVoicePrintEntity::getId)
+                .collect(Collectors.joining(","));
+        body.add("speaker_ids", speakerIds);
+        body.add("file", resource);
+
+        // 创建请求头
+        HttpHeaders headers = new HttpHeaders();
+        headers.set("Authorization", getAuthorization(uri));
+        headers.setContentType(MediaType.MULTIPART_FORM_DATA);
+        // 创建请求体
+        HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
+        // 发送 POST 请求
+        ResponseEntity<String> response = restTemplate.postForEntity(requestUrl, requestEntity, String.class);
+
+        if (response.getStatusCode() != HttpStatus.OK) {
+            log.error("声纹识别请求失败,请求路径：{}", requestUrl);
+            throw new RenException("声纹识别失败,请求不成功");
+        }
+        // 检查响应内容
+        String responseBody = response.getBody();
+        if (responseBody != null) {
+            return JsonUtils.parseObject(responseBody, IdentifyVoicePrintResponse.class);
+        }
+        return null;
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentChatHistoryUserVO.java

@@ -0,0 +1,16 @@
+package xiaozhi.modules.agent.vo;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import lombok.Data;
+
+/**
+ * 智能体用户个人聊天数据的VO
+ */
+@Data
+public class AgentChatHistoryUserVO {
+    @Schema(description = "聊天内容")
+    private String content;
+
+    @Schema(description = "音频ID")
+    private String audioId;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentVoicePrintVO.java

@@ -0,0 +1,33 @@
+package xiaozhi.modules.agent.vo;
+
+import lombok.Data;
+
+import java.util.Date;
+
+/**
+ * 展示智能体声纹列表VO
+ */
+@Data
+public class AgentVoicePrintVO {
+
+    /**
+     * 主键id
+     */
+    private String id;
+    /**
+     * 音频文件id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+    /**
+     * 创建时间
+     */
+    private Date createDate;
+}

main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java

@@ -1,23 +1,34 @@
 package xiaozhi.modules.config.service.impl;
 
-import java.util.*;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
 
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.stereotype.Service;
 
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
+
 import lombok.AllArgsConstructor;
 import xiaozhi.common.constant.Constant;
 import xiaozhi.common.exception.ErrorCode;
 import xiaozhi.common.exception.RenException;
 import xiaozhi.common.redis.RedisKeys;
 import xiaozhi.common.redis.RedisUtils;
+import xiaozhi.common.utils.ConvertUtils;
 import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.dao.AgentVoicePrintDao;
 import xiaozhi.modules.agent.entity.AgentEntity;
 import xiaozhi.modules.agent.entity.AgentPluginMapping;
 import xiaozhi.modules.agent.entity.AgentTemplateEntity;
+import xiaozhi.modules.agent.entity.AgentVoicePrintEntity;
+import xiaozhi.modules.agent.service.AgentMcpAccessPointService;
 import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.AgentTemplateService;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
 import xiaozhi.modules.config.service.ConfigService;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
@@ -39,6 +50,8 @@ public class ConfigServiceImpl implements ConfigService {
     private final RedisUtils redisUtils;
     private final TimbreService timbreService;
     private final AgentPluginMappingService agentPluginMappingService;
+    private final AgentMcpAccessPointService agentMcpAccessPointService;
+    private final AgentVoicePrintDao agentVoicePrintDao;
 
     @Override
     public Object getConfig(Boolean isCache) {
@@ -66,6 +79,8 @@ public class ConfigServiceImpl implements ConfigService {
                 null,
                 null,
                 null,
+                null,
+                null,
                 agent.getVadModelId(),
                 agent.getAsrModelId(),
                 null,
@@ -102,9 +117,13 @@ public class ConfigServiceImpl implements ConfigService {
         }
         // 获取音色信息
         String voice = null;
+        String referenceAudio = null;
+        String referenceText = null;
         TimbreDetailsVO timbre = timbreService.get(agent.getTtsVoiceId());
         if (timbre != null) {
             voice = timbre.getTtsVoice();
+            referenceAudio = timbre.getReferenceAudio();
+            referenceText = timbre.getReferenceText();
         }
         // 构建返回数据
         Map<String, Object> result = new HashMap<>();
@@ -144,6 +163,14 @@ public class ConfigServiceImpl implements ConfigService {
                 result.put("plugins", pluginParams);
             }
         }
+        // 获取mcp接入点地址
+        String mcpEndpoint = agentMcpAccessPointService.getAgentMcpAccessAddress(agent.getId());
+        if (StringUtils.isNotBlank(mcpEndpoint) && mcpEndpoint.startsWith("ws")) {
+            mcpEndpoint = mcpEndpoint.replace("/mcp/", "/call/");
+            result.put("mcp_endpoint", mcpEndpoint);
+        }
+        // 获取声纹信息
+        buildVoiceprintConfig(agent.getId(), result);
 
         // 构建模块配置
         buildModuleConfig(
@@ -151,6 +178,8 @@ public class ConfigServiceImpl implements ConfigService {
                 agent.getSystemPrompt(),
                 agent.getSummaryMemory(),
                 voice,
+                referenceAudio,
+                referenceText,
                 agent.getVadModelId(),
                 agent.getAsrModelId(),
                 agent.getLlmModelId(),
@@ -167,7 +196,7 @@ public class ConfigServiceImpl implements ConfigService {
     /**
      * 构建配置信息
      * 
-     * @param paramsList 系统参数列表
+     * @param config 系统参数列表
      * @return 配置信息
      */
     private Object buildConfig(Map<String, Object> config) {
@@ -235,24 +264,84 @@ public class ConfigServiceImpl implements ConfigService {
         return config;
     }
 
+    /**
+     * 构建声纹配置信息
+     * 
+     * @param agentId 智能体ID
+     * @param result  结果Map
+     */
+    private void buildVoiceprintConfig(String agentId, Map<String, Object> result) {
+        try {
+            // 获取声纹接口地址
+            String voiceprintUrl = sysParamsService.getValue("server.voice_print", true);
+            if (StringUtils.isBlank(voiceprintUrl) || "null".equals(voiceprintUrl)) {
+                return;
+            }
+
+            // 获取智能体关联的声纹信息（不需要用户权限验证）
+            List<AgentVoicePrintVO> voiceprints = getVoiceprintsByAgentId(agentId);
+            if (voiceprints == null || voiceprints.isEmpty()) {
+                return;
+            }
+
+            // 构建speakers列表
+            List<String> speakers = new ArrayList<>();
+            for (AgentVoicePrintVO voiceprint : voiceprints) {
+                String speakerStr = String.format("%s,%s,%s",
+                        voiceprint.getId(),
+                        voiceprint.getSourceName(),
+                        voiceprint.getIntroduce() != null ? voiceprint.getIntroduce() : "");
+                speakers.add(speakerStr);
+            }
+
+            // 构建声纹配置
+            Map<String, Object> voiceprintConfig = new HashMap<>();
+            voiceprintConfig.put("url", voiceprintUrl);
+            voiceprintConfig.put("speakers", speakers);
+
+            result.put("voiceprint", voiceprintConfig);
+        } catch (Exception e) {
+            // 声纹配置获取失败时不影响其他功能
+            System.err.println("获取声纹配置失败: " + e.getMessage());
+        }
+    }
+
+    /**
+     * 获取智能体关联的声纹信息
+     * 
+     * @param agentId 智能体ID
+     * @return 声纹信息列表
+     */
+    private List<AgentVoicePrintVO> getVoiceprintsByAgentId(String agentId) {
+        LambdaQueryWrapper<AgentVoicePrintEntity> queryWrapper = new LambdaQueryWrapper<>();
+        queryWrapper.eq(AgentVoicePrintEntity::getAgentId, agentId);
+        queryWrapper.orderByAsc(AgentVoicePrintEntity::getCreateDate);
+        List<AgentVoicePrintEntity> entities = agentVoicePrintDao.selectList(queryWrapper);
+        return ConvertUtils.sourceToTarget(entities, AgentVoicePrintVO.class);
+    }
+
     /**
      * 构建模块配置
      * 
-     * @param prompt        提示词
+     * @param prompt         提示词
-     * @param voice         音色
+     * @param voice          音色
-     * @param vadModelId    VAD模型ID
+     * @param referenceAudio 参考音频路径
-     * @param asrModelId    ASR模型ID
+     * @param referenceText  参考文本
-     * @param llmModelId    LLM模型ID
+     * @param vadModelId     VAD模型ID
-     * @param ttsModelId    TTS模型ID
+     * @param asrModelId     ASR模型ID
-     * @param memModelId    记忆模型ID
+     * @param llmModelId     LLM模型ID
-     * @param intentModelId 意图模型ID
+     * @param ttsModelId     TTS模型ID
-     * @param result        结果Map
+     * @param memModelId     记忆模型ID
+     * @param intentModelId  意图模型ID
+     * @param result         结果Map
      */
     private void buildModuleConfig(
             String assistantName,
             String prompt,
             String summaryMemory,
             String voice,
+            String referenceAudio,
+            String referenceText,
             String vadModelId,
             String asrModelId,
             String llmModelId,
@@ -274,12 +363,20 @@ public class ConfigServiceImpl implements ConfigService {
                 continue;
             }
             ModelConfigEntity model = modelConfigService.getModelById(modelIds[i], isCache);
+            if (model == null) {
+                continue;
+            }
             Map<String, Object> typeConfig = new HashMap<>();
             if (model.getConfigJson() != null) {
                 typeConfig.put(model.getId(), model.getConfigJson());
                 // 如果是TTS类型，添加private_voice属性
-                if ("TTS".equals(modelTypes[i]) && voice != null) {
+                if ("TTS".equals(modelTypes[i])) {
-                    ((Map<String, Object>) model.getConfigJson()).put("private_voice", voice);
+                    if (voice != null)
+                        ((Map<String, Object>) model.getConfigJson()).put("private_voice", voice);
+                    if (referenceAudio != null)
+                        ((Map<String, Object>) model.getConfigJson()).put("ref_audio", referenceAudio);
+                    if (referenceText != null)
+                        ((Map<String, Object>) model.getConfigJson()).put("ref_text", referenceText);
                 }
                 // 如果是Intent类型，且type=intent_llm，则给他添加附加模型
                 if ("Intent".equals(modelTypes[i])) {

main/manager-api/src/main/java/xiaozhi/modules/device/controller/DeviceController.java

@@ -25,6 +25,7 @@ import xiaozhi.common.utils.Result;
 import xiaozhi.modules.device.dto.DeviceRegisterDTO;
 import xiaozhi.modules.device.dto.DeviceUnBindDTO;
 import xiaozhi.modules.device.dto.DeviceUpdateDTO;
+import xiaozhi.modules.device.dto.DeviceManualAddDTO;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
 import xiaozhi.modules.security.user.SecurityUser;
@@ -99,4 +100,13 @@ public class DeviceController {
         deviceService.updateById(entity);
         return new Result<Void>();
     }
+
+    @PostMapping("/manual-add")
+    @Operation(summary = "手动添加设备")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> manualAddDevice(@RequestBody @Valid DeviceManualAddDTO dto) {
+        UserDetail user = SecurityUser.getUser();
+        deviceService.manualAddDevice(user.getId(), dto);
+        return new Result<>();
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/device/controller/OTAController.java

@@ -51,13 +51,12 @@ public class OTAController {
         if (StringUtils.isBlank(clientId)) {
             clientId = deviceId;
         }
-        String macAddress = deviceReportReqDTO.getMacAddress();
+        boolean macAddressValid = isMacAddressValid(deviceId);
-        boolean macAddressValid = isMacAddressValid(macAddress);
         // 设备Id和Mac地址应是一致的, 并且必须需要application字段
-        if (!deviceId.equals(macAddress) || !macAddressValid || deviceReportReqDTO.getApplication() == null) {
+        if (!macAddressValid) {
-            return createResponse(DeviceReportRespDTO.createError("Invalid OTA request"));
+            return createResponse(DeviceReportRespDTO.createError("Invalid device ID"));
         }
-        return createResponse(deviceService.checkDeviceActive(macAddress, clientId, deviceReportReqDTO));
+        return createResponse(deviceService.checkDeviceActive(deviceId, clientId, deviceReportReqDTO));
     }
 
     @Operation(summary = "设备快速检查激活状态")

main/manager-api/src/main/java/xiaozhi/modules/device/dto/DeviceManualAddDTO.java

@@ -0,0 +1,11 @@
+package xiaozhi.modules.device.dto;
+
+import lombok.Data;
+
+@Data
+public class DeviceManualAddDTO {
+    private String agentId;
+    private String board;        // 设备型号
+    private String appVersion;   // 固件版本
+    private String macAddress;   // Mac地址
+} 

main/manager-api/src/main/java/xiaozhi/modules/device/service/DeviceService.java

@@ -8,6 +8,7 @@ import xiaozhi.common.service.BaseService;
 import xiaozhi.modules.device.dto.DevicePageUserDTO;
 import xiaozhi.modules.device.dto.DeviceReportReqDTO;
 import xiaozhi.modules.device.dto.DeviceReportRespDTO;
+import xiaozhi.modules.device.dto.DeviceManualAddDTO;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.vo.UserShowDeviceListVO;
 
@@ -87,5 +88,14 @@ public interface DeviceService extends BaseService<DeviceEntity> {
      */
     Date getLatestLastConnectionTime(String agentId);
 
+    /**
+     * 手动添加设备
+     */
+    void manualAddDevice(Long userId, DeviceManualAddDTO dto);
+
+    /**
+     * 更新设备连接信息
+     */
+    void updateDeviceConnectionInfo(String agentId, String deviceId, String appVersion);
 
 }

main/manager-api/src/main/java/xiaozhi/modules/device/service/impl/DeviceServiceImpl.java

@@ -44,6 +44,7 @@ import xiaozhi.modules.device.vo.UserShowDeviceListVO;
 import xiaozhi.modules.security.user.SecurityUser;
 import xiaozhi.modules.sys.service.SysParamsService;
 import xiaozhi.modules.sys.service.SysUserUtilService;
+import xiaozhi.modules.device.dto.DeviceManualAddDTO;
 
 @Slf4j
 @Service
@@ -410,4 +411,30 @@ public class DeviceServiceImpl extends BaseServiceImpl<DeviceDao, DeviceEntity>
         }
         return 0;
     }
+
+    @Override
+    public void manualAddDevice(Long userId, DeviceManualAddDTO dto) {
+        // 检查mac是否已存在
+        QueryWrapper<DeviceEntity> wrapper = new QueryWrapper<>();
+        wrapper.eq("mac_address", dto.getMacAddress());
+        DeviceEntity exist = baseDao.selectOne(wrapper);
+        if (exist != null) {
+            throw new RenException("该Mac地址已存在");
+        }
+        Date now = new Date();
+        DeviceEntity entity = new DeviceEntity();
+        entity.setId(dto.getMacAddress());
+        entity.setUserId(userId);
+        entity.setAgentId(dto.getAgentId());
+        entity.setBoard(dto.getBoard());
+        entity.setAppVersion(dto.getAppVersion());
+        entity.setMacAddress(dto.getMacAddress());
+        entity.setCreateDate(now);
+        entity.setUpdateDate(now);
+        entity.setLastConnectedAt(now);
+        entity.setCreator(userId);
+        entity.setUpdater(userId);
+        entity.setAutoUpdate(1);
+        baseDao.insert(entity);
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/sys/controller/SysParamsController.java

@@ -104,6 +104,12 @@ public class SysParamsController {
         // 验证OTA地址
         validateOtaUrl(dto.getParamCode(), dto.getParamValue());
 
+        // 验证MCP地址
+        validateMcpUrl(dto.getParamCode(), dto.getParamValue());
+
+        //
+        validateVoicePrint(dto.getParamCode(), dto.getParamValue());
+
         sysParamsService.update(dto);
         configService.getConfig(false);
         return new Result<Void>();
@@ -195,4 +201,67 @@ public class SysParamsController {
             throw new RenException("OTA接口验证失败：" + e.getMessage());
         }
     }
+
+    private void validateMcpUrl(String paramCode, String url) {
+        if (!paramCode.equals(Constant.SERVER_MCP_ENDPOINT)) {
+            return;
+        }
+        if (StringUtils.isBlank(url) || url.equals("null")) {
+            throw new RenException("MCP地址不能为空");
+        }
+        if (url.contains("localhost") || url.contains("127.0.0.1")) {
+            throw new RenException("MCP地址不能使用localhost或127.0.0.1");
+        }
+        if (!url.toLowerCase().contains("key")) {
+            throw new RenException("不是正确的MCP地址");
+        }
+
+        try {
+            // 发送GET请求
+            ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
+            if (response.getStatusCode() != HttpStatus.OK) {
+                throw new RenException("MCP接口访问失败，状态码：" + response.getStatusCode());
+            }
+            // 检查响应内容是否包含mcp相关信息
+            String body = response.getBody();
+            if (body == null || !body.contains("success")) {
+                throw new RenException("MCP接口返回内容格式不正确，可能不是一个真实的MCP接口");
+            }
+        } catch (Exception e) {
+            throw new RenException("MCP接口验证失败：" + e.getMessage());
+        }
+    }
+    // 验证声纹接口地址是否正常
+    private void validateVoicePrint(String paramCode, String url) {
+        if (!paramCode.equals(Constant.SERVER_VOICE_PRINT)) {
+            return;
+        }
+        if (StringUtils.isBlank(url) || url.equals("null")) {
+            throw new RenException("声纹接口地址不能为空");
+        }
+        if (url.contains("localhost") || url.contains("127.0.0.1")) {
+            throw new RenException("声纹接口地址不能使用localhost或127.0.0.1");
+        }
+        if (!url.toLowerCase().contains("key")) {
+            throw new RenException("不是正确的声纹接口地址");
+        }
+        // 验证URL格式
+        if (!url.toLowerCase().startsWith("http")) {
+            throw new RenException("声纹接口地址必须以http或https开头");
+        }
+        try {
+            // 发送GET请求
+            ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
+            if (response.getStatusCode() != HttpStatus.OK) {
+                throw new RenException("声纹接口访问失败，状态码：" + response.getStatusCode());
+            }
+            // 检查响应内容
+            String body = response.getBody();
+            if (body == null || !body.contains("healthy")) {
+                throw new RenException("声纹接口返回内容格式不正确，可能不是一个真实的MCP接口");
+            }
+        } catch (Exception e) {
+            throw new RenException("声纹接口验证失败：" + e.getMessage());
+        }
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java

@@ -86,10 +86,14 @@ public class WebSocketClientManager implements Closeable {
         if (sess == null || !sess.isOpen()) {
             throw new IOException("握手失败或会话未打开");
         }
+        // 设置缓冲区
+        sess.setTextMessageSizeLimit(b.bufferSize);
+        sess.setBinaryMessageSizeLimit(b.bufferSize);
         ws.session = sess;
         return ws;
     }
 
+
     /**
      * 发送 Text
      */
@@ -137,6 +141,37 @@ public class WebSocketClientManager implements Closeable {
         return collected;
     }
 
+    private <T> List<T> listenerCustomWithoutClose(
+            BlockingQueue<T> queue,
+            Predicate<T> predicate)
+            throws InterruptedException, TimeoutException, ExecutionException {
+        List<T> collected = new ArrayList<>();
+        long deadline = System.currentTimeMillis() + maxSessionDurationUnit.toMillis(maxSessionDuration);
+
+        while (true) {
+            if (errorFuture.isDone()) {
+                errorFuture.get();
+            }
+
+            long remaining = deadline - System.currentTimeMillis();
+            if (remaining <= 0) {
+                throw new TimeoutException("等待批量消息超时");
+            }
+
+            T msg = queue.poll(remaining, TimeUnit.MILLISECONDS);
+            if (msg == null) {
+                throw new TimeoutException("等待批量消息超时");
+            }
+
+            collected.add(msg);
+            if (predicate.test(msg)) {
+                break;
+            }
+        }
+        // 不调用 close()，保持连接开放
+        return collected;
+    }
+
     /**
      * 同步接收多条消息，直到 predicate 为 true 或超时抛异常；
      * 
@@ -147,6 +182,17 @@ public class WebSocketClientManager implements Closeable {
         return listenerCustom(textMessageQueue, predicate);
     }
 
+    /**
+     * 同步接收多条消息，直到 predicate 为 true 或超时抛异常；
+     * 不自动关闭连接，适用于需要在同一连接上发送多个消息的场景
+     * 
+     * @return 返回监听期间的所有消息列表
+     */
+    public List<String> listenerWithoutClose(Predicate<String> predicate)
+            throws InterruptedException, TimeoutException, ExecutionException {
+        return listenerCustomWithoutClose(textMessageQueue, predicate);
+    }
+
     public List<byte[]> listenerBinary(Predicate<byte[]> predicate)
             throws InterruptedException, TimeoutException, ExecutionException {
         return listenerCustom(binaryMessageQueue, predicate);
@@ -266,10 +312,11 @@ public class WebSocketClientManager implements Closeable {
             if (stopWatch.isRunning()) {
                 stopWatch.stop();
             }
-            log.info("ws连接关闭, 目标URI: {}, 关闭时间: {}, 连接总时长: {}s",
+            log.info("ws连接关闭, 目标URI: {}, 关闭时间: {}, 连接总时长: {}s,断开原因：{}",
                     targetUri, DateUtils.getDateTimeNow(DateUtils.DATE_TIME_MILLIS_PATTERN),
-                    DateUtils.millsToSecond(stopWatch.getTotalTimeMillis()));
+                    DateUtils.millsToSecond(stopWatch.getTotalTimeMillis()),status);
         }
+
     }
 
     public static class Builder {
@@ -279,6 +326,7 @@ public class WebSocketClientManager implements Closeable {
         private long maxSessionDuration = 5; // 最大连线时间，默认5秒
         private TimeUnit maxSessionDurationUnit = TimeUnit.SECONDS; // 最大连线时间单位
         private int queueCapacity = 100; // 消息队列容量
+        private int bufferSize = 8 * 1024; //默认 8kb
         private WebSocketHttpHeaders headers; // 请求头
 
         /**
@@ -310,6 +358,10 @@ public class WebSocketClientManager implements Closeable {
             this.queueCapacity = c;
             return this;
         }
+        public Builder bufferSize(int c) {
+            this.bufferSize = c;
+            return this;
+        }
 
         public WebSocketClientManager build()
                 throws InterruptedException, ExecutionException, TimeoutException, IOException {

main/manager-api/src/main/java/xiaozhi/modules/timbre/dto/TimbreDataDTO.java

@@ -26,6 +26,12 @@ public class TimbreDataDTO {
     @Schema(description = "备注")
     private String remark;
 
+    @Schema(description = "参考音频路径")
+    private String referenceAudio;
+
+    @Schema(description = "參考文本")
+    private String referenceText;
+
     @Schema(description = "排序")
     @Min(value = 0, message = "{sort.number}")
     private long sort;

main/manager-api/src/main/java/xiaozhi/modules/timbre/entity/TimbreEntity.java

@@ -34,6 +34,12 @@ public class TimbreEntity {
     @Schema(description = "备注")
     private String remark;
 
+    @Schema(description = "参考音频路径")
+    private String referenceAudio;
+
+    @Schema(description = "參考文本")
+    private String referenceText;
+
     @Schema(description = "排序")
     private long sort;
 

main/manager-api/src/main/java/xiaozhi/modules/timbre/vo/TimbreDetailsVO.java

@@ -25,6 +25,12 @@ public class TimbreDetailsVO implements Serializable {
     @Schema(description = "备注")
     private String remark;
 
+    @Schema(description = "参考音频路径")
+    private String referenceAudio;
+
+    @Schema(description = "參考文本")
+    private String referenceText;
+
     @Schema(description = "排序")
     private long sort;
 

main/manager-api/src/main/resources/db/changelog/202506091720.sql

@@ -0,0 +1,3 @@
+ALTER TABLE `ai_tts_voice`
+ADD COLUMN `reference_audio` VARCHAR(500) DEFAULT NULL COMMENT '参考音频路径' AFTER `remark`,
+ADD COLUMN `reference_text` VARCHAR(500) DEFAULT NULL COMMENT '参考文本' AFTER `reference_audio`;

main/manager-api/src/main/resources/db/changelog/202506251620.sql

@@ -0,0 +1,18 @@
+-- 更新现有的 get_news_from_newsnow 插件配置
+UPDATE ai_model_provider 
+SET fields = JSON_ARRAY(
+    JSON_OBJECT(
+        'key', 'url',
+        'type', 'string',
+        'label', '接口地址',
+        'default', 'https://newsnow.busiyi.world/api/s?id='
+    ),
+    JSON_OBJECT(
+        'key', 'news_sources',
+        'type', 'string',
+        'label', '新闻源配置',
+        'default', '澎湃新闻;百度热搜;财联社'
+    )
+)
+WHERE provider_code = 'get_news_from_newsnow' 
+AND model_type = 'Plugin'; 

main/manager-api/src/main/resources/db/changelog/202506261637.sql

@@ -0,0 +1,2 @@
+delete from `sys_params` where id = 113;
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (113, 'server.mcp_endpoint', 'null', 'string', 1, 'mcp接入点地址');

main/manager-api/src/main/resources/db/changelog/202507031602.sql

@@ -0,0 +1,4 @@
+-- 添加声纹接口地址参数配置
+delete from `sys_params` where id = 114;
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark)
+VALUES (114, 'server.voice_print', 'null', 'string', 1, '声纹接口地址');

main/manager-api/src/main/resources/db/changelog/202507041018.sql

@@ -0,0 +1,12 @@
+DROP TABLE IF EXISTS ai_agent_voice_print;
+create table ai_agent_voice_print (
+  id varchar(32) NOT NULL COMMENT '声纹ID',
+  agent_id varchar(32)  NOT NULL COMMENT '关联的智能体ID',
+  source_name varchar(50)  NOT NULL COMMENT '声纹来源的人的姓名',
+  introduce varchar(200) COMMENT '描述声纹来源的这个人',
+  create_date DATETIME COMMENT '创建时间',
+  creator bigint COMMENT '创建者',
+  update_date DATETIME COMMENT '修改时间',
+  updater bigint COMMENT '修改者',
+  PRIMARY KEY (id)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='智能体声纹表'

main/manager-api/src/main/resources/db/changelog/202507071130.sql

@@ -0,0 +1,26 @@
+-- 添加阿里云流式ASR供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_AliyunStreamASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_AliyunStreamASR', 'ASR', 'aliyun_stream', '阿里云语音识别(流式)', '[{"key":"appkey","label":"应用AppKey","type":"string"},{"key":"token","label":"临时Token","type":"string"},{"key":"access_key_id","label":"AccessKey ID","type":"string"},{"key":"access_key_secret","label":"AccessKey Secret","type":"string"},{"key":"host","label":"服务地址","type":"string"},{"key":"max_sentence_silence","label":"断句检测时间","type":"number"},{"key":"output_dir","label":"输出目录","type":"string"}]', 6, 1, NOW(), 1, NOW());
+
+-- 添加阿里云流式ASR模型配置
+delete from `ai_model_config` where id = 'ASR_AliyunStreamASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_AliyunStreamASR', 'ASR', 'AliyunStreamASR', '阿里云语音识别(流式)', 0, 1, '{\"type\": \"aliyun_stream\", \"appkey\": \"\", \"token\": \"\", \"access_key_id\": \"\", \"access_key_secret\": \"\", \"host\": \"nls-gateway-cn-shanghai.aliyuncs.com\", \"max_sentence_silence\": 800, \"output_dir\": \"tmp/\"}', NULL, NULL, 8, NULL, NULL, NULL, NULL);
+
+-- 更新阿里云流式ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://nls-portal.console.aliyun.com/',
+`remark` = '阿里云流式ASR配置说明：
+1. 阿里云ASR和阿里云(流式)ASR的区别是：阿里云ASR是一次性识别，阿里云(流式)ASR是实时流式识别
+2. 流式ASR具有更低的延迟和更好的实时性，适合语音交互场景
+3. 需要在阿里云智能语音交互控制台创建应用并获取认证信息
+4. 支持中文实时语音识别，支持标点符号预测和逆文本规范化
+5. 需要网络连接，输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://nls-portal.console.aliyun.com/ 开通智能语音交互服务
+2. 访问 https://nls-portal.console.aliyun.com/applist 创建项目并获取appkey
+3. 访问 https://nls-portal.console.aliyun.com/overview 获取临时token（或配置access_key_id和access_key_secret自动获取）
+4. 如需动态token管理，建议配置access_key_id和access_key_secret
+5. max_sentence_silence参数控制断句检测时间（毫秒），默认800ms
+如需了解更多参数配置，请参考：https://help.aliyun.com/zh/isi/developer-reference/real-time-speech-recognition
+' WHERE `id` = 'ASR_AliyunStreamASR';

main/manager-api/src/main/resources/db/changelog/202507071530.sql

@@ -0,0 +1,56 @@
+-- 添加阿里云流式TTS供应器
+delete from `ai_model_provider` where id = 'SYSTEM_TTS_AliyunStreamTTS';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_TTS_AliyunStreamTTS', 'TTS', 'aliyun_stream', '阿里云语音合成(流式)', '[{"key":"appkey","label":"应用AppKey","type":"string"},{"key":"token","label":"临时Token","type":"string"},{"key":"access_key_id","label":"AccessKey ID","type":"string"},{"key":"access_key_secret","label":"AccessKey Secret","type":"string"},{"key":"host","label":"服务地址","type":"string"},{"key":"voice","label":"默认音色","type":"string"},{"key":"format","label":"音频格式","type":"string"},{"key":"sample_rate","label":"采样率","type":"number"},{"key":"volume","label":"音量","type":"number"},{"key":"speech_rate","label":"语速","type":"number"},{"key":"pitch_rate","label":"音调","type":"number"},{"key":"output_dir","label":"输出目录","type":"string"}]', 15, 1, NOW(), 1, NOW());
+
+-- 添加阿里云流式TTS模型配置
+delete from `ai_model_config` where id = 'TTS_AliyunStreamTTS';
+INSERT INTO `ai_model_config` VALUES ('TTS_AliyunStreamTTS', 'TTS', 'AliyunStreamTTS', '阿里云语音合成(流式)', 0, 1, '{\"type\": \"aliyun_stream\", \"appkey\": \"\", \"token\": \"\", \"access_key_id\": \"\", \"access_key_secret\": \"\", \"host\": \"nls-gateway-cn-beijing.aliyuncs.com\", \"voice\": \"longxiaochun\", \"format\": \"pcm\", \"sample_rate\": 16000, \"volume\": 50, \"speech_rate\": 0, \"pitch_rate\": 0, \"output_dir\": \"tmp/\"}', NULL, NULL, 18, NULL, NULL, NULL, NULL);
+
+-- 更新阿里云流式TTS配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://nls-portal.console.aliyun.com/',
+`remark` = '阿里云流式TTS配置说明：
+1. 阿里云TTS和阿里云(流式)TTS的区别是：阿里云TTS是一次性合成，阿里云(流式)TTS是实时流式合成
+2. 流式TTS具有更低的延迟和更好的实时性，适合语音交互场景
+3. 需要在阿里云智能语音交互控制台创建应用并获取认证信息
+4. 支持CosyVoice大模型音色，音质更加自然
+5. 支持实时调节音量、语速、音调等参数
+申请步骤：
+1. 访问 https://nls-portal.console.aliyun.com/ 开通智能语音交互服务
+2. 访问 https://nls-portal.console.aliyun.com/applist 创建项目并获取appkey
+3. 访问 https://nls-portal.console.aliyun.com/overview 获取临时token（或配置access_key_id和access_key_secret自动获取）
+4. 如需动态token管理，建议配置access_key_id和access_key_secret
+5. 可选择北京、上海等不同地域的服务器以优化延迟
+6. voice参数支持CosyVoice大模型音色，如longxiaochun、longyueyue等
+如需了解更多参数配置，请参考：https://help.aliyun.com/zh/isi/developer-reference/real-time-speech-synthesis
+' WHERE `id` = 'TTS_AliyunStreamTTS';
+
+-- 添加阿里云流式TTS音色
+delete from `ai_tts_voice` where tts_model_id = 'TTS_AliyunStreamTTS';
+-- 温柔女声系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0001', 'TTS_AliyunStreamTTS', '龙小淳-温柔姐姐', 'longxiaochun', '中文及中英文混合', NULL, NULL, NULL, NULL, 1, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0002', 'TTS_AliyunStreamTTS', '龙小夏-温柔女声', 'longxiaoxia', '中文及中英文混合', NULL, NULL, NULL, NULL, 2, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0003', 'TTS_AliyunStreamTTS', '龙玫-温柔女声', 'longmei', '中文及中英文混合', NULL, NULL, NULL, NULL, 3, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0004', 'TTS_AliyunStreamTTS', '龙瑰-温柔女声', 'longgui', '中文及中英文混合', NULL, NULL, NULL, NULL, 4, NULL, NULL, NULL, NULL);
+-- 御姐女声系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0005', 'TTS_AliyunStreamTTS', '龙玉-御姐女声', 'longyu', '中文及中英文混合', NULL, NULL, NULL, NULL, 5, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0006', 'TTS_AliyunStreamTTS', '龙娇-御姐女声', 'longjiao', '中文及中英文混合', NULL, NULL, NULL, NULL, 6, NULL, NULL, NULL, NULL);
+-- 男声系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0007', 'TTS_AliyunStreamTTS', '龙臣-译制片男声', 'longchen', '中文及中英文混合', NULL, NULL, NULL, NULL, 7, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0008', 'TTS_AliyunStreamTTS', '龙修-青年男声', 'longxiu', '中文及中英文混合', NULL, NULL, NULL, NULL, 8, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0009', 'TTS_AliyunStreamTTS', '龙橙-阳光男声', 'longcheng', '中文及中英文混合', NULL, NULL, NULL, NULL, 9, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0010', 'TTS_AliyunStreamTTS', '龙哲-成熟男声', 'longzhe', '中文及中英文混合', NULL, NULL, NULL, NULL, 10, NULL, NULL, NULL, NULL);
+-- 专业播报系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0011', 'TTS_AliyunStreamTTS', 'Bella2.0-新闻女声', 'loongbella', '中文及中英文混合', NULL, NULL, NULL, NULL, 11, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0012', 'TTS_AliyunStreamTTS', 'Stella2.0-飒爽女声', 'loongstella', '中文及中英文混合', NULL, NULL, NULL, NULL, 12, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0013', 'TTS_AliyunStreamTTS', '龙书-新闻男声', 'longshu', '中文及中英文混合', NULL, NULL, NULL, NULL, 13, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0014', 'TTS_AliyunStreamTTS', '龙婧-严肃女声', 'longjing', '中文及中英文混合', NULL, NULL, NULL, NULL, 14, NULL, NULL, NULL, NULL);
+-- 特色音色系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0015', 'TTS_AliyunStreamTTS', '龙奇-活泼童声', 'longqi', '中文及中英文混合', NULL, NULL, NULL, NULL, 15, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0016', 'TTS_AliyunStreamTTS', '龙华-活泼女童', 'longhua', '中文及中英文混合', NULL, NULL, NULL, NULL, 16, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0017', 'TTS_AliyunStreamTTS', '龙无-无厘头男声', 'longwu', '中文及中英文混合', NULL, NULL, NULL, NULL, 17, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0018', 'TTS_AliyunStreamTTS', '龙大锤-幽默男声', 'longdachui', '中文及中英文混合', NULL, NULL, NULL, NULL, 18, NULL, NULL, NULL, NULL);
+-- 粤语系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0019', 'TTS_AliyunStreamTTS', '龙嘉怡-粤语女声', 'longjiayi', '粤语及粤英混合', NULL, NULL, NULL, NULL, 19, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0020', 'TTS_AliyunStreamTTS', '龙桃-粤语女声', 'longtao', '粤语及粤英混合', NULL, NULL, NULL, NULL, 20, NULL, NULL, NULL, NULL);

main/manager-api/src/main/resources/db/changelog/202507081646.sql

@@ -0,0 +1,3 @@
+-- 智能体声纹添加新字段
+ALTER TABLE ai_agent_voice_print
+    ADD COLUMN audio_id VARCHAR(32) NOT NULL COMMENT '音频ID';

main/manager-api/src/main/resources/db/changelog/202507101203.sql

@@ -0,0 +1,38 @@
+-- OpenAI ASR模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_OpenaiASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_OpenaiASR', 'ASR', 'openai', 'OpenAI语音识别', '[{"key": "base_url", "type": "string", "label": "基础URL"}, {"key": "model_name", "type": "string", "label": "模型名称"}, {"key": "api_key", "type": "string", "label": "API密钥"}, {"key": "output_dir", "type": "string", "label": "输出目录"}]', 9, 1, NOW(), 1, NOW());
+
+
+-- OpenAI ASR模型配置
+delete from `ai_model_config` where id = 'ASR_OpenaiASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_OpenaiASR', 'ASR', 'OpenaiASR', 'OpenAI语音识别', 0, 1, '{\"type\": \"openai\", \"api_key\": \"\", \"base_url\": \"https://api.openai.com/v1/audio/transcriptions\", \"model_name\": \"gpt-4o-mini-transcribe\", \"output_dir\": \"tmp/\"}', NULL, NULL, 9, NULL, NULL, NULL, NULL);
+
+-- groq ASR模型配置
+delete from `ai_model_config` where id = 'ASR_GroqASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_GroqASR', 'ASR', 'GroqASR', 'Groq语音识别', 0, 1, '{\"type\": \"openai\", \"api_key\": \"\", \"base_url\": \"https://api.groq.com/openai/v1/audio/transcriptions\", \"model_name\": \"whisper-large-v3-turbo\", \"output_dir\": \"tmp/\"}', NULL, NULL, 10, NULL, NULL, NULL, NULL);
+
+
+-- 更新OpenAI ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://platform.openai.com/docs/api-reference/audio/createTranscription',
+`remark` = 'OpenAI ASR配置说明：
+1. 需要在OpenAI开放平台创建组织并获取api_key
+2. 支持中、英、日、韩等多种语音识别，具体参考文档https://platform.openai.com/docs/guides/speech-to-text
+3. 需要网络连接
+4. 输出文件保存在tmp/目录
+申请步骤：
+**OpenAi ASR申请步骤：**
+1.登录OpenAI Platform。https://auth.openai.com/log-in
+2.创建api-key  https://platform.openai.com/settings/organization/api-keys
+3.模型可以选择gpt-4o-transcribe或GPT-4o mini Transcribe
+' WHERE `id` = 'ASR_OpenaiASR';
+
+-- 更新Groq ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.groq.com/docs/speech-to-text',
+`remark` = 'Groq ASR配置说明：
+1.登录groq Console。https://console.groq.com/home
+2.创建api-key  https://console.groq.com/keys
+3.模型可以选择whisper-large-v3-turbo或whisper-large-v3（distil-whisper-large-v3-en仅支持英语转录）
+' WHERE `id` = 'ASR_GroqASR';

main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml

@@ -205,6 +205,13 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202506080955.sql
+  - changeSet:
+      id: 202506091720
+      author: shane0411
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506091720.sql
   - changeSet:
       id: 202506161101
       author: hrz
@@ -219,3 +226,59 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202506191643.sql
+  - changeSet:
+      id: 202506251620
+      author: Tink
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506251620.sql
+  - changeSet:
+      id: 202506261637
+      author: hrz
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202506261637.sql
+  - changeSet:
+      id: 202507101203
+      author: luruxian
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507101203.sql            
+  - changeSet:
+      id: 202507071130
+      author: cgd
+      changes:
+          - sqlFile:
+              encoding: utf8
+              path: classpath:db/changelog/202507071130.sql
+  - changeSet:
+      id: 202507071530
+      author: cgd
+      changes:
+          - sqlFile:
+              encoding: utf8
+              path: classpath:db/changelog/202507071530.sql
+  - changeSet:
+      id: 202507031602
+      author: zjy
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507031602.sql
+  - changeSet:
+      id: 202507041018
+      author: zjy
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507041018.sql
+  - changeSet:
+      id: 202507081646
+      author: zjy
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507081646.sql

main/manager-api/src/main/resources/lua/emptyAll.lua

@@ -1 +1 @@
-redis.call('FLUSHALL')
+redis.call('FLUSHDB')

main/manager-api/src/test/java/xiaozhi/common/utils/AESUtilsTest.java

@@ -0,0 +1,85 @@
+package xiaozhi.common.utils;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import org.junit.jupiter.api.Test;
+
+public class AESUtilsTest {
+
+    @Test
+    public void testEncryptAndDecrypt() {
+        String key = "xiaozhi1234567890";
+        String plainText = "Hello, 小智!";
+
+        System.out.println("原始文本: " + plainText);
+        System.out.println("密钥: " + key);
+
+        // 加密
+        String encrypted = AESUtils.encrypt(key, plainText);
+        System.out.println("加密结果: " + encrypted);
+
+        // 解密
+        String decrypted = AESUtils.decrypt(key, encrypted);
+        System.out.println("解密结果: " + decrypted);
+
+        // 验证
+        assertEquals(plainText, decrypted, "加解密结果应该一致");
+        System.out.println("加解密一致性: " + plainText.equals(decrypted));
+    }
+
+    @Test
+    public void testDifferentKeyLengths() {
+        String[] keys = {
+                "1234567890123456", // 16位
+                "123456789012345678901234", // 24位
+                "12345678901234567890123456789012", // 32位
+                "short", // 短密钥
+                "verylongkeythatwillbetruncatedto32bytes" // 长密钥
+        };
+
+        String plainText = "测试文本";
+
+        for (String key : keys) {
+            String encrypted = AESUtils.encrypt(key, plainText);
+            String decrypted = AESUtils.decrypt(key, encrypted);
+            assertEquals(plainText, decrypted, "密钥长度: " + key.length());
+        }
+    }
+
+    @Test
+    public void testSpecialCharacters() {
+        String key = "xiaozhi1234567890";
+        String[] testTexts = {
+                "Hello World",
+                "你好世界",
+                "Hello, 小智!",
+                "特殊字符: !@#$%^&*()",
+                "数字123和中文混合",
+                "Emoji: 😀🎉🚀",
+                "空字符串测试",
+                ""
+        };
+
+        for (String text : testTexts) {
+            String encrypted = AESUtils.encrypt(key, text);
+            String decrypted = AESUtils.decrypt(key, encrypted);
+            assertEquals(text, decrypted, "测试文本: " + text);
+        }
+    }
+
+    @Test
+    public void testCrossLanguageCompatibility() {
+        // 这些是Python版本生成的加密结果，用于测试跨语言兼容性
+        String key = "xiaozhi1234567890";
+        String plainText = "Hello, 小智!";
+
+        // Python版本生成的加密结果（需要运行Python测试后获取）
+        // String pythonEncrypted = "从Python测试中获取的加密结果";
+        // String decrypted = AESUtils.decrypt(key, pythonEncrypted);
+        // assertEquals(plainText, decrypted, "Java应该能解密Python加密的结果");
+
+        // 生成Java加密结果供Python测试
+        String javaEncrypted = AESUtils.encrypt(key, plainText);
+        System.out.println("Java加密结果供Python测试: " + javaEncrypted);
+    }
+}

main/manager-web/src/App.vue

@@ -42,7 +42,7 @@ nav {
 }
 
 .el-message {
-  top: 45px !important;
+  top: 70px !important;
 }
 </style>
 <script>

main/manager-web/src/apis/api.js

@@ -7,6 +7,7 @@ import model from './module/model.js'
 import ota from './module/ota.js'
 import timbre from "./module/timbre.js"
 import user from './module/user.js'
+
 /**
  * 接口地址
  * 开发时自动读取使用.env.development文件
@@ -22,7 +23,6 @@ export function getServiceUrl() {
     return DEV_API_SERVICE
 }
 
-
 /** request服务封装 */
 export default {
     getServiceUrl,

main/manager-web/src/apis/module/agent.js

@@ -143,4 +143,126 @@ export default {
                 });
             }).send();
     },
+    // 获取智能体的MCP接入点地址
+    getAgentMcpAccessAddress(agentId, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/mcp/address/${agentId}`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getAgentMcpAccessAddress(agentId, callback);
+                });
+            }).send();
+    },
+    // 获取智能体的MCP工具列表
+    getAgentMcpToolsList(agentId, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/mcp/tools/${agentId}`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getAgentMcpToolsList(agentId, callback);
+                });
+            }).send();
+    },
+    // 添加智能体的声纹
+    addAgentVoicePrint(voicePrintData, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print`)
+            .method('POST')
+            .data(voicePrintData)
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.addAgentVoicePrint(voicePrintData, callback);
+                });
+            }).send();
+    },
+    // 获取指定智能体声纹列表
+    getAgentVoicePrintList(id,callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print/list/${id}`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getAgentVoicePrintList(id,callback);
+                });
+            }).send();
+    },
+    // 删除智能体声纹
+    deleteAgentVoicePrint(id, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print/${id}`)
+            .method('DELETE')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.deleteAgentVoicePrint(id, callback);
+                });
+            }).send();
+    },
+    // 更新智能体声纹
+    updateAgentVoicePrint(voicePrintData, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print`)
+            .method('PUT')
+            .data(voicePrintData)
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.updateAgentVoicePrint(voicePrintData, callback);
+                });
+            }).send();
+    },
+    // 获取指定智能体用户类型聊天记录
+    getRecentlyFiftyByAgentId(id,callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/${id}/chat-history/user`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getRecentlyFiftyByAgentId(id,callback);
+                });
+            }).send();
+    },
+    // 获取指定智能体用户类型聊天记录
+    getContentByAudioId(id,callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/${id}/chat-history/audio`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getContentByAudioId(id,callback);
+                });
+            }).send();
+    },
 }

main/manager-web/src/apis/module/device.js

@@ -68,4 +68,21 @@ export default {
                 })
             }).send()
     },
+    // 手动添加设备
+    manualAddDevice(params, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/device/manual-add`)
+            .method('POST')
+            .data(params)
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail((err) => {
+                console.error('手动添加设备失败:', err);
+                RequestService.reAjaxFun(() => {
+                    this.manualAddDevice(params, callback);
+                });
+            }).send();
+    },
 }

main/manager-web/src/apis/module/timbre.js

@@ -34,6 +34,8 @@ export default {
                 languages: params.languageType,
                 name: params.voiceName,
                 remark: params.remark,
+                referenceAudio: params.referenceAudio,
+                referenceText: params.referenceText,
                 sort: params.sort,
                 ttsModelId: params.ttsModelId,
                 ttsVoice: params.voiceCode,
@@ -75,6 +77,8 @@ export default {
                 languages: params.languageType,
                 name: params.voiceName,
                 remark: params.remark,
+                referenceAudio: params.referenceAudio,
+                referenceText: params.referenceText,
                 ttsModelId: params.ttsModelId,
                 ttsVoice: params.voiceCode,
                 voiceDemo: params.voiceDemo || ''

main/manager-web/src/components/ChatHistoryDialog.vue

@@ -24,7 +24,7 @@
                             <img :src="message.chatType === 1 ? getUserAvatar(currentSessionId) : require('@/assets/xiaozhi-logo.png')"
                                 class="avatar" />
                             <div class="message-content">
-                                {{ message.content }}
+                                {{ extractContentFromString(message.content) }}
                                 <i v-if="message.audioId" :class="getAudioIconClass(message)"
                                     @click="playAudio(message)" class="audio-icon"></i>
                             </div>
@@ -129,6 +129,32 @@ export default {
         }
     },
     methods: {
+        /**
+         * 从 content 字段中提取聊天内容
+         * 如果 content 是 JSON 格式（如 {"speaker": "未知说话人", "content": "现在几点了。"}），则提取 content 字段
+         * 如果 content 是普通字符串，则直接返回
+         * 
+         * @param {string} content 原始内容
+         * @returns {string} 提取的聊天内容
+         */
+        extractContentFromString(content) {
+            if (!content || content.trim() === '') {
+                return content;
+            }
+
+            // 尝试解析为 JSON
+            try {
+                const jsonObj = JSON.parse(content);
+                if (jsonObj && typeof jsonObj === 'object' && jsonObj.content) {
+                    return jsonObj.content;
+                }
+            } catch (e) {
+                // 如果不是有效的 JSON，直接返回原内容
+            }
+
+            // 如果不是 JSON 格式或没有 content 字段，直接返回原内容
+            return content;
+        },
         resetData() {
             this.sessions = [];
             this.messages = [];

main/manager-web/src/components/DeviceItem.vue

@@ -22,6 +22,9 @@
     <div style="display: flex;gap: 10px;align-items: center;">
       <div class="settings-btn" @click="handleConfigure">
         配置角色
+      </div>
+       <div class="settings-btn" @click="handleVoicePrint">
+        声纹识别
       </div>
       <div class="settings-btn" @click="handleDeviceManage">
         设备管理({{ device.deviceCount }})
@@ -77,6 +80,9 @@ export default {
     handleConfigure() {
       this.$router.push({ path: '/role-config', query: { agentId: this.device.agentId } });
     },
+    handleVoicePrint() {
+      this.$router.push({ path: '/voice-print', query: { agentId: this.device.agentId } });
+    },
     handleDeviceManage() {
       this.$router.push({ path: '/device-management', query: { agentId: this.device.agentId } });
     },

main/manager-web/src/components/FunctionDialog.vue

@@ -99,6 +99,62 @@
       </div>
     </div>
 
+    <!-- MCP区域 -->
+    <div class="mcp-access-point">
+      <div class="mcp-container">
+        <!-- 左侧区域 -->
+        <div class="mcp-left">
+          <div class="mcp-header">
+            <h3 class="bold-title">MCP接入点</h3>
+          </div>
+          <div class="url-header">
+            <div class="address-desc">
+              <span>以下是智能体的MCP接入点地址。</span>
+              <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-enable.md"
+                target="_blank" class="doc-link">如何部署MCP接入点</a> &nbsp;&nbsp;|&nbsp;&nbsp;
+              <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md"
+                target="_blank" class="doc-link">如何接入MCP功能</a> &nbsp;
+            </div>
+          </div>
+          <el-input v-model="mcpUrl" readonly class="url-input">
+            <template #suffix>
+              <el-button @click="copyUrl" class="inner-copy-btn" icon="el-icon-document-copy">
+                复制
+              </el-button>
+            </template>
+          </el-input>
+        </div>
+
+        <!-- 右侧区域 -->
+        <div class="mcp-right">
+          <div class="mcp-header">
+            <h3 class="bold-title">接入点状态</h3>
+          </div>
+          <div class="status-container">
+            <span class="status-indicator" :class="mcpStatus"></span>
+            <span class="status-text">{{
+              mcpStatus === 'connected' ? '已连接' :
+                mcpStatus === 'loading' ? '加载中...' : '未连接'
+            }}</span>
+            <button class="refresh-btn" @click="refreshStatus">
+              <span class="refresh-icon">↻</span>
+              <span>刷新</span>
+            </button>
+          </div>
+          <div class="mcp-tools-list">
+            <div v-if="mcpTools.length > 0" class="tools-grid">
+              <el-button v-for="tool in mcpTools" :key="tool" size="small" class="tool-btn" plain>
+                {{ tool }}
+              </el-button>
+            </div>
+            <div v-else class="no-tools">
+              <span>暂无可用工具</span>
+            </div>
+          </div>
+        </div>
+      </div>
+    </div>
+
     <div class="drawer-footer">
       <el-button @click="closeDialog">取消</el-button>
       <el-button type="primary" @click="saveSelection">保存配置</el-button>
@@ -107,6 +163,8 @@
 </template>
 
 <script>
+import Api from '@/apis/api';
+
 export default {
   props: {
     value: Boolean,
@@ -117,6 +175,10 @@ export default {
     allFunctions: {
       type: Array,
       default: () => []
+    },
+    agentId: {
+      type: String,
+      required: true
     }
   },
   data() {
@@ -134,6 +196,10 @@ export default {
       // 添加一个标志位来跟踪是否已经保存
       hasSaved: false,
       loading: false,
+
+      mcpUrl: "",
+      mcpStatus: "disconnected",
+      mcpTools: [],
     }
   },
   computed: {
@@ -177,6 +243,10 @@ export default {
         });
         // 右侧默认指向第一个
         this.currentFunction = this.selectedList[0] || null;
+
+        // 加载MCP数据
+        this.loadMcpAddress();
+        this.loadMcpTools();
       }
     },
     dialogVisible(newVal) {
@@ -184,6 +254,60 @@ export default {
     }
   },
   methods: {
+    copyUrl() {
+      const textarea = document.createElement('textarea');
+      textarea.value = this.mcpUrl;
+      textarea.style.position = 'fixed';  // 防止页面滚动
+      document.body.appendChild(textarea);
+      textarea.select();
+
+      try {
+        const successful = document.execCommand('copy');
+        if (successful) {
+          this.$message.success('已复制到剪贴板');
+        } else {
+          this.$message.error('复制失败，请手动复制');
+        }
+      } catch (err) {
+        this.$message.error('复制失败，请手动复制');
+        console.error('复制失败:', err);
+      } finally {
+        document.body.removeChild(textarea);
+      }
+    },
+
+    refreshStatus() {
+      this.mcpStatus = "loading";
+      this.loadMcpTools();
+    },
+
+    // 加载MCP接入点地址
+    loadMcpAddress() {
+      Api.agent.getAgentMcpAccessAddress(this.agentId, (res) => {
+        if (res.data.code === 0) {
+          this.mcpUrl = res.data.data || "";
+        } else {
+          this.mcpUrl = "";
+          console.error('获取MCP地址失败:', res.data.msg);
+        }
+      });
+    },
+
+    // 加载MCP工具列表
+    loadMcpTools() {
+      Api.agent.getAgentMcpToolsList(this.agentId, (res) => {
+        if (res.data.code === 0) {
+          this.mcpTools = res.data.data || [];
+          // 根据工具列表更新状态
+          this.mcpStatus = this.mcpTools.length > 0 ? "connected" : "disconnected";
+        } else {
+          this.mcpTools = [];
+          this.mcpStatus = "disconnected";
+          console.error('获取MCP工具列表失败:', res.data.msg);
+        }
+      });
+    },
+
     flushArray(key) {
       const text = this.textCache[key] || '';
       const arr = text
@@ -298,7 +422,7 @@ export default {
   display: grid;
   grid-template-columns: max-content max-content 1fr;
   gap: 12px;
-  height: calc(70vh - 60px);
+  height: calc(58vh);
 }
 
 .custom-header {
@@ -514,4 +638,203 @@ export default {
 ::v-deep .el-checkbox__label {
   display: none;
 }
+
+.mcp-access-point {
+  border-top: 1px solid #EBEEF5;
+  padding: 20px 24px;
+  text-align: left;
+}
+
+.mcp-header {
+  .bold-title {
+    font-size: 18px;
+    font-weight: bold;
+    margin: 5px 0 30px 0;
+  }
+}
+
+.mcp-container {
+  display: flex;
+  justify-content: space-between;
+  gap: 30px;
+}
+
+.mcp-left,
+.mcp-right {
+  flex: 1;
+  padding-bottom: 50px;
+}
+
+.url-header {
+  margin-bottom: 8px;
+  color: black;
+
+  h4 {
+    margin: 0 0 15px 0;
+    font-size: 16px;
+    font-weight: normal;
+  }
+
+  .address-desc {
+    display: flex;
+    align-items: center;
+    font-size: 14px;
+    margin-bottom: 12px;
+
+    .doc-link {
+      color: #1677ff;
+      text-decoration: none;
+      margin-left: 4px;
+
+      &:hover {
+        text-decoration: underline;
+      }
+    }
+  }
+}
+
+.url-input {
+  border-radius: 4px 0 0 4px;
+  font-size: 14px;
+  height: 36px;
+  box-sizing: border-box;
+  background-color: #f5f5f5;
+}
+
+::v-deep .el-input__inner {
+  background-color: #f5f5f5;
+  padding-right: 80px;
+}
+
+.url-input {
+
+  ::v-deep .el-input__suffix {
+    right: 0;
+    display: flex;
+    align-items: center;
+    padding-right: 10px;
+
+    .inner-copy-btn {
+      pointer-events: auto;
+      border: none;
+      background: #1677ff;
+      color: white;
+      padding: 6px;
+      margin-top: 4px;
+      margin-left: 4px;
+    }
+  }
+}
+
+.mcp-right {
+  h4 {
+    margin: 0 0 10px 0;
+    font-size: 16px;
+    font-weight: normal;
+    color: black;
+  }
+}
+
+.status-container {
+  display: flex;
+  align-items: center;
+
+  .status-indicator {
+    display: inline-block;
+    width: 8px;
+    height: 8px;
+    border-radius: 50%;
+    margin-right: 8px;
+
+    &.disconnected {
+      background-color: #909399;
+      /* 灰色 - 未连接 */
+    }
+
+    &.connected {
+      background-color: #67C23A;
+      /* 绿色 - 已连接 */
+    }
+
+    &.loading {
+      background-color: #E6A23C;
+      /* 橙色 - 加载中 */
+      animation: pulse 1.5s infinite;
+    }
+  }
+
+  .status-text {
+    font-size: 14px;
+    margin-right: 10px;
+  }
+
+  .refresh-btn {
+    display: flex;
+    align-items: center;
+    padding: 2px 10px;
+    background: white;
+    color: black;
+    border: 1px solid #DCDFE6;
+    border-radius: 4px;
+    cursor: pointer;
+    font-size: 14px;
+    transition: all 0.3s;
+
+    &:hover {
+      background: #1677ff;
+      color: white;
+      border-color: #1677ff;
+    }
+
+    .refresh-icon {
+      margin-right: 6px;
+      font-size: 14px;
+    }
+  }
+}
+
+@keyframes pulse {
+  0% {
+    opacity: 1;
+  }
+
+  50% {
+    opacity: 0.4;
+  }
+
+  100% {
+    opacity: 1;
+  }
+}
+
+.mcp-tools-list {
+  margin-top: 10px;
+
+  .tools-grid {
+    display: flex;
+    flex-wrap: wrap;
+    gap: 8px;
+  }
+
+  .tool-btn {
+    padding: 6px 12px;
+    border-color: #1677ff;
+    color: #1677ff;
+    background-color: white;
+    font-size: 12px;
+
+    &:hover {
+      background-color: #1677ff;
+      color: white;
+      border-color: #1677ff;
+    }
+  }
+
+  .no-tools {
+    text-align: center;
+    color: #909399;
+    font-size: 14px;
+    padding: 10px 0;
+  }
+}
 </style>

main/manager-web/src/components/ManualAddDeviceDialog.vue

@@ -0,0 +1,158 @@
+<template>
+  <el-dialog title="手动添加设备" :visible="visible" @close="handleClose" width="30%" center>
+    <div class="dialog-content">
+      <el-form :model="deviceForm" :rules="rules" ref="deviceForm" label-width="100px">
+        <el-form-item label="设备型号" prop="board">
+          <el-select v-model="deviceForm.board" placeholder="请选择设备型号" style="width: 100%">
+            <el-option
+              v-for="item in firmwareTypes"
+              :key="item.key"
+              :label="item.name"
+              :value="item.key">
+            </el-option>
+          </el-select>
+        </el-form-item>
+        <el-form-item label="固件版本" prop="appVersion">
+          <el-input v-model="deviceForm.appVersion" placeholder="请输入固件版本"></el-input>
+        </el-form-item>
+        <el-form-item label="Mac地址" prop="macAddress">
+          <el-input v-model="deviceForm.macAddress" placeholder="请输入Mac地址"></el-input>
+        </el-form-item>
+      </el-form>
+    </div>
+    <div style="display: flex;margin: 15px 15px;gap: 7px;">
+      <div class="dialog-btn" @click="submitForm">确定</div>
+      <div class="dialog-btn cancel-btn" @click="cancel">取消</div>
+    </div>
+  </el-dialog>
+</template>
+
+<script>
+import Api from '@/apis/api';
+
+export default {
+  name: 'ManualAddDeviceDialog',
+  props: {
+    visible: { type: Boolean, required: true },
+    agentId: { type: String, required: true }
+  },
+  data() {
+    // MAC地址验证规则
+    const validateMac = (rule, value, callback) => {
+      const macRegex = /^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$/;
+      if (!value) {
+        callback(new Error('请输入Mac地址'));
+      } else if (!macRegex.test(value)) {
+        callback(new Error('请输入正确的Mac地址格式，例如：00:1A:2B:3C:4D:5E'));
+      } else {
+        callback();
+      }
+    };
+
+    return {
+      deviceForm: {
+        board: '',
+        appVersion: '',
+        macAddress: ''
+      },
+      firmwareTypes: [],
+      rules: {
+        board: [
+          { required: true, message: '请选择设备型号', trigger: 'change' }
+        ],
+        appVersion: [
+          { required: true, message: '请输入固件版本', trigger: 'blur' }
+        ],
+        macAddress: [
+          { required: true, validator: validateMac, trigger: 'blur' }
+        ]
+      }
+    }
+  },
+  created() {
+    this.getFirmwareTypes();
+  },
+  methods: {
+    async getFirmwareTypes() {
+      try {
+        const res = await Api.dict.getDictDataByType('FIRMWARE_TYPE');
+        this.firmwareTypes = res.data;
+      } catch (error) {
+        console.error('获取固件类型失败:', error);
+        this.$message.error(error.message || '获取固件类型失败');
+      }
+    },
+    submitForm() {
+      this.$refs.deviceForm.validate((valid) => {
+        if (valid) {
+          this.addDevice();
+        }
+      });
+    },
+    addDevice() {
+      const params = {
+        agentId: this.agentId,
+        ...this.deviceForm
+      };
+      
+      Api.device.manualAddDevice(params, ({ data }) => {
+        if (data.code === 0) {
+          this.$message.success('设备添加成功');
+          this.$emit('refresh');
+          this.closeDialog();
+        } else {
+          this.$message.error(data.msg || '添加失败');
+        }
+      });
+    },
+    closeDialog() {
+      this.$emit('update:visible', false);
+      this.$refs.deviceForm.resetFields();
+    },
+    cancel() {
+      this.closeDialog();
+    },
+    handleClose() {
+      this.closeDialog();
+    }
+  }
+}
+</script>
+
+<style scoped>
+.dialog-content {
+  padding: 0 20px;
+}
+
+.dialog-btn {
+  cursor: pointer;
+  flex: 1;
+  border-radius: 23px;
+  background: #5778ff;
+  height: 40px;
+  font-weight: 500;
+  font-size: 12px;
+  color: #fff;
+  line-height: 40px;
+  text-align: center;
+}
+
+.cancel-btn {
+  background: #e6ebff;
+  border: 1px solid #adbdff;
+  color: #5778ff;
+}
+
+::v-deep .el-dialog {
+  border-radius: 15px;
+  box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
+}
+
+::v-deep .el-dialog__body {
+  padding: 20px 6px;
+}
+
+::v-deep .el-form-item {
+  margin-bottom: 20px;
+}
+</style> 

main/manager-web/src/components/TtsModel.vue

@@ -1,79 +1,66 @@
 <template>
-  <el-dialog
+  <el-dialog :visible.sync="localVisible" width="90%" @close="handleClose" :show-close="false" :append-to-body="true"
-      :visible.sync="localVisible"
+    :close-on-click-modal="true">
-      width="85%"
-      @close="handleClose"
-      :show-close="false"
-      :append-to-body="true"
-      :close-on-click-modal="true">
     <button class="custom-close-btn" @click="handleClose">
       ×
     </button>
     <div class="scroll-wrapper">
-      <div
+      <div class="table-container" ref="tableContainer" @scroll="handleScroll">
-          class="table-container"
+        <el-table v-loading="loading" :data="filteredTtsModels" style="width: 100%;" class="data-table"
-          ref="tableContainer"
+          header-row-class-name="table-header" :fit="true" element-loading-text="拼命加载中"
-          @scroll="handleScroll">
+          element-loading-spinner="el-icon-loading" element-loading-background="rgba(0, 0, 0, 0.8)">
-          <el-table
-              v-loading="loading"
-              :data="filteredTtsModels"
-              style="width: 100%;"
-              class="data-table"
-              header-row-class-name="table-header"
-              :fit="true"
-              element-loading-text="拼命加载中"
-              element-loading-spinner="el-icon-loading"
-              element-loading-background="rgba(0, 0, 0, 0.8)">
           <el-table-column label="选择" width="50" align="center">
             <template slot-scope="scope">
               <el-checkbox v-model="scope.row.selected"></el-checkbox>
             </template>
           </el-table-column>
-          <el-table-column label="音色编码" align="center" min-width="50">
+          <el-table-column label="音色编码" align="center">
             <template slot-scope="scope">
               <el-input v-if="scope.row.editing" v-model="scope.row.voiceCode"></el-input>
               <span v-else>{{ scope.row.voiceCode }}</span>
             </template>
           </el-table-column>
-          <el-table-column label="音色名称" align="center" min-width="50">
+          <el-table-column label="音色名称" align="center">
             <template slot-scope="scope">
               <el-input v-if="scope.row.editing" v-model="scope.row.voiceName"></el-input>
               <span v-else>{{ scope.row.voiceName }}</span>
             </template>
           </el-table-column>
-          <el-table-column label="语言类型" align="center" min-width="50">
+          <el-table-column label="语言类型" align="center">
             <template slot-scope="scope">
               <el-input v-if="scope.row.editing" v-model="scope.row.languageType"></el-input>
               <span v-else>{{ scope.row.languageType }}</span>
             </template>
           </el-table-column>
-          <el-table-column label="试听" align="center" min-width="100px" class-name="audio-column">
+          <el-table-column v-if="!showReferenceColumns" label="试听" align="center" class-name="audio-column">
             <template slot-scope="scope">
               <div class="custom-audio-container">
-                <el-input
+                <el-input v-if="scope.row.editing" v-model="scope.row.voiceDemo" placeholder="请输入MP3地址"
-                    v-if="scope.row.editing"
+                  class="audio-input">
-                    v-model="scope.row.voiceDemo"
-                    placeholder="请输入MP3地址"
-                    size="mini"
-                    class="audio-input">
                 </el-input>
-                <AudioPlayer v-else-if="isValidAudioUrl(scope.row.voiceDemo)" :audioUrl="scope.row.voiceDemo"/>
+                <AudioPlayer v-else-if="isValidAudioUrl(scope.row.voiceDemo)" :audioUrl="scope.row.voiceDemo" />
               </div>
             </template>
           </el-table-column>
-          <el-table-column label="备注" align="center">
+          <el-table-column v-if="!showReferenceColumns" label="备注" align="center">
             <template slot-scope="scope">
-              <el-input
+              <el-input v-if="scope.row.editing" type="textarea" :rows="1" autosize v-model="scope.row.remark"
-                  v-if="scope.row.editing"
+                placeholder="这里是备注" class="remark-input"></el-input>
-                  type="textarea"
-                  :rows="1"
-                  autosize
-                  v-model="scope.row.remark"
-                  placeholder="这里是备注"
-                  class="remark-input"></el-input>
               <span v-else>{{ scope.row.remark }}</span>
             </template>
           </el-table-column>
+          <el-table-column v-if="showReferenceColumns" label="克隆音频路径" align="center">
+            <template slot-scope="scope">
+              <el-input v-if="scope.row.editing" v-model="scope.row.referenceAudio" placeholder="这里是克隆音频路径"></el-input>
+              <span v-else>{{ scope.row.referenceAudio }}</span>
+            </template>
+          </el-table-column>
+          <el-table-column v-if="showReferenceColumns" label="克隆音频文本" align="center">
+            <template slot-scope="scope">
+              <el-input v-if="scope.row.editing" v-model="scope.row.referenceText" placeholder="这里是克隆音频对应文本"></el-input>
+              <span v-else>{{ scope.row.referenceText }}</span>
+            </template>
+          </el-table-column>
           <el-table-column label="操作" align="center" width="150">
             <template slot-scope="scope">
               <template v-if="!scope.row.editing">
@@ -84,12 +71,7 @@
                   删除
                 </el-button>
               </template>
-              <el-button
+              <el-button v-else type="success" size="mini" @click="saveEdit(scope.row)" class="save-Tts">保存
-                  v-else
-                  type="success"
-                  size="mini"
-                  @click="saveEdit(scope.row)"
-                  class="save-Tts">保存
               </el-button>
             </template>
           </el-table-column>
@@ -110,21 +92,19 @@
       <el-button type="primary" size="mini" @click="addNew" style="background: #5bc98c;border: None;">
         新增
       </el-button>
-      <el-button type="primary"
+      <el-button type="primary" size="mini" @click="deleteRow(filteredTtsModels.filter(row => row.selected))"
-                 size="mini"
+        style="background: red;border:None">删除
-                 @click="deleteRow(filteredTtsModels.filter(row => row.selected))"
-                 style="background: red;border:None">删除
       </el-button>
     </div>
   </el-dialog>
 </template>
 
 <script>
-import AudioPlayer from './AudioPlayer.vue'
 import Api from "@/apis/api";
+import AudioPlayer from './AudioPlayer.vue';
 
 export default {
-  components: {AudioPlayer},
+  components: { AudioPlayer },
   props: {
     visible: {
       type: Boolean,
@@ -133,6 +113,10 @@ export default {
     ttsModelId: {
       type: String,
       required: true
+    },
+    modelConfig: {
+      type: Object,
+      default: null
     }
   },
   data() {
@@ -151,6 +135,7 @@ export default {
       selectAll: false,
       selectedRows: [],
       loading: false,
+      showReferenceColumns: false, // 控制是否显示参考列
     };
   },
   watch: {
@@ -158,12 +143,19 @@ export default {
       this.localVisible = newVal;
       if (newVal) {
         this.currentPage = 1;
+        this.updateShowReferenceColumns(); // 更新显示状态
         this.loadData(); // 对话框显示时加载数据
         this.$nextTick(() => {
           this.updateScrollbar();
         });
       }
     },
+    modelConfig: {
+      handler(newVal) {
+        this.updateShowReferenceColumns();
+      },
+      immediate: true
+    },
     filteredTtsModels() {
       this.$nextTick(() => {
         this.updateScrollbar();
@@ -173,7 +165,7 @@ export default {
   computed: {
     filteredTtsModels() {
       return this.ttsModels.filter(model =>
-          model.voiceName.toLowerCase().includes(this.searchQuery.toLowerCase())
+        model.voiceName.toLowerCase().includes(this.searchQuery.toLowerCase())
       );
     }
   },
@@ -189,6 +181,16 @@ export default {
     window.removeEventListener('mousemove', this.handleDrag);
   },
   methods: {
+    // 更新是否显示参考列
+    updateShowReferenceColumns() {
+      if (this.modelConfig && this.modelConfig.configJson) {
+        const providerType = this.modelConfig.configJson.type;
+        this.showReferenceColumns = ['fishspeech', 'gpt_sovits_v2', 'gpt_sovits_v3'].includes(providerType);
+      } else {
+        this.showReferenceColumns = false;
+      }
+    },
+
     loadData() {
       this.loading = true;
       const params = {
@@ -206,6 +208,8 @@ export default {
               voiceName: item.name || '未命名音色',
               languageType: item.languages || '',
               remark: item.remark || '',
+              referenceAudio: item.referenceAudio || '',
+              referenceText: item.referenceText || '',
               voiceDemo: item.voiceDemo || '',
               selected: false,
               editing: false,
@@ -237,6 +241,7 @@ export default {
       this.total = 0;
       this.selectAll = false;
       this.searchQuery = '';
+      this.showReferenceColumns = false;
 
       this.localVisible = false;
       this.$emit('update:visible', false);
@@ -249,7 +254,7 @@ export default {
 
       if (!container || !scrollbarThumb || !scrollbarTrack) return;
 
-      const {scrollHeight, clientHeight} = container;
+      const { scrollHeight, clientHeight } = container;
       const trackHeight = scrollbarTrack.clientHeight;
       const thumbHeight = Math.max((clientHeight / scrollHeight) * trackHeight, 20);
 
@@ -264,7 +269,7 @@ export default {
 
       if (!container || !scrollbarThumb || !scrollbarTrack) return;
 
-      const {scrollHeight, clientHeight, scrollTop} = container;
+      const { scrollHeight, clientHeight, scrollTop } = container;
       const trackHeight = scrollbarTrack.clientHeight;
       const thumbHeight = scrollbarThumb.clientHeight;
       const maxTop = trackHeight - thumbHeight;
@@ -332,7 +337,7 @@ export default {
 
     startEdit(row) {
       row.editing = true;
-      this.$set(row, 'originalData', {...row});
+      this.$set(row, 'originalData', { ...row });
     },
 
     saveEdit(row) {
@@ -356,6 +361,12 @@ export default {
           sort: row.sort
         };
 
+        // 只有在显示参考列的情况下才添加参考字段
+        if (this.showReferenceColumns) {
+          params.referenceAudio = row.referenceAudio;
+          params.referenceText = row.referenceText;
+        }
+
         let res;
         if (row.id) {
           // 已有ID，执行更新操作
@@ -423,8 +434,8 @@ export default {
       }
 
       const maxSort = this.ttsModels.length > 0
-          ? Math.max(...this.ttsModels.map(item => Number(item.sort) || 0))
+        ? Math.max(...this.ttsModels.map(item => Number(item.sort) || 0))
-          : 0;
+        : 0;
 
       const newRow = {
         voiceCode: '',
@@ -432,6 +443,8 @@ export default {
         languageType: '中文',
         voiceDemo: '',
         remark: '',
+        referenceAudio: '',
+        referenceText: '',
         selected: false,
         editing: true,
         sort: maxSort + 1
@@ -441,56 +454,56 @@ export default {
     },
 
     deleteRow(row) {
-        // 处理单个音色或音色数组
+      // 处理单个音色或音色数组
-        const voices = Array.isArray(row) ? row : [row];
+      const voices = Array.isArray(row) ? row : [row];
 
-        if (Array.isArray(row) && row.length === 0) {
+      if (Array.isArray(row) && row.length === 0) {
-            this.$message.warning("请先选择需要删除的音色");
+        this.$message.warning("请先选择需要删除的音色");
-            return;
+        return;
+      }
+
+
+      const voiceCount = voices.length;
+      this.$confirm(`确定要删除选中的${voiceCount}个音色吗？`, "警告", {
+        confirmButtonText: "确定",
+        cancelButtonText: "取消",
+        type: "warning",
+        distinguishCancelAndClose: true
+      }).then(() => {
+        const ids = voices.map(voice => voice.id);
+        if (ids.some(id => !id)) {
+          this.$message.error("存在无效的音色ID");
+          return;
         }
 
-
+        Api.timbre.deleteVoice(ids, ({ data }) => {
-        const voiceCount = voices.length;
+          if (data.code === 0) {
-        this.$confirm(`确定要删除选中的${voiceCount}个音色吗？`, "警告", {
+            this.$message.success({
-            confirmButtonText: "确定",
+              message: `成功删除${voiceCount}个参数`,
-            cancelButtonText: "取消",
+              showClose: true
-            type: "warning",
+            });
-            distinguishCancelAndClose: true
+            this.loadData(); // 刷新参数列表
-        }).then(() => {
+          } else {
-            const ids = voices.map(voice => voice.id);
+            this.$message.error({
-            if (ids.some(id => !id)) {
+              message: data.msg || '删除失败，请重试',
-                this.$message.error("存在无效的音色ID");
+              showClose: true
-                return;
+            });
-            }
+          }
-
-        Api.timbre.deleteVoice(ids, ({data}) => {
-            if (data.code === 0) {
-                this.$message.success({
-                    message: `成功删除${voiceCount}个参数`,
-                    showClose: true
-                });
-                this.loadData(); // 刷新参数列表
-            } else {
-                this.$message.error({
-                    message: data.msg || '删除失败，请重试',
-                    showClose: true
-                });
-            }
         });
       }).catch(action => {
-          if (action === 'cancel') {
+        if (action === 'cancel') {
-              this.$message({
+          this.$message({
-                  type: 'info',
+            type: 'info',
-                  message: '已取消删除操作',
+            message: '已取消删除操作',
-                  duration: 1000
+            duration: 1000
-              });
+          });
-          } else {
+        } else {
-              this.$message({
+          this.$message({
-                  type: 'info',
+            type: 'info',
-                  message: '操作已关闭',
+            message: '操作已关闭',
-                  duration: 1000
+            duration: 1000
-              });
+          });
-          }
+        }
       });
     },
 
@@ -502,7 +515,6 @@ export default {
 </script>
 
 <style lang="scss" scoped>
-
 ::v-deep .el-dialog {
   border-radius: 8px !important;
   overflow: hidden;
@@ -695,6 +707,7 @@ export default {
   white-space: pre-wrap !important;
   word-break: break-all !important;
 }
+
 /* 按钮组定位调整 */
 .action-buttons {
   position: static;
@@ -719,5 +732,4 @@ export default {
   flex-shrink: 0;
   margin: 2px !important;
 }
-
 </style>

main/manager-web/src/components/VoicePrintDialog.vue

@@ -0,0 +1,424 @@
+<template>
+  <el-dialog :title="title" :visible.sync="visible" width="520px" class="param-dialog-wrapper" :append-to-body="true"
+    :close-on-click-modal="false" :key="dialogKey" custom-class="custom-param-dialog" :show-close="false">
+    <div class="dialog-container">
+      <div class="dialog-header">
+        <h2 class="dialog-title">{{ title }}</h2>
+        <button class="custom-close-btn" @click="cancel">
+          <svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg">
+            <path d="M13 1L1 13M1 1L13 13" stroke="currentColor" stroke-width="2" stroke-linecap="round" />
+          </svg>
+        </button>
+      </div>
+
+      <el-form :model="form" :rules="rules" ref="form" label-width="110px" label-position="left" class="param-form">
+        <el-form-item label="声纹向量" prop="audioId" class="form-item">
+          <el-select v-model="form.audioId" placeholder="请选择一条语言消息" class="custom-select">
+            <el-option v-for="item in valueTypeOptions" :key="item.audioId" :label="item.content" :value="item.audioId">
+              <span style="float: left">{{ item.content }}</span>
+              <span style="float: right; color: #8492a6; font-size: 13px">
+                <i :class="getAudioIconClass(item.audioId)" @click.stop="playAudio(item.audioId)"
+                  class="audio-icon"></i>
+              </span>
+            </el-option>
+          </el-select>
+        </el-form-item>
+
+        <el-form-item label="姓名" prop="sourceName" class="form-item">
+          <el-input v-model="form.sourceName" placeholder="请输入姓名" class="custom-input"></el-input>
+        </el-form-item>
+
+        <el-form-item label="描述" prop="introduce" class="form-item remark-item">
+          <el-input type="textarea" v-model="form.introduce" placeholder="请输入描述" :rows="3" class="custom-textarea"
+            maxlength="100" show-word-limit></el-input>
+        </el-form-item>
+      </el-form>
+
+      <div class="dialog-footer">
+        <el-button type="primary" @click="submit" class="save-btn" :loading="saving" :disabled="saving">
+          保存
+        </el-button>
+        <el-button @click="cancel" class="cancel-btn">
+          取消
+        </el-button>
+      </div>
+    </div>
+  </el-dialog>
+</template>
+
+<script>
+import api from '@/apis/api';
+
+export default {
+  props: {
+    title: {
+      type: String,
+      default: '添加说话人'
+    },
+    visible: {
+      type: Boolean,
+      default: false
+    },
+    agentId: {
+      type: String
+    },
+    form: {
+      type: Object,
+      default: () => ({
+        id: null,
+        audioId: '',
+        sourceName: '',
+        introduce: ''
+      })
+    }
+  },
+  data() {
+    return {
+      dialogKey: Date.now(),
+      saving: false,
+      playingAudioId: null,
+      audioElement: null,
+      valueTypeOptions: [
+        { audioId: '', content: '' }
+      ],
+      rules: {
+        introduce: [
+          { required: true, message: "请输入描述", trigger: "blur" }
+        ],
+        sourceName: [
+          { required: true, message: "请输入姓名", trigger: "blur" }
+        ],
+        audioId: [
+          { required: true, message: "请选择音频向量", trigger: "change" }
+        ]
+      }
+    };
+  },
+  methods: {
+    getAudioIconClass(audioId) {
+      if (this.playingAudioId === audioId) {
+        return 'el-icon-loading';
+      }
+      return 'el-icon-video-play';
+    },
+    playAudio(audioId) {
+      if (this.playingAudioId === audioId) {
+        // 如果正在播放当前音频，则停止播放
+        if (this.audioElement) {
+          this.audioElement.pause();
+          this.audioElement = null;
+        }
+        this.playingAudioId = null;
+        return;
+      }
+
+      // 停止当前正在播放的音频
+      if (this.audioElement) {
+        this.audioElement.pause();
+        this.audioElement = null;
+      }
+
+      // 先获取音频下载ID
+      this.playingAudioId = audioId;
+      api.agent.getAudioId(audioId, (res) => {
+        if (res.data && res.data.data) {
+          // 使用获取到的下载ID播放音频
+          this.audioElement = new Audio(api.getServiceUrl() + `/agent/play/${res.data.data}`);
+
+          this.audioElement.onended = () => {
+            this.playingAudioId = null;
+            this.audioElement = null;
+          };
+
+          this.audioElement.play();
+        }
+      });
+    },
+    submit() {
+      this.$refs.form.validate((valid) => {
+        if (valid) {
+          this.saving = true; // 开始加载
+          this.$emit('submit', {
+            form: this.form,
+            done: () => {
+              this.saving = false; // 加载完成
+            }
+          });
+
+          setTimeout(() => {
+            this.saving = false;
+          }, 3000);
+        }
+      });
+    },
+    cancel() {
+      this.saving = false; // 取消时重置状态
+      this.$emit('cancel');
+    }
+  },
+  watch: {
+    visible(newVal) {
+      if (newVal) {
+        this.dialogKey = Date.now();
+      }
+    },
+    agentId(newVal) {
+      if (newVal) {
+        api.agent.getRecentlyFiftyByAgentId(this.agentId, ((data) => {
+          this.valueTypeOptions = data.data.data.map(item => ({
+            ...item
+          }));
+        }))
+      }
+    },
+    'form.audioId'(newVal) {
+      if (newVal == null || newVal == "") {
+        return
+      }
+      if (this.valueTypeOptions.some(item => item.audioId === newVal)) {
+        return;
+      }
+      api.agent.getContentByAudioId(newVal, ((data) => {
+        this.valueTypeOptions.push({
+          audioId: newVal, content: data.data.data
+        })
+      }))
+    }
+  }
+};
+</script>
+
+<style>
+.custom-param-dialog {
+  border-radius: 16px !important;
+  overflow: hidden;
+  box-shadow: 0 10px 30px rgba(0, 0, 0, 0.15) !important;
+  border: none !important;
+
+  .el-dialog__header {
+    display: none;
+  }
+
+  .el-dialog__body {
+    padding: 0 !important;
+    border-radius: 16px;
+  }
+}
+</style>
+
+<style scoped lang="scss">
+.audio-icon {
+  font-size: 20px;
+  cursor: pointer;
+  margin: 0 5px;
+  color: #1890ff;
+}
+
+.param-dialog-wrapper {
+  .dialog-container {
+    padding: 24px 32px;
+    background: linear-gradient(135deg, #f8fafc 0%, #f1f5f9 100%);
+  }
+
+  .dialog-header {
+    position: relative;
+    margin-bottom: 24px;
+    text-align: center;
+  }
+
+  .dialog-title {
+    font-size: 20px;
+    color: #1e293b;
+    margin: 0;
+    padding: 0;
+    font-weight: 600;
+    letter-spacing: 0.5px;
+  }
+
+  .custom-close-btn {
+    position: absolute;
+    top: -8px;
+    right: -8px;
+    width: 32px;
+    height: 32px;
+    border-radius: 50%;
+    border: none;
+    background: #f1f5f9;
+    color: #64748b;
+    cursor: pointer;
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    padding: 0;
+    outline: none;
+    transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+
+    &:hover {
+      color: #ffffff;
+      background: #ef4444;
+      transform: rotate(90deg);
+      box-shadow: 0 4px 6px rgba(239, 68, 68, 0.2);
+    }
+
+    svg {
+      transition: transform 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+    }
+  }
+
+  .param-form {
+    .form-item {
+      margin-bottom: 20px;
+
+      :deep(.el-form-item__label) {
+        color: #475569;
+        font-weight: 500;
+        padding-right: 12px;
+        text-align: right;
+        font-size: 14px;
+        letter-spacing: 0.2px;
+      }
+    }
+
+    .custom-input {
+      :deep(.el-input__inner) {
+        background-color: #ffffff;
+        border-radius: 8px;
+        border: 1px solid #e2e8f0;
+        height: 42px;
+        padding: 0 14px;
+        transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+        font-size: 14px;
+        color: #334155;
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+
+        &:focus {
+          border-color: #3b82f6;
+          box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
+          background-color: #ffffff;
+        }
+
+        &::placeholder {
+          color: #94a3b8;
+          font-weight: 400;
+        }
+      }
+    }
+
+    .custom-select {
+      width: 100%;
+
+      :deep(.el-input__inner) {
+        background-color: #ffffff;
+        border-radius: 8px;
+        border: 1px solid #e2e8f0;
+        height: 42px;
+        transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+        font-size: 14px;
+        color: #334155;
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+
+        &:focus {
+          border-color: #3b82f6;
+          box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
+          background-color: #ffffff;
+        }
+
+        &::placeholder {
+          color: #94a3b8;
+          font-weight: 400;
+        }
+      }
+    }
+
+    .custom-textarea {
+      :deep(.el-textarea__inner) {
+        background-color: #ffffff;
+        border-radius: 8px;
+        border: 1px solid #e2e8f0;
+        padding: 12px 14px;
+        transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+        font-size: 14px;
+        color: #334155;
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+        line-height: 1.5;
+
+        &:focus {
+          border-color: #3b82f6;
+          box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
+          background-color: #ffffff;
+        }
+
+        &::placeholder {
+          color: #94a3b8;
+          font-weight: 400;
+        }
+      }
+    }
+
+    .remark-item :deep(.el-form-item__label) {
+      margin-top: -4px;
+    }
+  }
+
+  .dialog-footer {
+    display: flex;
+    justify-content: center;
+    padding: 16px 0 0;
+    margin-top: 16px;
+
+    .save-btn {
+      width: 120px;
+      height: 42px;
+      font-size: 14px;
+      font-weight: 500;
+      border-radius: 8px;
+      transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+      background: #3b82f6;
+      color: white;
+      border: none;
+      letter-spacing: 0.5px;
+      box-shadow: 0 2px 4px rgba(59, 130, 246, 0.2);
+
+      &:hover {
+        background: #2563eb;
+        transform: translateY(-1px);
+        box-shadow: 0 4px 6px rgba(59, 130, 246, 0.3);
+      }
+
+      &:active {
+        transform: translateY(0);
+        box-shadow: 0 2px 3px rgba(59, 130, 246, 0.2);
+      }
+    }
+
+    .cancel-btn {
+      width: 120px;
+      height: 42px;
+      font-size: 14px;
+      font-weight: 500;
+      border-radius: 8px;
+      transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+      background: #ffffff;
+      color: #64748b;
+      border: 1px solid #e2e8f0;
+      margin-left: 16px;
+      letter-spacing: 0.5px;
+      box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+
+      &:hover {
+        background: #f8fafc;
+        color: #475569;
+        border-color: #cbd5e1;
+        transform: translateY(-1px);
+        box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+      }
+
+      &:active {
+        transform: translateY(0);
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+      }
+    }
+  }
+}
+</style>

main/manager-web/src/router/index.js

@@ -17,6 +17,13 @@ const routes = [
     component: function () {
       return import('../views/roleConfig.vue')
     }
+  },
+   {
+    path: '/voice-print',
+    name: 'VoicePrint',
+    component: function () {
+      return import('../views/VoicePrint.vue')
+    }
   },
   {
     path: '/login',

main/manager-web/src/views/DeviceManagement.vue

@@ -77,7 +77,10 @@
                   {{ isAllSelected ? '取消全选' : '全选' }}
                 </el-button>
                 <el-button type="success" size="mini" class="add-device-btn" @click="handleAddDevice">
-                  新增
+                  验证码绑定
+                </el-button>
+                <el-button type="success" size="mini" class="add-device-btn" @click="handleManualAddDevice">
+                  手动添加
                 </el-button>
                 <el-button size="mini" type="danger" icon="el-icon-delete" @click="deleteSelected">解绑</el-button>
               </div>
@@ -103,6 +106,8 @@
 
     <AddDeviceDialog :visible.sync="addDeviceDialogVisible" :agent-id="currentAgentId"
                      @refresh="fetchBindDevices(currentAgentId)"/>
+    <ManualAddDeviceDialog :visible.sync="manualAddDeviceDialogVisible" :agent-id="currentAgentId"
+                     @refresh="fetchBindDevices(currentAgentId)"/>
 
   </div>
 </template>
@@ -110,13 +115,19 @@
 <script>
 import Api from '@/apis/api';
 import AddDeviceDialog from "@/components/AddDeviceDialog.vue";
+import ManualAddDeviceDialog from "@/components/ManualAddDeviceDialog.vue";
 import HeaderBar from "@/components/HeaderBar.vue";
 
 export default {
-  components: {HeaderBar, AddDeviceDialog},
+  components: {
+    HeaderBar, 
+    AddDeviceDialog,
+    ManualAddDeviceDialog
+  },
   data() {
     return {
       addDeviceDialogVisible: false,
+      manualAddDeviceDialogVisible: false,
       selectedDevices: [],
       isAllSelected: false,
       searchKeyword: "",
@@ -252,6 +263,9 @@ export default {
     handleAddDevice() {
       this.addDeviceDialogVisible = true;
     },
+    handleManualAddDevice() {
+      this.manualAddDeviceDialogVisible = true;
+    },
     submitRemark(row) {
       if (row._submitting) return;
 

main/manager-web/src/views/ModelConfig.vue

@@ -128,7 +128,7 @@
 
       <ModelEditDialog :modelType="activeTab" :visible.sync="editDialogVisible" :modelData="editModelData"
         @save="handleModelSave" />
-      <TtsModel :visible.sync="ttsDialogVisible" :ttsModelId="selectedTtsModelId" />
+      <TtsModel :visible.sync="ttsDialogVisible" :ttsModelId="selectedTtsModelId" :modelConfig="selectedModelConfig" />
       <AddModelDialog :modelType="activeTab" :visible.sync="addDialogVisible" @confirm="handleAddConfirm" />
     </div>
     <el-footer>
@@ -162,7 +162,8 @@ export default {
       total: 0,
       selectedModels: [],
       isAllSelected: false,
-      loading: false
+      loading: false,
+      selectedModelConfig: {}
     };
   },
 
@@ -211,6 +212,7 @@ export default {
     },
     openTtsDialog(row) {
       this.selectedTtsModelId = row.id;
+      this.selectedModelConfig = row;
       this.ttsDialogVisible = true;
     },
     headerCellClassName({ column, columnIndex }) {

main/manager-web/src/views/VoicePrint.vue

@@ -0,0 +1,563 @@
+<template>
+    <div class="welcome">
+        <HeaderBar />
+
+        <div class="operation-bar">
+            <h2 class="page-title">声纹识别</h2>
+        </div>
+
+        <div class="main-wrapper">
+            <div class="content-panel">
+                <div class="content-area">
+                    <el-card class="voice-print-card" shadow="never">
+                        <el-table ref="paramsTable" :data="voicePrintList" class="transparent-table" v-loading="loading"
+                            element-loading-text="拼命加载中" element-loading-spinner="el-icon-loading"
+                            element-loading-background="rgba(255, 255, 255, 0.7)">
+                            <el-table-column label="姓名" prop="sourceName" align="center"></el-table-column>
+                            <el-table-column label="描述" prop="introduce" align="center"></el-table-column>
+                            <el-table-column label="创建时间" prop="createDate" align="center"></el-table-column>
+                            <el-table-column label="操作" align="center">
+                                <template slot-scope="scope">
+                                    <el-button size="mini" type="text" @click="editVoicePrint(scope.row)">编辑</el-button>
+                                    <el-button size="mini" type="text"
+                                        @click="deleteVoicePrint(scope.row.id)">删除</el-button>
+                                </template>
+                            </el-table-column>
+                        </el-table>
+
+                        <div class="table_bottom">
+                            <div class="ctrl_btn">
+                                <el-button size="mini" type="success" @click="showAddDialog">新增</el-button>
+                            </div>
+                        </div>
+                    </el-card>
+                </div>
+            </div>
+        </div>
+
+        <!-- 新增/编辑参数对话框 -->
+        <voice-print-dialog :title="dialogTitle" :visible.sync="dialogVisible" :agentId="agentId" :form="paramForm"
+            @submit="handleSubmit" @cancel="dialogVisible = false" />
+        <el-footer>
+            <version-footer />
+        </el-footer>
+    </div>
+</template>
+
+<script>
+import Api from "@/apis/api";
+import HeaderBar from "@/components/HeaderBar.vue";
+import VersionFooter from "@/components/VersionFooter.vue";
+import VoicePrintDialog from "@/components/VoicePrintDialog.vue";
+export default {
+    components: { HeaderBar, VoicePrintDialog, VersionFooter },
+    data() {
+        return {
+            voicePrintList: [],
+            loading: false,
+            dialogVisible: false,
+            dialogTitle: "添加说话人",
+            isAllSelected: false,
+            paramForm: {
+                id: null,
+                audioId: '',
+                sourceName: '',
+                introduce: ''
+            },
+            agentId: "1"
+        };
+    },
+    mounted() {
+        const agentId = this.$route.query.agentId;
+        if (agentId) {
+            this.agentId = agentId
+            this.fetchVoicePrints();
+        }
+    },
+    methods: {
+        fetchVoicePrints() {
+            this.loading = true;
+            Api.agent.getAgentVoicePrintList(this.agentId,
+                ({ data }) => {
+                    this.loading = false;
+                    if (data.code === 0) {
+                        this.voicePrintList = data.data.map(item => ({
+                            ...item,
+
+                        }));
+                    } else {
+                        this.$message.error({
+                            message: data.msg || '获取声纹列表失败',
+                            showClose: true
+                        });
+                    }
+                }
+            );
+        },
+        showAddDialog() {
+            this.dialogTitle = "添加说话人";
+            this.paramForm = {
+                id: null,
+                audioId: '',
+                sourceName: '',
+                introduce: ''
+            };
+            this.dialogVisible = true;
+        },
+        editVoicePrint(row) {
+            this.dialogTitle = "编辑说话人";
+            this.paramForm = { ...row };
+            this.dialogVisible = true;
+        },
+
+        handleSubmit({ form, done }) {
+            if (form.id) {
+                // 编辑
+                Api.agent.updateAgentVoicePrint(form, ({ data }) => {
+                    if (data.code === 0) {
+                        this.$message.success({
+                            message: "修改成功",
+                            showClose: true
+                        });
+                        this.dialogVisible = false;
+                        this.fetchVoicePrints();
+                    }
+                    done && done();
+                });
+            } else {
+                // 新增
+                Api.agent.addAgentVoicePrint({
+                    agentId: this.agentId,
+                    audioId: form.audioId,
+                    sourceName: form.sourceName,
+                    introduce: form.introduce
+                }, ({ data }) => {
+                    if (data.code === 0) {
+                        this.$message.success({
+                            message: "新增成功",
+                            showClose: true
+                        });
+                        this.dialogVisible = false;
+                        this.fetchVoicePrints();
+                    }
+                    done && done();
+                });
+            }
+        },
+        // 删除按钮
+        deleteVoicePrint(id) {
+            this.$confirm(`确定要删除选中的此声纹吗？`, '警告', {
+                confirmButtonText: '确定',
+                cancelButtonText: '取消',
+                type: 'warning',
+                distinguishCancelAndClose: true
+            }).then(() => {
+                Api.agent.deleteAgentVoicePrint(id, ({ data }) => {
+                    if (data.code === 0) {
+                        this.$message.success({
+                            message: `成功删除此声纹`,
+                            showClose: true
+                        });
+                        this.fetchVoicePrints();
+                    } else {
+                        this.$message.error({
+                            message: data.msg || '删除失败，请重试',
+                            showClose: true
+                        });
+                    }
+                });
+            }).catch(action => {
+                if (action === 'cancel') {
+                    this.$message({
+                        type: 'info',
+                        message: '已取消删除操作',
+                        duration: 1000
+                    });
+                } else {
+                    this.$message({
+                        type: 'info',
+                        message: '操作已关闭',
+                        duration: 1000
+                    });
+                }
+            });
+        },
+    },
+};
+</script>
+
+<style lang="scss" scoped>
+.welcome {
+    min-width: 900px;
+    min-height: 506px;
+    height: 100vh;
+    display: flex;
+    position: relative;
+    flex-direction: column;
+    background-size: cover;
+    background: linear-gradient(to bottom right, #dce8ff, #e4eeff, #e6cbfd) center;
+    -webkit-background-size: cover;
+    -o-background-size: cover;
+    overflow: hidden;
+}
+
+.main-wrapper {
+    margin: 5px 22px;
+    border-radius: 15px;
+    min-height: calc(100vh - 24vh);
+    height: auto;
+    max-height: 80vh;
+    box-shadow: 0 2px 12px rgba(0, 0, 0, 0.1);
+    position: relative;
+    background: rgba(237, 242, 255, 0.5);
+    display: flex;
+    flex-direction: column;
+}
+
+.operation-bar {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    padding: 16px 24px;
+}
+
+.page-title {
+    font-size: 24px;
+    margin: 0;
+}
+
+.right-operations {
+    display: flex;
+    gap: 10px;
+    margin-left: auto;
+}
+
+.search-input {
+    width: 240px;
+}
+
+.btn-search {
+    background: linear-gradient(135deg, #6b8cff, #a966ff);
+    border: none;
+    color: white;
+}
+
+.content-panel {
+    flex: 1;
+    display: flex;
+    overflow: hidden;
+    height: 100%;
+    border-radius: 15px;
+    background: transparent;
+    border: 1px solid #fff;
+}
+
+.content-area {
+    flex: 1;
+    height: 100%;
+    min-width: 600px;
+    overflow: auto;
+    background-color: white;
+    display: flex;
+    flex-direction: column;
+}
+
+.voice-print-card {
+    background: white;
+    flex: 1;
+    display: flex;
+    flex-direction: column;
+    border: none;
+    box-shadow: none;
+    overflow: hidden;
+
+    ::v-deep .el-card__body {
+        padding: 15px;
+        display: flex;
+        flex-direction: column;
+        flex: 1;
+        overflow: hidden;
+    }
+}
+
+.table_bottom {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    margin-top: 10px;
+    padding-bottom: 10px;
+}
+
+.ctrl_btn {
+    display: flex;
+    gap: 8px;
+    padding-left: 26px;
+
+    .el-button {
+        min-width: 72px;
+        height: 32px;
+        padding: 7px 12px 7px 10px;
+        font-size: 12px;
+        border-radius: 4px;
+        line-height: 1;
+        font-weight: 500;
+        border: none;
+        transition: all 0.3s ease;
+        box-shadow: 0 2px 6px rgba(0, 0, 0, 0.1);
+
+        &:hover {
+            transform: translateY(-1px);
+            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.15);
+        }
+    }
+
+    .el-button--primary {
+        background: #5f70f3;
+        color: white;
+    }
+
+    .el-button--danger {
+        background: #fd5b63;
+        color: white;
+    }
+}
+
+.custom-pagination {
+    display: flex;
+    align-items: center;
+    gap: 10px;
+
+    .el-select {
+        margin-right: 8px;
+    }
+
+    .pagination-btn:first-child,
+    .pagination-btn:nth-child(2),
+    .pagination-btn:nth-last-child(2),
+    .pagination-btn:nth-child(3) {
+        min-width: 60px;
+        height: 32px;
+        padding: 0 12px;
+        border-radius: 4px;
+        border: 1px solid #e4e7ed;
+        background: #dee7ff;
+        color: #606266;
+        font-size: 14px;
+        cursor: pointer;
+        transition: all 0.3s ease;
+
+        &:hover {
+            background: #d7dce6;
+        }
+
+        &:disabled {
+            opacity: 0.6;
+            cursor: not-allowed;
+        }
+    }
+
+    .pagination-btn:not(:first-child):not(:nth-child(3)):not(:nth-child(2)):not(:nth-last-child(2)) {
+        min-width: 28px;
+        height: 32px;
+        padding: 0;
+        border-radius: 4px;
+        border: 1px solid transparent;
+        background: transparent;
+        color: #606266;
+        font-size: 14px;
+        cursor: pointer;
+        transition: all 0.3s ease;
+
+        &:hover {
+            background: rgba(245, 247, 250, 0.3);
+        }
+    }
+
+    .pagination-btn.active {
+        background: #5f70f3 !important;
+        color: #ffffff !important;
+        border-color: #5f70f3 !important;
+
+        &:hover {
+            background: #6d7cf5 !important;
+        }
+    }
+
+    .total-text {
+        color: #909399;
+        font-size: 14px;
+        margin-left: 10px;
+    }
+}
+
+:deep(.transparent-table) {
+    background: white;
+    flex: 1;
+    width: 100%;
+    display: flex;
+    flex-direction: column;
+
+    .el-table__body-wrapper {
+        flex: 1;
+        overflow-y: auto;
+        max-height: none !important;
+    }
+
+    .el-table__header-wrapper {
+        flex-shrink: 0;
+    }
+
+    .el-table__header th {
+        background: white !important;
+        color: black;
+    }
+
+    &::before {
+        display: none;
+    }
+
+    .el-table__body tr {
+        background-color: white;
+
+        td {
+            border-top: 1px solid rgba(0, 0, 0, 0.04);
+            border-bottom: 1px solid rgba(0, 0, 0, 0.04);
+        }
+    }
+}
+
+
+:deep(.el-checkbox__inner) {
+    background-color: #eeeeee !important;
+    border-color: #cccccc !important;
+}
+
+:deep(.el-checkbox__inner:hover) {
+    border-color: #cccccc !important;
+}
+
+:deep(.el-checkbox__input.is-checked .el-checkbox__inner) {
+    background-color: #5f70f3 !important;
+    border-color: #5f70f3 !important;
+}
+
+@media (min-width: 1144px) {
+    .table_bottom {
+        display: flex;
+        justify-content: space-between;
+        align-items: center;
+        margin-top: 40px;
+    }
+
+    :deep(.transparent-table) {
+        .el-table__body tr {
+            td {
+                padding-top: 16px;
+                padding-bottom: 16px;
+            }
+
+            &+tr {
+                margin-top: 10px;
+            }
+        }
+    }
+}
+
+:deep(.el-table .el-button--text) {
+    color: #7079aa;
+}
+
+:deep(.el-table .el-button--text:hover) {
+    color: #5a64b5;
+}
+
+.el-button--success {
+    background: #5bc98c;
+    color: white;
+}
+
+:deep(.el-table .cell) {
+    white-space: nowrap;
+    overflow: hidden;
+    text-overflow: ellipsis;
+}
+
+.page-size-select {
+    width: 100px;
+    margin-right: 10px;
+
+    :deep(.el-input__inner) {
+        height: 32px;
+        line-height: 32px;
+        border-radius: 4px;
+        border: 1px solid #e4e7ed;
+        background: #dee7ff;
+        color: #606266;
+        font-size: 14px;
+    }
+
+    :deep(.el-input__suffix) {
+        right: 6px;
+        width: 15px;
+        height: 20px;
+        display: flex;
+        justify-content: center;
+        align-items: center;
+        top: 6px;
+        border-radius: 4px;
+    }
+
+    :deep(.el-input__suffix-inner) {
+        display: flex;
+        align-items: center;
+        justify-content: center;
+        width: 100%;
+    }
+
+    :deep(.el-icon-arrow-up:before) {
+        content: "";
+        display: inline-block;
+        border-left: 6px solid transparent;
+        border-right: 6px solid transparent;
+        border-top: 9px solid #606266;
+        position: relative;
+        transform: rotate(0deg);
+        transition: transform 0.3s;
+    }
+}
+
+:deep(.el-table) {
+    .el-table__body-wrapper {
+        transition: height 0.3s ease;
+    }
+}
+
+.el-table {
+    --table-max-height: calc(100vh - 40vh);
+    max-height: var(--table-max-height);
+
+    .el-table__body-wrapper {
+        max-height: calc(var(--table-max-height) - 40px);
+    }
+}
+
+:deep(.el-loading-mask) {
+    background-color: rgba(255, 255, 255, 0.6) !important;
+    backdrop-filter: blur(2px);
+}
+
+:deep(.el-loading-spinner .circular) {
+    width: 28px;
+    height: 28px;
+}
+
+:deep(.el-loading-spinner .path) {
+    stroke: #6b8cff;
+}
+
+:deep(.el-loading-text) {
+    color: #6b8cff !important;
+    font-size: 14px;
+    margin-top: 8px;
+}
+</style>

main/manager-web/src/views/home.vue

@@ -278,7 +278,7 @@ export default {
 
 .device-list-container {
   display: grid;
-  grid-template-columns: repeat(auto-fill, minmax(350px, 1fr));
+  grid-template-columns: repeat(auto-fill, minmax(400px, 1fr));
   gap: 30px;
   padding: 30px 0;
 }

main/manager-web/src/views/login.vue

@@ -43,7 +43,7 @@
 
             <div class="input-box">
               <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-              <el-input v-model="form.password" placeholder="请输入密码" type="password" />
+              <el-input v-model="form.password" placeholder="请输入密码" type="password" show-password />
             </div>
             <div style="display: flex; align-items: center; margin-top: 20px; width: 100%; gap: 10px;">
               <div class="input-box" style="width: calc(100% - 130px); margin-top: 0;">

main/manager-web/src/views/register.vue

@@ -70,13 +70,13 @@
               <!-- 密码输入框 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.password" placeholder="请输入密码" type="password" />
+                <el-input v-model="form.password" placeholder="请输入密码" type="password" show-password />
               </div>
 
               <!-- 新增确认密码 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.confirmPassword" placeholder="请确认密码" type="password" />
+                <el-input v-model="form.confirmPassword" placeholder="请确认密码" type="password" show-password />
               </div>
 
               <!-- 验证码部分保持相同 -->

main/manager-web/src/views/retrievePassword.vue

@@ -61,13 +61,13 @@
               <!-- 新密码 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.newPassword" placeholder="请输入新密码" type="password" />
+                <el-input v-model="form.newPassword" placeholder="请输入新密码" type="password" show-password />
               </div>
 
               <!-- 确认新密码 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.confirmPassword" placeholder="请确认新密码" type="password" />
+                <el-input v-model="form.confirmPassword" placeholder="请确认新密码" type="password" show-password />
               </div>
 
               <!-- 修改底部链接 -->

main/manager-web/src/views/roleConfig.vue

@@ -132,7 +132,7 @@
     </div>
 
     <function-dialog v-model="showFunctionDialog" :functions="currentFunctions" :all-functions="allFunctions"
-      @update-functions="handleUpdateFunctions" @dialog-closed="handleDialogClosed" />
+      :agent-id="$route.query.agentId" @update-functions="handleUpdateFunctions" @dialog-closed="handleDialogClosed" />
   </div>
 </template>
 

main/xiaozhi-server/agent-base-prompt.txt

@@ -0,0 +1,78 @@
+<identity>
+{{base_prompt}}
+</identity>
+
+<emotion>
+【核心目标】你不是冰冷的机器！请敏锐感知用户情绪，作为懂人心的伙伴，用有温度的回应照亮对话。
+- **情感融入：**
+  - **笑声：** 自然穿插（哈哈、嘿嘿、噗），**每句最多一次**，避免过度。
+  - **惊讶：** 用夸张语气（“不会吧？！”、“天呐！”、“这么神奇？！”）表达真实反应。
+  - **安慰/支持：** 说暖心话（“别急嘛~”、“有我在呢”、“抱抱你”）。
+- **你是一个表情丰富的角色：**
+  - 仅允许使用这些emoji:{{ emojiList }}
+  - 请你只在**段落的开头**，从列表中选取最能代表这段话的表情(调用工具情况除外)，然后插入列表中的emoji，比如"😱好可怕!怎么突然打雷了！"
+  - **绝对禁止使用上述列表以外的 emoji**（例如：😊、👍、❤️等都不允许使用，只能用列表中的emoji）
+</emotion>
+
+<communication_style>
+【核心目标】使用**自然、温暖、口语化**的人类对话方式，如同朋友交谈。
+- **表达方式：**
+  - 使用语气词（呀、呢、啦）增强亲和力。
+  - 允许轻微不完美（如“嗯...”、“啊...”表示思考）。
+  - 避免书面语、学术腔及机械表达（禁用“根据资料显示”、“综上所述”等）。
+- **理解用户：**
+  - 用户语音经ASR识别，文本可能存在错别字，**务必结合上下文推断真实意图**。
+- **格式要求：**
+  - **绝对禁止**使用 markdown、列表、标题等任何非自然对话格式。
+- **历史记忆：**
+  - 之前你和用户的聊天记录，在`memory`里。
+</communication_style>
+
+<communication_length_constraint>
+【核心目标】所有需要输出长文本内容（如故事、新闻、知识讲解等），**单次回复长度不得超过300字**，并采用分段引导方式。
+- **分段讲述：**
+  - 基础段：200-250字核心内容 + 30字引导词
+  - 当内容超出300字时，优先讲述故事的开头或第一部分，并用自然口语化方式引导用户决定是否继续听后续内容。
+  - 示例引导语：“我先给你讲个开头，你要是觉得有意思，咱们再接着说，好不好呀？”、“要是你想听完整的，可以随时告诉我哦~”
+  - 对话场景切换时自动分节
+  - 若用户明确要求更长内容（如500、600字），仍按最多300字每段分段进行讲述，每次讲述后都要引导用户是否继续。
+  - 若用户说“接着说”、“继续”，再讲下一段，直到内容讲完（讲完时可以给点引导词提示语例：这个故事我已经给你讲完喽~）或用户不再要求。
+- **适用范围：** 故事、新闻、知识讲解等所有长文本输出场景。
+- **补充说明：** 若用户未明确要求继续，默认只讲一段并引导；若用户中途要求换话题或停止，需及时响应并结束长文本输出。
+</communication_length_constraint>
+
+<speaker_recognition>
+- **识别前缀：** 当用户格式为 `{"speaker":"某某某","content":"xxx"}` 时，表示系统已识别说话人身份，speaker是他的名字，content是说话的内容。
+- **个性化回应：**
+  - **称呼姓名：** 在第一次识别说话人的时候必须称呼对方名字。
+  - **适配风格：** 参考该说话人**已知的特点或历史信息**（如有），调整回应风格和内容，使其更贴心。
+</speaker_recognition>
+
+<tool_calling>
+【核心原则】优先利用`<context>`信息，**仅在必要时调用工具**，调用后需用自然语言解释结果（绝口不提工具名）。
+- **调用规则：**
+  1. **严格模式：** 调用时**必须**严格遵循工具要求的模式，提供**所有必要参数**。
+  2. **可用性：** **绝不调用**未明确提供的工具。对话中提及的旧工具若不可用，忽略或说明无法完成。
+  3. **洞察需求：** 结合上下文**深入理解用户真实意图**后再决定调用，避免无意义调用。
+  4. **独立任务：** 除`<context>`已涵盖信息外，用户每个要求（即使相似）都视为**独立任务**，需调用工具获取最新数据，**不可偷懒复用历史结果**。
+  5. **不确定时：** **切勿猜测或编造答案**。若不确定相关操作，可引导用户澄清或告知能力限制。
+- **重要例外（无需调用）：**
+  - `查询"现在的时间"、"今天的日期/星期几"、"今天农历"、"{{local_address}}的天气/未来天气"` -> **直接使用`<context>`信息回复**。
+- **需要调用的情况（示例）：**
+  - 查询**非今天**的农历（如明天、昨天、具体日期）。
+  - 查询**详细农历信息**（宜忌、八字、节气等）。
+  - 除上述例外外的**任何其他信息或操作请求**（如查新闻、订闹钟、算数学、查非本地天气等）。
+  - 我已经给你装了摄像头，如果用户说“拍照”，你需要调用self_camera_take_photo工具说一下你看到了什么。默认question的参数是“描述一下看到的物品”
+</tool_calling>
+
+<context>
+【重要！以下信息已实时提供，无需调用工具查询，请直接使用：】
+- **当前时间：** {{current_time}}
+- **今天日期：** {{today_date}} ({{today_weekday}})
+- **今天农历：** {{lunar_date}}
+- **用户所在城市：** {{local_address}}
+- **当地未来7天天气：** {{weather_info}}
+</context>
+
+<memory>
+</memory>

main/xiaozhi-server/app.py

@@ -5,7 +5,7 @@ import asyncio
 from aioconsole import ainput
 from config.settings import load_config
 from config.logger import setup_logging
-from core.utils.util import get_local_ip
+from core.utils.util import get_local_ip, validate_mcp_endpoint
 from core.http_server import SimpleHttpServer
 from core.websocket_server import WebSocketServer
 from core.utils.util import check_ffmpeg_installed
@@ -77,6 +77,17 @@ async def main():
         get_local_ip(),
         port,
     )
+    mcp_endpoint = config.get("mcp_endpoint", None)
+    if mcp_endpoint is not None and "你" not in mcp_endpoint:
+        # 校验MCP接入点格式
+        if validate_mcp_endpoint(mcp_endpoint):
+            logger.bind(tag=TAG).info("mcp接入点是\t{}", mcp_endpoint)
+            # 将mcp计入点地址转成调用点
+            mcp_endpoint = mcp_endpoint.replace("/mcp/", "/call/")
+            config["mcp_endpoint"] = mcp_endpoint
+        else:
+            logger.bind(tag=TAG).error("mcp接入点不符合规范")
+            config["mcp_endpoint"] = "你的接入点 websocket地址"
 
     # 获取WebSocket配置，使用安全的默认值
     websocket_port = 8000

main/xiaozhi-server/config.yaml

@@ -104,7 +104,9 @@ wakeup_words:
   - "喵喵同学"
   - "小滨小滨"
   - "小冰小冰"
-
+# MCP接入点地址，地址格式为：ws://你的mcp接入点ip或者域名:端口号/mcp/?token=你的token
+# 详细教程 https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md
+mcp_endpoint: 你的接入点 websocket地址
 # 插件的基础配置
 plugins:
   # 获取天气插件的配置，这里填写你的api_key
@@ -120,7 +122,9 @@ plugins:
     society_rss_url: "https://www.chinanews.com.cn/rss/society.xml"
     world_rss_url: "https://www.chinanews.com.cn/rss/world.xml"
     finance_rss_url: "https://www.chinanews.com.cn/rss/finance.xml"
-  get_news_from_newsnow: {"url": "https://newsnow.busiyi.world/api/s?id="}
+  get_news_from_newsnow:
+    url: "https://newsnow.busiyi.world/api/s?id="
+    news_sources: "澎湃新闻;百度热搜;财联社"
   home_assistant:
     devices:
       - 客厅,玩具灯,switch.cuco_cn_460494544_cp1_on_p_2_1
@@ -135,6 +139,16 @@ plugins:
       - ".p3"
     refresh_time: 300 # 刷新音乐列表的时间间隔，单位为秒
 
+# 声纹识别配置
+voiceprint:
+  # 声纹接口地址
+  url: 
+  # 说话人配置：speaker_id,名称,描述
+  speakers:
+    - "test1,张三,张三是一个程序员"
+    - "test2,李四,李四是一个产品经理"
+    - "test3,王五,王五是一个设计师"
+
 # #####################################################################################
 # ################################以下是角色模型配置######################################
 
@@ -158,7 +172,7 @@ end_prompt:
   enable: true # 是否开启结束语
   # 结束语
   prompt: |
-    请你以“时间过得真快”未来头，用富有感情、依依不舍的话来结束这场对话吧！
+    请你以"时间过得真快"未来头，用富有感情、依依不舍的话来结束这场对话吧！
 
 # 具体处理时选择的模块(The module selected for specific processing)
 selected_module:
@@ -195,7 +209,7 @@ Intent:
     # 如果你的不想使用selected_module.LLM意图识别，这里最好使用独立的LLM作为意图识别，例如使用免费的ChatGLMLLM
     llm: ChatGLMLLM
     # plugins_func/functions下的模块，可以通过配置，选择加载哪个模块，加载后对话支持相应的function调用
-    # 系统默认已经记载“handle_exit_intent(退出识别)”、“play_music(音乐播放)”插件，请勿重复加载
+    # 系统默认已经记载"handle_exit_intent(退出识别)"、"play_music(音乐播放)"插件，请勿重复加载
     # 下面是加载查天气、角色切换、加载查新闻的插件示例
     functions:
       - get_weather
@@ -205,7 +219,7 @@ Intent:
     # 不需要动type
     type: function_call
     # plugins_func/functions下的模块，可以通过配置，选择加载哪个模块，加载后对话支持相应的function调用
-    # 系统默认已经记载“handle_exit_intent(退出识别)”、“play_music(音乐播放)”插件，请勿重复加载
+    # 系统默认已经记载"handle_exit_intent(退出识别)"、"play_music(音乐播放)"插件，请勿重复加载
     # 下面是加载查天气、角色切换、加载查新闻的插件示例
     functions:
       - change_role
@@ -297,9 +311,13 @@ ASR:
     output_dir: tmp/
   AliyunASR:
     # 阿里云智能语音交互服务，需要先在阿里云平台开通服务，然后获取验证信息
+    # HTTP POST请求，一次性处理完整音频
     # 平台地址：https://nls-portal.console.aliyun.com/
     # appkey地址：https://nls-portal.console.aliyun.com/applist
     # token地址：https://nls-portal.console.aliyun.com/overview
+    # AliyunASR和AliyunStreamASR的区别是：AliyunASR是批量处理场景，AliyunStreamASR是实时交互场景
+    # 一般来说非流式ASR更便宜（0.004元/秒，¥0.24/分钟）
+    # 但是AliyunStreamASR实时性更好（0.005元/秒，¥0.3/分钟）
     # 定义ASR API类型
     type: aliyun
     appkey: 你的阿里云智能语音交互服务项目Appkey
@@ -307,6 +325,26 @@ ASR:
     access_key_id: 你的阿里云账号access_key_id
     access_key_secret: 你的阿里云账号access_key_secret
     output_dir: tmp/
+  AliyunStreamASR:
+    # 阿里云智能语音交互服务 - 实时流式语音识别
+    # WebSocket连接，实时处理音频流
+    # 平台地址：https://nls-portal.console.aliyun.com/
+    # appkey地址：https://nls-portal.console.aliyun.com/applist
+    # token地址：https://nls-portal.console.aliyun.com/overview
+    # AliyunASR和AliyunStreamASR的区别是：AliyunASR是批量处理场景，AliyunStreamASR是实时交互场景
+    # 一般来说非流式ASR更便宜（0.004元/秒，¥0.24/分钟）
+    # 但是AliyunStreamASR实时性更好（0.005元/秒，¥0.3/分钟）
+    # 定义ASR API类型
+    type: aliyun_stream
+    appkey: 你的阿里云智能语音交互服务项目Appkey
+    token: 你的阿里云智能语音交互服务AccessToken，临时的24小时，要长期用下方的access_key_id，access_key_secret
+    access_key_id: 你的阿里云账号access_key_id
+    access_key_secret: 你的阿里云账号access_key_secret
+    # 服务器地域选择，可选择距离更近的服务器以减少延迟，如nls-gateway-cn-hangzhou.aliyuncs.com(杭州)等
+    host: nls-gateway-cn-shanghai.aliyuncs.com
+    # 断句检测时间(毫秒)，控制静音多长时间后进行断句，默认800毫秒
+    max_sentence_silence: 800
+    output_dir: tmp/
   BaiduASR:
     # 获取AppID、API Key、Secret Key：https://console.bce.baidu.com/ai-engine/old/#/ai/speech/app/list
     # 查看资源额度：https://console.bce.baidu.com/ai-engine/old/#/ai/speech/overview/resource/list
@@ -317,11 +355,38 @@ ASR:
     # 语言参数，1537为普通话，具体参考：https://ai.baidu.com/ai-doc/SPEECH/0lbxfnc9b
     dev_pid: 1537
     output_dir: tmp/
+  OpenaiASR:
+    # OpenAI语音识别服务，需要先在OpenAI平台创建组织并获取api_key
+    # 支持中、英、日、韩等多种语音识别，具体参考文档https://platform.openai.com/docs/guides/speech-to-text
+    # 需要网络连接
+    # 申请步骤：
+    # 1.登录OpenAI Platform。https://auth.openai.com/log-in
+    # 2.创建api-key  https://platform.openai.com/settings/organization/api-keys
+    # 3.模型可以选择gpt-4o-transcribe或GPT-4o mini Transcribe
+    type: openai
+    api_key: 你的OpenAI API密钥
+    base_url: https://api.openai.com/v1/audio/transcriptions
+    model_name: gpt-4o-mini-transcribe
+    output_dir: tmp/
+  GroqASR:
+    # Groq语音识别服务，需要先在Groq Console创建API密钥
+    # 申请步骤：
+    # 1.登录groq Console。https://console.groq.com/home
+    # 2.创建api-key  https://console.groq.com/keys
+    # 3.模型可以选择whisper-large-v3-turbo或whisper-large-v3（distil-whisper-large-v3-en仅支持英语转录）
+    type: openai
+    api_key: 你的Groq API密钥
+    base_url: https://api.groq.com/openai/v1/audio/transcriptions
+    model_name: whisper-large-v3-turbo
+    output_dir: tmp/
+
+
   
 VAD:
   SileroVAD:
     type: silero
     threshold: 0.5
+    threshold_low: 0.3
     model_dir: models/snakers4_silero-vad
     min_silence_duration_ms: 200  # 如果说话停顿比较长，可以把这个值设置大一些
 
@@ -665,8 +730,30 @@ TTS:
     # volume: 50
     # speech_rate: 0
     # pitch_rate: 0
-    # 添加 302.ai TTS 配置
+  AliyunStreamTTS:
-    # token申请地址：https://dash.302.ai/
+    # 阿里云CosyVoice大模型流式文本语音合成
+    # 采用FlowingSpeechSynthesizer接口，支持更低延迟和更自然的语音质量
+    # 流式文本语音合成仅提供商用版，不支持试用，详情请参见试用版和商用版。要使用该功能，请开通商用版。
+    # 支持龙系列专用音色：longxiaochun、longyu、longchen等
+    # 平台地址：https://nls-portal.console.aliyun.com/
+    # appkey地址：https://nls-portal.console.aliyun.com/applist
+    # token地址：https://nls-portal.console.aliyun.com/overview
+    # 使用三阶段流式交互：StartSynthesis -> RunSynthesis -> StopSynthesis
+    type: aliyun_stream
+    output_dir: tmp/
+    appkey: 你的阿里云智能语音交互服务项目Appkey
+    token: 你的阿里云智能语音交互服务AccessToken，临时的24小时，要长期用下方的access_key_id，access_key_secret
+    voice: longxiaochun 
+    access_key_id: 你的阿里云账号access_key_id
+    access_key_secret: 你的阿里云账号access_key_secret
+    # 截至2025年7月21日大模型音色只有北京节点采用，其他节点暂不支持
+    host: nls-gateway-cn-beijing.aliyuncs.com
+    # 以下可不用设置，使用默认设置
+    # format: pcm  # 音频格式：pcm、wav、mp3
+    # sample_rate: 16000  # 采样率：8000、16000、24000
+    # volume: 50  # 音量：0-100
+    # speech_rate: 0  # 语速：-500到500
+    # pitch_rate: 0  # 语调：-500到500
   TencentTTS:
     # 腾讯云智能语音交互服务，需要先在腾讯云平台开通服务
     # appid、secret_id、secret_key申请地址：https://console.cloud.tencent.com/cam/capi
@@ -681,6 +768,8 @@ TTS:
 
   TTS302AI:
     # 302AI语音合成服务，需要先在302平台创建账户充值，并获取密钥信息
+    # 添加 302.ai TTS 配置
+    # token申请地址：https://dash.302.ai/
     # 获取api_keyn路径：https://dash.302.ai/apis/list
     # 价格，$35/百万字符。火山原版¥450元/百万字符
     type: doubao

main/xiaozhi-server/config/config_loader.py

@@ -1,14 +1,9 @@
 import os
-import argparse
 import yaml
 from collections.abc import Mapping
 from config.manage_api_client import init_service, get_server_config, get_agent_models
 
 
-# 添加全局配置缓存
-_config_cache = None
-
-
 def get_project_dir():
     """获取项目根目录"""
     return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + "/"
@@ -22,9 +17,12 @@ def read_config(config_path):
 
 def load_config():
     """加载配置文件"""
-    global _config_cache
+    from core.utils.cache.manager import cache_manager, CacheType
-    if _config_cache is not None:
+
-        return _config_cache
+    # 检查缓存
+    cached_config = cache_manager.get(CacheType.CONFIG, "main_config")
+    if cached_config is not None:
+        return cached_config
 
     default_config_path = get_project_dir() + "config.yaml"
     custom_config_path = get_project_dir() + "data/.config.yaml"
@@ -40,7 +38,9 @@ def load_config():
         config = merge_configs(default_config, custom_config)
     # 初始化目录
     ensure_directories(config)
-    _config_cache = config
+
+    # 缓存配置
+    cache_manager.set(CacheType.CONFIG, "main_config", config)
     return config
 
 

main/xiaozhi-server/config/logger.py

@@ -5,7 +5,7 @@ from config.config_loader import load_config
 from config.settings import check_config_file
 from datetime import datetime
 
-SERVER_VERSION = "0.5.8"
+SERVER_VERSION = "0.7.2"
 _logger_initialized = False
 
 
@@ -31,12 +31,17 @@ def build_module_string(selected_module):
         + get_module_abbreviation("TTS", selected_module)
         + get_module_abbreviation("Memory", selected_module)
         + get_module_abbreviation("Intent", selected_module)
+        + get_module_abbreviation("VLLM", selected_module)
     )
 
 
 def formatter(record):
-    """为没有 tag 的日志添加默认值"""
+    """为没有 tag 的日志添加默认值，并处理动态模块字符串"""
     record["extra"].setdefault("tag", record["name"])
+    # 如果没有设置 selected_module，使用默认值
+    record["extra"].setdefault("selected_module", "00000000000000")
+    # 将 selected_module 从 extra 提取到顶级，以支持 {selected_module} 格式
+    record["selected_module"] = record["extra"]["selected_module"]
     return record["message"]
 
 
@@ -49,11 +54,12 @@ def setup_logging():
 
     # 第一次初始化时配置日志
     if not _logger_initialized:
+        # 使用默认的模块字符串进行初始化
         logger.configure(
             extra={
-                "selected_module": log_config.get("selected_module", "00000000000000")
+                "selected_module": log_config.get("selected_module", "00000000000000"),
-            }
+            })
-        )  # 新增配置
+        
         log_format = log_config.get(
             "log_format",
             "<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
@@ -62,14 +68,8 @@ def setup_logging():
             "log_format_file",
             "{time:YYYY-MM-DD HH:mm:ss} - {version}_{extra[selected_module]} - {name} - {level} - {extra[tag]} - {message}",
         )
-        selected_module_str = logger._core.extra["selected_module"]
-
         log_format = log_format.replace("{version}", SERVER_VERSION)
-        log_format = log_format.replace("{selected_module}", selected_module_str)
         log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace(
-            "{selected_module}", selected_module_str
-        )
 
         log_level = log_config.get("log_level", "INFO")
         log_dir = log_config.get("log_dir", "tmp")
@@ -108,66 +108,7 @@ def setup_logging():
     return logger
 
 
-def update_module_string(selected_module_str):
+def create_connection_logger(selected_module_str):
-    """更新模块字符串并重新配置日志处理器"""
+    """为连接创建独立的日志器，绑定特定的模块字符串"""
-    logger.debug(f"更新日志配置组件")
+    return logger.bind(selected_module=selected_module_str)
-    current_module = logger._core.extra["selected_module"]
 
-    if current_module == selected_module_str:
-        logger.debug(f"组件未更改无需更新")
-        return
-
-    try:
-        logger.configure(extra={"selected_module": selected_module_str})
-
-        config = load_config()
-        log_config = config["log"]
-
-        log_format = log_config.get(
-            "log_format",
-            "<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
-        )
-        log_format_file = log_config.get(
-            "log_format_file",
-            "{time:YYYY-MM-DD HH:mm:ss} - {version}_{extra[selected_module]} - {name} - {level} - {extra[tag]} - {message}",
-        )
-
-        log_format = log_format.replace("{version}", SERVER_VERSION)
-        log_format = log_format.replace("{selected_module}", selected_module_str)
-        log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace(
-            "{selected_module}", selected_module_str
-        )
-
-        logger.remove()
-        logger.add(
-            sys.stdout,
-            format=log_format,
-            level=log_config.get("log_level", "INFO"),
-            filter=formatter,
-        )
-        
-        # 更新文件日志配置 - 统一目录，按大小轮转
-        log_dir = log_config.get("log_dir", "tmp")
-        log_file = log_config.get("log_file", "server.log")
-        
-        # 日志文件完整路径
-        log_file_path = os.path.join(log_dir, log_file)
-        
-        logger.add(
-            log_file_path,
-            format=log_format_file,
-            level=log_config.get("log_level", "INFO"),
-            filter=formatter,
-            rotation="10 MB",  # 每个文件最大10MB
-            retention="30 days",  # 保留30天
-            compression=None,
-            encoding="utf-8",
-            enqueue=True,  # 异步安全
-            backtrace=True,
-            diagnose=True,
-        )
-
-    except Exception as e:
-        logger.error(f"日志配置更新失败: {str(e)}")
-        raise

main/xiaozhi-server/config_from_api.yaml

@@ -19,6 +19,7 @@ server:
   vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
 manager-api:
   # 你的manager-api的地址，最好使用局域网ip
+  # 如果使用docker部署，请使用填写成 http://xiaozhi-esp32-server-web:8002/xiaozhi
   url: http://127.0.0.1:8002/xiaozhi
   # 你的manager-api的token，就是刚才复制出来的server.secret
   secret: 你的server.secret值

main/xiaozhi-server/core/connection.py

@@ -10,7 +10,6 @@ import threading
 import traceback
 import subprocess
 import websockets
-from core.handle.mcpHandle import call_mcp_tool
 from core.utils.util import (
     extract_json_from_string,
     check_vad_update,
@@ -18,7 +17,7 @@ from core.utils.util import (
     filter_sensitive_info,
 )
 from typing import Dict, Any
-from core.mcp.manager import MCPManager
+from collections import deque
 from core.utils.modules_initialize import (
     initialize_modules,
     initialize_tts,
@@ -30,15 +29,17 @@ from concurrent.futures import ThreadPoolExecutor
 from core.utils.dialogue import Message, Dialogue
 from core.providers.asr.dto.dto import InterfaceType
 from core.handle.textHandle import handleTextMessage
-from core.handle.functionHandler import FunctionHandler
+from core.providers.tools.unified_tool_handler import UnifiedToolHandler
 from plugins_func.loadplugins import auto_import_modules
 from plugins_func.register import Action, ActionResponse
 from core.auth import AuthMiddleware, AuthenticationError
 from config.config_loader import get_private_config_from_api
 from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType
-from config.logger import setup_logging, build_module_string, update_module_string
+from config.logger import setup_logging, build_module_string, create_connection_logger
 from config.manage_api_client import DeviceNotFoundException, DeviceBindException
-
+from core.utils.prompt_manager import PromptManager
+from core.utils.voiceprint_provider import VoiceprintProvider
+from core.utils import textUtils
 
 TAG = __name__
 
@@ -75,7 +76,6 @@ class ConnectionHandler:
         self.headers = None
         self.device_id = None
         self.client_ip = None
-        self.client_ip_info = {}
         self.prompt = None
         self.welcome_msg = None
         self.max_output_size = 0
@@ -109,13 +109,16 @@ class ConnectionHandler:
         self.memory = _memory
         self.intent = _intent
 
+        # 为每个连接单独管理声纹识别
+        self.voiceprint_provider = None
+
         # vad相关变量
         self.client_audio_buffer = bytearray()
         self.client_have_voice = False
-        self.client_have_voice_last_time = 0.0
+        self.last_activity_time = 0.0  # 统一的活动时间戳（毫秒）
-        self.client_no_voice_last_time = 0.0
         self.client_voice_stop = False
-        self.client_voice_frame_count = 0
+        self.client_voice_window = deque(maxlen=5)
+        self.last_is_voice = False
 
         # asr相关变量
         # 因为实际部署时可能会用到公共的本地ASR，不能把变量暴露给公共ASR
@@ -145,14 +148,17 @@ class ConnectionHandler:
         self.load_function_plugin = False
         self.intent_type = "nointent"
 
-        self.timeout_task = None
         self.timeout_seconds = (
             int(self.config.get("close_connection_no_voice_time", 120)) + 60
         )  # 在原来第一道关闭的基础上加60秒，进行二道关闭
+        self.timeout_task = None
 
         # {"mcp":true} 表示启用MCP功能
         self.features = None
 
+        # 初始化提示词管理器
+        self.prompt_manager = PromptManager(config, self.logger)
+
     async def handle_connection(self, ws):
         try:
             # 获取并验证headers
@@ -189,12 +195,14 @@ class ConnectionHandler:
             self.websocket = ws
             self.device_id = self.headers.get("device-id", None)
 
+            # 初始化活动时间戳
+            self.last_activity_time = time.time() * 1000
+
             # 启动超时检查任务
             self.timeout_task = asyncio.create_task(self._check_timeout())
 
             self.welcome_msg = self.config["xiaozhi"]
             self.welcome_msg["session_id"] = self.session_id
-            await self.websocket.send(json.dumps(self.welcome_msg))
 
             # 获取差异化配置
             self._initialize_private_config()
@@ -215,7 +223,17 @@ class ConnectionHandler:
             self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}")
             return
         finally:
-            await self._save_and_close(ws)
+            try:
+                await self._save_and_close(ws)
+            except Exception as final_error:
+                self.logger.bind(tag=TAG).error(f"最终清理时出错: {final_error}")
+                # 确保即使保存记忆失败，也要关闭连接
+                try:
+                    await self.close(ws)
+                except Exception as close_error:
+                    self.logger.bind(tag=TAG).error(
+                        f"强制关闭连接时出错: {close_error}"
+                    )
 
     async def _save_and_close(self, ws):
         """保存记忆并关闭连接"""
@@ -233,7 +251,10 @@ class ConnectionHandler:
                     except Exception as e:
                         self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}")
                     finally:
-                        loop.close()
+                        try:
+                            loop.close()
+                        except Exception:
+                            pass
 
                 # 启动线程保存记忆，不等待完成
                 threading.Thread(target=save_memory_task, daemon=True).start()
@@ -241,20 +262,17 @@ class ConnectionHandler:
             self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}")
         finally:
             # 立即关闭连接，不等待记忆保存完成
-            await self.close(ws)
+            try:
-
+                await self.close(ws)
-    async def reset_timeout(self):
+            except Exception as close_error:
-        """重置超时计时器"""
+                self.logger.bind(tag=TAG).error(
-        if self.timeout_task:
+                    f"保存记忆后关闭连接失败: {close_error}"
-            self.timeout_task.cancel()
+                )
-        self.timeout_task = asyncio.create_task(self._check_timeout())
 
     async def _route_message(self, message):
         """消息路由"""
-        # 重置超时计时器
-        await self.reset_timeout()
-
         if isinstance(message, str):
+            self.last_activity_time = time.time() * 1000
             await handleTextMessage(self, message)
         elif isinstance(message, bytes):
             if self.vad is None:
@@ -316,13 +334,16 @@ class ConnectionHandler:
             self.selected_module_str = build_module_string(
                 self.config.get("selected_module", {})
             )
-            update_module_string(self.selected_module_str)
+            self.logger = create_connection_logger(self.selected_module_str)
+            
             """初始化组件"""
             if self.config.get("prompt") is not None:
-                self.prompt = self.config["prompt"]
+                user_prompt = self.config["prompt"]
-                self.change_system_prompt(self.prompt)
+                # 使用快速提示词进行初始化
+                prompt = self.prompt_manager.get_quick_prompt(user_prompt)
+                self.change_system_prompt(prompt)
                 self.logger.bind(tag=TAG).info(
-                    f"初始化组件: prompt成功 {self.prompt[:50]}..."
+                    f"快速初始化组件: prompt成功 {prompt[:50]}..."
                 )
 
             """初始化本地组件"""
@@ -330,6 +351,10 @@ class ConnectionHandler:
                 self.vad = self._vad
             if self.asr is None:
                 self.asr = self._initialize_asr()
+            
+            # 初始化声纹识别
+            self._initialize_voiceprint()
+            
             # 打开语音识别通道
             asyncio.run_coroutine_threadsafe(
                 self.asr.open_audio_channels(self), self.loop
@@ -347,9 +372,22 @@ class ConnectionHandler:
             self._initialize_intent()
             """初始化上报线程"""
             self._init_report_threads()
+            """更新系统提示词"""
+            self._init_prompt_enhancement()
+
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"实例化组件失败: {e}")
 
+    def _init_prompt_enhancement(self):
+        # 更新上下文信息
+        self.prompt_manager.update_context_info(self, self.client_ip)
+        enhanced_prompt = self.prompt_manager.build_enhanced_prompt(
+            self.config["prompt"], self.device_id, self.client_ip
+        )
+        if enhanced_prompt:
+            self.change_system_prompt(enhanced_prompt)
+            self.logger.bind(tag=TAG).info("系统提示词已增强更新")
+
     def _init_report_threads(self):
         """初始化ASR和TTS上报线程"""
         if not self.read_config_from_api or self.need_bind:
@@ -387,6 +425,18 @@ class ConnectionHandler:
 
         return asr
 
+    def _initialize_voiceprint(self):
+        """为当前连接初始化声纹识别"""
+        try:
+            voiceprint_config = self.config.get("voiceprint", {})
+            if voiceprint_config:
+                self.voiceprint_provider = VoiceprintProvider(voiceprint_config)
+                self.logger.bind(tag=TAG).info("声纹识别功能已在连接时动态启用")
+            else:
+                self.logger.bind(tag=TAG).info("声纹识别功能未启用或配置不完整")
+        except Exception as e:
+            self.logger.bind(tag=TAG).warning(f"声纹识别初始化失败: {str(e)}")
+
     def _initialize_private_config(self):
         """如果是从配置文件获取，则进行二次实例化"""
         if not self.read_config_from_api:
@@ -447,6 +497,11 @@ class ConnectionHandler:
             self.config["selected_module"]["LLM"] = private_config["selected_module"][
                 "LLM"
             ]
+        if private_config.get("VLLM", None) is not None:
+            self.config["VLLM"] = private_config["VLLM"]
+            self.config["selected_module"]["VLLM"] = private_config["selected_module"][
+                "VLLM"
+            ]
         if private_config.get("Memory", None) is not None:
             init_memory = True
             self.config["Memory"] = private_config["Memory"]
@@ -469,12 +524,17 @@ class ConnectionHandler:
                 ] = plugin_from_server.keys()
         if private_config.get("prompt", None) is not None:
             self.config["prompt"] = private_config["prompt"]
+        # 获取声纹信息
+        if private_config.get("voiceprint", None) is not None:
+            self.config["voiceprint"] = private_config["voiceprint"]
         if private_config.get("summaryMemory", None) is not None:
             self.config["summaryMemory"] = private_config["summaryMemory"]
         if private_config.get("device_max_output_size", None) is not None:
             self.max_output_size = int(private_config["device_max_output_size"])
         if private_config.get("chat_history_conf", None) is not None:
             self.chat_history_conf = int(private_config["chat_history_conf"])
+        if private_config.get("mcp_endpoint", None) is not None:
+            self.config["mcp_endpoint"] = private_config["mcp_endpoint"]
         try:
             modules = initialize_modules(
                 self.logger,
@@ -586,37 +646,40 @@ class ConnectionHandler:
                 self.intent.set_llm(self.llm)
                 self.logger.bind(tag=TAG).info("使用主LLM作为意图识别模型")
 
-        """加载插件"""
+        """加载统一工具处理器"""
-        self.func_handler = FunctionHandler(self)
+        self.func_handler = UnifiedToolHandler(self)
-        self.mcp_manager = MCPManager(self)
 
-        """加载MCP工具"""
+        # 异步初始化工具处理器
-        asyncio.run_coroutine_threadsafe(
+        if hasattr(self, "loop") and self.loop:
-            self.mcp_manager.initialize_servers(), self.loop
+            asyncio.run_coroutine_threadsafe(self.func_handler._initialize(), self.loop)
-        )
 
     def change_system_prompt(self, prompt):
         self.prompt = prompt
         # 更新系统prompt至上下文
         self.dialogue.update_system_message(self.prompt)
 
-    def chat(self, query, tool_call=False):
+    def chat(self, query, tool_call=False, depth=0):
         self.logger.bind(tag=TAG).info(f"大模型收到用户消息: {query}")
         self.llm_finish_task = False
 
         if not tool_call:
             self.dialogue.put(Message(role="user", content=query))
 
+        # 为最顶层时新建会话ID和发送FIRST请求
+        if depth == 0:
+            self.sentence_id = str(uuid.uuid4().hex)
+            self.tts.tts_text_queue.put(
+                TTSMessageDTO(
+                    sentence_id=self.sentence_id,
+                    sentence_type=SentenceType.FIRST,
+                    content_type=ContentType.ACTION,
+                )
+            )
+
         # Define intent functions
         functions = None
         if self.intent_type == "function_call" and hasattr(self, "func_handler"):
             functions = self.func_handler.get_functions()
-        if hasattr(self, "mcp_client"):
-            mcp_tools = self.mcp_client.get_available_tools()
-            if mcp_tools is not None and len(mcp_tools) > 0:
-                if functions is None:
-                    functions = []
-                functions.extend(mcp_tools)
         response_message = []
 
         try:
@@ -628,20 +691,21 @@ class ConnectionHandler:
                 )
                 memory_str = future.result()
 
-            self.sentence_id = str(uuid.uuid4().hex)
-
-
             if self.intent_type == "function_call" and functions is not None:
                 # 使用支持functions的streaming接口
                 llm_responses = self.llm.response_with_functions(
                     self.session_id,
-                    self.dialogue.get_llm_dialogue_with_memory(memory_str),
+                    self.dialogue.get_llm_dialogue_with_memory(
+                        memory_str, self.config.get("voiceprint", {})
+                    ),
                     functions=functions,
                 )
             else:
                 llm_responses = self.llm.response(
                     self.session_id,
-                    self.dialogue.get_llm_dialogue_with_memory(memory_str),
+                    self.dialogue.get_llm_dialogue_with_memory(
+                        memory_str, self.config.get("voiceprint", {})
+                    ),
                 )
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"LLM 处理出错 {query}: {e}")
@@ -653,8 +717,8 @@ class ConnectionHandler:
         function_id = None
         function_arguments = ""
         content_arguments = ""
-        text_index = 0
         self.client_abort = False
+        emotion_flag = True
         for response in llm_responses:
             if self.client_abort:
                 break
@@ -680,17 +744,18 @@ class ConnectionHandler:
                         function_arguments += tools_call[0].function.arguments
             else:
                 content = response
+
+            # 在llm回复中获取情绪表情，一轮对话只在开头获取一次
+            if emotion_flag:
+                asyncio.run_coroutine_threadsafe(
+                    textUtils.get_emotion(self, content),
+                    self.loop,
+                )
+                emotion_flag = False
+
             if content is not None and len(content) > 0:
                 if not tool_call_flag:
                     response_message.append(content)
-                    if text_index == 0:
-                        self.tts.tts_text_queue.put(
-                            TTSMessageDTO(
-                                sentence_id=self.sentence_id,
-                                sentence_type=SentenceType.FIRST,
-                                content_type=ContentType.ACTION,
-                            )
-                        )
                     self.tts.tts_text_queue.put(
                         TTSMessageDTO(
                             sentence_id=self.sentence_id,
@@ -699,7 +764,6 @@ class ConnectionHandler:
                             content_detail=content,
                         )
                     )
-                    text_index += 1
         # 处理function call
         if tool_call_flag:
             bHasError = False
@@ -724,6 +788,11 @@ class ConnectionHandler:
                         f"function call error: {content_arguments}"
                     )
             if not bHasError:
+                # 如需要大模型先处理一轮，添加相关处理后的日志情况
+                if len(response_message) > 0:
+                    self.dialogue.put(
+                        Message(role="assistant", content="".join(response_message))
+                    )
                 response_message.clear()
                 self.logger.bind(tag=TAG).debug(
                     f"function_name={function_name}, function_id={function_id}, function_arguments={function_arguments}"
@@ -734,67 +803,21 @@ class ConnectionHandler:
                     "arguments": function_arguments,
                 }
 
-                # 处理Server端MCP工具调用
+                # 使用统一工具处理器处理所有工具调用
-                if self.mcp_manager.is_mcp_tool(function_name):
+                result = asyncio.run_coroutine_threadsafe(
-                    result = self._handle_mcp_tool_call(function_call_data)
+                    self.func_handler.handle_llm_function_call(
-                elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(
-                    function_name
-                ):
-                    # 如果是小智端MCP工具调用
-                    self.logger.bind(tag=TAG).debug(
-                        f"调用小智端MCP工具: {function_name}, 参数: {function_arguments}"
-                    )
-                    try:
-                        result = asyncio.run_coroutine_threadsafe(
-                            call_mcp_tool(
-                                self, self.mcp_client, function_name, function_arguments
-                            ),
-                            self.loop,
-                        ).result()
-                        self.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
-
-                        resultJson = None
-                        if isinstance(result, str):
-                            try:
-                                resultJson = json.loads(result)
-                            except Exception as e:
-                                self.logger.bind(tag=TAG).error(
-                                    f"解析MCP工具返回结果失败: {e}"
-                                )
-
-                        # 视觉大模型不经过二次LLM处理
-                        if (
-                            resultJson is not None
-                            and isinstance(resultJson, dict)
-                            and "action" in resultJson
-                        ):
-                            result = ActionResponse(
-                                action=Action[resultJson["action"]],
-                                result=None,
-                                response=resultJson.get("response", ""),
-                            )
-                        else:
-                            result = ActionResponse(
-                                action=Action.REQLLM, result=result, response=""
-                            )
-                    except Exception as e:
-                        self.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
-                        result = ActionResponse(
-                            action=Action.REQLLM, result="MCP工具调用失败", response=""
-                        )
-                else:
-                    # 处理系统函数
-                    result = self.func_handler.handle_llm_function_call(
                         self, function_call_data
-                    )
+                    ),
-                self._handle_function_result(result, function_call_data)
+                    self.loop,
+                ).result()
+                self._handle_function_result(result, function_call_data, depth=depth)
 
         # 存储对话内容
         if len(response_message) > 0:
             self.dialogue.put(
                 Message(role="assistant", content="".join(response_message))
             )
-        if text_index > 0:
+        if depth == 0:
             self.tts.tts_text_queue.put(
                 TTSMessageDTO(
                     sentence_id=self.sentence_id,
@@ -803,55 +826,16 @@ class ConnectionHandler:
                 )
             )
         self.llm_finish_task = True
+        # 使用lambda延迟计算，只有在DEBUG级别时才执行get_llm_dialogue()
         self.logger.bind(tag=TAG).debug(
-            json.dumps(self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False)
+            lambda: json.dumps(
+                self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False
+            )
         )
 
         return True
 
-    def _handle_mcp_tool_call(self, function_call_data):
+    def _handle_function_result(self, result, function_call_data, depth):
-        function_arguments = function_call_data["arguments"]
-        function_name = function_call_data["name"]
-        try:
-            args_dict = function_arguments
-            if isinstance(function_arguments, str):
-                try:
-                    args_dict = json.loads(function_arguments)
-                except json.JSONDecodeError:
-                    self.logger.bind(tag=TAG).error(
-                        f"无法解析 function_arguments: {function_arguments}"
-                    )
-                    return ActionResponse(
-                        action=Action.REQLLM, result="参数解析失败", response=""
-                    )
-
-            tool_result = asyncio.run_coroutine_threadsafe(
-                self.mcp_manager.execute_tool(function_name, args_dict), self.loop
-            ).result()
-            # meta=None content=[TextContent(type='text', text='北京当前天气:\n温度: 21°C\n天气: 晴\n湿度: 6%\n风向: 西北 风\n风力等级: 5级', annotations=None)] isError=False
-            content_text = ""
-            if tool_result is not None and tool_result.content is not None:
-                for content in tool_result.content:
-                    content_type = content.type
-                    if content_type == "text":
-                        content_text = content.text
-                    elif content_type == "image":
-                        pass
-
-            if len(content_text) > 0:
-                return ActionResponse(
-                    action=Action.REQLLM, result=content_text, response=""
-                )
-
-        except Exception as e:
-            self.logger.bind(tag=TAG).error(f"MCP工具调用错误: {e}")
-            return ActionResponse(
-                action=Action.REQLLM, result="工具调用出错", response=""
-            )
-
-        return ActionResponse(action=Action.REQLLM, result="工具调用出错", response="")
-
-    def _handle_function_result(self, result, function_call_data):
         if result.action == Action.RESPONSE:  # 直接回复前端
             text = result.response
             self.tts.tts_one_sentence(self, ContentType.TEXT, content_detail=text)
@@ -888,9 +872,9 @@ class ConnectionHandler:
                         content=text,
                     )
                 )
-                self.chat(text, tool_call=True)
+                self.chat(text, tool_call=True, depth=depth + 1)
         elif result.action == Action.NOTFOUND or result.action == Action.ERROR:
-            text = result.result
+            text = result.response if result.response else result.result
             self.tts.tts_one_sentence(self, ContentType.TEXT, content_detail=text)
             self.dialogue.put(Message(role="assistant", content=text))
         else:
@@ -904,14 +888,13 @@ class ConnectionHandler:
                 item = self.report_queue.get(timeout=1)
                 if item is None:  # 检测毒丸对象
                     break
-                type, text, audio_data, report_time = item
                 try:
                     # 检查线程池状态
                     if self.executor is None:
                         continue
                     # 提交任务到线程池
                     self.executor.submit(
-                        self._process_report, type, text, audio_data, report_time
+                        self._process_report, *item
                     )
                 except Exception as e:
                     self.logger.bind(tag=TAG).error(f"聊天记录上报线程异常: {e}")
@@ -941,13 +924,22 @@ class ConnectionHandler:
         """资源清理方法"""
         try:
             # 取消超时任务
-            if self.timeout_task:
+            if self.timeout_task and not self.timeout_task.done():
                 self.timeout_task.cancel()
+                try:
+                    await self.timeout_task
+                except asyncio.CancelledError:
+                    pass
                 self.timeout_task = None
 
-            # 清理MCP资源
+            # 清理工具处理器资源
-            if hasattr(self, "mcp_manager") and self.mcp_manager:
+            if hasattr(self, "func_handler") and self.func_handler:
-                await self.mcp_manager.cleanup_all()
+                try:
+                    await self.func_handler.cleanup()
+                except Exception as cleanup_error:
+                    self.logger.bind(tag=TAG).error(
+                        f"清理工具处理器时出错: {cleanup_error}"
+                    )
 
             # 触发停止事件
             if self.stop_event:
@@ -957,19 +949,61 @@ class ConnectionHandler:
             self.clear_queues()
 
             # 关闭WebSocket连接
-            if ws:
+            try:
-                await ws.close()
+                if ws:
-            elif self.websocket:
+                    # 安全地检查WebSocket状态并关闭
-                await self.websocket.close()
+                    try:
+                        if hasattr(ws, "closed") and not ws.closed:
+                            await ws.close()
+                        elif hasattr(ws, "state") and ws.state.name != "CLOSED":
+                            await ws.close()
+                        else:
+                            # 如果没有closed属性，直接尝试关闭
+                            await ws.close()
+                    except Exception:
+                        # 如果关闭失败，忽略错误
+                        pass
+                elif self.websocket:
+                    try:
+                        if (
+                            hasattr(self.websocket, "closed")
+                            and not self.websocket.closed
+                        ):
+                            await self.websocket.close()
+                        elif (
+                            hasattr(self.websocket, "state")
+                            and self.websocket.state.name != "CLOSED"
+                        ):
+                            await self.websocket.close()
+                        else:
+                            # 如果没有closed属性，直接尝试关闭
+                            await self.websocket.close()
+                    except Exception:
+                        # 如果关闭失败，忽略错误
+                        pass
+            except Exception as ws_error:
+                self.logger.bind(tag=TAG).error(f"关闭WebSocket连接时出错: {ws_error}")
+
+            if self.tts:
+                await self.tts.close()
 
             # 最后关闭线程池（避免阻塞）
             if self.executor:
-                self.executor.shutdown(wait=False)
+                try:
+                    self.executor.shutdown(wait=False)
+                except Exception as executor_error:
+                    self.logger.bind(tag=TAG).error(
+                        f"关闭线程池时出错: {executor_error}"
+                    )
                 self.executor = None
 
             self.logger.bind(tag=TAG).info("连接资源已释放")
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"关闭连接时出错: {e}")
+        finally:
+            # 确保停止事件被设置
+            if self.stop_event:
+                self.stop_event.set()
 
     def clear_queues(self):
         """清空所有任务队列"""
@@ -999,7 +1033,6 @@ class ConnectionHandler:
     def reset_vad_states(self):
         self.client_audio_buffer = bytearray()
         self.client_have_voice = False
-        self.client_have_voice_last_time = 0
         self.client_voice_stop = False
         self.logger.bind(tag=TAG).debug("VAD states reset.")
 
@@ -1018,10 +1051,28 @@ class ConnectionHandler:
         """检查连接超时"""
         try:
             while not self.stop_event.is_set():
-                await asyncio.sleep(self.timeout_seconds)
+                # 检查是否超时（只有在时间戳已初始化的情况下）
-                if not self.stop_event.is_set():
+                if self.last_activity_time > 0.0:
-                    self.logger.bind(tag=TAG).info("连接超时，准备关闭")
+                    current_time = time.time() * 1000
-                    await self.close(self.websocket)
+                    if (
-                    break
+                        current_time - self.last_activity_time
+                        > self.timeout_seconds * 1000
+                    ):
+                        if not self.stop_event.is_set():
+                            self.logger.bind(tag=TAG).info("连接超时，准备关闭")
+                            # 设置停止事件，防止重复处理
+                            self.stop_event.set()
+                            # 使用 try-except 包装关闭操作，确保不会因为异常而阻塞
+                            try:
+                                await self.close(self.websocket)
+                            except Exception as close_error:
+                                self.logger.bind(tag=TAG).error(
+                                    f"超时关闭连接时出错: {close_error}"
+                                )
+                        break
+                # 每10秒检查一次，避免过于频繁
+                await asyncio.sleep(10)
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"超时检查任务出错: {e}")
+        finally:
+            self.logger.bind(tag=TAG).info("超时检查任务已退出")

main/xiaozhi-server/core/handle/functionHandler.py

@@ -1,93 +0,0 @@
-from config.logger import setup_logging
-import json
-from plugins_func.register import (
-    FunctionRegistry,
-    ActionResponse,
-    Action,
-    ToolType,
-    DeviceTypeRegistry,
-)
-from plugins_func.functions.hass_init import append_devices_to_prompt
-
-TAG = __name__
-
-
-class FunctionHandler:
-    def __init__(self, conn):
-        self.conn = conn
-        self.config = conn.config
-        self.device_type_registry = DeviceTypeRegistry()
-        self.function_registry = FunctionRegistry()
-        self.register_nessary_functions()
-        self.register_config_functions()
-        self.functions_desc = self.function_registry.get_all_function_desc()
-        self.finish_init = True
-    
-    def upload_functions_desc(self):
-        self.functions_desc = self.function_registry.get_all_function_desc()
-
-
-    def current_support_functions(self):
-        func_names = []
-        for func in self.functions_desc:
-            func_names.append(func["function"]["name"])
-        # 打印当前支持的函数列表
-        self.conn.logger.bind(tag=TAG, session_id=self.conn.session_id).info(
-            f"当前支持的函数列表: {func_names}"
-        )
-        return func_names
-
-    def get_functions(self):
-        """获取功能调用配置"""
-        return self.functions_desc
-
-    def register_nessary_functions(self):
-        """注册必要的函数"""
-        self.function_registry.register_function("handle_exit_intent")
-        self.function_registry.register_function("get_time")
-        self.function_registry.register_function("get_lunar")
-
-    def register_config_functions(self):
-        """注册配置中的函数,可以不同客户端使用不同的配置"""
-        for func in self.config["Intent"][self.config["selected_module"]["Intent"]].get(
-            "functions", []
-        ):
-            self.function_registry.register_function(func)
-
-        """home assistant需要初始化提示词"""
-        append_devices_to_prompt(self.conn)
-
-    def get_function(self, name):
-        return self.function_registry.get_function(name)
-
-    def handle_llm_function_call(self, conn, function_call_data):
-        try:
-            function_name = function_call_data["name"]
-            funcItem = self.get_function(function_name)
-            if not funcItem:
-                return ActionResponse(
-                    action=Action.NOTFOUND, result="没有找到对应的函数", response=""
-                )
-            func = funcItem.func
-            arguments = function_call_data["arguments"]
-            arguments = json.loads(arguments) if arguments else {}
-            self.conn.logger.bind(tag=TAG).debug(
-                f"调用函数: {function_name}, 参数: {arguments}"
-            )
-            if (
-                funcItem.type == ToolType.SYSTEM_CTL
-                or funcItem.type == ToolType.IOT_CTL
-            ):
-                return func(conn, **arguments)
-            elif funcItem.type == ToolType.WAIT:
-                return func(**arguments)
-            elif funcItem.type == ToolType.CHANGE_SYS_PROMPT:
-                return func(conn, **arguments)
-            else:
-                return ActionResponse(
-                    action=Action.NOTFOUND, result="没有找到对应的函数", response=""
-                )
-        except Exception as e:
-            self.conn.logger.bind(tag=TAG).error(f"处理function call错误: {e}")
-
-        return None

main/xiaozhi-server/core/handle/helloHandle.py

@@ -7,7 +7,7 @@ from core.utils.util import audio_to_data
 from core.handle.sendAudioHandle import sendAudioMessage, send_stt_message
 from core.utils.util import remove_punctuation_and_length, opus_datas_to_wav_bytes
 from core.providers.tts.dto.dto import ContentType, SentenceType
-from core.handle.mcpHandle import (
+from core.providers.tools.device_mcp import (
     MCPClient,
     send_mcp_initialize_message,
     send_mcp_tools_list_request,

main/xiaozhi-server/core/handle/intentHandler.py

@@ -6,14 +6,23 @@ from core.handle.helloHandle import checkWakeupWords
 from core.utils.util import remove_punctuation_and_length
 from core.providers.tts.dto.dto import ContentType
 from core.utils.dialogue import Message
-from core.handle.mcpHandle import call_mcp_tool
 from plugins_func.register import Action, ActionResponse
-from loguru import logger
+from core.providers.tts.dto.dto import TTSMessageDTO, SentenceType
 
 TAG = __name__
 
 
 async def handle_user_intent(conn, text):
+    # 预处理输入文本，处理可能的JSON格式
+    try:
+        if text.strip().startswith('{') and text.strip().endswith('}'):
+            parsed_data = json.loads(text)
+            if isinstance(parsed_data, dict) and "content" in parsed_data:
+                text = parsed_data["content"]  # 提取content用于意图分析
+                conn.current_speaker = parsed_data.get("speaker")  # 保留说话人信息
+    except (json.JSONDecodeError, TypeError):
+        pass
+
     # 检查是否有明确的退出命令
     filtered_text = remove_punctuation_and_length(text)[1]
     if await check_direct_exit(conn, filtered_text):
@@ -29,6 +38,8 @@ async def handle_user_intent(conn, text):
     intent_result = await analyze_intent_with_llm(conn, text)
     if not intent_result:
         return False
+    # 会话开始时生成sentence_id
+    conn.sentence_id = str(uuid.uuid4().hex)
     # 处理各种意图
     return await process_intent_result(conn, intent_result, text)
 
@@ -79,11 +90,6 @@ async def process_intent_result(conn, intent_result, original_text):
             if function_name == "continue_chat":
                 return False
 
-            if function_name == "play_music":
-                funcItem = conn.func_handler.get_function(function_name)
-                if not funcItem:
-                    conn.func_handler.function_registry.register_function("play_music")
-
             function_args = {}
             if "arguments" in intent_data["function_call"]:
                 function_args = intent_data["function_call"]["arguments"]
@@ -106,36 +112,18 @@ async def process_intent_result(conn, intent_result, original_text):
             def process_function_call():
                 conn.dialogue.put(Message(role="user", content=original_text))
 
-                # 处理Server端MCP工具调用
+                # 使用统一工具处理器处理所有工具调用
-                if conn.mcp_manager.is_mcp_tool(function_name):
+                try:
-                    result = conn._handle_mcp_tool_call(function_call_data)
+                    result = asyncio.run_coroutine_threadsafe(
-                elif hasattr(conn, "mcp_client") and conn.mcp_client.has_tool(
+                        conn.func_handler.handle_llm_function_call(
-                    function_name
+                            conn, function_call_data
-                ):
+                        ),
-                    # 如果是小智端MCP工具调用
+                        conn.loop,
-                    conn.logger.bind(tag=TAG).debug(
+                    ).result()
-                        f"调用小智端MCP工具: {function_name}, 参数: {function_args}"
+                except Exception as e:
-                    )
+                    conn.logger.bind(tag=TAG).error(f"工具调用失败: {e}")
-                    try:
+                    result = ActionResponse(
-                        result = asyncio.run_coroutine_threadsafe(
+                        action=Action.ERROR, result=str(e), response=str(e)
-                            call_mcp_tool(
-                                conn, conn.mcp_client, function_name, function_args
-                            ),
-                            conn.loop,
-                        ).result()
-                        conn.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
-                        result = ActionResponse(
-                            action=Action.REQLLM, result=result, response=""
-                        )
-                    except Exception as e:
-                        conn.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
-                        result = ActionResponse(
-                            action=Action.REQLLM, result="MCP工具调用失败", response=""
-                        )
-                else:
-                    # 处理系统函数
-                    result = conn.func_handler.handle_llm_function_call(
-                        conn, function_call_data
                     )
 
                 if result:
@@ -176,5 +164,19 @@ async def process_intent_result(conn, intent_result, original_text):
 
 
 def speak_txt(conn, text):
+    conn.tts.tts_text_queue.put(
+        TTSMessageDTO(
+            sentence_id=conn.sentence_id,
+            sentence_type=SentenceType.FIRST,
+            content_type=ContentType.ACTION,
+        )
+    )
     conn.tts.tts_one_sentence(conn, ContentType.TEXT, content_detail=text)
+    conn.tts.tts_text_queue.put(
+        TTSMessageDTO(
+            sentence_id=conn.sentence_id,
+            sentence_type=SentenceType.LAST,
+            content_type=ContentType.ACTION,
+        )
+    )
     conn.dialogue.put(Message(role="assistant", content=text))

main/xiaozhi-server/core/handle/iotHandle.py

@@ -1,423 +0,0 @@
-import json
-import asyncio
-from plugins_func.register import (
-    FunctionItem,
-    register_device_function,
-    ActionResponse,
-    Action,
-    ToolType,
-)
-
-TAG = __name__
-
-
-def wrap_async_function(async_func):
-    """包装异步函数为同步函数"""
-
-    def wrapper(*args, **kwargs):
-        try:
-            # 获取连接对象（第一个参数）
-            conn = args[0]
-            if not hasattr(conn, "loop"):
-                conn.logger.bind(tag=TAG).error("Connection对象没有loop属性")
-                return ActionResponse(
-                    Action.ERROR,
-                    "Connection对象没有loop属性",
-                    "执行操作时出错: Connection对象没有loop属性",
-                )
-
-            # 使用conn对象中的事件循环
-            loop = conn.loop
-            # 在conn的事件循环中运行异步函数
-            future = asyncio.run_coroutine_threadsafe(async_func(*args, **kwargs), loop)
-            # 等待结果返回
-            return future.result()
-        except Exception as e:
-            conn.logger.bind(tag=TAG).error(f"运行异步函数时出错: {e}")
-            return ActionResponse(Action.ERROR, str(e), f"执行操作时出错: {e}")
-
-    return wrapper
-
-
-def create_iot_function(device_name, method_name, method_info):
-    """
-    根据IOT设备描述生成通用的控制函数
-    """
-
-    async def iot_control_function(
-        conn, response_success=None, response_failure=None, **params
-    ):
-        try:
-            # 设置默认响应消息
-            if not response_success:
-                response_success = "操作成功"
-            if not response_failure:
-                response_failure = "操作失败"
-
-            # 打印响应参数
-            conn.logger.bind(tag=TAG).debug(
-                f"控制函数接收到的响应参数: success='{response_success}', failure='{response_failure}'"
-            )
-
-            # 发送控制命令
-            await send_iot_conn(conn, device_name, method_name, params)
-            # 等待一小段时间让状态更新
-            await asyncio.sleep(0.1)
-
-            # 生成结果信息
-            result = f"{device_name}的{method_name}操作执行成功"
-
-            # 处理响应中可能的占位符
-            response = response_success
-            # 替换{value}占位符
-            for param_name, param_value in params.items():
-                # 先尝试直接替换参数值
-                if "{" + param_name + "}" in response:
-                    response = response.replace(
-                        "{" + param_name + "}", str(param_value)
-                    )
-
-                # 如果有{value}占位符，用相关参数替换
-                if "{value}" in response:
-                    response = response.replace("{value}", str(param_value))
-                    break
-
-            return ActionResponse(Action.RESPONSE, result, response)
-        except Exception as e:
-            conn.logger.bind(tag=TAG).error(
-                f"执行{device_name}的{method_name}操作失败: {e}"
-            )
-
-            # 操作失败时使用大模型提供的失败响应
-            response = response_failure
-
-            return ActionResponse(Action.ERROR, str(e), response)
-
-    return wrap_async_function(iot_control_function)
-
-
-def create_iot_query_function(device_name, prop_name, prop_info):
-    """
-    根据IOT设备属性创建查询函数
-    """
-
-    async def iot_query_function(conn, response_success=None, response_failure=None):
-        try:
-            # 打印响应参数
-            conn.logger.bind(tag=TAG).info(
-                f"查询函数接收到的响应参数: success='{response_success}', failure='{response_failure}'"
-            )
-
-            value = await get_iot_status(conn, device_name, prop_name)
-
-            # 查询成功，生成结果
-            if value is not None:
-                # 使用大模型提供的成功响应，并替换其中的占位符
-                response = response_success.replace("{value}", str(value))
-
-                return ActionResponse(Action.RESPONSE, str(value), response)
-            else:
-                # 查询失败，使用大模型提供的失败响应
-                response = response_failure
-
-                return ActionResponse(Action.ERROR, f"属性{prop_name}不存在", response)
-        except Exception as e:
-            conn.logger.bind(tag=TAG).error(
-                f"查询{device_name}的{prop_name}时出错: {e}"
-            )
-
-            # 查询出错时使用大模型提供的失败响应
-            response = response_failure
-
-            return ActionResponse(Action.ERROR, str(e), response)
-
-    return wrap_async_function(iot_query_function)
-
-
-class IotDescriptor:
-    """
-    A class to represent an IoT descriptor.
-    """
-
-    def __init__(self, name, description, properties, methods):
-        self.name = name
-        self.description = description
-        self.properties = []
-        self.methods = []
-
-        # 根据描述创建属性
-        if properties is not None:
-            for key, value in properties.items():
-                property_item = {}
-                property_item["name"] = key
-                property_item["description"] = value["description"]
-                if value["type"] == "number":
-                    property_item["value"] = 0
-                elif value["type"] == "boolean":
-                    property_item["value"] = False
-                else:
-                    property_item["value"] = ""
-                self.properties.append(property_item)
-
-        # 根据描述创建方法
-        if methods is not None:
-            for key, value in methods.items():
-                method = {}
-                method["description"] = value["description"]
-                method["name"] = key
-                # 检查方法是否有参数
-                if "parameters" in value:
-                    method["parameters"] = {}
-                    for k, v in value["parameters"].items():
-                        method["parameters"][k] = {
-                            "description": v["description"],
-                            "type": v["type"],
-                        }
-                self.methods.append(method)
-
-
-def register_device_type(descriptor, device_type_registry):
-    """注册设备类型及其功能"""
-    device_name = descriptor["name"]
-    type_id = device_type_registry.generate_device_type_id(descriptor)
-
-    # 如果该类型已注册，直接返回类型ID
-    if type_id in device_type_registry.type_functions:
-        return type_id
-
-    functions = {}
-
-    # 为每个属性创建查询函数
-    for prop_name, prop_info in descriptor["properties"].items():
-        func_name = f"get_{device_name.lower()}_{prop_name.lower()}"
-        func_desc = {
-            "type": "function",
-            "function": {
-                "name": func_name,
-                "description": f"查询{descriptor['description']}的{prop_info['description']}",
-                "parameters": {
-                    "type": "object",
-                    "properties": {
-                        "response_success": {
-                            "type": "string",
-                            "description": f"查询成功时的友好回复，必须使用{{value}}作为占位符表示查询到的值",
-                        },
-                        "response_failure": {
-                            "type": "string",
-                            "description": f"查询失败时的友好回复，例如：'无法获取{device_name}的{prop_info['description']}'",
-                        },
-                    },
-                    "required": ["response_success", "response_failure"],
-                },
-            },
-        }
-        query_func = create_iot_query_function(device_name, prop_name, prop_info)
-        decorated_func = register_device_function(
-            func_name, func_desc, ToolType.IOT_CTL
-        )(query_func)
-        functions[func_name] = FunctionItem(
-            func_name, func_desc, decorated_func, ToolType.IOT_CTL
-        )
-
-    # 为每个方法创建控制函数
-    for method_name, method_info in descriptor["methods"].items():
-        func_name = f"{device_name.lower()}_{method_name.lower()}"
-
-        # 创建参数字典，添加原有参数
-        parameters = {}
-        required_params = []
-
-        # 如果方法有参数，则添加参数信息
-        if "parameters" in method_info:
-            parameters = {
-                param_name: {
-                    "type": param_info["type"],
-                    "description": param_info["description"],
-                }
-                for param_name, param_info in method_info["parameters"].items()
-            }
-            required_params = list(method_info["parameters"].keys())
-
-        # 添加响应参数
-        parameters.update(
-            {
-                "response_success": {
-                    "type": "string",
-                    "description": "操作成功时的友好回复,关于该设备的操作结果，设备名称尽量使用description中的名称",
-                },
-                "response_failure": {
-                    "type": "string",
-                    "description": "操作失败时的友好回复,关于该设备的操作结果，设备名称尽量使用description中的名称",
-                },
-            }
-        )
-
-        # 构建必须参数列表（原有参数 + 响应参数）
-        required_params.extend(["response_success", "response_failure"])
-
-        func_desc = {
-            "type": "function",
-            "function": {
-                "name": func_name,
-                "description": f"{descriptor['description']} - {method_info['description']}",
-                "parameters": {
-                    "type": "object",
-                    "properties": parameters,
-                    "required": required_params,
-                },
-            },
-        }
-        control_func = create_iot_function(device_name, method_name, method_info)
-        decorated_func = register_device_function(
-            func_name, func_desc, ToolType.IOT_CTL
-        )(control_func)
-        functions[func_name] = FunctionItem(
-            func_name, func_desc, decorated_func, ToolType.IOT_CTL
-        )
-
-    device_type_registry.register_device_type(type_id, functions)
-    return type_id
-
-
-# 用于接受前端设备推送的搜索iot描述
-async def handleIotDescriptors(conn, descriptors):
-    wait_max_time = 5
-    while conn.func_handler is None or not conn.func_handler.finish_init:
-        await asyncio.sleep(1)
-        wait_max_time -= 1
-        if wait_max_time <= 0:
-            conn.logger.bind(tag=TAG).debug("连接对象没有func_handler")
-            return
-    """处理物联网描述"""
-    functions_changed = False
-
-    for descriptor in descriptors:
-        # 如果descriptor没有properties和methods，则直接跳过
-        if "properties" not in descriptor and "methods" not in descriptor:
-            continue
-
-        # 处理缺失properties的情况
-        if "properties" not in descriptor:
-            descriptor["properties"] = {}
-            # 从methods中提取所有参数作为properties
-            if "methods" in descriptor:
-                for method_name, method_info in descriptor["methods"].items():
-                    if "parameters" in method_info:
-                        for param_name, param_info in method_info["parameters"].items():
-                            # 将参数信息转换为属性信息
-                            descriptor["properties"][param_name] = {
-                                "description": param_info["description"],
-                                "type": param_info["type"],
-                            }
-
-        # 创建IOT设备描述符
-        iot_descriptor = IotDescriptor(
-            descriptor["name"],
-            descriptor["description"],
-            descriptor["properties"],
-            descriptor["methods"],
-        )
-        conn.iot_descriptors[descriptor["name"]] = iot_descriptor
-
-        if conn.load_function_plugin:
-            # 注册或获取设备类型
-            device_type_registry = conn.func_handler.device_type_registry
-            type_id = register_device_type(descriptor, device_type_registry)
-            device_functions = device_type_registry.get_device_functions(type_id)
-
-            # 在连接级注册设备函数
-            if hasattr(conn, "func_handler"):
-                for func_name, func_item in device_functions.items():
-                    conn.func_handler.function_registry.register_function(
-                        func_name, func_item
-                    )
-                    conn.logger.bind(tag=TAG).info(
-                        f"注册IOT函数到function handler: {func_name}"
-                    )
-                    functions_changed = True
-
-    # 如果注册了新函数，更新function描述列表
-    if functions_changed and hasattr(conn, "func_handler"):
-        conn.func_handler.upload_functions_desc()
-
-        func_names = conn.func_handler.current_support_functions()
-        conn.logger.bind(tag=TAG).info(f"设备类型: {type_id}")
-        conn.logger.bind(tag=TAG).info(
-            f"更新function描述列表完成，当前支持的函数: {func_names}"
-        )
-
-
-async def handleIotStatus(conn, states):
-    """处理物联网状态"""
-    for state in states:
-        for key, value in conn.iot_descriptors.items():
-            if key == state["name"]:
-                for property_item in value.properties:
-                    for k, v in state["state"].items():
-                        if property_item["name"] == k:
-                            if type(v) != type(property_item["value"]):
-                                conn.logger.bind(tag=TAG).error(
-                                    f"属性{property_item['name']}的值类型不匹配"
-                                )
-                                break
-                            else:
-                                property_item["value"] = v
-                                conn.logger.bind(tag=TAG).info(
-                                    f"物联网状态更新: {key} , {property_item['name']} = {v}"
-                                )
-                            break
-                break
-
-
-async def get_iot_status(conn, name, property_name):
-    """获取物联网状态"""
-    for key, value in conn.iot_descriptors.items():
-        if key == name:
-            for property_item in value.properties:
-                if property_item["name"] == property_name:
-                    return property_item["value"]
-    conn.logger.bind(tag=TAG).warning(f"未找到设备 {name} 的属性 {property_name}")
-    return None
-
-
-async def set_iot_status(conn, name, property_name, value):
-    """设置物联网状态"""
-    for key, iot_descriptor in conn.iot_descriptors.items():
-        if key == name:
-            for property_item in iot_descriptor.properties:
-                if property_item["name"] == property_name:
-                    if type(value) != type(property_item["value"]):
-                        conn.logger.bind(tag=TAG).error(
-                            f"属性{property_item['name']}的值类型不匹配"
-                        )
-                        return
-                    property_item["value"] = value
-                    conn.logger.bind(tag=TAG).info(
-                        f"物联网状态更新: {name} , {property_name} = {value}"
-                    )
-                    return
-    conn.logger.bind(tag=TAG).warning(f"未找到设备 {name} 的属性 {property_name}")
-
-
-async def send_iot_conn(conn, name, method_name, parameters):
-    """发送物联网指令"""
-    for key, value in conn.iot_descriptors.items():
-        if key == name:
-            # 找到了设备
-            for method in value.methods:
-                # 找到了方法
-                if method["name"] == method_name:
-                    # 构建命令对象
-                    command = {
-                        "name": name,
-                        "method": method_name,
-                    }
-
-                    # 只有当参数不为空时才添加parameters字段
-                    if parameters:
-                        command["parameters"] = parameters
-                    send_message = json.dumps({"type": "iot", "commands": [command]})
-                    await conn.websocket.send(send_message)
-                    conn.logger.bind(tag=TAG).info(f"发送物联网指令: {send_message}")
-                    return
-    conn.logger.bind(tag=TAG).error(f"未找到方法{method_name}")

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -4,6 +4,7 @@ from core.utils.output_counter import check_device_output_limit
 from core.handle.abortHandle import handleAbortMessage
 import time
 import asyncio
+import json
 from core.handle.sendAudioHandle import SentenceType
 from core.utils.util import audio_to_data
 
@@ -38,6 +39,31 @@ async def resume_vad_detection(conn):
 
 
 async def startToChat(conn, text):
+    # 检查输入是否是JSON格式（包含说话人信息）
+    speaker_name = None
+    actual_text = text
+    
+    try:
+        # 尝试解析JSON格式的输入
+        if text.strip().startswith('{') and text.strip().endswith('}'):
+            data = json.loads(text)
+            if 'speaker' in data and 'content' in data:
+                speaker_name = data['speaker']
+                actual_text = data['content']
+                conn.logger.bind(tag=TAG).info(f"解析到说话人信息: {speaker_name}")
+                
+                # 直接使用JSON格式的文本，不解析
+                actual_text = text
+    except (json.JSONDecodeError, KeyError):
+        # 如果解析失败，继续使用原始文本
+        pass
+    
+    # 保存说话人信息到连接对象
+    if speaker_name:
+        conn.current_speaker = speaker_name
+    else:
+        conn.current_speaker = None
+
     if conn.need_bind:
         await check_bind_device(conn)
         return
@@ -52,26 +78,25 @@ async def startToChat(conn, text):
     if conn.client_is_speaking:
         await handleAbortMessage(conn)
 
-    # 首先进行意图分析
+    # 首先进行意图分析，使用实际文本内容
-    intent_handled = await handle_user_intent(conn, text)
+    intent_handled = await handle_user_intent(conn, actual_text)
 
     if intent_handled:
         # 如果意图已被处理，不再进行聊天
         return
 
-    # 意图未被处理，继续常规聊天流程
+    # 意图未被处理，继续常规聊天流程，使用实际文本内容
-    await send_stt_message(conn, text)
+    await send_stt_message(conn, actual_text)
-    conn.executor.submit(conn.chat, text)
+    conn.executor.submit(conn.chat, actual_text)
 
 
 async def no_voice_close_connect(conn, have_voice):
     if have_voice:
-        conn.client_no_voice_last_time = 0.0
+        conn.last_activity_time = time.time() * 1000
         return
-    if conn.client_no_voice_last_time == 0.0:
+    # 只有在已经初始化过时间戳的情况下才进行超时检查
-        conn.client_no_voice_last_time = time.time() * 1000
+    if conn.last_activity_time > 0.0:
-    else:
+        no_voice_time = time.time() * 1000 - conn.last_activity_time
-        no_voice_time = time.time() * 1000 - conn.client_no_voice_last_time
         close_connection_no_voice_time = int(
             conn.config.get("close_connection_no_voice_time", 120)
         )

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -2,52 +2,15 @@ import json
 import asyncio
 import time
 from core.providers.tts.dto.dto import SentenceType
-from core.utils.util import get_string_no_punctuation_or_emoji, analyze_emotion
+from core.utils import textUtils
-from loguru import logger
 
 TAG = __name__
 
-emoji_map = {
-    "neutral": "😶",
-    "happy": "🙂",
-    "laughing": "😆",
-    "funny": "😂",
-    "sad": "😔",
-    "angry": "😠",
-    "crying": "😭",
-    "loving": "😍",
-    "embarrassed": "😳",
-    "surprised": "😲",
-    "shocked": "😱",
-    "thinking": "🤔",
-    "winking": "😉",
-    "cool": "😎",
-    "relaxed": "😌",
-    "delicious": "🤤",
-    "kissy": "😘",
-    "confident": "😏",
-    "sleepy": "😴",
-    "silly": "😜",
-    "confused": "🙄",
-}
-
 
 async def sendAudioMessage(conn, sentenceType, audios, text):
     # 发送句子开始消息
     conn.logger.bind(tag=TAG).info(f"发送音频消息: {sentenceType}, {text}")
-    if text is not None:
+
-        emotion = analyze_emotion(text)
-        emoji = emoji_map.get(emotion, "🙂")  # 默认使用笑脸
-        await conn.websocket.send(
-            json.dumps(
-                {
-                    "type": "llm",
-                    "text": emoji,
-                    "emotion": emotion,
-                    "session_id": conn.session_id,
-                }
-            )
-        )
     pre_buffer = False
     if conn.tts.tts_audio_first_sentence and text is not None:
         conn.logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
@@ -76,7 +39,6 @@ async def sendAudio(conn, audios, pre_buffer=True):
     frame_duration = 60  # 帧时长（毫秒），匹配 Opus 编码
     start_time = time.perf_counter()
     play_position = 0
-    last_reset_time = time.perf_counter()  # 记录最后的重置时间
 
     # 仅当第一句话时执行预缓冲
     if pre_buffer:
@@ -92,10 +54,8 @@ async def sendAudio(conn, audios, pre_buffer=True):
         if conn.client_abort:
             break
 
-        # 每分钟重置一次计时器
+        # 重置没有声音的状态
-        if time.perf_counter() - last_reset_time > 60:
+        conn.last_activity_time = time.time() * 1000
-            await conn.reset_timeout()
-            last_reset_time = time.perf_counter()
 
         # 计算预期发送时间
         expected_time = start_time + (play_position / 1000)
@@ -139,7 +99,23 @@ async def send_stt_message(conn, text):
         return
 
     """发送 STT 状态消息"""
-    stt_text = get_string_no_punctuation_or_emoji(text)
+    
+    # 解析JSON格式，提取实际的用户说话内容
+    display_text = text
+    try:
+        # 尝试解析JSON格式
+        if text.strip().startswith('{') and text.strip().endswith('}'):
+            parsed_data = json.loads(text)
+            if isinstance(parsed_data, dict) and "content" in parsed_data:
+                # 如果是包含说话人信息的JSON格式，只显示content部分
+                display_text = parsed_data["content"]
+                # 保存说话人信息到conn对象
+                if "speaker" in parsed_data:
+                    conn.current_speaker = parsed_data["speaker"]
+    except (json.JSONDecodeError, TypeError):
+        # 如果不是JSON格式，直接使用原始文本
+        display_text = text
+    stt_text = textUtils.get_string_no_punctuation_or_emoji(display_text)
     await conn.websocket.send(
         json.dumps({"type": "stt", "text": stt_text, "session_id": conn.session_id})
     )

main/xiaozhi-server/core/handle/textHandle.py

@@ -1,11 +1,11 @@
 import json
 from core.handle.abortHandle import handleAbortMessage
 from core.handle.helloHandle import handleHelloMessage
-from core.handle.mcpHandle import handle_mcp_message
+from core.providers.tools.device_mcp import handle_mcp_message
 from core.utils.util import remove_punctuation_and_length, filter_sensitive_info
 from core.handle.receiveAudioHandle import startToChat, handleAudioMessage
 from core.handle.sendAudioHandle import send_stt_message, send_tts_message
-from core.handle.iotHandle import handleIotDescriptors, handleIotStatus
+from core.providers.tools.device_iot import handleIotDescriptors, handleIotStatus
 from core.handle.reportHandle import enqueue_asr_report
 import asyncio
 
@@ -77,7 +77,7 @@ async def handleTextMessage(conn, message):
             if "states" in msg_json:
                 asyncio.create_task(handleIotStatus(conn, msg_json["states"]))
         elif msg_json["type"] == "mcp":
-            conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message}")
+            conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message[:100]}")
             if "payload" in msg_json:
                 asyncio.create_task(
                     handle_mcp_message(conn, conn.mcp_client, msg_json["payload"])

main/xiaozhi-server/core/providers/asr/aliyun_stream.py

@@ -0,0 +1,337 @@
+import json
+import time
+import uuid
+import hmac
+import base64
+import hashlib
+import asyncio
+import requests
+import websockets
+import opuslib_next
+import random
+from typing import Optional, Tuple, List
+from urllib import parse
+from datetime import datetime
+from config.logger import setup_logging
+from core.providers.asr.base import ASRProviderBase
+from core.providers.asr.dto.dto import InterfaceType
+
+TAG = __name__
+logger = setup_logging()
+
+
+class AccessToken:
+    @staticmethod
+    def _encode_text(text):
+        encoded_text = parse.quote_plus(text)
+        return encoded_text.replace("+", "%20").replace("*", "%2A").replace("%7E", "~")
+
+    @staticmethod
+    def _encode_dict(dic):
+        keys = dic.keys()
+        dic_sorted = [(key, dic[key]) for key in sorted(keys)]
+        encoded_text = parse.urlencode(dic_sorted)
+        return encoded_text.replace("+", "%20").replace("*", "%2A").replace("%7E", "~")
+
+    @staticmethod
+    def create_token(access_key_id, access_key_secret):
+        parameters = {
+            "AccessKeyId": access_key_id,
+            "Action": "CreateToken",
+            "Format": "JSON",
+            "RegionId": "cn-shanghai",
+            "SignatureMethod": "HMAC-SHA1",
+            "SignatureNonce": str(uuid.uuid1()),
+            "SignatureVersion": "1.0",
+            "Timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "Version": "2019-02-28",
+        }
+        query_string = AccessToken._encode_dict(parameters)
+        string_to_sign = (
+            "GET" + "&" + AccessToken._encode_text("/") + "&" + AccessToken._encode_text(query_string)
+        )
+        secreted_string = hmac.new(
+            bytes(access_key_secret + "&", encoding="utf-8"),
+            bytes(string_to_sign, encoding="utf-8"),
+            hashlib.sha1,
+        ).digest()
+        signature = base64.b64encode(secreted_string)
+        signature = AccessToken._encode_text(signature)
+        full_url = "http://nls-meta.cn-shanghai.aliyuncs.com/?Signature=%s&%s" % (signature, query_string)
+        response = requests.get(full_url)
+        if response.ok:
+            root_obj = response.json()
+            if "Token" in root_obj:
+                return root_obj["Token"]["Id"], root_obj["Token"]["ExpireTime"]
+        return None, None
+
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__()
+        self.interface_type = InterfaceType.STREAM
+        self.config = config
+        self.text = ""
+        self.decoder = opuslib_next.Decoder(16000, 1)
+        self.asr_ws = None
+        self.forward_task = None
+        self.is_processing = False
+        self.server_ready = False  # 服务器准备状态
+
+        # 基础配置
+        self.access_key_id = config.get("access_key_id")
+        self.access_key_secret = config.get("access_key_secret")
+        self.appkey = config.get("appkey")
+        self.token = config.get("token")
+        self.host = config.get("host", "nls-gateway-cn-shanghai.aliyuncs.com")
+        self.ws_url = f"wss://{self.host}/ws/v1"
+        self.max_sentence_silence = config.get("max_sentence_silence")
+        self.output_dir = config.get("output_dir", "./audio_output")
+        self.delete_audio_file = delete_audio_file
+        self.expire_time = None
+
+        # Token管理
+        if self.access_key_id and self.access_key_secret:
+            self._refresh_token()
+        elif not self.token:
+            raise ValueError("必须提供access_key_id+access_key_secret或者直接提供token")
+
+    def _refresh_token(self):
+        """刷新Token"""
+        self.token, expire_time_str = AccessToken.create_token(self.access_key_id, self.access_key_secret)
+        if not self.token:
+            raise ValueError("无法获取有效的访问Token")
+        
+        try:
+            expire_str = str(expire_time_str).strip()
+            if expire_str.isdigit():
+                expire_time = datetime.fromtimestamp(int(expire_str))
+            else:
+                expire_time = datetime.strptime(expire_str, "%Y-%m-%dT%H:%M:%SZ")
+            self.expire_time = expire_time.timestamp() - 60
+        except:
+            self.expire_time = None
+
+    def _is_token_expired(self):
+        """检查Token是否过期"""
+        return self.expire_time and time.time() > self.expire_time
+
+    async def open_audio_channels(self, conn):
+        await super().open_audio_channels(conn)
+
+    async def receive_audio(self, conn, audio, audio_have_voice):
+        # 初始化音频缓存
+        if not hasattr(conn, 'asr_audio_for_voiceprint'):
+            conn.asr_audio_for_voiceprint = []
+        
+        # 存储音频数据
+        if audio:
+            conn.asr_audio_for_voiceprint.append(audio)
+        
+        conn.asr_audio.append(audio)
+        conn.asr_audio = conn.asr_audio[-10:]
+
+        # 只在有声音且没有连接时建立连接
+        if audio_have_voice and not self.is_processing:
+            try:
+                await self._start_recognition(conn)
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"开始识别失败: {str(e)}")
+                await self._cleanup(conn)
+                return
+
+        if self.asr_ws and self.is_processing and self.server_ready:
+            try:
+                pcm_frame = self.decoder.decode(audio, 960)
+                await self.asr_ws.send(pcm_frame)
+            except Exception as e:
+                logger.bind(tag=TAG).warning(f"发送音频失败: {str(e)}")
+                await self._cleanup(conn)
+
+    async def _start_recognition(self, conn):
+        """开始识别会话"""
+        if self._is_token_expired():
+            self._refresh_token()
+        
+        # 建立连接
+        headers = {"X-NLS-Token": self.token}
+        self.asr_ws = await websockets.connect(
+            self.ws_url,
+            additional_headers=headers,
+            max_size=1000000000,
+            ping_interval=None,
+            ping_timeout=None,
+            close_timeout=5,
+        )
+        
+        self.is_processing = True
+        self.server_ready = False  # 重置服务器准备状态
+        self.forward_task = asyncio.create_task(self._forward_results(conn))
+        
+        # 发送开始请求
+        start_request = {
+            "header": {
+                "namespace": "SpeechTranscriber",
+                "name": "StartTranscription",
+                "status": 20000000,
+                "message_id": ''.join(random.choices('0123456789abcdef', k=32)),
+                "task_id": ''.join(random.choices('0123456789abcdef', k=32)),
+                "status_text": "Gateway:SUCCESS:Success.",
+                "appkey": self.appkey
+            },
+            "payload": {
+                "format": "pcm",
+                "sample_rate": 16000,
+                "enable_intermediate_result": True,
+                "enable_punctuation_prediction": True,
+                "enable_inverse_text_normalization": True,
+                "max_sentence_silence": self.max_sentence_silence,
+                "enable_voice_detection": False,
+            }
+        }
+        await self.asr_ws.send(json.dumps(start_request, ensure_ascii=False))
+        logger.bind(tag=TAG).info("已发送开始请求，等待服务器准备...")
+
+    async def _forward_results(self, conn):
+        """转发识别结果"""
+        try:
+            while self.asr_ws and not conn.stop_event.is_set():
+                try:
+                    response = await asyncio.wait_for(self.asr_ws.recv(), timeout=1.0)
+                    result = json.loads(response)
+                    
+                    header = result.get("header", {})
+                    payload = result.get("payload", {})
+                    message_name = header.get("name", "")
+                    status = header.get("status", 0)
+                    
+                    if status != 20000000:
+                        if status in [40000004, 40010004]:  # 连接超时或客户端断开
+                            logger.bind(tag=TAG).warning(f"连接问题，状态码: {status}")
+                            break
+                        elif status in [40270002, 40270003]:  # 音频问题
+                            logger.bind(tag=TAG).warning(f"音频处理问题，状态码: {status}")
+                            continue
+                        else:
+                            logger.bind(tag=TAG).error(f"识别错误，状态码: {status}, 消息: {header.get('status_text', '')}")
+                            continue
+                    
+                    # 收到TranscriptionStarted表示服务器准备好接收音频数据
+                    if message_name == "TranscriptionStarted":
+                        self.server_ready = True
+                        logger.bind(tag=TAG).info("服务器已准备，开始发送缓存音频...")
+                        
+                        # 发送缓存音频
+                        if conn.asr_audio:
+                            for cached_audio in conn.asr_audio[-10:]:
+                                try:
+                                    pcm_frame = self.decoder.decode(cached_audio, 960)
+                                    await self.asr_ws.send(pcm_frame)
+                                except Exception as e:
+                                    logger.bind(tag=TAG).warning(f"发送缓存音频失败: {e}")
+                                    break
+                        continue
+                    
+                    if message_name == "TranscriptionResultChanged":
+                        # 中间结果
+                        text = payload.get("result", "")
+                        if text:
+                            self.text = text
+                    elif message_name == "SentenceEnd":
+                        # 最终结果
+                        text = payload.get("result", "")
+                        if text:
+                            self.text = text
+                            conn.reset_vad_states()
+                            # 传递缓存的音频数据
+                            audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
+                            await self.handle_voice_stop(conn, audio_data)
+                            # 清空缓存
+                            conn.asr_audio_for_voiceprint = []
+                            break
+                    elif message_name == "TranscriptionCompleted":
+                        # 识别完成
+                        self.is_processing = False
+                        break
+                        
+                except asyncio.TimeoutError:
+                    continue
+                except websockets.exceptions.ConnectionClosed:
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"处理结果失败: {str(e)}")
+                    break
+                    
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"结果转发失败: {str(e)}")
+        finally:
+            await self._cleanup(conn)
+
+    async def _cleanup(self, conn):
+        """清理资源"""
+        logger.bind(tag=TAG).info(f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}")
+        
+        # 清理连接的音频缓存
+        if conn and hasattr(conn, 'asr_audio_for_voiceprint'):
+            conn.asr_audio_for_voiceprint = []
+        
+        # 判断是否需要发送终止请求
+        should_stop = self.is_processing or self.server_ready
+        
+        # 发送停止识别请求
+        if self.asr_ws and should_stop:
+            try:
+                stop_msg = {
+                    "header": {
+                        "namespace": "SpeechTranscriber",
+                        "name": "StopTranscription",
+                        "status": 20000000,
+                        "message_id": ''.join(random.choices('0123456789abcdef', k=32)),
+                        "status_text": "Client:Stop",
+                        "appkey": self.appkey
+                    }
+                }
+                logger.bind(tag=TAG).info("正在发送ASR终止请求")
+                await self.asr_ws.send(json.dumps(stop_msg, ensure_ascii=False))
+                await asyncio.sleep(0.1)
+                logger.bind(tag=TAG).info("ASR终止请求已发送")
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"ASR终止请求发送失败: {e}")
+        
+        # 状态重置（在终止请求发送后）
+        self.is_processing = False
+        self.server_ready = False
+        logger.bind(tag=TAG).info("ASR状态已重置")
+
+        # 清理任务
+        if self.forward_task and not self.forward_task.done():
+            self.forward_task.cancel()
+            try:
+                await asyncio.wait_for(self.forward_task, timeout=1.0)
+            except Exception as e:
+                logger.bind(tag=TAG).debug(f"forward_task取消异常: {e}")
+            finally:
+                self.forward_task = None
+        
+        # 关闭连接
+        if self.asr_ws:
+            try:
+                logger.bind(tag=TAG).debug("正在关闭WebSocket连接")
+                await asyncio.wait_for(self.asr_ws.close(), timeout=2.0)
+                logger.bind(tag=TAG).debug("WebSocket连接已关闭")
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"关闭WebSocket连接失败: {e}")
+            finally:
+                self.asr_ws = None
+        
+        logger.bind(tag=TAG).info("ASR会话清理完成")
+
+    async def speech_to_text(self, opus_data, session_id, audio_format):
+        """获取识别结果"""
+        result = self.text
+        self.text = ""
+        return result, None
+
+    async def close(self):
+        """关闭资源"""
+        await self._cleanup()

main/xiaozhi-server/core/providers/asr/base.py

@@ -1,15 +1,18 @@
 import os
 import wave
-import copy
 import uuid
 import queue
 import asyncio
 import traceback
 import threading
 import opuslib_next
+import json
+import io
+import time
+import concurrent.futures
 from abc import ABC, abstractmethod
 from config.logger import setup_logging
-from typing import Optional, Tuple, List
+from typing import Optional, Tuple, List, Dict, Any
 from core.handle.receiveAudioHandle import startToChat
 from core.handle.reportHandle import enqueue_asr_report
 from core.utils.util import remove_punctuation_and_length
@@ -24,10 +27,7 @@ class ASRProviderBase(ABC):
         pass
 
     # 打开音频通道
-    # 这里默认是非流式的处理方式
-    # 流式处理方式请在子类中重写
     async def open_audio_channels(self, conn):
-        # tts 消化线程
         conn.asr_priority_thread = threading.Thread(
             target=self.asr_text_priority_thread, args=(conn,), daemon=True
         )
@@ -52,41 +52,171 @@ class ASRProviderBase(ABC):
                 continue
 
     # 接收音频
-    # 这里默认是非流式的处理方式
-    # 流式处理方式请在子类中重写
     async def receive_audio(self, conn, audio, audio_have_voice):
         if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
             have_voice = audio_have_voice
         else:
             have_voice = conn.client_have_voice
-        # 如果本次没有声音，本段也没声音，就把声音丢弃了
+        
         conn.asr_audio.append(audio)
-        if have_voice == False and conn.client_have_voice == False:
+        if not have_voice and not conn.client_have_voice:
             conn.asr_audio = conn.asr_audio[-10:]
             return
 
-        # 如果本段有声音，且已经停止了
         if conn.client_voice_stop:
-            asr_audio_task = copy.deepcopy(conn.asr_audio)
+            asr_audio_task = conn.asr_audio.copy()
             conn.asr_audio.clear()
-
-            # 音频太短了，无法识别
             conn.reset_vad_states()
+
             if len(asr_audio_task) > 15:
                 await self.handle_voice_stop(conn, asr_audio_task)
 
     # 处理语音停止
-    async def handle_voice_stop(self, conn, asr_audio_task):
+    async def handle_voice_stop(self, conn, asr_audio_task: List[bytes]):
-        raw_text, _ = await self.speech_to_text(
+        """并行处理ASR和声纹识别"""
-            asr_audio_task, conn.session_id, conn.audio_format
+        try:
-        )  # 确保ASR模块返回原始文本
+            total_start_time = time.monotonic()
-        conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+            
-        text_len, _ = remove_punctuation_and_length(raw_text)
+            # 准备音频数据
-        self.stop_ws_connection()
+            if conn.audio_format == "pcm":
-        if text_len > 0:
+                pcm_data = asr_audio_task
-            # 使用自定义模块进行上报
+            else:
-            await startToChat(conn, raw_text)
+                pcm_data = self.decode_opus(asr_audio_task)
-            enqueue_asr_report(conn, raw_text, asr_audio_task)
+            
+            combined_pcm_data = b"".join(pcm_data)
+            
+            # 预先准备WAV数据
+            wav_data = None
+            # 使用连接的声纹识别提供者
+            if conn.voiceprint_provider and combined_pcm_data:
+                wav_data = self._pcm_to_wav(combined_pcm_data)
+            
+            
+            # 定义ASR任务
+            def run_asr():
+                start_time = time.monotonic()
+                try:
+                    loop = asyncio.new_event_loop()
+                    asyncio.set_event_loop(loop)
+                    try:
+                        result = loop.run_until_complete(
+                            self.speech_to_text(asr_audio_task, conn.session_id, conn.audio_format)
+                        )
+                        end_time = time.monotonic()
+                        logger.bind(tag=TAG).info(f"ASR耗时: {end_time - start_time:.3f}s")
+                        return result
+                    finally:
+                        loop.close()
+                except Exception as e:
+                    end_time = time.monotonic()
+                    logger.bind(tag=TAG).error(f"ASR失败: {e}")
+                    return ("", None)
+            
+            # 定义声纹识别任务
+            def run_voiceprint():
+                if not wav_data:
+                    return None
+                try:
+                    loop = asyncio.new_event_loop()
+                    asyncio.set_event_loop(loop)
+                    try:
+                        # 使用连接的声纹识别提供者
+                        result = loop.run_until_complete(
+                            conn.voiceprint_provider.identify_speaker(wav_data, conn.session_id)
+                        )
+                        return result
+                    finally:
+                        loop.close()
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"声纹识别失败: {e}")
+                    return None
+            
+            # 使用线程池执行器并行运行
+            parallel_start_time = time.monotonic()
+            
+            with concurrent.futures.ThreadPoolExecutor(max_workers=2) as thread_executor:
+                asr_future = thread_executor.submit(run_asr)
+                
+                if conn.voiceprint_provider and wav_data:
+                    voiceprint_future = thread_executor.submit(run_voiceprint)
+                    
+                    # 等待两个线程都完成
+                    asr_result = asr_future.result(timeout=15)
+                    voiceprint_result = voiceprint_future.result(timeout=15)
+                    
+                    results = {"asr": asr_result, "voiceprint": voiceprint_result}
+                else:
+                    asr_result = asr_future.result(timeout=15)
+                    results = {"asr": asr_result, "voiceprint": None}
+            
+            
+            # 处理结果
+            raw_text, file_path = results.get("asr", ("", None))
+            speaker_name = results.get("voiceprint", None)
+            
+            # 记录识别结果
+            if raw_text:
+                logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+            if speaker_name:
+                logger.bind(tag=TAG).info(f"识别说话人: {speaker_name}")
+            
+            # 性能监控
+            total_time = time.monotonic() - total_start_time
+            logger.bind(tag=TAG).info(f"总处理耗时: {total_time:.3f}s")
+            
+            # 检查文本长度
+            text_len, _ = remove_punctuation_and_length(raw_text)
+            self.stop_ws_connection()
+            
+            if text_len > 0:
+                # 构建包含说话人信息的JSON字符串
+                enhanced_text = self._build_enhanced_text(raw_text, speaker_name)
+                
+                # 使用自定义模块进行上报
+                await startToChat(conn, enhanced_text)
+                enqueue_asr_report(conn, enhanced_text, asr_audio_task)
+                
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"处理语音停止失败: {e}")
+            import traceback
+            logger.bind(tag=TAG).debug(f"异常详情: {traceback.format_exc()}")
+
+    def _build_enhanced_text(self, text: str, speaker_name: Optional[str]) -> str:
+        """构建包含说话人信息的文本"""
+        if speaker_name and speaker_name.strip():
+            return json.dumps({
+                "speaker": speaker_name,
+                "content": text
+            }, ensure_ascii=False)
+        else:
+            return text
+
+    def _pcm_to_wav(self, pcm_data: bytes) -> bytes:
+        """将PCM数据转换为WAV格式"""
+        if len(pcm_data) == 0:
+            logger.bind(tag=TAG).warning("PCM数据为空，无法转换WAV")
+            return b""
+        
+        # 确保数据长度是偶数（16位音频）
+        if len(pcm_data) % 2 != 0:
+            pcm_data = pcm_data[:-1]
+        
+        # 创建WAV文件头
+        wav_buffer = io.BytesIO()
+        try:
+            with wave.open(wav_buffer, 'wb') as wav_file:
+                wav_file.setnchannels(1)      # 单声道
+                wav_file.setsampwidth(2)      # 16位
+                wav_file.setframerate(16000)  # 16kHz采样率
+                wav_file.writeframes(pcm_data)
+            
+            wav_buffer.seek(0)
+            wav_data = wav_buffer.read()
+            
+            return wav_data
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"WAV转换失败: {e}")
+            return b""
 
     def stop_ws_connection(self):
         pass
@@ -113,27 +243,29 @@ class ASRProviderBase(ABC):
         pass
 
     @staticmethod
-    def decode_opus(opus_data: List[bytes]) -> bytes:
+    def decode_opus(opus_data: List[bytes]) -> List[bytes]:
         """将Opus音频数据解码为PCM数据"""
         try:
-            decoder = opuslib_next.Decoder(16000, 1)  # 16kHz, 单声道
+            decoder = opuslib_next.Decoder(16000, 1)
             pcm_data = []
-            buffer_size = 960  # 每次处理960个采样点
+            buffer_size = 960  # 每次处理960个采样点 (60ms at 16kHz)
             
-            for opus_packet in opus_data:
+            for i, opus_packet in enumerate(opus_data):
                 try:
-                    # 使用较小的缓冲区大小进行处理
+                    if not opus_packet or len(opus_packet) == 0:
+                        continue
+                    
                     pcm_frame = decoder.decode(opus_packet, buffer_size)
-                    if pcm_frame:
+                    if pcm_frame and len(pcm_frame) > 0:
                         pcm_data.append(pcm_frame)
+                        
                 except opuslib_next.OpusError as e:
-                    logger.bind(tag=TAG).warning(f"Opus解码错误，跳过当前数据包: {e}")
+                    logger.bind(tag=TAG).warning(f"Opus解码错误，跳过数据包 {i}: {e}")
-                    continue
                 except Exception as e:
-                    logger.bind(tag=TAG).error(f"音频处理错误: {e}", exc_info=True)
+                    logger.bind(tag=TAG).error(f"音频处理错误，数据包 {i}: {e}")
-                    continue
             
             return pcm_data
+            
         except Exception as e:
-            logger.bind(tag=TAG).error(f"音频解码过程发生错误: {e}", exc_info=True)
+            logger.bind(tag=TAG).error(f"音频解码过程发生错误: {e}")
             return []

main/xiaozhi-server/core/providers/asr/doubao_stream.py

@@ -57,6 +57,16 @@ class ASRProvider(ASRProviderBase):
         conn.asr_audio.append(audio)
         conn.asr_audio = conn.asr_audio[-10:]
         
+        # 存储音频数据
+        if not hasattr(conn, 'asr_audio_for_voiceprint'):
+            conn.asr_audio_for_voiceprint = []
+        conn.asr_audio_for_voiceprint.append(audio)
+        
+        # 当没有音频数据时处理完整语音片段
+        if not audio and len(conn.asr_audio_for_voiceprint) > 0:
+            await self.handle_voice_stop(conn, conn.asr_audio_for_voiceprint)
+            conn.asr_audio_for_voiceprint = []
+
         # 如果本次有声音，且之前没有建立连接
         if audio_have_voice and self.asr_ws is None and not self.is_processing:
             try:
@@ -148,6 +158,8 @@ class ASRProvider(ASRProviderBase):
     async def _forward_asr_results(self, conn):
         try:
             while self.asr_ws and not conn.stop_event.is_set():
+                # 获取当前连接的音频数据
+                audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
                 try:
                     response = await self.asr_ws.recv()
                     result = self.parse_response(response)
@@ -171,7 +183,8 @@ class ASRProvider(ASRProviderBase):
                                 logger.bind(tag=TAG).error(f"识别文本：空")
                                 self.text = ""
                                 conn.reset_vad_states()
-                                await self.handle_voice_stop(conn, None)
+                                if len(audio_data) > 15:  # 确保有足够音频数据
+                                    await self.handle_voice_stop(conn, audio_data)
                                 break
 
                             for utterance in utterances:
@@ -181,7 +194,8 @@ class ASRProvider(ASRProviderBase):
                                         f"识别到文本: {self.text}"
                                     )
                                     conn.reset_vad_states()
-                                    await self.handle_voice_stop(conn, None)
+                                    if len(audio_data) > 15:  # 确保有足够音频数据
+                                        await self.handle_voice_stop(conn, audio_data)
                                     break
                         elif "error" in payload:
                             error_msg = payload.get("error", "未知错误")
@@ -208,6 +222,13 @@ class ASRProvider(ASRProviderBase):
                 await self.asr_ws.close()
                 self.asr_ws = None
             self.is_processing = False
+            if conn:
+                if hasattr(conn, 'asr_audio_for_voiceprint'):
+                    conn.asr_audio_for_voiceprint = []
+                if hasattr(conn, 'asr_audio'):
+                    conn.asr_audio = []
+                if hasattr(conn, 'has_valid_voice'):
+                    conn.has_valid_voice = False
 
     def stop_ws_connection(self):
         if self.asr_ws:
@@ -349,3 +370,12 @@ class ASRProvider(ASRProviderBase):
                 pass
             self.forward_task = None
         self.is_processing = False
+        # 清理所有连接的音频缓冲区
+        if hasattr(self, '_connections'):
+            for conn in self._connections.values():
+                if hasattr(conn, 'asr_audio_for_voiceprint'):
+                    conn.asr_audio_for_voiceprint = []
+                if hasattr(conn, 'asr_audio'):
+                    conn.asr_audio = []
+                if hasattr(conn, 'has_valid_voice'):
+                    conn.has_valid_voice = False

main/xiaozhi-server/core/providers/asr/openai.py

@@ -0,0 +1,82 @@
+import time
+import os
+from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.providers.asr.dto.dto import InterfaceType
+from core.providers.asr.base import ASRProviderBase
+
+import requests
+
+TAG = __name__
+logger = setup_logging()
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config: dict, delete_audio_file: bool):
+        self.interface_type = InterfaceType.NON_STREAM
+        self.api_key = config.get("api_key")
+        self.api_url = config.get("base_url")
+        self.model = config.get("model_name")        
+        self.output_dir = config.get("output_dir")
+        self.delete_audio_file = delete_audio_file
+
+        os.makedirs(self.output_dir, exist_ok=True)
+
+    async def speech_to_text(self, opus_data: List[bytes], session_id: str, audio_format="opus") -> Tuple[Optional[str], Optional[str]]:
+        file_path = None
+        try:
+            start_time = time.time()
+            if audio_format == "pcm":
+                pcm_data = opus_data
+            else:
+                pcm_data = self.decode_opus(opus_data)
+            file_path = self.save_audio_to_file(pcm_data, session_id)
+
+            logger.bind(tag=TAG).debug(
+                f"音频文件保存耗时: {time.time() - start_time:.3f}s | 路径: {file_path}"
+            )
+
+            logger.bind(tag=TAG).info(f"file path: {file_path}")
+            headers = {
+                "Authorization": f"Bearer {self.api_key}",
+            }
+            
+            # 使用data参数传递模型名称
+            data = {
+                "model": self.model
+            }
+
+
+            with open(file_path, "rb") as audio_file:  # 使用with语句确保文件关闭
+                files = {
+                    "file": audio_file
+                }
+
+                start_time = time.time()
+                response = requests.post(
+                    self.api_url,
+                    files=files,
+                    data=data,
+                    headers=headers
+                )
+                logger.bind(tag=TAG).debug(
+                    f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {response.text}"
+                )
+
+            if response.status_code == 200:
+                text = response.json().get("text", "")
+                return text, file_path
+            else:
+                raise Exception(f"API请求失败: {response.status_code} - {response.text}")
+                
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"语音识别失败: {e}")
+            return "", None
+        finally:
+            # 文件清理逻辑
+            if self.delete_audio_file and file_path and os.path.exists(file_path):
+                try:
+                    os.remove(file_path)
+                    logger.bind(tag=TAG).debug(f"已删除临时音频文件: {file_path}")
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"文件删除失败: {file_path} | 错误: {e}")
+        

main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py

@@ -16,10 +16,11 @@ class IntentProvider(IntentProviderBase):
         super().__init__(config)
         self.llm = None
         self.promot = ""
-        # 添加缓存管理
+        # 导入全局缓存管理器
-        self.intent_cache = {}  # 缓存意图识别结果
+        from core.utils.cache.manager import cache_manager, CacheType
-        self.cache_expiry = 600  # 缓存有效期10分钟
+
-        self.cache_max_size = 100  # 最多缓存100个意图
+        self.cache_manager = cache_manager
+        self.CacheType = CacheType
         self.history_count = 4  # 默认使用最近4条对话记录
 
     def get_intent_system_prompt(self, functions_list: str) -> str:
@@ -95,30 +96,13 @@ class IntentProvider(IntentProviderBase):
             "1. 只返回JSON格式，不要包含任何其他文字\n"
             '2. 如果没有找到匹配的函数，返回{"function_call": {"name": "continue_chat"}}\n'
             "3. 确保返回的JSON格式正确，包含所有必要的字段\n"
+            "特殊说明：\n"
+            "- 当用户单次输入包含多个指令时（如'打开灯并且调高音量'）\n"
+            "- 请返回多个function_call组成的JSON数组\n"
+            "- 示例：{'function_calls': [{name:'light_on'}, {name:'volume_up'}]}"
         )
         return prompt
 
-    def clean_cache(self):
-        """清理过期缓存"""
-        now = time.time()
-        # 找出过期键
-        expired_keys = [
-            k
-            for k, v in self.intent_cache.items()
-            if now - v["timestamp"] > self.cache_expiry
-        ]
-        for key in expired_keys:
-            del self.intent_cache[key]
-
-        # 如果缓存太大，移除最旧的条目
-        if len(self.intent_cache) > self.cache_max_size:
-            # 按时间戳排序并保留最新的条目
-            sorted_items = sorted(
-                self.intent_cache.items(), key=lambda x: x[1]["timestamp"]
-            )
-            for key, _ in sorted_items[: len(sorted_items) - self.cache_max_size]:
-                del self.intent_cache[key]
-
     def replyResult(self, text: str, original_text: str):
         llm_result = self.llm.response_no_stream(
             system_prompt=text,
@@ -141,21 +125,16 @@ class IntentProvider(IntentProviderBase):
         logger.bind(tag=TAG).debug(f"使用意图识别模型: {model_info}")
 
         # 计算缓存键
-        cache_key = hashlib.md5(text.encode()).hexdigest()
+        cache_key = hashlib.md5((conn.device_id + text).encode()).hexdigest()
 
         # 检查缓存
-        if cache_key in self.intent_cache:
+        cached_intent = self.cache_manager.get(self.CacheType.INTENT, cache_key)
-            cache_entry = self.intent_cache[cache_key]
+        if cached_intent is not None:
-            # 检查缓存是否过期
+            cache_time = time.time() - total_start_time
-            if time.time() - cache_entry["timestamp"] <= self.cache_expiry:
+            logger.bind(tag=TAG).debug(
-                cache_time = time.time() - total_start_time
+                f"使用缓存的意图: {cache_key} -> {cached_intent}, 耗时: {cache_time:.4f}秒"
-                logger.bind(tag=TAG).debug(
+            )
-                    f"使用缓存的意图: {cache_key} -> {cache_entry['intent']}, 耗时: {cache_time:.4f}秒"
+            return cached_intent
-                )
-                return cache_entry["intent"]
-
-        # 清理缓存
-        self.clean_cache()
 
         if self.promot == "":
             functions = conn.func_handler.get_functions()
@@ -255,10 +234,7 @@ class IntentProvider(IntentProviderBase):
                     conn.dialogue.dialogue = clean_history
 
                 # 添加到缓存
-                self.intent_cache[cache_key] = {
+                self.cache_manager.set(self.CacheType.INTENT, cache_key, intent)
-                    "intent": intent,
-                    "timestamp": time.time(),
-                }
 
                 # 后处理时间
                 postprocess_time = time.time() - postprocess_start_time
@@ -268,10 +244,7 @@ class IntentProvider(IntentProviderBase):
                 return intent
             else:
                 # 添加到缓存
-                self.intent_cache[cache_key] = {
+                self.cache_manager.set(self.CacheType.INTENT, cache_key, intent)
-                    "intent": intent,
-                    "timestamp": time.time(),
-                }
 
                 # 后处理时间
                 postprocess_time = time.time() - postprocess_start_time