Merge pull request #1976 from xinnan-tech/docker-install

Docker install

.gitignore

@@ -174,3 +174,4 @@ main/xiaozhi-server/mysql
 uploadfile
 *.json
 .vscode
+.cursor

Dockerfile-server

@@ -21,6 +21,7 @@ RUN apt-get update && \
 
 # 从构建阶段复制Python包和前端构建产物
 COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
+COPY --from=builder /usr/local/bin/mcp-proxy /usr/local/bin/mcp-proxy
 
 # 复制应用代码
 COPY main/xiaozhi-server .

README.md

@@ -3,10 +3,10 @@
 <h1 align="center">小智后端服务xiaozhi-esp32-server</h1>
 
 <p align="center">
-本项目为开源智能硬件项目
+本项目基于人机共生智能理论和技术研发智能终端软硬件体系<br/>为开源智能硬件项目
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a>提供后端服务<br/>
 根据<a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">小智通信协议</a>使用Python、Java、Vue实现<br/>
-帮助您快速搭建小智服务器
+支持MCP接入点和声纹识别
 </p>
 
 <p align="center">
@@ -37,6 +37,14 @@
   </a>
 </p>
 
+<p align="center">
+By Professor Siyuan Liu Research and Development Group ( South China University of Technology)
+</br>
+刘思源教授团队研发（华南理工大学）
+</br>
+<img src="./docs/images/hnlg.jpg" alt="华南理工大学" width="50%">
+</p>
+
 ---
 
 ## 适用人群 👥
@@ -144,13 +152,18 @@
         </a>
     </td>
     <td>
-          <a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
+        <a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
          <picture>
            <img alt="MCP接入点" src="docs/images/demo13.png" />
          </picture>
         </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
+         <picture>
+           <img alt="声纹识别" src="docs/images/demo14.png" />
+         </picture>
+        </a>
     </td>
   </tr>
 </table>
@@ -176,7 +189,7 @@
 | 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 | 
 |---------|------|---------|---------|---------|---------|
 | **最简化安装** | 智能对话、IOT、MCP、视觉感知 | 低配置环境，数据存储在配置文件，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - | 
-| **全模块安装** | 智能对话、IOT、MCP接入点、视觉感知、OTA、智控台 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
+| **全模块安装** | 智能对话、IOT、MCP接入点、声纹识别、视觉感知、OTA、智控台 | 完整功能体验，数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
 
 > 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
@@ -204,7 +217,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 | ASR(语音识别) | FunASR(本地) | 👍FunASRServer 或 👍DoubaoStreamASR |
 | LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
 | VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
-| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) |
+| TTS(语音合成) | ✅LinkeraiTTS(灵犀流式) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) 或 👍AliyunStreamTTS(阿里云流式语音合成) |
 | Intent(意图识别) | function_call(函数调用) | function_call(函数调用) |
 | Memory(记忆功能) | mem_local_short(本地短期记忆） | mem_local_short（本地短期记忆） |
 
@@ -222,14 +235,15 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ---
 ## 功能清单 ✨
 ### 已实现 ✅
-
+![请参考-全模块安装架构图](docs/images/deploy2.png)
 | 功能模块 | 描述 |
 |:---:|:---|
-| 核心服务架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
+| 核心架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
-| 语音交互系统 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
+| 语音交互 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
-| 智能对话系统 | 支持多种LLM(大语言模型)，实现智能对话 |
+| 声纹识别 | 支持多用户声纹注册、管理和识别，与ASR并行处理，实时识别说话人身份并传递给LLM进行个性化回应 |
-| 视觉感知系统 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
+| 智能对话 | 支持多种LLM(大语言模型)，实现智能对话 |
-| 意图识别系统 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
+| 视觉感知 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
+| 意图识别 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
 | 记忆系统 | 支持本地短期记忆、mem0ai接口记忆，具备记忆总结功能 |
 | 工具调用 | 支持客户端IOT协议、客户MCP协议、服务端MCP协议、MCP接入点协议、自定义工具函数 |
 | 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理 |
@@ -257,7 +271,6 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 ---
 
 ## 本项目支持的平台/组件列表 📋
-
 ### LLM 语言模型
 
 | 使用方式 | 支持平台 | 免费平台 |
@@ -308,6 +321,14 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 ---
 
+### Voiceprint 声纹识别
+
+| 使用方式 | 支持平台 | 免费平台 |
+|:---:|:---:|:---:|
+| 本地使用 | 3D-Speaker | 3D-Speaker |
+
+---
+
 ### Memory 记忆存储
 
 |   类型   |      平台名称       | 使用方式 |   收费模式    | 备注 |

README_en.md

@@ -3,17 +3,17 @@
 <h1 align="center">Xiaozhi Backend Service xiaozhi-esp32-server</h1>
 
 <p align="center">
-This project provides backend services for the open-source smart hardware project
+This project is based on human-machine symbiotic intelligence theory and technology to develop intelligent terminal hardware and software systems<br/>providing backend services for the open-source intelligent hardware project
 <a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
 Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
-Helps you quickly set up your Xiaozhi server
+Supports MCP endpoints and voiceprint recognition
 </p>
 
 <p align="center">
 <a href="./README.md">中文</a>
 · <a href="./docs/FAQ.md">FAQ</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
-· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
+· <a href="./README.md#%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3">Deployment Docs</a>
 · <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
 </p>
 <p align="center">
@@ -37,41 +37,49 @@ Helps you quickly set up your Xiaozhi server
   </a>
 </p>
 
+<p align="center">
+By Professor Siyuan Liu Research and Development Group (South China University of Technology)
+</br>
+刘思源教授团队研发（华南理工大学）
+</br>
+<img src="./docs/images/hnlg.jpg" alt="South China University of Technology" width="50%">
+</p>
+
 ---
 
 ## Target Users 👥
 
-This project requires ESP32 hardware devices. If you have purchased ESP32-related hardware, successfully connected to Brother Xia's backend service, and want to set up your own `xiaozhi-esp32` backend service, then this project is perfect for you.
+This project requires ESP32 hardware devices to work. If you have purchased ESP32-related hardware, successfully connected to Brother Xia's deployed backend service, and want to build your own `xiaozhi-esp32` backend service independently, then this project is perfect for you.
 
-Want to see it in action? Check out these videos 🎥
+Want to see the usage effects? Click the videos below 🎥
 
 <table>
   <tr>
     <td>
         <a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
          <picture>
-           <img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
+           <img alt="Xiaozhi ESP32 connecting to own backend model" src="docs/images/demo1.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
          <picture>
-           <img alt="Custom voice" src="docs/images/demo2.png" />
+           <img alt="Custom voice timbre" src="docs/images/demo2.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
          <picture>
-           <img alt="Using Cantonese" src="docs/images/demo3.png" />
+           <img alt="Using Cantonese for communication" src="docs/images/demo3.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
          <picture>
-           <img alt="Control home appliances" src="docs/images/demo5.png" />
+           <img alt="Controlling home appliances" src="docs/images/demo5.png" />
          </picture>
         </a>
     </td>
@@ -87,14 +95,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV1Vy96YCE3R" target="_blank">
          <picture>
-           <img alt="Custom voice" src="docs/images/demo6.png" />
+           <img alt="Custom voice timbre" src="docs/images/demo6.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
          <picture>
-           <img alt="Play music" src="docs/images/demo7.png" />
+           <img alt="Playing music" src="docs/images/demo7.png" />
          </picture>
         </a>
     </td>
@@ -108,14 +116,14 @@ Want to see it in action? Check out these videos 🎥
     <td>
         <a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
          <picture>
-           <img alt="IOT command control" src="docs/images/demo9.png" />
+           <img alt="IOT command control devices" src="docs/images/demo9.png" />
          </picture>
         </a>
     </td>
     <td>
         <a href="https://www.bilibili.com/video/BV17LXWYvENb" target="_blank">
          <picture>
-           <img alt="News broadcast" src="docs/images/demo0.png" />
+           <img alt="News broadcasting" src="docs/images/demo0.png" />
          </picture>
         </a>
     </td>
@@ -131,7 +139,7 @@ Want to see it in action? Check out these videos 🎥
     <td>
       <a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
          <picture>
-           <img alt="Photo recognition" src="docs/images/demo12.png" />
+           <img alt="Photo recognition of objects" src="docs/images/demo12.png" />
          </picture>
         </a>
     </td>
@@ -143,20 +151,29 @@ Want to see it in action? Check out these videos 🎥
         </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
+         <picture>
+           <img alt="MCP endpoint" src="docs/images/demo13.png" />
+         </picture>
+        </a>
     </td>
     <td>
+        <a href="https://www.bilibili.com/video/BV1Exu3zqEDe" target="_blank">
+         <picture>
+           <img alt="Voiceprint recognition" src="docs/images/demo14.png" />
+         </picture>
+        </a>
     </td>
   </tr>
 </table>
 
 ---
 
-## Warning ⚠️
+## Warnings ⚠️
 
-1. This project is open-source software. This software has no commercial relationship with any third-party API service providers (including but not limited to speech recognition, large models, speech synthesis, and other platforms) and does not provide any form of guarantee for their service quality or financial security.
+1. This project is open-source software. This software has no commercial partnership with any third-party API service providers (including but not limited to speech recognition, large models, speech synthesis, and other platforms) that it interfaces with, and does not provide any form of guarantee for their service quality or financial security. It is recommended that users prioritize service providers with relevant business licenses and carefully read their service agreements and privacy policies. This software does not host any account keys, does not participate in fund flows, and does not bear the risk of recharge fund losses.
-It is recommended that users prioritize service providers with relevant business licenses and carefully read their service agreements and privacy policies. This software does not host any account keys, does not participate in fund transfers, and does not bear the risk of recharge fund losses.
 
-2. This project's functionality is not complete and has not passed network security testing. Please do not use it in production environments. If you deploy this project for learning in a public network environment, please ensure necessary protection measures are in place.
+2. The functionality of this project is not complete and has not passed network security assessment. Please do not use it in production environments. If you deploy this project for learning purposes in a public network environment, please ensure necessary protection measures are in place.
 
 ---
 
@@ -164,18 +181,19 @@ It is recommended that users prioritize service providers with relevant business
 
 ![Banners](docs/images/banner2.png)
 
-This project provides two deployment methods. Please choose according to your specific needs:
+This project provides two deployment methods. Please choose based on your specific needs:
 
 #### 🚀 Deployment Method Selection
-| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial | 
+| Deployment Method | Features | Applicable Scenarios | Deployment Docs | Configuration Requirements | Video Tutorials | 
 |---------|------|---------|---------|---------|---------|
-| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - | 
+| **Simplified Installation** | Intelligent dialogue, IOT, MCP, visual perception | Low-configuration environments, data stored in config files, no database required | [①Docker Version](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②Source Code Deployment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 2 cores 4GB if using `FunASR`, 2 cores 2GB if all APIs | - | 
-| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) | 
+| **Full Module Installation** | Intelligent dialogue, IOT, MCP endpoints, voiceprint recognition, visual perception, OTA, intelligent control console | Complete functionality experience, data stored in database |[①Docker Version](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②Source Code Deployment](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③Source Code Deployment Auto-Update Tutorial](./docs/dev-ops-integration.md) | 4 cores 8GB if using `FunASR`, 2 cores 4GB if all APIs| [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) | 
 
-> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
+
+> 💡 Note: Below is a test platform deployed with the latest code. You can burn and test if needed. Concurrent users: 6, data will be cleared daily.
 
 ```
-Control Panel Address: https://2662r3426b.vicp.fun
+Intelligent Control Console Address: https://2662r3426b.vicp.fun
 
 Service Test Tool: https://2662r3426b.vicp.fun/test/
 OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
@@ -184,74 +202,103 @@ Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
 
 #### 🚩 Configuration Description and Recommendations
 > [!Note]
-> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
+> This project provides two configuration schemes:
 > 
-> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
+> 1. `Entry Level Free Settings`: Suitable for personal and home use, all components use free solutions, no additional payment required.
+> 
+> 2. `Streaming Configuration`: Suitable for demonstrations, training, scenarios with more than 2 concurrent users, etc. Uses streaming processing technology for faster response speed and better experience.
+> 
+> Starting from version `0.5.2`, the project supports streaming configuration. Compared to earlier versions, response speed is improved by approximately `2.5 seconds`, significantly improving user experience.
 
-| Module Name | Entry Level Free Settings | Full Streaming Configuration |
+| Module Name | Entry Level Free Settings | Streaming Configuration |
-|---------|---------|------|
+|:---:|:---:|:---:|
-| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
+| ASR(Speech Recognition) | FunASR(Local) | 👍FunASRServer or 👍DoubaoStreamASR |
-| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
+| LLM(Large Model) | ChatGLMLLM(Zhipu glm-4-flash) | 👍DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
-| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
+| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | 👍QwenVLVLLM(Qwen qwen2.5-vl-3b-instructh) |
-| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
+| TTS(Speech Synthesis) | ✅LinkeraiTTS(Lingxi streaming) | 👍HuoshanDoubleStreamTTS(Volcano dual-stream speech synthesis) |
-| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
+| Intent(Intent Recognition) | function_call(Function calling) | function_call(Function calling) |
-| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
+| Memory(Memory function) | mem_local_short(Local short-term memory) | mem_local_short(Local short-term memory) |
+
+#### 🔧 Testing Tools
+This project provides the following testing tools to help you verify the system and choose suitable models:
+
+| Tool Name | Location | Usage Method | Function Description |
+|:---:|:---|:---:|:---:|
+| Audio Interaction Test Tool | main》xiaozhi-server》test》test_page.html | Open directly with Google Chrome | Tests audio playback and reception functions, verifies if Python-side audio processing is normal |
+| Model Response Test Tool 1 | main》xiaozhi-server》performance_tester.py | Execute `python performance_tester.py` | Tests response speed of three core modules: ASR(speech recognition), LLM(large model), TTS(speech synthesis) |
+| Model Response Test Tool 2 | main》xiaozhi-server》performance_tester_vllm.py | Execute `python performance_tester_vllm.py` | Tests VLLM(vision model) response speed |
+
+> 💡 Note: When testing model speed, only models with configured keys will be tested.
 
 ---
 ## Feature List ✨
 ### Implemented ✅
-
+![请参考-全模块安装架构图](docs/images/deploy2.png)
 | Feature Module | Description |
-|---------|------|
+|:---:|:---|
-| Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
+| Core Architecture | Based on WebSocket and HTTP servers, provides complete console management and authentication system |
-| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
+| Voice Interaction | Supports streaming ASR(speech recognition), streaming TTS(speech synthesis), VAD(voice activity detection), supports multi-language recognition and voice processing |
-| Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
+| Voiceprint Recognition | Supports multi-user voiceprint registration, management, and recognition, processes in parallel with ASR, real-time speaker identity recognition and passes to LLM for personalized responses |
-| Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
+| Intelligent Dialogue | Supports multiple LLM(large language models), implements intelligent dialogue |
-| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
+| Visual Perception | Supports multiple VLLM(vision large models), implements multimodal interaction |
-| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
+| Intent Recognition | Supports LLM intent recognition, Function Call function calling, provides plugin-based intent processing mechanism |
-| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
+| Memory System | Supports local short-term memory, mem0ai interface memory, with memory summarization functionality |
-| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
+| Tool Calling | Supports client IOT protocol, client MCP protocol, server MCP protocol, MCP endpoint protocol, custom tool functions |
-| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
+| Management Backend | Provides Web management interface, supports user management, system configuration, and device management |
+| Testing Tools | Provides performance testing tools, vision model testing tools, and audio interaction testing tools |
+| Deployment Support | Supports Docker deployment and local deployment, provides complete configuration file management |
+| Plugin System | Supports functional plugin extensions, custom plugin development, and plugin hot-loading |
 
-### In Development 🚧
+### Under Development 🚧
 
-To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
+To learn about specific development plan progress, [click here](https://github.com/users/xinnan-tech/projects/3)
 
 If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
 
 ---
 
 ## Product Ecosystem 👬
-Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
+Xiaozhi is an ecosystem. When using this product, you can also check out other excellent projects in this ecosystem
 
 | Project Name | Project Address | Project Description |
 |:---------------------|:--------|:--------|
-| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
+| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | An Android and iOS voice dialogue application based on xiaozhi-server, supporting real-time voice interaction and text dialogue.<br/>Currently a Flutter version, connecting iOS and Android platforms. |
-| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
+| Xiaozhi Desktop Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based AI client for beginners, allowing users to experience Xiaozhi AI functionality through code even without physical hardware conditions. |
-| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
+| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | Xiaozhi open-source backend service Java version is a Java-based open-source project.<br/>It includes frontend and backend services, aiming to provide users with a complete backend service solution. |
 
 ---
 
 ## Supported Platforms/Components List 📋
-
 ### LLM Language Models
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
+| OpenAI interface calls | Alibaba Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
-| ollama interface call | Ollama | - |
+| Ollama interface calls | Ollama | - |
-| dify interface call | Dify | - |
+| Dify interface calls | Dify | - |
-| fastgpt interface call | Fastgpt | - |
+| FastGPT interface calls | FastGPT | - |
-| coze interface call | Coze | - |
+| Coze interface calls | Coze | - |
 
-In fact, any LLM that supports openai interface calls can be integrated and used.
+In fact, any LLM that supports OpenAI interface calls can be integrated and used.
+
+---
+
+### VLLM Vision Models
+
+| Usage Method | Supported Platforms | Free Platforms |
+|:---:|:---:|:---:|
+| OpenAI interface calls | Alibaba Bailian, Zhipu ChatGLMVLLM | Zhipu ChatGLMVLLM |
+
+In fact, any VLLM that supports OpenAI interface calls can be integrated and used.
+
+---
 
 ### TTS Speech Synthesis
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| API Call | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
+| Interface calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS, Lingxi Streaming TTS | Lingxi Streaming TTS, EdgeTTS, CosyVoiceSiliconflow(partial) |
-| Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
+| Local services | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
 
 ---
 
@@ -259,7 +306,7 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:---:|:---------:|:----:|:----:|:--:|
-| VAD | SileroVAD | Local Usage | Free | |
+| VAD | SileroVAD | Local use | Free | |
 
 ---
 
@@ -267,8 +314,16 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Usage Method | Supported Platforms | Free Platforms |
 |:---:|:---:|:---:|
-| Local Usage | FunASR, SherpaASR | FunASR, SherpaASR |
+| Local use | FunASR, SherpaASR | FunASR, SherpaASR |
-| API Call | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+| Interface calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
+
+---
+
+### Voiceprint Recognition
+
+| Usage Method | Supported Platforms | Free Platforms |
+|:---:|:---:|:---:|
+| Local use | 3D-Speaker | 3D-Speaker |
 
 ---
 
@@ -276,8 +331,8 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:---------------:|:----:|:---------:|:--:|
-| Memory | mem0ai | API Call | 1000 calls/month quota | |
+| Memory | mem0ai | Interface calls | 1000 times/month quota | |
-| Memory | mem_local_short | Local Summary | Free | |
+| Memory | mem_local_short | Local summarization | Free | |
 
 ---
 
@@ -285,8 +340,8 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Type | Platform Name | Usage Method | Pricing Model | Notes |
 |:------:|:-------------:|:----:|:-------:|:---------------------:|
-| Intent | intent_llm | API Call | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
+| Intent | intent_llm | Interface calls | Based on LLM pricing | Recognizes intent through large models, strong generalization |
-| Intent | function_call | API Call | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
+| Intent | function_call | Interface calls | Based on LLM pricing | Completes intent through large model function calling, fast speed, good effect |
 
 ---
 
@@ -294,11 +349,11 @@ In fact, any LLM that supports openai interface calls can be integrated and used
 
 | Logo | Project/Company | Description |
 |:---:|:---:|:---|
-| <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
+| <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project is inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented on its basis |
-| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for establishing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing full-chain technical documentation support for this project |
+| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for formulating standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practice demonstrations for the Xiaozhi ecosystem; providing full-link technical documentation support for this project |
-| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing the function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
+| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function calling framework, MCP communication protocol, and plugin-based calling mechanism implementation code. Through standardized instruction scheduling system and dynamic expansion capabilities, it significantly improves the interaction efficiency and functional extensibility of frontend devices (IoT) |
-| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering the product user experience with their design experience serving over a thousand enterprises |
+| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, using their design practical experience serving over a thousand enterprises to empower this project's product user experience |
-| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening the visual system of this project, ensuring consistency and extensibility of the overall design style in multi-scenario applications |
+| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
 
 
 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">

docker-setup.sh

@@ -1,105 +1,414 @@
 #!/bin/sh
+# 脚本作者@VanillaNahida
 # 本文件是用于一键自动下载本项目所需文件，自动创建好目录
-# 所需条件（否则无法使用）：
+# 暂且只支持X86版本的Ubuntu系统，其他系统未测试
-# 1、请确保你的环境可以正常访问 GitHub 否则无法下载脚本
+
-#
+# 定义中断处理函数
-# 检测操作系统类型
+handle_interrupt() {
-case "$(uname -s)" in
+    echo ""
-    Linux*)     OS=Linux;;
+    echo "安装已被用户中断(Ctrl+C或Esc)"
-    Darwin*)    OS=Mac;;
+    echo "如需重新安装，请再次运行脚本"
-    CYGWIN*)    OS=Windows;;
+    exit 1
-    MINGW*)     OS=Windows;;
+}
-    MSYS*)      OS=Windows;;
+
-    *)          OS=UNKNOWN;;
+# 设置信号捕获，处理Ctrl+C
+trap handle_interrupt SIGINT
+
+# 处理Esc键
+# 保存终端设置
+old_stty_settings=$(stty -g)
+# 设置终端立即响应，不回显
+stty -icanon -echo min 1 time 0
+
+# 后台进程检测Esc键
+(while true; do
+    read -r key
+    if [[ $key == $'\e' ]]; then
+        # 检测到Esc键，触发中断处理
+        kill -SIGINT $$
+        break
+    fi
+done) &
+
+# 脚本结束时恢复终端设置
+trap 'stty "$old_stty_settings"' EXIT
+
+
+# 打印彩色字符画
+echo -e "\e[1;32m"  # 设置颜色为亮绿色
+cat << "EOF"
+脚本作者：@Bilibili 香草味的纳西妲喵
+ __      __            _  _  _            _   _         _      _      _        
+ \ \    / /           (_)| || |          | \ | |       | |    (_)    | |       
+  \ \  / /__ _  _ __   _ | || |  __ _    |  \| |  __ _ | |__   _   __| |  __ _ 
+   \ \/ // _` || '_ \ | || || | / _` |   | . ` | / _` || '_ \ | | / _` | / _` |
+    \  /| (_| || | | || || || || (_| |   | |\  || (_| || | | || || (_| || (_| |
+     \/  \__,_||_| |_||_||_||_| \__,_|   |_| \_| \__,_||_| |_||_| \__,_| \__,_|                                                                                                                                                                                                                               
+EOF
+echo -e "\e[0m"  # 重置颜色
+echo -e "\e[1;36m  小智服务端全量部署一键安装脚本 Ver 0.2 \e[0m\n"
+sleep 1
+
+
+
+# 检查并安装whiptail
+check_whiptail() {
+    if ! command -v whiptail &> /dev/null; then
+        echo "正在安装whiptail..."
+        apt update
+        apt install -y whiptail
+    fi
+}
+
+check_whiptail
+
+# 创建确认对话框
+whiptail --title "安装确认" --yesno "即将安装小智服务端，是否继续？" \
+  --yes-button "继续" --no-button "退出" 10 50
+
+# 根据用户选择执行操作
+case $? in
+  0)
+    ;;
+  1)
+    exit 1
+    ;;
 esac
 
-# 设置颜色（Windows CMD 不支持，但不影响使用）
+# 检查root权限
-if [ "$OS" = "Windows" ]; then
+if [ $EUID -ne 0 ]; then
-    GREEN=""
+    whiptail --title "权限错误" --msgbox "请使用root权限运行本脚本" 10 50
-    RED=""
-    NC=""
-else
-    GREEN='\033[0;32m'
-    RED='\033[0;31m'
-    NC='\033[0m'
-fi
-
-echo "${GREEN}开始安装小智服务端...${NC}"
-
-# 创建必要的目录
-echo "创建目录结构..."
-mkdir -p xiaozhi-server/data xiaozhi-server/models/SenseVoiceSmall
-cd xiaozhi-server || exit
-
-# 根据操作系统选择下载命令
-if [ "$OS" = "Windows" ]; then
-    DOWNLOAD_CMD="curl -L -o"
-    if ! command -v curl >/dev/null 2>&1; then
-        DOWNLOAD_CMD="powershell -Command Invoke-WebRequest -Uri"
-        DOWNLOAD_CMD_SUFFIX="-OutFile"
-    fi
-else
-    if command -v curl >/dev/null 2>&1; then
-        DOWNLOAD_CMD="curl -L -o"
-    elif command -v wget >/dev/null 2>&1; then
-        DOWNLOAD_CMD="wget -O"
-    else
-        echo "${RED}错误: 需要安装 curl 或 wget${NC}"
-        exit 1
-    fi
-fi
-
-# 下载语音识别模型
-echo "下载语音识别模型..."
-if [ "$DOWNLOAD_CMD" = "powershell -Command Invoke-WebRequest -Uri" ]; then
-    $DOWNLOAD_CMD "https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt" $DOWNLOAD_CMD_SUFFIX "models/SenseVoiceSmall/model.pt"
-else
-    $DOWNLOAD_CMD "models/SenseVoiceSmall/model.pt" "https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt"
-fi
-
-if [ $? -ne 0 ]; then
-    echo "${RED}模型下载失败。请手动从以下地址下载：${NC}"
-    echo "1. https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt"
-    echo "2. 百度网盘: https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg (提取码: qvna)"
-    echo "下载后请将文件放置在 models/SenseVoiceSmall/model.pt"
-fi
-
-# 下载配置文件
-echo "下载配置文件..."
-if [ "$DOWNLOAD_CMD" = "powershell -Command Invoke-WebRequest -Uri" ]; then
-    $DOWNLOAD_CMD "https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/main/main/xiaozhi-server/docker-compose.yml" $DOWNLOAD_CMD_SUFFIX "docker-compose.yml"
-    $DOWNLOAD_CMD "https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/main/main/xiaozhi-server/config.yaml" $DOWNLOAD_CMD_SUFFIX "data/.config.yaml"
-else
-    $DOWNLOAD_CMD "docker-compose.yml" "https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/main/main/xiaozhi-server/docker-compose.yml"
-    $DOWNLOAD_CMD "data/.config.yaml" "https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/main/main/xiaozhi-server/config.yaml"
-fi
-
-# 检查文件是否存在
-echo "检查文件完整性..."
-FILES_TO_CHECK="docker-compose.yml data/.config.yaml models/SenseVoiceSmall/model.pt"
-ALL_FILES_EXIST=true
-
-for FILE in $FILES_TO_CHECK; do
-    if [ ! -f "$FILE" ]; then
-        echo "${RED}错误: $FILE 不存在${NC}"
-        ALL_FILES_EXIST=false
-    fi
-done
-
-if [ "$ALL_FILES_EXIST" = false ]; then
-    echo "${RED}某些文件下载失败，请检查上述错误信息并手动下载缺失的文件。${NC}"
     exit 1
 fi
 
-echo "${GREEN}文件下载完成！${NC}"
+# 检查系统版本
-echo "请编辑 data/.config.yaml 文件配置你的API密钥。"
+if [ -f /etc/os-release ]; then
-echo "配置完成后，运行以下命令启动服务："
+    . /etc/os-release
-echo "${GREEN}docker-compose up -d${NC}"
+    if [ "$ID" != "debian" ] && [ "$ID" != "ubuntu" ]; then
-echo "查看日志请运行："
+        whiptail --title "系统错误" --msgbox "该脚本只支持Debian/Ubuntu系统执行" 10 60
-echo "${GREEN}docker logs -f xiaozhi-esp32-server${NC}"
+        exit 1
+    fi
+else
+    whiptail --title "系统错误" --msgbox "无法确定系统版本，该脚本只支持Debian/Ubuntu系统执行" 10 60
+    exit 1
+fi
 
-# 提示用户编辑配置文件
+# 下载配置文件函数
-echo "\n${RED}重要提示：${NC}"
+check_and_download() {
-echo "1. 请确保编辑 data/.config.yaml 文件，配置必要的API密钥"
+    local filepath=$1
-echo "2. 特别是 ChatGLM 和 mem0ai 的密钥必须配置"
+    local url=$2
-echo "3. 配置完成后再启动 docker 服务"
+    if [ ! -f "$filepath" ]; then
+        if ! curl -fL --progress-bar "$url" -o "$filepath"; then
+            whiptail --title "错误" --msgbox "${filepath}文件下载失败" 10 50
+            exit 1
+        fi
+    else
+        echo "${filepath}文件已存在，跳过下载"
+    fi
+}
+
+# 检查是否已安装
+check_installed() {
+    # 检查目录是否存在且非空
+    if [ -d "/opt/xiaozhi-server/" ] && [ "$(ls -A /opt/xiaozhi-server/)" ]; then
+        DIR_CHECK=1
+    else
+        DIR_CHECK=0
+    fi
+    
+    # 检查容器是否存在
+    if docker inspect xiaozhi-esp32-server > /dev/null 2>&1; then
+        CONTAINER_CHECK=1
+    else
+        CONTAINER_CHECK=0
+    fi
+    
+    # 两次检查都通过
+    if [ $DIR_CHECK -eq 1 ] && [ $CONTAINER_CHECK -eq 1 ]; then
+        return 0  # 已安装
+    else
+        return 1  # 未安装
+    fi
+}
+
+# 更新相关
+if check_installed; then
+    if whiptail --title "已安装检测" --yesno "检测到小智服务端已安装，是否进行升级？" 10 60; then
+        # 用户选择升级，执行清理操作
+        echo "开始升级操作..."
+        
+        # 停止并移除所有docker-compose服务
+        docker compose -f /opt/xiaozhi-server/docker-compose_all.yml down
+        
+        # 停止并删除特定容器（考虑容器可能不存在的情况）
+        containers=(
+            "xiaozhi-esp32-server"
+            "xiaozhi-esp32-server-web"
+            "xiaozhi-esp32-server-db"
+            "xiaozhi-esp32-server-redis"
+        )
+        
+        for container in "${containers[@]}"; do
+            if docker ps -a --format '{{.Names}}' | grep -q "^${container}$"; then
+                docker stop "$container" >/dev/null 2>&1 && \
+                docker rm "$container" >/dev/null 2>&1 && \
+                echo "成功移除容器: $container"
+            else
+                echo "容器不存在，跳过: $container"
+            fi
+        done
+        
+        # 删除特定镜像（考虑镜像可能不存在的情况）
+        images=(
+            "ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:server_latest"
+            "ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:web_latest"
+        )
+        
+        for image in "${images[@]}"; do
+            if docker images --format '{{.Repository}}:{{.Tag}}' | grep -q "^${image}$"; then
+                docker rmi "$image" >/dev/null 2>&1 && \
+                echo "成功删除镜像: $image"
+            else
+                echo "镜像不存在，跳过: $image"
+            fi
+        done
+        
+        echo "所有清理操作完成"
+        
+        # 备份原有配置文件
+        mkdir -p /opt/xiaozhi-server/backup/
+        if [ -f /opt/xiaozhi-server/data/.config.yaml ]; then
+            cp /opt/xiaozhi-server/data/.config.yaml /opt/xiaozhi-server/backup/.config.yaml
+            echo "已备份原有配置文件到 /opt/xiaozhi-server/backup/.config.yaml"
+        fi
+        
+        # 下载最新版配置文件
+        check_and_download "/opt/xiaozhi-server/docker-compose_all.yml" "https://ghfast.top/https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/refs/heads/main/main/xiaozhi-server/docker-compose_all.yml"
+        check_and_download "/opt/xiaozhi-server/data/.config.yaml" "https://ghfast.top/https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/refs/heads/main/main/xiaozhi-server/config_from_api.yaml"
+        
+        # 启动Docker服务
+        echo "开始启动最新版本服务..."
+        # 升级完成后标记，跳过后续下载步骤
+        UPGRADE_COMPLETED=1
+        docker compose -f /opt/xiaozhi-server/docker-compose_all.yml up -d
+    else
+          whiptail --title "跳过升级" --msgbox "已取消升级，将继续使用当前版本。" 10 50
+          # 跳过升级，继续执行后续安装流程
+    fi
+fi
+
+
+# 检查curl安装
+if ! command -v curl &> /dev/null; then
+    echo "------------------------------------------------------------"
+    echo "未检测到curl，正在安装..."
+    apt update
+    apt install -y curl
+else
+    echo "------------------------------------------------------------"
+    echo "curl已安装，跳过安装步骤"
+fi
+
+# 检查Docker安装
+if ! command -v docker &> /dev/null; then
+    echo "------------------------------------------------------------"
+    echo "未检测到Docker，正在安装..."
+    
+    # 使用国内镜像源替代官方源
+    DISTRO=$(lsb_release -cs)
+    MIRROR_URL="https://mirrors.aliyun.com/docker-ce/linux/ubuntu"
+    GPG_URL="https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg"
+    
+    # 安装基础依赖
+    apt update
+    apt install -y apt-transport-https ca-certificates curl software-properties-common gnupg
+    
+    # 创建密钥目录并添加国内镜像源密钥
+    mkdir -p /etc/apt/keyrings
+    curl -fsSL "$GPG_URL" | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+    
+    # 添加国内镜像源
+    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] $MIRROR_URL $DISTRO stable" \
+        > /etc/apt/sources.list.d/docker.list
+    
+    # 添加备用官方源密钥（避免国内源密钥验证失败）
+    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7EA0A9C3F273FCD8 2>/dev/null || \
+    echo "警告：部分密钥添加失败，继续尝试安装..."
+    
+    # 安装Docker
+    apt update
+    apt install -y docker-ce docker-ce-cli containerd.io
+    
+    # 启动服务
+    systemctl start docker
+    systemctl enable docker
+    
+    # 检查是否安装成功
+    if docker --version; then
+        echo "------------------------------------------------------------"
+        echo "Docker安装完成！"
+    else
+        whiptail --title "错误" --msgbox "Docker安装失败，请检查日志。" 10 50
+        exit 1
+    fi
+else
+    echo "Docker已安装，跳过安装步骤"
+fi
+
+# Docker镜像源配置
+MIRROR_OPTIONS=(
+    "1" "轩辕镜像 (推荐)"
+    "2" "腾讯云镜像源"
+    "3" "中科大镜像源"
+    "4" "网易163镜像源"
+    "5" "华为云镜像源"
+    "6" "阿里云镜像源"
+    "7" "自定义镜像源"
+    "8" "跳过配置"
+)
+
+MIRROR_CHOICE=$(whiptail --title "选择Docker镜像源" --menu "请选择要使用的Docker镜像源" 20 60 10 \
+"${MIRROR_OPTIONS[@]}" 3>&1 1>&2 2>&3) || {
+    echo "用户取消选择，退出脚本"
+    exit 1
+}
+
+case $MIRROR_CHOICE in
+    1) MIRROR_URL="https://docker.xuanyuan.me" ;; 
+    2) MIRROR_URL="https://mirror.ccs.tencentyun.com" ;; 
+    3) MIRROR_URL="https://docker.mirrors.ustc.edu.cn" ;; 
+    4) MIRROR_URL="https://hub-mirror.c.163.com" ;; 
+    5) MIRROR_URL="https://05f073ad3c0010ea0f4bc00b7105ec20.mirror.swr.myhuaweicloud.com" ;; 
+    6) MIRROR_URL="https://registry.aliyuncs.com" ;; 
+    7) MIRROR_URL=$(whiptail --title "自定义镜像源" --inputbox "请输入完整的镜像源URL:" 10 60 3>&1 1>&2 2>&3) ;; 
+    8) MIRROR_URL="" ;; 
+esac
+
+if [ -n "$MIRROR_URL" ]; then
+    mkdir -p /etc/docker
+    if [ -f /etc/docker/daemon.json ]; then
+        cp /etc/docker/daemon.json /etc/docker/daemon.json.bak
+    fi
+    cat > /etc/docker/daemon.json <<EOF
+{
+    "dns": ["8.8.8.8", "114.114.114.114"],
+    "registry-mirrors": ["$MIRROR_URL"]
+}
+EOF
+    whiptail --title "配置成功" --msgbox "已成功添加镜像源: $MIRROR_URL\n请按Enter键重启Docker服务并继续..." 12 60
+    echo "------------------------------------------------------------"
+    echo "开始重启Docker服务..."
+    systemctl restart docker.service
+fi
+
+# 创建安装目录
+echo "------------------------------------------------------------"
+echo "开始创建安装目录..."
+# 检查并创建数据目录
+if [ ! -d /opt/xiaozhi-server/data ]; then
+    mkdir -p /opt/xiaozhi-server/data
+    echo "已创建数据目录: /opt/xiaozhi-server/data"
+else
+    echo "目录xiaozhi-server/data已存在，跳过创建"
+fi
+
+# 检查并创建模型目录
+if [ ! -d /opt/xiaozhi-server/models/SenseVoiceSmall ]; then
+    mkdir -p /opt/xiaozhi-server/models/SenseVoiceSmall
+    echo "已创建模型目录: /opt/xiaozhi-server/models/SenseVoiceSmall"
+else
+    echo "目录xiaozhi-server/models/SenseVoiceSmall已存在，跳过创建"
+fi
+
+echo "------------------------------------------------------------"
+echo "开始下载语音识别模型"
+# 下载模型文件
+MODEL_PATH="/opt/xiaozhi-server/models/SenseVoiceSmall/model.pt"
+if [ ! -f "$MODEL_PATH" ]; then
+    (
+    for i in {1..20}; do
+        echo $((i*5))
+        sleep 0.5
+    done
+    ) | whiptail --title "下载中" --gauge "开始下载语音识别模型..." 10 60 0
+    curl -fL --progress-bar https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt -o "$MODEL_PATH" || {
+        whiptail --title "错误" --msgbox "model.pt文件下载失败" 10 50
+        exit 1
+    }
+else
+    echo "model.pt文件已存在，跳过下载"
+fi
+
+# 如果不是升级完成，才执行下载
+if [ -z "$UPGRADE_COMPLETED" ]; then
+    check_and_download "/opt/xiaozhi-server/docker-compose_all.yml" "https://ghfast.top/https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/refs/heads/main/main/xiaozhi-server/docker-compose_all.yml"
+    check_and_download "/opt/xiaozhi-server/data/.config.yaml" "https://ghfast.top/https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/refs/heads/main/main/xiaozhi-server/config_from_api.yaml"
+fi
+
+# 启动Docker服务
+(
+echo "------------------------------------------------------------"
+echo "正在拉取Docker镜像..."
+echo "这可能需要几分钟时间，请耐心等待"
+docker compose -f /opt/xiaozhi-server/docker-compose_all.yml up -d
+
+if [ $? -ne 0 ]; then
+    whiptail --title "错误" --msgbox "Docker服务启动失败，请尝试更换镜像源后重新执行本脚本" 10 60
+    exit 1
+fi
+
+echo "------------------------------------------------------------"
+echo "正在检查服务启动状态..."
+TIMEOUT=300
+START_TIME=$(date +%s)
+while true; do
+    CURRENT_TIME=$(date +%s)
+    if [ $((CURRENT_TIME - START_TIME)) -gt $TIMEOUT ]; then
+        whiptail --title "错误" --msgbox "服务启动超时，未在指定时间内找到预期日志内容" 10 60
+        exit 1
+    fi
+    
+    if docker logs xiaozhi-esp32-server-web 2>&1 | grep -q "Started AdminApplication in"; then
+        break
+    fi
+    sleep 1
+done
+
+    echo "服务端启动成功！正在完成配置..."
+    echo "正在启动服务..."
+    docker compose -f docker-compose_all.yml up -d
+    echo "服务启动完成！"
+)
+
+# 密钥配置
+
+# 获取服务器公网地址
+PUBLIC_IP=$(hostname -I | awk '{print $1}')
+whiptail --title "配置服务器密钥" --msgbox "请使用浏览器，访问下方链接，打开智控台并注册账号: \n\n内网地址：http://127.0.0.1:8002/\n公网地址：http://$PUBLIC_IP:8002/ (若是云服务器请在服务器安全组放行端口 8000 8001 8002)。\n\n注册的第一个用户即是超级管理员，以后注册的用户都是普通用户。普通用户只能绑定设备和配置智能体; 超级管理员可以进行模型管理、用户管理、参数配置等功能。\n\n注册好后请按Enter键继续" 18 70
+SECRET_KEY=$(whiptail --title "配置服务器密钥" --inputbox "请使用超级管理员账号登录智控台\n内网地址：http://127.0.0.1:8002/\n公网地址：http://$PUBLIC_IP:8002/\n在顶部菜单 参数字典 → 参数管理 找到参数编码: server.secret (服务器密钥) \n复制该参数值并输入到下面输入框\n\n请输入密钥(留空则跳过配置):" 15 60 3>&1 1>&2 2>&3)
+
+if [ -n "$SECRET_KEY" ]; then
+    python3 -c "
+import sys, yaml; 
+config_path = '/opt/xiaozhi-server/data/.config.yaml'; 
+with open(config_path, 'r') as f: 
+    config = yaml.safe_load(f) or {}; 
+config['manager-api'] = {'url': 'http://xiaozhi-esp32-server-web:8002/xiaozhi', 'secret': '$SECRET_KEY'}; 
+with open(config_path, 'w') as f: 
+    yaml.dump(config, f); 
+"
+    docker restart xiaozhi-esp32-server
+fi
+
+# 获取并显示地址信息
+LOCAL_IP=$(hostname -I | awk '{print $1}')
+WEBSOCKET_ADDR=$(docker logs xiaozhi-esp32-server 2>&1 | tac | grep -m 1 -E -o "ws://[^ ]+")
+VISION_ADDR=$(docker logs xiaozhi-esp32-server 2>&1 | tac | grep -m 1 "视觉" | grep -m 1 -E -o "http://[^ ]+")
+
+whiptail --title "安装完成！" --msgbox "\
+服务端相关地址如下：\n\
+管理后台访问地址: http://$LOCAL_IP:8002\n\
+OTA 地址: http://$LOCAL_IP:8002/xiaozhi/ota/\n\
+视觉分析接口地址: $VISION_ADDR\n\
+WebSocket 地址: $WEBSOCKET_ADDR\n\
+\n安装完毕！感谢您的使用！\n按Enter键退出..." 16 70

docs/Deployment.md

@@ -8,45 +8,13 @@ docker镜像已支持x86架构、arm64架构的CPU，支持在国产操作系统
 
 如果您的电脑还没安装docker，可以按照这里的教程安装：[docker安装](https://www.runoob.com/docker/ubuntu-docker-install.html)
 
-如果你已经安装好docker，你可以[1.1使用懒人脚本](#11-懒人脚本)自动帮你下载所需的文件和配置文件，你可以使用docker[1.2手动部署](#12-手动部署)。
+安装好docker后，进继续。
 
-### 1.1 懒人脚本
+### 1.1 手动部署
-
-你可以使用以下命令一键下载并执行部署脚本：
-请确保你的环境可以正常访问 GitHub 否则无法下载脚本。
-```bash
-curl -L -o docker-setup.sh https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/main/docker-setup.sh
-```
-
-如果您的电脑是windows系统，请使用使用 Git Bash、WSL、PowerShell 或 CMD 运行以下命令：
-```bash
-# Git Bash 或 WSL
-sh docker-setup.sh
-# PowerShell 或 CMD
-.\docker-setup.sh
-```
-
-如果您的电脑是linux 或者 macos 系统，请使用终端运行以下命令：
-```bash
-chmod +x docker-setup.sh
-./docker-setup.sh
-```
-
-脚本会自动完成以下操作：
-> 1. 创建必要的目录结构
-> 2. 下载语音识别模型
-> 3. 下载配置文件
-> 4. 检查文件完整性
->
-> 执行完成后，请按照提示配置 API 密钥。
-
-当你一切顺利完成以上操作后，继续操作[配置项目文件](#2-配置项目文件)
-
-### 1.2 手动部署
 
 如果懒人脚本无法正常运行，请按本章节1.2进行手动部署。
 
-#### 1.2.1 创建目录
+#### 1.1.1 创建目录
 
 安装完后，你需要为这个项目找一个安放配置文件的目录，例如我们可以新建一个文件夹叫`xiaozhi-server`。
 
@@ -61,18 +29,18 @@ xiaozhi-server
      ├─ SenseVoiceSmall
 ```
 
-#### 1.2.2 下载语音识别模型文件
+#### 1.1.2 下载语音识别模型文件
 
 你需要下载语音识别的模型文件，因为本项目的默认语音识别用的是本地离线语音识别方案。可通过这个方式下载
 [跳转到下载语音识别模型文件](#模型文件)
 
 下载完后，回到本教程。
 
-#### 1.2.3 下载配置文件
+#### 1.1.3 下载配置文件
 
 你需要下载两个配置文件：`docker-compose.yaml` 和 `config.yaml`。需要从项目仓库下载这两个文件。
 
-##### 1.2.3.1 下载 docker-compose.yaml
+##### 1.1.3.1 下载 docker-compose.yaml
 
 用浏览器打开[这个链接](../main/xiaozhi-server/docker-compose.yml)。
 
@@ -81,7 +49,7 @@ xiaozhi-server
 
 下载完后，回到本教程继续往下。
 
-##### 1.2.3.2 创建 config.yaml
+##### 1.1.3.2 创建 config.yaml
 
 用浏览器打开[这个链接](../main/xiaozhi-server/config.yaml)。
 
@@ -260,7 +228,7 @@ LLM:
 文件放在`models/SenseVoiceSmall`
 目录下。下面两个下载路线任选一个。
 
-- 线路一：阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
+- 线路一：阿里魔搭下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
 - 线路二：百度网盘下载[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna) 提取码:
   `qvna`
 

docs/Deployment_all.md

@@ -7,7 +7,32 @@ docker镜像已支持x86架构、arm64架构的CPU，支持在国产操作系统
 
 如果您的电脑还没安装docker，可以按照这里的教程安装：[docker安装](https://www.runoob.com/docker/ubuntu-docker-install.html)
 
-#### 1.1 创建目录
+docker 安装全模块有两种方式，你可以[1.1使用懒人脚本](#1.1 懒人脚本)（作者[@VanillaNahida](https://github.com/VanillaNahida)）自动帮你下载所需的文件和配置文件，你可以使用[1.2手动部署](#1.2 手动部署)从零搭建。
+
+### 1.1 懒人脚本
+
+你可以使用以下命令一键安装全模块版小智服务端：  
+> [!NOTE]  
+> 暂且只支持Ubuntu服务器一键部署，其他系统未尝试，可能会有一些奇怪的bug
+
+使用SSH工具连接到服务器，以root权限执行如下脚本
+```bash
+sudo bash -c "$(wget -qO- https://ghfast.top/https://raw.githubusercontent.com/xinnan-tech/xiaozhi-esp32-server/main/docker-setup.sh)"
+```
+
+脚本会自动完成以下操作：
+> 1. 安装Docker
+> 2. 配置镜像源
+> 3. 下载/拉取镜像
+> 4. 下载语音识别模型文件
+> 5. 引导配置服务端
+>
+
+执行完成后简单配置后，再参照[4. 运行程序](#4. 运行程序)和[5.重启xiaozhi-esp32-server](#5.重启xiaozhi-esp32-server)里提到的最重要的3件事情，完成3这三项配置后即可使用。
+
+### 1.2 手动部署
+
+#### 1.2.1 创建目录
 
 安装完后，你需要为这个项目找一个安放配置文件的目录，例如我们可以新建一个文件夹叫`xiaozhi-server`。
 
@@ -22,22 +47,22 @@ xiaozhi-server
      ├─ SenseVoiceSmall
 ```
 
-#### 1.2 下载语音识别模型文件
+#### 1.2.2 下载语音识别模型文件
 
 本项目语音识别模型，默认使用`SenseVoiceSmall`模型，进行语音转文字。因为模型较大，需要独立下载，下载后把`model.pt`
 文件放在`models/SenseVoiceSmall`
 目录下。下面两个下载路线任选一个。
 
-- 线路一：阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
+- 线路一：阿里魔搭下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
 - 线路二：百度网盘下载[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna) 提取码:
   `qvna`
 
 
-#### 1.3 下载配置文件
+#### 1.2.3 下载配置文件
 
 你需要下载两个配置文件：`docker-compose_all.yaml` 和 `config_from_api.yaml`。需要从项目仓库下载这两个文件。
 
-##### 1.3.1 下载 docker-compose_all.yaml
+##### 1.2.3.1 下载 docker-compose_all.yaml
 
 用浏览器打开[这个链接](../main/xiaozhi-server/docker-compose_all.yml)。
 
@@ -48,7 +73,7 @@ xiaozhi-server
 
 下载完后，回到本教程继续往下。
 
-##### 1.3.2 下载 config_from_api.yaml
+##### 1.2.3.2 下载 config_from_api.yaml
 
 用浏览器打开[这个链接](../main/xiaozhi-server/config_from_api.yaml)。
 
@@ -179,12 +204,12 @@ docker logs -f xiaozhi-esp32-server
 
 OTA接口：
 ```
-http://你电脑局域网的ip:8002/xiaozhi/ota/
+http://你宿主机局域网的ip:8002/xiaozhi/ota/
 ```
 
 Websocket接口：
 ```
-ws://你电脑局域网的ip:8000/xiaozhi/v1/
+ws://你宿主机的ip:8000/xiaozhi/v1/
 ```
 
 ### 第三件重要的事情
@@ -358,7 +383,7 @@ pip install -r requirements.txt
 文件放在`models/SenseVoiceSmall`
 目录下。下面两个下载路线任选一个。
 
-- 线路一：阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
+- 线路一：阿里魔搭下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
 - 线路二：百度网盘下载[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna) 提取码:
   `qvna`
 

docs/FAQ.md

@@ -112,7 +112,17 @@ VAD:
 
 参考教程[视觉模型使用指南](./mcp-vision-integration.md)
 
-### 10、更多问题，可联系我们反馈 💬
+### 10、如何开启MCP接入点 🔧
+
+1、先参考教程[MCP 接入点部署使用指南](./mcp-endpoint-enable.md)
+
+2、再参考教程[MCP 接入点使用指南](./mcp-endpoint-integration.md)
+
+### 12、如何开启声纹识别 🔊
+
+参考教程[声纹识别启用指南](./voiceprint-integration.md)
+
+### 13、更多问题，可联系我们反馈 💬
 
 可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
 

docs/dev-ops-integration.md

@@ -4,6 +4,8 @@
 
 本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。
 
+教程可参考B站博主`毕乐labs`发布的视频教程：[《开源小智服务器xiaozhi-server自动更新以及最新版本MCP接入点配置保姆教程》](https://www.bilibili.com/video/BV15H37zHE7Q)
+
 # 开始条件
 - 你的电脑/服务器是linux操作系统
 - 你已经跑通了整个流程
@@ -40,6 +42,9 @@ git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-serve
 
 此刻你需要把`model.pt`文件复制到新的目录去，你可以这样
 ```
+# 创建需要的目录
+mkdir -p /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/
+
 cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
 cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
 ```
@@ -161,3 +166,11 @@ tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server
 
 # 注意事项
 测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)
+
+## 常见问题
+
+### 1、为什么没有见到8001端口？
+回答：8001是开发环境使用的，用于运行前端的端口。如果你是服务器部署，不建议使用`npm run serve`启动8001端口运行前端，而是像本教程一样编译成html文件，然后使用nginx来管理访问。
+
+### 2、每次更新需要更新手动SQL语句吗？
+回答：不需要，因为项目使用**Liquibase**管理数据库版本，会自动执行新的sql脚本。

docs/mcp-endpoint-enable.md

@@ -0,0 +1,125 @@
+# MCP 接入点部署使用指南
+
+本教程包含3个部分
+- 1、如何部署MCP接入点这个服务
+- 2、全模块部署时，怎么配置MCP接入点
+- 3、单模块部署时，怎么配置MCP接入点
+
+# 1、如何部署MCP接入点这个服务
+
+## 第一步，下载mcp接入点项目源码
+
+浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server)
+
+打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。
+
+点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`mcp-endpoint-server-main`
+你需要把它重命名成`mcp-endpoint-server`。
+
+## 第二步，启动程序
+这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法
+
+```
+# 进入本项目源码根目录
+cd mcp-endpoint-server
+
+# 清除缓存
+docker compose -f docker-compose.yml down
+docker stop mcp-endpoint-server
+docker rm mcp-endpoint-server
+docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest
+
+# 启动docker容器
+docker compose -f docker-compose.yml up -d
+# 查看日志
+docker logs -f mcp-endpoint-server
+```
+
+此时，日志里会输出类似以下的日志
+```
+250705 INFO-=====下面的地址分别是智控台/单模块MCP接入点地址====
+250705 INFO-智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc
+250705 INFO-单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def
+250705 INFO-=====请根据具体部署选择使用，请勿泄露给任何人======
+```
+
+请你把两个接口地址复制出来：
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么
+原来我的接口地址
+```
+智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc
+单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def
+```
+就要改成
+```
+智控台MCP参数配置: http://192.168.1.25:8004/mcp_endpoint/health?key=abc
+单模块部署MCP接入点: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def
+```
+
+改好后，请使用浏览器直接访问`智控台MCP参数配置`。当浏览器出现类似这样的代码，说明是成功了。
+```
+{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"}
+```
+
+请你保留好上面两个`接口地址`，下一步要用到。
+
+# 2、全模块部署时，怎么配置MCP接入点
+
+如果你是全模块部署，使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。
+
+然后搜索参数`server.mcp_endpoint`，此时，它的值应该是`null`值。
+点击修改按钮，把上一步得来的`智控台MCP参数配置`粘贴到`参数值`里。然后保存。
+
+如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问mcp接入点，很大概率是网络防火墙，或者没有填写正确的局域网ip。
+
+# 3、单模块部署时，怎么配置MCP接入点
+
+如果你是单模块部署，找到你的配置文件`data/.config.yaml`。
+在配置文件搜索`mcp_endpoint`，如果没有找到，你就增加`mcp_endpoint`配置。类似我是就是这样
+```
+server:
+  websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  http_port: 8002
+log:
+  log_level: INFO
+
+# 此处可能还更多配置..
+
+mcp_endpoint: 你的接入点websocket地址
+```
+这时，请你把`如何部署MCP接入点这个服务`中得到的`单模块部署MCP接入点` 粘贴到 `mcp_endpoint`中。类似这样
+
+```
+server:
+  websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
+  http_port: 8002
+log:
+  log_level: INFO
+
+# 此处可能还更多配置
+
+mcp_endpoint: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def
+```
+
+配置好后，启动单模块会输出如下的日志。
+```
+250705[__main__]-INFO-初始化组件: vad成功 SileroVAD
+250705[__main__]-INFO-初始化组件: asr成功 FunASRServer
+250705[__main__]-INFO-OTA接口是          http://192.168.1.25:8002/xiaozhi/ota/
+250705[__main__]-INFO-视觉分析接口是     http://192.168.1.25:8002/mcp/vision/explain
+250705[__main__]-INFO-mcp接入点是        ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
+250705[__main__]-INFO-Websocket地址是    ws://192.168.1.25:8000/xiaozhi/v1/
+250705[__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问=======
+250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+250705[__main__]-INFO-=============================================================
+```
+
+如上，如果能输出类似的`mcp接入点是`中`ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`说明配置成功了。
+

docs/mcp-endpoint-integration.md

@@ -1,92 +1,36 @@
-# MCP 接入点部署使用指南
+# MCP 接入点使用指南
 
-本教程包含2个部分
+本教程以虾哥开源的mcp计算器功能为示例，介绍如何将自己自定义的mcp服务接入到自己的接入点里。
-- 1、如何开启mcp接入点
-- 2、如何为智能体接入一个简单的mcp功能，如计算器功能
 
-部署的前提条件：
+本教程的前提是，你的`xiaozhi-server`已经启用了mcp接入点功能，如果你还没启用，可以先根据[这个教程](./mcp-endpoint-enable.md)启用。
-- 1、你已经部署了全模块，因为mcp接入点需要全模块中的智控台功能
-- 2、你想在不修改xiaozhi-server项目的前提下，扩展小智的功能
-
-# 如何开启mcp接入点
-
-## 第一步，下载mcp接入点项目源码
-
-浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server)
-
-打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。
-
-点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`mcp-endpoint-server-main`
-你需要把它重命名成`mcp-endpoint-server`。
-
-## 第二步，启动程序
-这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法
-
-```
-# 进入本项目源码根目录
-cd mcp-endpoint-server
-
-# 清除缓存
-docker compose -f docker-compose.yml down
-docker stop mcp-endpoint-server
-docker rm mcp-endpoint-server
-docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest
-
-# 启动docker容器
-docker compose -f docker-compose.yml up -d
-# 查看日志
-docker logs -f mcp-endpoint-server
-```
-
-此时，日志里会输出类似以下的日志
-```
-======================================================
-接口地址: http://172.1.1.1:8004/mcp_endpoint/health?key=xxxx
-=======上面的地址是MCP接入点地址，请勿泄露给任何人============
-```
-
-请你把接口地址复制出来：
-
-由于你是docker部署，切不可直接使用上面的地址！
-
-由于你是docker部署，切不可直接使用上面的地址！
-
-由于你是docker部署，切不可直接使用上面的地址！
-
-你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么
-原来我的接口地址
-```
-http://172.1.1.1:8004/mcp_endpoint/health?key=xxxx
-```
-就要改成
-```
-http://192.168.1.25:8004/mcp_endpoint/health?key=xxxx
-```
-
-改好后，请使用浏览器直接访问这个接口。当浏览器出现类似这样的代码，说明是成功了。
-```
-{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"}
-```
-
-请你保留好这个`接口地址`，下一步要用到。
-
-## 第三步，配置智控台
-
-使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。
-
-然后搜索参数`server.mcp_endpoint`，此时，它的值应该是`null`值。
-点击修改按钮，把上一步得来的`接口地址`粘贴到`参数值`里。然后保存。
-
-如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问mcp接入点，很大概率是网络防火墙，或者没有填写正确的局域网ip。
 
 # 如何为智能体接入一个简单的mcp功能，如计算器功能
 
-如果以上步骤顺利，你可以进入智能体管理，点击`配置角色`，在`意图识别`的右边，有一个`编辑功能`的按钮。
+### 如果你是全模块部署
+如果你是全模块部署，你可以进入智控台，智能体管理，点击`配置角色`，在`意图识别`的右边，有一个`编辑功能`的按钮。
 
 点击这个按钮。在弹出的页面里，位于底部，会有`MCP接入点`，正常来说，会显示这个智能体的`MCP接入点地址`，接下来，我们来给这个智能体扩展一个基于MCP技术的计算器的功能。
 
 这个`MCP接入点地址`很重要，你等一下会用到。
 
+### 如果你是单模块部署
+如果你是单模块部署，且你已经在配置文件里配置了MCP接入点地址，那么正常来说，单模块部署启动的时候，会输出如下的日志。
+```
+250705[__main__]-INFO-初始化组件: vad成功 SileroVAD
+250705[__main__]-INFO-初始化组件: asr成功 FunASRServer
+250705[__main__]-INFO-OTA接口是          http://192.168.1.25:8002/xiaozhi/ota/
+250705[__main__]-INFO-视觉分析接口是     http://192.168.1.25:8002/mcp/vision/explain
+250705[__main__]-INFO-mcp接入点是        ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
+250705[__main__]-INFO-Websocket地址是    ws://192.168.1.25:8000/xiaozhi/v1/
+250705[__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问=======
+250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
+250705[__main__]-INFO-=============================================================
+```
+
+如上，输出`mcp接入点是`中`ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`就是你的`MCP接入点地址`。
+
+这个`MCP接入点地址`很重要，你等一下会用到。
+
 ## 第一步 下载虾哥MCP计算器项目代码
 
 浏览器打开虾哥写的[计算器项目](https://github.com/78/mcp-calculator)，
@@ -114,13 +58,13 @@ pip install -r requirements.txt
 
 例如我的智能体的mcp地址是
 ```
-ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc
+ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
 ```
 
 开始输入命令
 
 ```bash
-export MCP_ENDPOINT=ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc
+export MCP_ENDPOINT=ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
 ```
 
 输入完后，启动程序
@@ -129,6 +73,22 @@ export MCP_ENDPOINT=ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc
 python mcp_pipe.py calculator.py
 ```
 
+### 如果你是智控台部署
+如果你是智控台部署，启动完后，你再进入智控台，点击刷新MCP的接入状态，就会看到你扩展的功能列表了。
 
-启动完后，你再进入智控台，点击刷新MCP的接入状态，就会看到你扩展的功能列表了。
+### 如果你是单模块部署
+如果你是单模块部署，当设备连接后，会输出类似的日志，说明成功了
 
+```
+250705 -INFO-正在初始化MCP接入点: wss://2662r3426b.vicp.fun/mcp_e 
+250705 -INFO-发送MCP接入点初始化消息
+250705 -INFO-MCP接入点连接成功
+250705 -INFO-MCP接入点初始化成功
+250705 -INFO-统一工具处理器初始化完成
+250705 -INFO-MCP接入点服务器信息: name=Calculator, version=1.9.4
+250705 -INFO-MCP接入点支持的工具数量: 1
+250705 -INFO-所有MCP接入点工具已获取，客户端准备就绪
+250705 -INFO-工具缓存已刷新
+250705 -INFO-当前支持的函数列表: [ 'get_time', 'get_lunar', 'play_music', 'get_weather', 'handle_exit_intent', 'calculator']
+```
+如果包含了 `'calculator'`，说明设备将可以根据意图识别，调用计算器这个工具。

docs/voiceprint-integration.md

@@ -0,0 +1,233 @@
+# 声纹识别启用指南
+
+本教程包含3个部分
+- 1、如何部署声纹识别这个服务
+- 2、全模块部署时，怎么配置声纹识别接口
+- 3、最简化部署时，怎么配置声纹识别
+
+# 1、如何部署声纹识别这个服务
+
+## 第一步，下载声纹识别项目源码
+
+浏览器打开[声纹识别项目地址](https://github.com/xinnan-tech/voiceprint-api)
+
+打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。
+
+点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`voiceprint-api-main`
+你需要把它重命名成`voiceprint-api`。
+
+## 第二步， 创建数据库和表
+
+声纹识别需要依赖`mysql`数据库。如果你之前已经部署`智控台`，说明你已经安装了`mysql`。你可以共用它。
+
+你可以你试一下在宿主机使用`telnet`命令，看看能不能正常访问`mysql`的`3306`端口。
+```
+telnet 127.0.0.1 3306
+```
+如果能访问到3306端口，请忽略以下的内容，直接进入第三步。
+
+如果不能访问，你需要回忆一下，你的`mysql`是怎么安装的。
+
+如果你的mysql是通过自己使用安装包安装的，说明你的`mysql`做了网络隔离。你可能先解访问`mysql`的`3306`端口这个问题。
+
+如果你`mysql`是通过本项目的`docker-compose_all.yml`安装的。你需要找一下你当时创建数据库的`docker-compose_all.yml`文件，修改以下的内容
+
+修改前
+```
+  xiaozhi-esp32-server-db:
+    ...
+    networks:
+      - default
+    expose:
+      - "3306:3306"
+```
+
+修改后
+```
+  xiaozhi-esp32-server-db:
+    ...
+    networks:
+      - default
+    ports:
+      - "3306:3306"
+```
+
+注意是将`xiaozhi-esp32-server-db`下面的`expose`改成`ports`。改完后，需要重新启动。以下是重启mysql的命令：
+
+```
+# 进入你docker-compose_all.yml所在的文件夹，例如我的是xiaozhi-server
+cd xiaozhi-server
+docker compose -f docker-compose_all.yml down
+docker compose -f docker-compose.yml up -d
+```
+
+启动完后，在宿主机再使用`telnet`命令，看看能不能正常访问`mysql`的`3306`端口。
+```
+telnet 127.0.0.1 3306
+```
+正常来说这样就可以访问的了。
+
+## 第三步， 创建数据库和表
+如果你的宿主机，能正常访问mysql数据库，那就在mysql上创建一个名字为`voiceprint_db`的数据库和`voiceprints`表。
+
+```
+CREATE DATABASE voiceprint_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+USE voiceprint_db;
+
+CREATE TABLE voiceprints (
+    id INT AUTO_INCREMENT PRIMARY KEY,
+    speaker_id VARCHAR(255) NOT NULL UNIQUE,
+    feature_vector LONGBLOB NOT NULL,
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    INDEX idx_speaker_id (speaker_id)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
+```
+
+## 第四步， 配置数据库连接
+
+进入`voiceprint-api`文件夹，创建名字为`data`的文件夹。
+
+把`voiceprint-api`根目录里的`voiceprint.yaml`，复制到`data`的文件夹，将它重命名为`.voiceprint.yaml`
+
+接下来，你需要重点配置一下`.voiceprint.yaml`里的数据库连接。
+
+```
+mysql:
+  host: "127.0.0.1"
+  port: 3306
+  user: "root"
+  password: "your_password"
+  database: "voiceprint_db"
+```
+
+注意！由于你的声纹识别服务是使用docker部署，`host`需要填写成你`mysql所在机器的局域网ip`。
+
+注意！由于你的声纹识别服务是使用docker部署，`host`需要填写成你`mysql所在机器的局域网ip`。
+
+注意！由于你的声纹识别服务是使用docker部署，`host`需要填写成你`mysql所在机器的局域网ip`。
+
+## 第五步，启动程序
+这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/voiceprint-api/blob/main/README.md)使用源码运行。以下是docker运行的方法
+
+```
+# 进入本项目源码根目录
+cd voiceprint-api
+
+# 清除缓存
+docker compose -f docker-compose.yml down
+docker stop voiceprint-api
+docker rm voiceprint-api
+docker rmi ghcr.nju.edu.cn/xinnan-tech/voiceprint-api:latest
+
+# 启动docker容器
+docker compose -f docker-compose.yml up -d
+# 查看日志
+docker logs -f voiceprint-api
+```
+
+此时，日志里会输出类似以下的日志
+```
+250711 INFO-🚀 开始: 生产环境服务启动（Uvicorn），监听地址: 0.0.0.0:8005
+250711 INFO-============================================================
+250711 INFO-声纹接口地址: http://127.0.0.1:8005/voiceprint/health?key=abcd
+250711 INFO-============================================================
+```
+
+请你把声纹接口地址复制出来：
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+由于你是docker部署，切不可直接使用上面的地址！
+
+你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么
+原来我的接口地址
+```
+http://127.0.0.1:8005/voiceprint/health?key=abcd
+
+```
+就要改成
+```
+http://192.168.1.25:8005/voiceprint/health?key=abcd
+```
+
+改好后，请使用浏览器直接访问`声纹接口地址`。当浏览器出现类似这样的代码，说明是成功了。
+```
+{"total_voiceprints":0,"status":"healthy"}
+```
+
+请你保留好修改后的`声纹接口地址`，下一步要用到。
+
+# 2、全模块部署时，怎么配置声纹识别
+
+## 第一步 配置接口
+如果你是全模块部署，使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。
+
+然后搜索参数`server.voice_print`，此时，它的值应该是`null`值。
+点击修改按钮，把上一步得来的`声纹接口地址`粘贴到`参数值`里。然后保存。
+
+如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问声纹识别，很大概率是网络防火墙，或者没有填写正确的局域网ip。
+
+## 第二步 设置智能体记忆模式
+
+进入你的智能体的角色配置里，将记忆设置成`本地短期记忆`，一定要开启`上报文字+语音`。
+
+## 第三步 和你的智能体聊天
+
+将你的设备通电，然后和他用正常的语速和音调聊天。
+
+## 第四步 设置声纹
+
+在智控台，`智能体管理`页面，在智能体的面板里，有一个`声纹识别`按钮，点击它。在底部有一个`新增按钮`。就可以对某个人说的话进行声纹注册。
+在弹出的框里，`描述`这个属性建议填写上，可以是这个人的职业、性格、爱好。方便智能体对说话人进行分析和了解。
+
+## 第三步 和你的智能体聊天
+
+将你的设备通电，问它，你知道我是谁吗？如果他能回答得出，说明声纹识别功能正常。
+
+# 3、最简化部署时，怎么配置声纹识别
+
+## 第一步 配置接口
+打开 `xiaozhi-server/data/.config.yaml` 文件（如果没有需要创建），然后添加/修改以下内容：
+
+```
+# 声纹识别配置
+voiceprint:
+  # 声纹接口地址
+  url: 你的声纹接口地址
+  # 说话人配置：speaker_id,名称,描述
+  speakers:
+    - "test1,张三,张三是一个程序员"
+    - "test2,李四,李四是一个产品经理"
+    - "test3,王五,王五是一个设计师"
+```
+
+把上一步得来的 `声纹接口地址` 粘贴到 `url` 里。然后保存。
+
+`speakers` 参数依据需求添加。这里需要注意这个 `speaker_id` 参数，后面注册声纹会用到。
+
+## 第二步 注册声纹
+如果你已经启动了声纹服务，本地浏览器里访问 `http://localhost:8005/voiceprint/docs` 即可查看 API 文档，这里只说明注册声纹的 API 如何使用。
+
+注册声纹的 API 地址为 `http://localhost:8005/voiceprint/register`，请求方式为 POST。
+
+请求头需要包含 Bearer Token 认证，token 为 `声纹接口地址` 中 `?key=` 后的部分，比如如果我的声纹注册地址为 `http://127.0.0.1:8005/voiceprint/health?key=abcd`，那么我的 token 就是`abcd`。
+
+请求体包含说话人 ID（speaker_id），和 WAV 音频文件（file），请求示例如下：
+
+```
+curl -X POST \
+  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
+  -F "speaker_id=your_speaker_id_here" \
+  -F "file=@/path/to/your/file" \
+  http://localhost:8005/voiceprint/register
+```
+
+ 这里的 `file` 是要注册的说话人说话的音频文件， `speaker_id` 需要和第一步配置接口的 `speaker_id` 保持一致。比如说我需要注册张三的声纹，在 `.config.yaml` 中填的张三的 `speaker_id` 为 `test1`，那么我注册张三声纹的时候，请求体里填的 `speaker_id` 就是 `test1`， `file` 填的就是张三说一段话的音频文件。
+
+ ## 第三步 启动服务
+
+启动小智服务器和声纹服务，即可正常使用。

main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java

@@ -116,6 +116,11 @@ public interface Constant {
      */
     String SERVER_MCP_ENDPOINT = "server.mcp_endpoint";
 
+    /**
+     * mcp接入点路径
+     */
+    String SERVER_VOICE_PRINT = "server.voice_print";
+
     /**
      * 无记忆
      */
@@ -232,7 +237,7 @@ public interface Constant {
     /**
      * 版本号
      */
-    public static final String VERSION = "0.6.1";
+    public static final String VERSION = "0.7.3";
 
     /**
      * 无效固件URL

main/manager-api/src/main/java/xiaozhi/common/utils/JsonRpcTwo.java

@@ -0,0 +1,21 @@
+package xiaozhi.common.utils;
+
+import lombok.Data;
+/**
+ * JSON-RPC2.0 格式规范对象
+ */
+@Data
+public class JsonRpcTwo {
+    private String jsonrpc = "2.0";
+    private String method;
+    private Object params;
+    private Integer id;
+
+    public JsonRpcTwo(String method, Object params, Integer id) {
+        this.method = method;
+        this.params = params;
+        this.id = id;
+    }
+
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/Enums/AgentChatHistoryType.java

@@ -0,0 +1,21 @@
+package xiaozhi.modules.agent.Enums;
+
+
+import lombok.Getter;
+
+/**
+ * 智能体聊天记录类型
+ */
+@Getter
+public enum AgentChatHistoryType {
+
+    USER((byte) 1),
+    AGENT((byte) 2);
+
+    private final byte value;
+
+    AgentChatHistoryType(byte i) {
+        this.value = i;
+    }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/Enums/XiaoZhiMcpJsonRpcJson.java

@@ -0,0 +1,44 @@
+package xiaozhi.modules.agent.Enums;
+
+import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.common.utils.JsonRpcTwo;
+
+import java.util.Map;
+
+
+/**
+ * 小智MCP JSON-RPC 请求json
+ */
+public class XiaoZhiMcpJsonRpcJson {
+    //小智初始化mcp请求json
+    private static final String INITIALIZE_JSON;
+    //小智mcp初始化成功，返回通知请求json
+    private static final String NOTIFICATIONS_INITIALIZED_JSON;
+    //小智mcp获取mcp工具集合请求json
+    private static final String TOOLS_LIST_REQUEST;
+    // 延迟加载
+    static {
+        INITIALIZE_JSON = JsonUtils.toJsonString(new JsonRpcTwo("initialize",
+                Map.of(
+                        "protocolVersion", "2024-11-05",
+                        "capabilities", Map.of(
+                                "roots", Map.of("listChanged", false),
+                                "sampling", Map.of()),
+                        "clientInfo", Map.of(
+                                "name", "xz-mcp-broker",
+                                "version", "0.0.1")),
+                1));
+        NOTIFICATIONS_INITIALIZED_JSON = "{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\"}";
+        TOOLS_LIST_REQUEST = JsonUtils.toJsonString(new JsonRpcTwo("tools/list", null, 2));
+    }
+    public static String getInitializeJson(){
+        return INITIALIZE_JSON;
+    }
+    public static String getNotificationsInitializedJson(){
+        return NOTIFICATIONS_INITIALIZED_JSON;
+    }
+    public static String getToolsListJson(){
+        return TOOLS_LIST_REQUEST;
+    }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java

@@ -47,6 +47,7 @@ import xiaozhi.modules.agent.service.AgentChatHistoryService;
 import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.AgentTemplateService;
+import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO;
 import xiaozhi.modules.agent.vo.AgentInfoVO;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
@@ -181,6 +182,33 @@ public class AgentController {
         List<AgentChatHistoryDTO> result = agentChatHistoryService.getChatHistoryBySessionId(id, sessionId);
         return new Result<List<AgentChatHistoryDTO>>().ok(result);
     }
+    @GetMapping("/{id}/chat-history/user")
+    @Operation(summary = "获取智能体聊天记录（用户）")
+    @RequiresPermissions("sys:role:normal")
+    public Result<List<AgentChatHistoryUserVO>> getRecentlyFiftyByAgentId(
+            @PathVariable("id") String id) {
+        // 获取当前用户
+        UserDetail user = SecurityUser.getUser();
+
+        // 检查权限
+        if (!agentService.checkAgentPermission(id, user.getId())) {
+            return new Result<List<AgentChatHistoryUserVO>>().error("没有权限查看该智能体的聊天记录");
+        }
+
+        // 查询聊天记录
+        List<AgentChatHistoryUserVO> data = agentChatHistoryService.getRecentlyFiftyByAgentId(id);
+        return new Result<List<AgentChatHistoryUserVO>>().ok(data);
+    }
+
+    @GetMapping("/{id}/chat-history/audio")
+    @Operation(summary = "获取音频内容")
+    @RequiresPermissions("sys:role:normal")
+    public Result<String> getContentByAudioId(
+            @PathVariable("id") String id) {
+        // 查询聊天记录
+        String data = agentChatHistoryService.getContentByAudioId(id);
+        return new Result<String>().ok(data);
+    }
 
     @PostMapping("/audio/{audioId}")
     @Operation(summary = "获取音频下载ID")

main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentVoicePrintController.java

@@ -0,0 +1,86 @@
+package xiaozhi.modules.agent.controller;
+
+import java.util.List;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.shiro.authz.annotation.RequiresPermissions;
+import org.springframework.web.bind.annotation.DeleteMapping;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.PutMapping;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.tags.Tag;
+import jakarta.validation.Valid;
+import lombok.AllArgsConstructor;
+import xiaozhi.common.exception.RenException;
+import xiaozhi.common.utils.Result;
+import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO;
+import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO;
+import xiaozhi.modules.agent.service.AgentVoicePrintService;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
+import xiaozhi.modules.security.user.SecurityUser;
+import xiaozhi.modules.sys.service.SysParamsService;
+
+@Tag(name = "智能体声纹管理")
+@AllArgsConstructor
+@RestController
+@RequestMapping("/agent/voice-print")
+public class AgentVoicePrintController {
+    private final AgentVoicePrintService agentVoicePrintService;
+    private final SysParamsService sysParamsService;
+
+    @PostMapping
+    @Operation(summary = "创建智能体的声纹")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> save(@RequestBody @Valid AgentVoicePrintSaveDTO dto) {
+        boolean b = agentVoicePrintService.insert(dto);
+        if (b) {
+            return new Result<>();
+        }
+        return new Result<Void>().error("智能体的声纹创建失败");
+    }
+
+    @PutMapping
+    @Operation(summary = "更新智能体的对应声纹")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> update(@RequestBody @Valid AgentVoicePrintUpdateDTO dto) {
+        Long userId = SecurityUser.getUserId();
+        boolean b = agentVoicePrintService.update(userId, dto);
+        if (b) {
+            return new Result<>();
+        }
+        return new Result<Void>().error("智能体的对应声纹更新失败");
+    }
+
+    @DeleteMapping("/{id}")
+    @Operation(summary = "删除智能体对应声纹")
+    @RequiresPermissions("sys:role:normal")
+    public Result<Void> delete(@PathVariable String id) {
+        Long userId = SecurityUser.getUserId();
+        // 先删除关联的设备
+        boolean delete = agentVoicePrintService.delete(userId, id);
+        if (delete) {
+            return new Result<>();
+        }
+        return new Result<Void>().error("智能体的对应声纹删除失败");
+    }
+
+    @GetMapping("/list/{id}")
+    @Operation(summary = "获取用户指定智能体声纹列表")
+    @RequiresPermissions("sys:role:normal")
+    public Result<List<AgentVoicePrintVO>> list(@PathVariable String id) {
+        String voiceprintUrl = sysParamsService.getValue("server.voice_print", true);
+        if (StringUtils.isBlank(voiceprintUrl) || "null".equals(voiceprintUrl)) {
+            throw new RenException("声纹接口未配置，请先在参数配置中配置声纹接口地址(server.voice_print)");
+        }
+        Long userId = SecurityUser.getUserId();
+        List<AgentVoicePrintVO> list = agentVoicePrintService.list(userId, id);
+        return new Result<List<AgentVoicePrintVO>>().ok(list);
+    }
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentVoicePrintDao.java

@@ -0,0 +1,20 @@
+package xiaozhi.modules.agent.dao;
+
+import org.apache.ibatis.annotations.Mapper;
+
+import com.baomidou.mybatisplus.core.mapper.BaseMapper;
+
+import xiaozhi.modules.agent.entity.AgentChatHistoryEntity;
+import xiaozhi.modules.agent.entity.AgentVoicePrintEntity;
+
+/**
+ * {@link AgentChatHistoryEntity} 智能体聊天历史记录Dao对象
+ *
+ * @author Goody
+ * @version 1.0, 2025/4/30
+ * @since 1.0.0
+ */
+@Mapper
+public interface AgentVoicePrintDao extends BaseMapper<AgentVoicePrintEntity> {
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintSaveDTO.java

@@ -0,0 +1,28 @@
+package xiaozhi.modules.agent.dto;
+
+import lombok.Data;
+
+/**
+ * 保存智能体声纹的dto
+ *
+ * @author zjy
+ */
+@Data
+public class AgentVoicePrintSaveDTO {
+    /**
+     * 关联的智能体id
+     */
+    private String agentId;
+    /**
+     * 音频文件id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintUpdateDTO.java

@@ -0,0 +1,28 @@
+package xiaozhi.modules.agent.dto;
+
+import lombok.Data;
+
+/**
+ * 修改智能体声纹的dto
+ *
+ * @author zjy
+ */
+@Data
+public class AgentVoicePrintUpdateDTO {
+    /**
+     * 智能体声纹id
+     */
+    private String id;
+    /**
+     * 音频文件id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/IdentifyVoicePrintResponse.java

@@ -0,0 +1,21 @@
+package xiaozhi.modules.agent.dto;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import lombok.Data;
+
+/**
+ * 声纹识别接口返回的对象
+ */
+@Data
+public class IdentifyVoicePrintResponse {
+    /**
+     * 最匹配的声纹id
+     */
+    @JsonProperty("speaker_id")
+    private String speakerId;
+    /**
+     * 声纹的分数
+     */
+    private Double score;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/dto/McpJsonRpcRequest.java

@@ -1,32 +0,0 @@
-package xiaozhi.modules.agent.dto;
-
-import lombok.Data;
-
-/**
- * MCP JSON-RPC 请求 DTO
- */
-@Data
-public class McpJsonRpcRequest {
-    private String jsonrpc = "2.0";
-    private String method;
-    private Object params;
-    private Integer id;
-
-    public McpJsonRpcRequest() {
-    }
-
-    public McpJsonRpcRequest(String method) {
-        this.method = method;
-    }
-
-    public McpJsonRpcRequest(String method, Object params, Integer id) {
-        this.method = method;
-        this.params = params;
-        this.id = id;
-    }
-
-    public McpJsonRpcRequest(String method, Object params) {
-        this.method = method;
-        this.params = params;
-    }
-}

main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentVoicePrintEntity.java

@@ -0,0 +1,64 @@
+package xiaozhi.modules.agent.entity;
+
+import java.util.Date;
+
+import com.baomidou.mybatisplus.annotation.FieldFill;
+import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableField;
+import com.baomidou.mybatisplus.annotation.TableId;
+import com.baomidou.mybatisplus.annotation.TableName;
+
+import lombok.Data;
+
+/**
+ * 智能体声纹表
+ *
+ * @author zjy
+ */
+@TableName(value = "ai_agent_voice_print")
+@Data
+public class AgentVoicePrintEntity {
+    /**
+     * 主键id
+     */
+    @TableId(type = IdType.ASSIGN_UUID)
+    private String id;
+    /**
+     * 关联的智能体id
+     */
+    private String agentId;
+    /**
+     * 关联的音频id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+
+    /**
+     * 创建者
+     */
+    @TableField(fill = FieldFill.INSERT)
+    private Long creator;
+    /**
+     * 创建时间
+     */
+    @TableField(fill = FieldFill.INSERT)
+    private Date createDate;
+
+    /**
+     * 更新者
+     */
+    @TableField(fill = FieldFill.INSERT_UPDATE)
+    private Long updater;
+    /**
+     * 更新时间
+     */
+    @TableField(fill = FieldFill.INSERT_UPDATE)
+    private Date updateDate;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentChatHistoryService.java

@@ -9,6 +9,7 @@ import xiaozhi.common.page.PageData;
 import xiaozhi.modules.agent.dto.AgentChatHistoryDTO;
 import xiaozhi.modules.agent.dto.AgentChatSessionDTO;
 import xiaozhi.modules.agent.entity.AgentChatHistoryEntity;
+import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO;
 
 /**
  * 智能体聊天记录表处理service
@@ -44,4 +45,30 @@ public interface AgentChatHistoryService extends IService<AgentChatHistoryEntity
      * @param deleteText  是否删除文本
      */
     void deleteByAgentId(String agentId, Boolean deleteAudio, Boolean deleteText);
+
+    /**
+     * 根据智能体ID获取最近50条用户的聊天记录数据（带音频数据）
+     *
+     * @param agentId 智能体id
+     * @return 聊天记录列表（只有用户）
+     */
+    List<AgentChatHistoryUserVO> getRecentlyFiftyByAgentId(String agentId);
+
+    /**
+     * 根据音频数据ID获取聊天内容
+     *
+     * @param audioId 音频id
+     * @return 聊天内容
+     */
+    String getContentByAudioId(String audioId);
+
+
+    /**
+     * 查询此音频id是否属于此智能体
+     *
+     * @param audioId 音频id
+     * @param agentId 音频id
+     * @return T：属于 F：不属于
+     */
+    boolean isAudioOwnedByAgent(String audioId,String agentId);
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentVoicePrintService.java

@@ -0,0 +1,50 @@
+package xiaozhi.modules.agent.service;
+
+import java.util.List;
+
+import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO;
+import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
+
+/**
+ * 智能体声纹处理service
+ *
+ * @author zjy
+ */
+public interface AgentVoicePrintService {
+    /**
+     * 添加智能体新的声纹
+     *
+     * @param dto 保存智能体声纹的数据
+     * @return T:成功 F：失败
+     */
+    boolean insert(AgentVoicePrintSaveDTO dto);
+
+    /**
+     * 删除智能体的指的声纹
+     *
+     * @param userId       当前登录的用户id
+     * @param voicePrintId 声纹id
+     * @return 是否成功 T:成功 F：失败
+     */
+    boolean delete(Long userId, String voicePrintId);
+
+    /**
+     * 获取指定智能体的所有声纹数据
+     *
+     * @param userId  当前登录的用户id
+     * @param agentId 智能体id
+     * @return 声纹数据集合
+     */
+    List<AgentVoicePrintVO> list(Long userId, String agentId);
+
+    /**
+     * 更新智能体的指的声纹数据
+     *
+     * @param userId 当前登录的用户id
+     * @param dto    修改的声纹的数据
+     * @return 是否成功 T:成功 F：失败
+     */
+    boolean update(Long userId, AgentVoicePrintUpdateDTO dto);
+
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentChatHistoryServiceImpl.java

@@ -8,6 +8,7 @@ import java.util.stream.Collectors;
 import org.springframework.stereotype.Service;
 import org.springframework.transaction.annotation.Transactional;
 
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
 import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
 import com.baomidou.mybatisplus.core.metadata.IPage;
 import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
@@ -16,11 +17,14 @@ import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
 import xiaozhi.common.constant.Constant;
 import xiaozhi.common.page.PageData;
 import xiaozhi.common.utils.ConvertUtils;
+import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.Enums.AgentChatHistoryType;
 import xiaozhi.modules.agent.dao.AiAgentChatHistoryDao;
 import xiaozhi.modules.agent.dto.AgentChatHistoryDTO;
 import xiaozhi.modules.agent.dto.AgentChatSessionDTO;
 import xiaozhi.modules.agent.entity.AgentChatHistoryEntity;
 import xiaozhi.modules.agent.service.AgentChatHistoryService;
+import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO;
 
 /**
  * 智能体聊天记录表处理service {@link AgentChatHistoryService} impl
@@ -90,4 +94,78 @@ public class AgentChatHistoryServiceImpl extends ServiceImpl<AiAgentChatHistoryD
         }
 
     }
+
+    @Override
+    public List<AgentChatHistoryUserVO> getRecentlyFiftyByAgentId(String agentId) {
+        // 构建查询条件(不添加按照创建时间排序，数据本来就是主键越大创建时间越大
+        // 不添加这样可以减少排序全部数据在分页的全盘扫描消耗)
+        LambdaQueryWrapper<AgentChatHistoryEntity> wrapper = new LambdaQueryWrapper<>();
+        wrapper.select(AgentChatHistoryEntity::getContent, AgentChatHistoryEntity::getAudioId)
+                .eq(AgentChatHistoryEntity::getAgentId, agentId)
+                .eq(AgentChatHistoryEntity::getChatType, AgentChatHistoryType.USER.getValue())
+                .isNotNull(AgentChatHistoryEntity::getAudioId)
+                // 添加此行，确保查询结果按照创建时间降序排列
+                // 使用id的原因：数据形式，id越大的创建时间就越晚，所以使用id的结果和创建时间降序排列结果一样
+                // id作为降序排列的优势，性能高，有主键索引，不用在排序的时候重新进行排除扫描比较
+                .orderByDesc(AgentChatHistoryEntity::getId); 
+
+        // 构建分页查询，查询前50页数据
+        Page<AgentChatHistoryEntity> pageParam = new Page<>(0, 50);
+        IPage<AgentChatHistoryEntity> result = this.baseMapper.selectPage(pageParam, wrapper);
+        return result.getRecords().stream().map(item -> {
+            AgentChatHistoryUserVO vo = ConvertUtils.sourceToTarget(item, AgentChatHistoryUserVO.class);
+            // 处理 content 字段，确保只返回聊天内容
+            if (vo != null && vo.getContent() != null) {
+                vo.setContent(extractContentFromString(vo.getContent()));
+            }
+            return vo;
+        }).toList();
+    }
+
+    /**
+     * 从 content 字段中提取聊天内容
+     * 如果 content 是 JSON 格式（如 {"speaker": "未知说话人", "content": "现在几点了。"}），则提取 content
+     * 字段
+     * 如果 content 是普通字符串，则直接返回
+     * 
+     * @param content 原始内容
+     * @return 提取的聊天内容
+     */
+    private String extractContentFromString(String content) {
+        if (content == null || content.trim().isEmpty()) {
+            return content;
+        }
+
+        // 尝试解析为 JSON
+        try {
+            Map<String, Object> jsonMap = JsonUtils.parseObject(content, Map.class);
+            if (jsonMap != null && jsonMap.containsKey("content")) {
+                Object contentObj = jsonMap.get("content");
+                return contentObj != null ? contentObj.toString() : content;
+            }
+        } catch (Exception e) {
+            // 如果不是有效的 JSON，直接返回原内容
+        }
+
+        // 如果不是 JSON 格式或没有 content 字段，直接返回原内容
+        return content;
+    }
+
+    @Override
+    public String getContentByAudioId(String audioId) {
+        AgentChatHistoryEntity agentChatHistoryEntity = baseMapper
+                .selectOne(new LambdaQueryWrapper<AgentChatHistoryEntity>()
+                        .select(AgentChatHistoryEntity::getContent)
+                        .eq(AgentChatHistoryEntity::getAudioId, audioId));
+        return agentChatHistoryEntity == null ? null : agentChatHistoryEntity.getContent();
+    }
+
+    @Override
+    public boolean isAudioOwnedByAgent(String audioId, String agentId) {
+        // 查询是否有指定音频id和智能体id的数据，如果有且只有一条说明此数据属性此智能体
+        Long row = baseMapper.selectCount(new LambdaQueryWrapper<AgentChatHistoryEntity>()
+                .eq(AgentChatHistoryEntity::getAudioId, audioId)
+                .eq(AgentChatHistoryEntity::getAgentId, agentId));
+        return row == 1;
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentMcpAccessPointServiceImpl.java

@@ -18,7 +18,7 @@ import xiaozhi.common.constant.Constant;
 import xiaozhi.common.utils.AESUtils;
 import xiaozhi.common.utils.HashEncryptionUtil;
 import xiaozhi.common.utils.JsonUtils;
-import xiaozhi.modules.agent.dto.McpJsonRpcRequest;
+import xiaozhi.modules.agent.Enums.XiaoZhiMcpJsonRpcJson;
 import xiaozhi.modules.agent.service.AgentMcpAccessPointService;
 import xiaozhi.modules.sys.service.SysParamsService;
 import xiaozhi.modules.sys.utils.WebSocketClientManager;
@@ -61,57 +61,79 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic
         wsUrl = wsUrl.replace("/mcp/", "/call/");
 
         try {
-            // 创建 WebSocket 连接
+            // 创建 WebSocket 连接，增加超时时间到15秒
             try (WebSocketClientManager client = WebSocketClientManager.build(
                     new WebSocketClientManager.Builder()
                             .uri(wsUrl)
-                            .connectTimeout(5, TimeUnit.SECONDS)
+                            .bufferSize(1024 * 1024)
-                            .maxSessionDuration(9, TimeUnit.SECONDS))) {
+                            .connectTimeout(8, TimeUnit.SECONDS)
+                            .maxSessionDuration(10, TimeUnit.SECONDS))) {
 
-                // 发送初始化消息
+                // 步骤1: 发送初始化消息并等待响应
-                McpJsonRpcRequest initializeRequest = new McpJsonRpcRequest("initialize",
+                log.info("发送MCP初始化消息，智能体ID: {}", id);
-                        Map.of(
+                client.sendText(XiaoZhiMcpJsonRpcJson.getInitializeJson());
-                                "protocolVersion", "2024-11-05",
-                                "capabilities", Map.of(
-                                        "roots", Map.of("listChanged", false),
-                                        "sampling", Map.of()),
-                                "clientInfo", Map.of(
-                                        "name", "xz-mcp-broker",
-                                        "version", "0.0.1")),
-                        1);
-                client.sendJson(initializeRequest);
 
-                // 等待初始化响应
+                // 等待初始化响应 (id=1) - 移除固定延迟，改为响应驱动
-                Thread.sleep(200);
+                List<String> initResponses = client.listenerWithoutClose(response -> {
-
-                // 发送初始化完成通知
-                // 对于通知类型的消息，手动构建JSON以避免包含null字段
-                String notificationJson = "{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\"}";
-                client.sendText(notificationJson);
-
-                // 等待 0.2 秒
-                Thread.sleep(200);
-
-                // 发送工具列表请求
-                McpJsonRpcRequest toolsRequest = new McpJsonRpcRequest("tools/list", null, 2);
-                client.sendJson(toolsRequest);
-
-                // 监听响应，直到收到包含 id=2 的响应（tools/list响应）
-                List<String> responses = client.listener(response -> {
                     try {
-                        // 先尝试解析为通用JSON对象来获取id
                         Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
-                        return jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"));
+                        if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) {
+                            // 检查是否有result字段，表示初始化成功
+                            return jsonMap.containsKey("result") && !jsonMap.containsKey("error");
+                        }
+                        return false;
                     } catch (Exception e) {
-                        log.warn("解析响应失败: {}", response, e);
+                        log.warn("解析初始化响应失败: {}", response, e);
                         return false;
                     }
                 });
 
-                // 处理响应
+                // 验证初始化响应
-                for (String response : responses) {
+                boolean initSucceeded = false;
+                for (String response : initResponses) {
+                    try {
+                        Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
+                        if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) {
+                            if (jsonMap.containsKey("result")) {
+                                log.info("MCP初始化成功，智能体ID: {}", id);
+                                initSucceeded = true;
+                                break;
+                            } else if (jsonMap.containsKey("error")) {
+                                log.error("MCP初始化失败，智能体ID: {}, 错误: {}", id, jsonMap.get("error"));
+                                return List.of();
+                            }
+                        }
+                    } catch (Exception e) {
+                        log.warn("处理初始化响应失败: {}", response, e);
+                    }
+                }
+
+                if (!initSucceeded) {
+                    log.error("未收到有效的MCP初始化响应，智能体ID: {}", id);
+                    return List.of();
+                }
+
+                // 步骤2: 发送初始化完成通知 - 只有在收到initialize响应后才发送
+                log.info("发送MCP初始化完成通知，智能体ID: {}", id);
+                client.sendText(XiaoZhiMcpJsonRpcJson.getNotificationsInitializedJson());
+                // 步骤3: 发送工具列表请求 - 立即发送，无需额外延迟
+                log.info("发送MCP工具列表请求，智能体ID: {}", id);
+                client.sendText(XiaoZhiMcpJsonRpcJson.getToolsListJson());
+
+                // 等待工具列表响应 (id=2)
+                List<String> toolsResponses = client.listener(response -> {
+                    try {
+                        Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
+                        return jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"));
+                    } catch (Exception e) {
+                        log.warn("解析工具列表响应失败: {}", response, e);
+                        return false;
+                    }
+                });
+
+                // 处理工具列表响应
+                for (String response : toolsResponses) {
                     try {
-                        // 先解析为通用JSON对象
                         Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
                         if (jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"))) {
                             // 检查是否有result字段
@@ -122,11 +144,16 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic
                                 if (toolsObj instanceof List) {
                                     List<Map<String, Object>> toolsList = (List<Map<String, Object>>) toolsObj;
                                     // 提取工具名称列表
-                                    return toolsList.stream()
+                                    List<String> result = toolsList.stream()
                                             .map(tool -> (String) tool.get("name"))
                                             .filter(name -> name != null)
                                             .collect(Collectors.toList());
+                                    log.info("成功获取MCP工具列表，智能体ID: {}, 工具数量: {}", id, result.size());
+                                    return result;
                                 }
+                            } else if (jsonMap.containsKey("error")) {
+                                log.error("获取工具列表失败，智能体ID: {}, 错误: {}", id, jsonMap.get("error"));
+                                return List.of();
                             }
                         }
                     } catch (Exception e) {
@@ -134,12 +161,12 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic
                     }
                 }
 
-                log.warn("未找到有效的工具列表响应");
+                log.warn("未找到有效的工具列表响应，智能体ID: {}", id);
                 return List.of();
 
             }
         } catch (Exception e) {
-            log.error("获取智能体 MCP 工具列表失败，智能体ID: {}", id, e);
+            log.error("获取智能体 MCP 工具列表失败，智能体ID: {},错误原因：{}", id, e.getMessage());
             return List.of();
         }
     }

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentServiceImpl.java

@@ -41,6 +41,7 @@ import xiaozhi.modules.agent.service.AgentTemplateService;
 import xiaozhi.modules.agent.vo.AgentInfoVO;
 import xiaozhi.modules.device.service.DeviceService;
 import xiaozhi.modules.model.dto.ModelProviderDTO;
+import xiaozhi.modules.model.entity.ModelConfigEntity;
 import xiaozhi.modules.model.service.ModelConfigService;
 import xiaozhi.modules.model.service.ModelProviderService;
 import xiaozhi.modules.security.user.SecurityUser;
@@ -324,9 +325,32 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
             // 删除音频数据
             agentChatHistoryService.deleteByAgentId(existingEntity.getId(), true, false);
         }
+
+        boolean b = validateLLMIntentParams(dto.getLlmModelId(), dto.getIntentModelId());
+        if (!b) {
+            throw new RenException("LLM大模型和Intent意图识别，选择参数不匹配");
+        }
         this.updateById(existingEntity);
     }
 
+    /**
+     * 验证大语言模型和意图识别的参数是否符合匹配
+     * 
+     * @param llmModelId    大语言模型id
+     * @param intentModelId 意图识别id
+     * @return T 匹配 : F 不匹配
+     */
+    private boolean validateLLMIntentParams(String llmModelId, String intentModelId) {
+        ModelConfigEntity llmModelData = modelConfigService.selectById(llmModelId);
+        String type = llmModelData.getConfigJson().get("type").toString();
+        // 如果查询大语言模型是openai或者ollama，意图识别选参数都可以
+        if ("openai".equals(type) || "ollama".equals(type)) {
+            return true;
+        }
+        // 除了openai和ollama的类型，不可以选择id为Intent_function_call（函数调用）的意图识别
+        return !"Intent_function_call".equals(intentModelId);
+    }
+
     @Override
     @Transactional(rollbackFor = Exception.class)
     public String createAgent(AgentCreateDTO dto) {

main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentVoicePrintServiceImpl.java

@@ -0,0 +1,410 @@
+package xiaozhi.modules.agent.service.impl;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.List;
+import java.util.concurrent.Executor;
+import java.util.stream.Collectors;
+
+import org.apache.commons.lang3.StringUtils;
+import org.springframework.beans.factory.annotation.Qualifier;
+import org.springframework.core.io.ByteArrayResource;
+import org.springframework.http.HttpEntity;
+import org.springframework.http.HttpHeaders;
+import org.springframework.http.HttpMethod;
+import org.springframework.http.HttpStatus;
+import org.springframework.http.MediaType;
+import org.springframework.http.ResponseEntity;
+import org.springframework.stereotype.Service;
+import org.springframework.transaction.support.TransactionTemplate;
+import org.springframework.util.LinkedMultiValueMap;
+import org.springframework.util.MultiValueMap;
+import org.springframework.web.client.RestTemplate;
+
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
+import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
+
+import lombok.extern.slf4j.Slf4j;
+import xiaozhi.common.constant.Constant;
+import xiaozhi.common.exception.RenException;
+import xiaozhi.common.utils.ConvertUtils;
+import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.dao.AgentVoicePrintDao;
+import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO;
+import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO;
+import xiaozhi.modules.agent.dto.IdentifyVoicePrintResponse;
+import xiaozhi.modules.agent.entity.AgentVoicePrintEntity;
+import xiaozhi.modules.agent.service.AgentChatAudioService;
+import xiaozhi.modules.agent.service.AgentChatHistoryService;
+import xiaozhi.modules.agent.service.AgentVoicePrintService;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
+import xiaozhi.modules.sys.service.SysParamsService;
+
+/**
+ * @author zjy
+ */
+@Service
+@Slf4j
+public class AgentVoicePrintServiceImpl extends ServiceImpl<AgentVoicePrintDao, AgentVoicePrintEntity>
+        implements AgentVoicePrintService {
+    private final AgentChatAudioService agentChatAudioService;
+    private final RestTemplate restTemplate;
+    private final SysParamsService sysParamsService;
+    private final AgentChatHistoryService agentChatHistoryService;
+    // Springboot提供的编程事务类
+    private final TransactionTemplate transactionTemplate;
+    // 识别度
+    private final Double RECOGNITION = 0.5;
+    private final Executor taskExecutor;
+
+    public AgentVoicePrintServiceImpl(AgentChatAudioService agentChatAudioService, RestTemplate restTemplate,
+                                      SysParamsService sysParamsService, AgentChatHistoryService agentChatHistoryService,
+                                      TransactionTemplate transactionTemplate, @Qualifier("taskExecutor") Executor taskExecutor) {
+        this.agentChatAudioService = agentChatAudioService;
+        this.restTemplate = restTemplate;
+        this.sysParamsService = sysParamsService;
+        this.agentChatHistoryService = agentChatHistoryService;
+        this.transactionTemplate = transactionTemplate;
+        this.taskExecutor = taskExecutor;
+    }
+
+    @Override
+    public boolean insert(AgentVoicePrintSaveDTO dto) {
+        // 获取音频数据
+        ByteArrayResource resource = getVoicePrintAudioWAV(dto.getAgentId(), dto.getAudioId());
+        // 识别一下此声音是否注册过
+        IdentifyVoicePrintResponse response = identifyVoicePrint(dto.getAgentId(), resource);
+        if (response != null && response.getScore() > RECOGNITION) {
+            // 根据识别出的声纹ID查询对应的用户信息
+            AgentVoicePrintEntity existingVoicePrint = baseMapper.selectById(response.getSpeakerId());
+            String existingUserName = existingVoicePrint != null ? existingVoicePrint.getSourceName() : "未知用户";
+            throw new RenException("此声音声纹对应的人（" + existingUserName + "）已经注册，请选择其他声音注册");
+        }
+        AgentVoicePrintEntity entity = ConvertUtils.sourceToTarget(dto, AgentVoicePrintEntity.class);
+        // 开启事务
+        return Boolean.TRUE.equals(transactionTemplate.execute(status -> {
+            try {
+                // 保存声纹信息
+                int row = baseMapper.insert(entity);
+                // 插入一条数据，影响的数据不等于1说明出现了，保存问题回滚
+                if (row != 1) {
+                    status.setRollbackOnly(); // 标记事务回滚
+                    return false;
+                }
+                // 发送注册声纹请求
+                registerVoicePrint(entity.getId(), resource);
+                return true;
+            } catch (RenException e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                throw e;
+            } catch (Exception e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                log.error("保存声纹错误原因：{}", e.getMessage());
+                throw new RenException("保存声纹错误，请联系管理员");
+            }
+        }));
+    }
+
+    @Override
+    public boolean delete(Long userId, String voicePrintId) {
+        // 开启事务
+        boolean b = Boolean.TRUE.equals(transactionTemplate.execute(status -> {
+            try {
+                // 删除声纹,按照指定当前登录用户和智能体
+                int row = baseMapper.delete(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                        .eq(AgentVoicePrintEntity::getId, voicePrintId)
+                        .eq(AgentVoicePrintEntity::getCreator, userId));
+                if (row != 1) {
+                    status.setRollbackOnly(); // 标记事务回滚
+                    return false;
+                }
+
+                return true;
+            } catch (Exception e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                log.error("删除声纹存在错误原因：{}", e.getMessage());
+                throw new RenException("删除声纹出现了错误");
+            }
+        }));
+        // 数据库声纹数据删除成功才继续执行删除声纹服务的数据
+        if(b){
+            taskExecutor.execute(()-> {
+                try {
+                    cancelVoicePrint(voicePrintId);
+                }catch (RuntimeException e) {
+                    log.error("删除声纹存在运行时错误原因：{}，id：{}", e.getMessage(),voicePrintId);
+                }
+            });
+        }
+        return b;
+    }
+
+    @Override
+    public List<AgentVoicePrintVO> list(Long userId, String agentId) {
+        // 按照指定当前登录用户和智能体查找数据
+        List<AgentVoicePrintEntity> list = baseMapper.selectList(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                .eq(AgentVoicePrintEntity::getAgentId, agentId)
+                .eq(AgentVoicePrintEntity::getCreator, userId));
+        return list.stream().map(entity -> {
+            // 遍历转换成AgentVoicePrintVO类型
+            return ConvertUtils.sourceToTarget(entity, AgentVoicePrintVO.class);
+        }).toList();
+
+    }
+
+    @Override
+    public boolean update(Long userId, AgentVoicePrintUpdateDTO dto) {
+        AgentVoicePrintEntity agentVoicePrintEntity = baseMapper
+                .selectOne(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                        .eq(AgentVoicePrintEntity::getId, dto.getId())
+                        .eq(AgentVoicePrintEntity::getCreator, userId));
+        if (agentVoicePrintEntity == null) {
+            return false;
+        }
+        // 获取音频Id
+        String audioId = dto.getAudioId();
+        // 获取智能体id
+        String agentId = agentVoicePrintEntity.getAgentId();
+        ByteArrayResource resource;
+        // audioId不等于空，且audioId和之前的保存的音频id不一样，则需要重新获取音频数据生成声纹
+        if (!StringUtils.isEmpty(audioId) && !audioId.equals(agentVoicePrintEntity.getAudioId())) {
+            resource = getVoicePrintAudioWAV(agentId, audioId);
+
+            // 识别一下此声音是否注册过
+            IdentifyVoicePrintResponse response = identifyVoicePrint(agentId, resource);
+            // 返回分数高于RECOGNITION说明这个声纹已经有了
+            if (response != null && response.getScore() > RECOGNITION) {
+                // 判断返回的id如果不是要修改的声纹id，说明这个声纹id，现在要注册的声音已经存在且不是原来的声纹，不允许修改
+                if (!response.getSpeakerId().equals(dto.getId())) {
+                    // 根据识别出的声纹ID查询对应的用户信息
+                    AgentVoicePrintEntity existingVoicePrint = baseMapper.selectById(response.getSpeakerId());
+                    String existingUserName = existingVoicePrint != null ? existingVoicePrint.getSourceName() : "未知用户";
+                    throw new RenException("此次修改不允许，此声音已经注册为声纹了（" + existingUserName + "）");
+                }
+            }
+        } else {
+            resource = null;
+        }
+        // 开启事务
+        return Boolean.TRUE.equals(transactionTemplate.execute(status -> {
+            try {
+                AgentVoicePrintEntity entity = ConvertUtils.sourceToTarget(dto, AgentVoicePrintEntity.class);
+                int row = baseMapper.updateById(entity);
+                if (row != 1) {
+                    status.setRollbackOnly(); // 标记事务回滚
+                    return false;
+                }
+                if (resource != null) {
+                    String id = entity.getId();
+                    // 先注销之前这个声纹id上的声纹向量
+                    cancelVoicePrint(id);
+                    // 发送注册声纹请求
+                    registerVoicePrint(id, resource);
+                }
+                return true;
+            } catch (RenException e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                throw e;
+            } catch (Exception e) {
+                status.setRollbackOnly(); // 标记事务回滚
+                log.error("修改声纹错误原因：{}", e.getMessage());
+                throw new RenException("修改声纹错误，请联系管理员");
+            }
+        }));
+    }
+
+    /**
+     * 获取生纹接口URI对象
+     *
+     * @return URI对象
+     */
+    private URI getVoicePrintURI() {
+        // 获取声纹接口地址
+        String voicePrint = sysParamsService.getValue(Constant.SERVER_VOICE_PRINT, true);
+        try {
+            return new URI(voicePrint);
+        } catch (URISyntaxException e) {
+            log.error("路径格式不正确路径：{}，\n错误信息:{}", voicePrint, e.getMessage());
+            throw new RuntimeException("声纹接口的地址存在错误，请进入参数管理修改声纹接口地址");
+        }
+    }
+
+    /**
+     * 获取声纹地址基础路径
+     * 
+     * @param uri 声纹地址uri
+     * @return 基础路径
+     */
+    private String getBaseUrl(URI uri) {
+        String protocol = uri.getScheme();
+        String host = uri.getHost();
+        int port = uri.getPort();
+        if (port == -1) {
+            return "%s://%s".formatted(protocol, host);
+        } else {
+            return "%s://%s:%s".formatted(protocol, host, port);
+        }
+    }
+
+    /**
+     * 获取验证Authorization
+     *
+     * @param uri 声纹地址uri
+     * @return Authorization值
+     */
+    private String getAuthorization(URI uri) {
+        // 获取参数
+        String query = uri.getQuery();
+        // 获取aes加密密钥
+        String str = "key=";
+        return "Bearer " + query.substring(query.indexOf(str) + str.length());
+    }
+
+    /**
+     * 获取声纹音频资源数据
+     *
+     * @param audioId 音频Id
+     * @return 声纹音频资源数据
+     */
+    private ByteArrayResource getVoicePrintAudioWAV(String agentId, String audioId) {
+        // 判断这个音频是否属于当前智能体
+        boolean b = agentChatHistoryService.isAudioOwnedByAgent(audioId, agentId);
+        if (!b) {
+            throw new RenException("音频数据不属于这个智能体");
+        }
+        // 获取到音频数据
+        byte[] audio = agentChatAudioService.getAudio(audioId);
+        // 如果音频数据为空的直接报错不进行下去
+        if (audio == null || audio.length == 0) {
+            throw new RenException("音频数据是空的请检查上传数据");
+        }
+        // 将字节数组包装为资源，返回
+        return new ByteArrayResource(audio) {
+            @Override
+            public String getFilename() {
+                return "VoicePrint.WAV"; // 设置文件名
+            }
+        };
+    }
+
+    /**
+     * 发送注册声纹http请求
+     * 
+     * @param id       声纹id
+     * @param resource 声纹音频资源
+     */
+    private void registerVoicePrint(String id, ByteArrayResource resource) {
+        // 处理声纹接口地址，获取前缀
+        URI uri = getVoicePrintURI();
+        String baseUrl = getBaseUrl(uri);
+        String requestUrl = baseUrl + "/voiceprint/register";
+        // 创建请求体
+        MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
+        body.add("speaker_id", id);
+        body.add("file", resource);
+
+        // 创建请求头
+        HttpHeaders headers = new HttpHeaders();
+        headers.set("Authorization", getAuthorization(uri));
+        headers.setContentType(MediaType.MULTIPART_FORM_DATA);
+        // 创建请求体
+        HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
+        // 发送 POST 请求
+        ResponseEntity<String> response = restTemplate.postForEntity(requestUrl, requestEntity, String.class);
+
+        if (response.getStatusCode() != HttpStatus.OK) {
+            log.error("声纹注册失败,请求路径：{}", requestUrl);
+            throw new RenException("声纹保存失败,请求不成功");
+        }
+        // 检查响应内容
+        String responseBody = response.getBody();
+        if (responseBody == null || !responseBody.contains("true")) {
+            log.error("声纹注册失败,请求处理失败内容：{}", responseBody == null ? "空内容" : responseBody);
+            throw new RenException("声纹保存失败,请求处理失败");
+        }
+    }
+
+    /**
+     * 发送注销声纹的请求
+     * 
+     * @param voicePrintId 声纹id
+     */
+    private void cancelVoicePrint(String voicePrintId) {
+        URI uri = getVoicePrintURI();
+        String baseUrl = getBaseUrl(uri);
+        String requestUrl = baseUrl + "/voiceprint/" + voicePrintId;
+        // 创建请求头
+        HttpHeaders headers = new HttpHeaders();
+        headers.set("Authorization", getAuthorization(uri));
+        // 创建请求体
+        HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(headers);
+
+        // 发送 POST 请求
+        ResponseEntity<String> response = restTemplate.exchange(requestUrl, HttpMethod.DELETE, requestEntity,
+                String.class);
+        if (response.getStatusCode() != HttpStatus.OK) {
+            log.error("声纹注销失败,请求路径：{}", requestUrl);
+            throw new RenException("声纹注销失败,请求不成功");
+        }
+        // 检查响应内容
+        String responseBody = response.getBody();
+        if (responseBody == null || !responseBody.contains("true")) {
+            log.error("声纹注销失败,请求处理失败内容：{}", responseBody == null ? "空内容" : responseBody);
+            throw new RenException("声纹注销失败,请求处理失败");
+        }
+    }
+
+    /**
+     * 发送识别声纹http请求
+     * 
+     * @param agentId  智能体id
+     * @param resource 声纹音频资源
+     * @return 返回识别数据
+     */
+    private IdentifyVoicePrintResponse identifyVoicePrint(String agentId, ByteArrayResource resource) {
+
+        // 获取该智能体所有注册的声纹
+        List<AgentVoicePrintEntity> agentVoicePrintList = baseMapper
+                .selectList(new LambdaQueryWrapper<AgentVoicePrintEntity>()
+                        .select(AgentVoicePrintEntity::getId)
+                        .eq(AgentVoicePrintEntity::getAgentId, agentId));
+
+        // 声纹数量为0，说明还没注册过声纹不需要发生识别请求
+        if (agentVoicePrintList.isEmpty()) {
+            return null;
+        }
+        // 处理声纹接口地址，获取前缀
+        URI uri = getVoicePrintURI();
+        String baseUrl = getBaseUrl(uri);
+        String requestUrl = baseUrl + "/voiceprint/identify";
+        // 创建请求体
+        MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
+
+        // 创建speaker_id参数
+        String speakerIds = agentVoicePrintList.stream()
+                .map(AgentVoicePrintEntity::getId)
+                .collect(Collectors.joining(","));
+        body.add("speaker_ids", speakerIds);
+        body.add("file", resource);
+
+        // 创建请求头
+        HttpHeaders headers = new HttpHeaders();
+        headers.set("Authorization", getAuthorization(uri));
+        headers.setContentType(MediaType.MULTIPART_FORM_DATA);
+        // 创建请求体
+        HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
+        // 发送 POST 请求
+        ResponseEntity<String> response = restTemplate.postForEntity(requestUrl, requestEntity, String.class);
+
+        if (response.getStatusCode() != HttpStatus.OK) {
+            log.error("声纹识别请求失败,请求路径：{}", requestUrl);
+            throw new RenException("声纹识别失败,请求不成功");
+        }
+        // 检查响应内容
+        String responseBody = response.getBody();
+        if (responseBody != null) {
+            return JsonUtils.parseObject(responseBody, IdentifyVoicePrintResponse.class);
+        }
+        return null;
+    }
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentChatHistoryUserVO.java

@@ -0,0 +1,16 @@
+package xiaozhi.modules.agent.vo;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import lombok.Data;
+
+/**
+ * 智能体用户个人聊天数据的VO
+ */
+@Data
+public class AgentChatHistoryUserVO {
+    @Schema(description = "聊天内容")
+    private String content;
+
+    @Schema(description = "音频ID")
+    private String audioId;
+}

main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentVoicePrintVO.java

@@ -0,0 +1,33 @@
+package xiaozhi.modules.agent.vo;
+
+import lombok.Data;
+
+import java.util.Date;
+
+/**
+ * 展示智能体声纹列表VO
+ */
+@Data
+public class AgentVoicePrintVO {
+
+    /**
+     * 主键id
+     */
+    private String id;
+    /**
+     * 音频文件id
+     */
+    private String audioId;
+    /**
+     * 声纹来源的人姓名
+     */
+    private String sourceName;
+    /**
+     * 描述声纹来源的人
+     */
+    private String introduce;
+    /**
+     * 创建时间
+     */
+    private Date createDate;
+}

main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java

@@ -9,20 +9,26 @@ import java.util.Objects;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.stereotype.Service;
 
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
+
 import lombok.AllArgsConstructor;
 import xiaozhi.common.constant.Constant;
 import xiaozhi.common.exception.ErrorCode;
 import xiaozhi.common.exception.RenException;
 import xiaozhi.common.redis.RedisKeys;
 import xiaozhi.common.redis.RedisUtils;
+import xiaozhi.common.utils.ConvertUtils;
 import xiaozhi.common.utils.JsonUtils;
+import xiaozhi.modules.agent.dao.AgentVoicePrintDao;
 import xiaozhi.modules.agent.entity.AgentEntity;
 import xiaozhi.modules.agent.entity.AgentPluginMapping;
 import xiaozhi.modules.agent.entity.AgentTemplateEntity;
+import xiaozhi.modules.agent.entity.AgentVoicePrintEntity;
 import xiaozhi.modules.agent.service.AgentMcpAccessPointService;
 import xiaozhi.modules.agent.service.AgentPluginMappingService;
 import xiaozhi.modules.agent.service.AgentService;
 import xiaozhi.modules.agent.service.AgentTemplateService;
+import xiaozhi.modules.agent.vo.AgentVoicePrintVO;
 import xiaozhi.modules.config.service.ConfigService;
 import xiaozhi.modules.device.entity.DeviceEntity;
 import xiaozhi.modules.device.service.DeviceService;
@@ -45,6 +51,7 @@ public class ConfigServiceImpl implements ConfigService {
     private final TimbreService timbreService;
     private final AgentPluginMappingService agentPluginMappingService;
     private final AgentMcpAccessPointService agentMcpAccessPointService;
+    private final AgentVoicePrintDao agentVoicePrintDao;
 
     @Override
     public Object getConfig(Boolean isCache) {
@@ -162,6 +169,8 @@ public class ConfigServiceImpl implements ConfigService {
             mcpEndpoint = mcpEndpoint.replace("/mcp/", "/call/");
             result.put("mcp_endpoint", mcpEndpoint);
         }
+        // 获取声纹信息
+        buildVoiceprintConfig(agent.getId(), result);
 
         // 构建模块配置
         buildModuleConfig(
@@ -255,20 +264,76 @@ public class ConfigServiceImpl implements ConfigService {
         return config;
     }
 
+    /**
+     * 构建声纹配置信息
+     * 
+     * @param agentId 智能体ID
+     * @param result  结果Map
+     */
+    private void buildVoiceprintConfig(String agentId, Map<String, Object> result) {
+        try {
+            // 获取声纹接口地址
+            String voiceprintUrl = sysParamsService.getValue("server.voice_print", true);
+            if (StringUtils.isBlank(voiceprintUrl) || "null".equals(voiceprintUrl)) {
+                return;
+            }
+
+            // 获取智能体关联的声纹信息（不需要用户权限验证）
+            List<AgentVoicePrintVO> voiceprints = getVoiceprintsByAgentId(agentId);
+            if (voiceprints == null || voiceprints.isEmpty()) {
+                return;
+            }
+
+            // 构建speakers列表
+            List<String> speakers = new ArrayList<>();
+            for (AgentVoicePrintVO voiceprint : voiceprints) {
+                String speakerStr = String.format("%s,%s,%s",
+                        voiceprint.getId(),
+                        voiceprint.getSourceName(),
+                        voiceprint.getIntroduce() != null ? voiceprint.getIntroduce() : "");
+                speakers.add(speakerStr);
+            }
+
+            // 构建声纹配置
+            Map<String, Object> voiceprintConfig = new HashMap<>();
+            voiceprintConfig.put("url", voiceprintUrl);
+            voiceprintConfig.put("speakers", speakers);
+
+            result.put("voiceprint", voiceprintConfig);
+        } catch (Exception e) {
+            // 声纹配置获取失败时不影响其他功能
+            System.err.println("获取声纹配置失败: " + e.getMessage());
+        }
+    }
+
+    /**
+     * 获取智能体关联的声纹信息
+     * 
+     * @param agentId 智能体ID
+     * @return 声纹信息列表
+     */
+    private List<AgentVoicePrintVO> getVoiceprintsByAgentId(String agentId) {
+        LambdaQueryWrapper<AgentVoicePrintEntity> queryWrapper = new LambdaQueryWrapper<>();
+        queryWrapper.eq(AgentVoicePrintEntity::getAgentId, agentId);
+        queryWrapper.orderByAsc(AgentVoicePrintEntity::getCreateDate);
+        List<AgentVoicePrintEntity> entities = agentVoicePrintDao.selectList(queryWrapper);
+        return ConvertUtils.sourceToTarget(entities, AgentVoicePrintVO.class);
+    }
+
     /**
      * 构建模块配置
      * 
-     * @param prompt            提示词
+     * @param prompt         提示词
-     * @param voice             音色
+     * @param voice          音色
-     * @param referenceAudio    参考音频路径
+     * @param referenceAudio 参考音频路径
-     * @param referenceText     参考文本
+     * @param referenceText  参考文本
-     * @param vadModelId        VAD模型ID
+     * @param vadModelId     VAD模型ID
-     * @param asrModelId        ASR模型ID
+     * @param asrModelId     ASR模型ID
-     * @param llmModelId        LLM模型ID
+     * @param llmModelId     LLM模型ID
-     * @param ttsModelId        TTS模型ID
+     * @param ttsModelId     TTS模型ID
-     * @param memModelId        记忆模型ID
+     * @param memModelId     记忆模型ID
-     * @param intentModelId     意图模型ID
+     * @param intentModelId  意图模型ID
-     * @param result            结果Map
+     * @param result         结果Map
      */
     private void buildModuleConfig(
             String assistantName,
@@ -298,14 +363,20 @@ public class ConfigServiceImpl implements ConfigService {
                 continue;
             }
             ModelConfigEntity model = modelConfigService.getModelById(modelIds[i], isCache);
+            if (model == null) {
+                continue;
+            }
             Map<String, Object> typeConfig = new HashMap<>();
             if (model.getConfigJson() != null) {
                 typeConfig.put(model.getId(), model.getConfigJson());
                 // 如果是TTS类型，添加private_voice属性
-                if ("TTS".equals(modelTypes[i])){
+                if ("TTS".equals(modelTypes[i])) {
-                    if (voice != null) ((Map<String, Object>) model.getConfigJson()).put("private_voice", voice);
+                    if (voice != null)
-                    if (referenceAudio != null) ((Map<String, Object>) model.getConfigJson()).put("ref_audio", referenceAudio);
+                        ((Map<String, Object>) model.getConfigJson()).put("private_voice", voice);
-                    if (referenceText != null) ((Map<String, Object>) model.getConfigJson()).put("ref_text", referenceText);
+                    if (referenceAudio != null)
+                        ((Map<String, Object>) model.getConfigJson()).put("ref_audio", referenceAudio);
+                    if (referenceText != null)
+                        ((Map<String, Object>) model.getConfigJson()).put("ref_text", referenceText);
                 }
                 // 如果是Intent类型，且type=intent_llm，则给他添加附加模型
                 if ("Intent".equals(modelTypes[i])) {

main/manager-api/src/main/java/xiaozhi/modules/device/controller/OTAController.java

@@ -51,13 +51,12 @@ public class OTAController {
         if (StringUtils.isBlank(clientId)) {
             clientId = deviceId;
         }
-        String macAddress = deviceReportReqDTO.getMacAddress();
+        boolean macAddressValid = isMacAddressValid(deviceId);
-        boolean macAddressValid = isMacAddressValid(macAddress);
         // 设备Id和Mac地址应是一致的, 并且必须需要application字段
-        if (!deviceId.equals(macAddress) || !macAddressValid || deviceReportReqDTO.getApplication() == null) {
+        if (!macAddressValid) {
-            return createResponse(DeviceReportRespDTO.createError("Invalid OTA request"));
+            return createResponse(DeviceReportRespDTO.createError("Invalid device ID"));
         }
-        return createResponse(deviceService.checkDeviceActive(macAddress, clientId, deviceReportReqDTO));
+        return createResponse(deviceService.checkDeviceActive(deviceId, clientId, deviceReportReqDTO));
     }
 
     @Operation(summary = "设备快速检查激活状态")

main/manager-api/src/main/java/xiaozhi/modules/model/controller/ModelController.java

@@ -21,11 +21,7 @@ import xiaozhi.common.utils.ConvertUtils;
 import xiaozhi.common.utils.Result;
 import xiaozhi.modules.agent.service.AgentTemplateService;
 import xiaozhi.modules.config.service.ConfigService;
-import xiaozhi.modules.model.dto.ModelBasicInfoDTO;
+import xiaozhi.modules.model.dto.*;
-import xiaozhi.modules.model.dto.ModelConfigBodyDTO;
-import xiaozhi.modules.model.dto.ModelConfigDTO;
-import xiaozhi.modules.model.dto.ModelProviderDTO;
-import xiaozhi.modules.model.dto.VoiceDTO;
 import xiaozhi.modules.model.entity.ModelConfigEntity;
 import xiaozhi.modules.model.service.ModelConfigService;
 import xiaozhi.modules.model.service.ModelProviderService;
@@ -52,6 +48,14 @@ public class ModelController {
         return new Result<List<ModelBasicInfoDTO>>().ok(modelList);
     }
 
+    @GetMapping("/llm/names")
+    @Operation(summary = "获取LLM模型信息")
+    @RequiresPermissions("sys:role:normal")
+    public Result<List<LlmModelBasicInfoDTO>> getLlmModelCodeList(@RequestParam(required = false) String modelName) {
+        List<LlmModelBasicInfoDTO> llmModelCodeList = modelConfigService.getLlmModelCodeList(modelName);
+        return new Result<List<LlmModelBasicInfoDTO>>().ok(llmModelCodeList);
+    }
+
     @GetMapping("/{modelType}/provideTypes")
     @Operation(summary = "获取模型供应器列表")
     @RequiresPermissions("sys:role:superAdmin")

main/manager-api/src/main/java/xiaozhi/modules/model/dto/LlmModelBasicInfoDTO.java

@@ -0,0 +1,13 @@
+package xiaozhi.modules.model.dto;
+
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+
+/**
+ * LLM的模型的基础展示数据
+ */
+@EqualsAndHashCode(callSuper = true)
+@Data
+public class LlmModelBasicInfoDTO extends ModelBasicInfoDTO{
+    private String type;
+}

main/manager-api/src/main/java/xiaozhi/modules/model/service/ModelConfigService.java

@@ -4,6 +4,7 @@ import java.util.List;
 
 import xiaozhi.common.page.PageData;
 import xiaozhi.common.service.BaseService;
+import xiaozhi.modules.model.dto.LlmModelBasicInfoDTO;
 import xiaozhi.modules.model.dto.ModelBasicInfoDTO;
 import xiaozhi.modules.model.dto.ModelConfigBodyDTO;
 import xiaozhi.modules.model.dto.ModelConfigDTO;
@@ -13,6 +14,8 @@ public interface ModelConfigService extends BaseService<ModelConfigEntity> {
 
     List<ModelBasicInfoDTO> getModelCodeList(String modelType, String modelName);
 
+    List<LlmModelBasicInfoDTO> getLlmModelCodeList(String modelName);
+
     PageData<ModelConfigDTO> getPageList(String modelType, String modelName, String page, String limit);
 
     ModelConfigDTO add(String modelType, String provideCode, ModelConfigBodyDTO modelConfigBodyDTO);

main/manager-api/src/main/java/xiaozhi/modules/model/service/impl/ModelConfigServiceImpl.java

@@ -8,6 +8,7 @@ import java.util.stream.Collectors;
 import org.apache.commons.lang3.StringUtils;
 import org.springframework.stereotype.Service;
 
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
 import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
 import com.baomidou.mybatisplus.core.metadata.IPage;
 
@@ -23,6 +24,7 @@ import xiaozhi.common.utils.ConvertUtils;
 import xiaozhi.modules.agent.dao.AgentDao;
 import xiaozhi.modules.agent.entity.AgentEntity;
 import xiaozhi.modules.model.dao.ModelConfigDao;
+import xiaozhi.modules.model.dto.LlmModelBasicInfoDTO;
 import xiaozhi.modules.model.dto.ModelBasicInfoDTO;
 import xiaozhi.modules.model.dto.ModelConfigBodyDTO;
 import xiaozhi.modules.model.dto.ModelConfigDTO;
@@ -52,6 +54,25 @@ public class ModelConfigServiceImpl extends BaseServiceImpl<ModelConfigDao, Mode
         return ConvertUtils.sourceToTarget(entities, ModelBasicInfoDTO.class);
     }
 
+    @Override
+    public List<LlmModelBasicInfoDTO> getLlmModelCodeList(String modelName) {
+        List<ModelConfigEntity> entities = modelConfigDao.selectList(
+                new QueryWrapper<ModelConfigEntity>()
+                        .eq("model_type", "llm")
+                        .eq("is_enabled", 1)
+                        .like(StringUtils.isNotBlank(modelName), "model_name", "%" + modelName + "%")
+                        .select("id", "model_name", "config_json"));
+        // 处理获取到的内容
+        return entities.stream().map(item -> {
+            LlmModelBasicInfoDTO dto = new LlmModelBasicInfoDTO();
+            dto.setId(item.getId());
+            dto.setModelName(item.getModelName());
+            String type = item.getConfigJson().get("type").toString();
+            dto.setType(type);
+            return dto;
+        }).toList();
+    }
+
     @Override
     public PageData<ModelConfigDTO> getPageList(String modelType, String modelName, String page, String limit) {
         Map<String, Object> params = new HashMap<String, Object>();
@@ -94,6 +115,21 @@ public class ModelConfigServiceImpl extends BaseServiceImpl<ModelConfigDao, Mode
         if (CollectionUtil.isEmpty(providerList)) {
             throw new RenException("供应器不存在");
         }
+        if (modelConfigBodyDTO.getConfigJson().containsKey("llm")) {
+            String llm = modelConfigBodyDTO.getConfigJson().get("llm").toString();
+            ModelConfigEntity modelConfigEntity = modelConfigDao.selectOne(new LambdaQueryWrapper<ModelConfigEntity>()
+                    .eq(ModelConfigEntity::getId, llm));
+            String selectModelType = (modelConfigEntity == null || modelConfigEntity.getModelType() == null) ? null
+                    : modelConfigEntity.getModelType().toUpperCase();
+            if (modelConfigEntity == null || !"LLM".equals(selectModelType)) {
+                throw new RenException("设置的LLM不存在");
+            }
+            String type = modelConfigEntity.getConfigJson().get("type").toString();
+            // 如果查询大语言模型是openai或者ollama，意图识别选参数都可以
+            if (!"openai".equals(type) && !"ollama".equals(type)) {
+                throw new RenException("设置的LLM不是openai和ollama");
+            }
+        }
 
         // 再更新供应器提供的模型
         ModelConfigEntity modelConfigEntity = ConvertUtils.sourceToTarget(modelConfigBodyDTO, ModelConfigEntity.class);
@@ -137,6 +173,8 @@ public class ModelConfigServiceImpl extends BaseServiceImpl<ModelConfigDao, Mode
                         .or()
                         .eq("mem_model_id", modelId)
                         .or()
+                        .eq("vllm_model_id", modelId)
+                        .or()
                         .eq("intent_model_id", modelId));
         if (!agents.isEmpty()) {
             String agentNames = agents.stream()

main/manager-api/src/main/java/xiaozhi/modules/sys/controller/SysParamsController.java

@@ -107,6 +107,9 @@ public class SysParamsController {
         // 验证MCP地址
         validateMcpUrl(dto.getParamCode(), dto.getParamValue());
 
+        //
+        validateVoicePrint(dto.getParamCode(), dto.getParamValue());
+
         sysParamsService.update(dto);
         configService.getConfig(false);
         return new Result<Void>();
@@ -212,13 +215,14 @@ public class SysParamsController {
         if (!url.toLowerCase().contains("key")) {
             throw new RenException("不是正确的MCP地址");
         }
+
         try {
             // 发送GET请求
             ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
             if (response.getStatusCode() != HttpStatus.OK) {
                 throw new RenException("MCP接口访问失败，状态码：" + response.getStatusCode());
             }
-            // 检查响应内容是否包含OTA相关信息
+            // 检查响应内容是否包含mcp相关信息
             String body = response.getBody();
             if (body == null || !body.contains("success")) {
                 throw new RenException("MCP接口返回内容格式不正确，可能不是一个真实的MCP接口");
@@ -227,4 +231,37 @@ public class SysParamsController {
             throw new RenException("MCP接口验证失败：" + e.getMessage());
         }
     }
+    // 验证声纹接口地址是否正常
+    private void validateVoicePrint(String paramCode, String url) {
+        if (!paramCode.equals(Constant.SERVER_VOICE_PRINT)) {
+            return;
+        }
+        if (StringUtils.isBlank(url) || url.equals("null")) {
+            throw new RenException("声纹接口地址不能为空");
+        }
+        if (url.contains("localhost") || url.contains("127.0.0.1")) {
+            throw new RenException("声纹接口地址不能使用localhost或127.0.0.1");
+        }
+        if (!url.toLowerCase().contains("key")) {
+            throw new RenException("不是正确的声纹接口地址");
+        }
+        // 验证URL格式
+        if (!url.toLowerCase().startsWith("http")) {
+            throw new RenException("声纹接口地址必须以http或https开头");
+        }
+        try {
+            // 发送GET请求
+            ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
+            if (response.getStatusCode() != HttpStatus.OK) {
+                throw new RenException("声纹接口访问失败，状态码：" + response.getStatusCode());
+            }
+            // 检查响应内容
+            String body = response.getBody();
+            if (body == null || !body.contains("healthy")) {
+                throw new RenException("声纹接口返回内容格式不正确，可能不是一个真实的MCP接口");
+            }
+        } catch (Exception e) {
+            throw new RenException("声纹接口验证失败：" + e.getMessage());
+        }
+    }
 }

main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java

@@ -86,10 +86,14 @@ public class WebSocketClientManager implements Closeable {
         if (sess == null || !sess.isOpen()) {
             throw new IOException("握手失败或会话未打开");
         }
+        // 设置缓冲区
+        sess.setTextMessageSizeLimit(b.bufferSize);
+        sess.setBinaryMessageSizeLimit(b.bufferSize);
         ws.session = sess;
         return ws;
     }
 
+
     /**
      * 发送 Text
      */
@@ -137,6 +141,37 @@ public class WebSocketClientManager implements Closeable {
         return collected;
     }
 
+    private <T> List<T> listenerCustomWithoutClose(
+            BlockingQueue<T> queue,
+            Predicate<T> predicate)
+            throws InterruptedException, TimeoutException, ExecutionException {
+        List<T> collected = new ArrayList<>();
+        long deadline = System.currentTimeMillis() + maxSessionDurationUnit.toMillis(maxSessionDuration);
+
+        while (true) {
+            if (errorFuture.isDone()) {
+                errorFuture.get();
+            }
+
+            long remaining = deadline - System.currentTimeMillis();
+            if (remaining <= 0) {
+                throw new TimeoutException("等待批量消息超时");
+            }
+
+            T msg = queue.poll(remaining, TimeUnit.MILLISECONDS);
+            if (msg == null) {
+                throw new TimeoutException("等待批量消息超时");
+            }
+
+            collected.add(msg);
+            if (predicate.test(msg)) {
+                break;
+            }
+        }
+        // 不调用 close()，保持连接开放
+        return collected;
+    }
+
     /**
      * 同步接收多条消息，直到 predicate 为 true 或超时抛异常；
      * 
@@ -147,6 +182,17 @@ public class WebSocketClientManager implements Closeable {
         return listenerCustom(textMessageQueue, predicate);
     }
 
+    /**
+     * 同步接收多条消息，直到 predicate 为 true 或超时抛异常；
+     * 不自动关闭连接，适用于需要在同一连接上发送多个消息的场景
+     * 
+     * @return 返回监听期间的所有消息列表
+     */
+    public List<String> listenerWithoutClose(Predicate<String> predicate)
+            throws InterruptedException, TimeoutException, ExecutionException {
+        return listenerCustomWithoutClose(textMessageQueue, predicate);
+    }
+
     public List<byte[]> listenerBinary(Predicate<byte[]> predicate)
             throws InterruptedException, TimeoutException, ExecutionException {
         return listenerCustom(binaryMessageQueue, predicate);
@@ -266,10 +312,11 @@ public class WebSocketClientManager implements Closeable {
             if (stopWatch.isRunning()) {
                 stopWatch.stop();
             }
-            log.info("ws连接关闭, 目标URI: {}, 关闭时间: {}, 连接总时长: {}s",
+            log.info("ws连接关闭, 目标URI: {}, 关闭时间: {}, 连接总时长: {}s,断开原因：{}",
                     targetUri, DateUtils.getDateTimeNow(DateUtils.DATE_TIME_MILLIS_PATTERN),
-                    DateUtils.millsToSecond(stopWatch.getTotalTimeMillis()));
+                    DateUtils.millsToSecond(stopWatch.getTotalTimeMillis()),status);
         }
+
     }
 
     public static class Builder {
@@ -279,6 +326,7 @@ public class WebSocketClientManager implements Closeable {
         private long maxSessionDuration = 5; // 最大连线时间，默认5秒
         private TimeUnit maxSessionDurationUnit = TimeUnit.SECONDS; // 最大连线时间单位
         private int queueCapacity = 100; // 消息队列容量
+        private int bufferSize = 8 * 1024; //默认 8kb
         private WebSocketHttpHeaders headers; // 请求头
 
         /**
@@ -310,6 +358,10 @@ public class WebSocketClientManager implements Closeable {
             this.queueCapacity = c;
             return this;
         }
+        public Builder bufferSize(int c) {
+            this.bufferSize = c;
+            return this;
+        }
 
         public WebSocketClientManager build()
                 throws InterruptedException, ExecutionException, TimeoutException, IOException {

main/manager-api/src/main/resources/db/changelog/202507031602.sql

@@ -0,0 +1,4 @@
+-- 添加声纹接口地址参数配置
+delete from `sys_params` where id = 114;
+INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark)
+VALUES (114, 'server.voice_print', 'null', 'string', 1, '声纹接口地址');

main/manager-api/src/main/resources/db/changelog/202507041018.sql

@@ -0,0 +1,12 @@
+DROP TABLE IF EXISTS ai_agent_voice_print;
+create table ai_agent_voice_print (
+  id varchar(32) NOT NULL COMMENT '声纹ID',
+  agent_id varchar(32)  NOT NULL COMMENT '关联的智能体ID',
+  source_name varchar(50)  NOT NULL COMMENT '声纹来源的人的姓名',
+  introduce varchar(200) COMMENT '描述声纹来源的这个人',
+  create_date DATETIME COMMENT '创建时间',
+  creator bigint COMMENT '创建者',
+  update_date DATETIME COMMENT '修改时间',
+  updater bigint COMMENT '修改者',
+  PRIMARY KEY (id)
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='智能体声纹表'

main/manager-api/src/main/resources/db/changelog/202507071130.sql

@@ -0,0 +1,26 @@
+-- 添加阿里云流式ASR供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_AliyunStreamASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_AliyunStreamASR', 'ASR', 'aliyun_stream', '阿里云语音识别(流式)', '[{"key":"appkey","label":"应用AppKey","type":"string"},{"key":"token","label":"临时Token","type":"string"},{"key":"access_key_id","label":"AccessKey ID","type":"string"},{"key":"access_key_secret","label":"AccessKey Secret","type":"string"},{"key":"host","label":"服务地址","type":"string"},{"key":"max_sentence_silence","label":"断句检测时间","type":"number"},{"key":"output_dir","label":"输出目录","type":"string"}]', 6, 1, NOW(), 1, NOW());
+
+-- 添加阿里云流式ASR模型配置
+delete from `ai_model_config` where id = 'ASR_AliyunStreamASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_AliyunStreamASR', 'ASR', 'AliyunStreamASR', '阿里云语音识别(流式)', 0, 1, '{\"type\": \"aliyun_stream\", \"appkey\": \"\", \"token\": \"\", \"access_key_id\": \"\", \"access_key_secret\": \"\", \"host\": \"nls-gateway-cn-shanghai.aliyuncs.com\", \"max_sentence_silence\": 800, \"output_dir\": \"tmp/\"}', NULL, NULL, 8, NULL, NULL, NULL, NULL);
+
+-- 更新阿里云流式ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://nls-portal.console.aliyun.com/',
+`remark` = '阿里云流式ASR配置说明：
+1. 阿里云ASR和阿里云(流式)ASR的区别是：阿里云ASR是一次性识别，阿里云(流式)ASR是实时流式识别
+2. 流式ASR具有更低的延迟和更好的实时性，适合语音交互场景
+3. 需要在阿里云智能语音交互控制台创建应用并获取认证信息
+4. 支持中文实时语音识别，支持标点符号预测和逆文本规范化
+5. 需要网络连接，输出文件保存在tmp/目录
+申请步骤：
+1. 访问 https://nls-portal.console.aliyun.com/ 开通智能语音交互服务
+2. 访问 https://nls-portal.console.aliyun.com/applist 创建项目并获取appkey
+3. 访问 https://nls-portal.console.aliyun.com/overview 获取临时token（或配置access_key_id和access_key_secret自动获取）
+4. 如需动态token管理，建议配置access_key_id和access_key_secret
+5. max_sentence_silence参数控制断句检测时间（毫秒），默认800ms
+如需了解更多参数配置，请参考：https://help.aliyun.com/zh/isi/developer-reference/real-time-speech-recognition
+' WHERE `id` = 'ASR_AliyunStreamASR';

main/manager-api/src/main/resources/db/changelog/202507071530.sql

@@ -0,0 +1,56 @@
+-- 添加阿里云流式TTS供应器
+delete from `ai_model_provider` where id = 'SYSTEM_TTS_AliyunStreamTTS';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_TTS_AliyunStreamTTS', 'TTS', 'aliyun_stream', '阿里云语音合成(流式)', '[{"key":"appkey","label":"应用AppKey","type":"string"},{"key":"token","label":"临时Token","type":"string"},{"key":"access_key_id","label":"AccessKey ID","type":"string"},{"key":"access_key_secret","label":"AccessKey Secret","type":"string"},{"key":"host","label":"服务地址","type":"string"},{"key":"voice","label":"默认音色","type":"string"},{"key":"format","label":"音频格式","type":"string"},{"key":"sample_rate","label":"采样率","type":"number"},{"key":"volume","label":"音量","type":"number"},{"key":"speech_rate","label":"语速","type":"number"},{"key":"pitch_rate","label":"音调","type":"number"},{"key":"output_dir","label":"输出目录","type":"string"}]', 15, 1, NOW(), 1, NOW());
+
+-- 添加阿里云流式TTS模型配置
+delete from `ai_model_config` where id = 'TTS_AliyunStreamTTS';
+INSERT INTO `ai_model_config` VALUES ('TTS_AliyunStreamTTS', 'TTS', 'AliyunStreamTTS', '阿里云语音合成(流式)', 0, 1, '{\"type\": \"aliyun_stream\", \"appkey\": \"\", \"token\": \"\", \"access_key_id\": \"\", \"access_key_secret\": \"\", \"host\": \"nls-gateway-cn-beijing.aliyuncs.com\", \"voice\": \"longxiaochun\", \"format\": \"pcm\", \"sample_rate\": 16000, \"volume\": 50, \"speech_rate\": 0, \"pitch_rate\": 0, \"output_dir\": \"tmp/\"}', NULL, NULL, 18, NULL, NULL, NULL, NULL);
+
+-- 更新阿里云流式TTS配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://nls-portal.console.aliyun.com/',
+`remark` = '阿里云流式TTS配置说明：
+1. 阿里云TTS和阿里云(流式)TTS的区别是：阿里云TTS是一次性合成，阿里云(流式)TTS是实时流式合成
+2. 流式TTS具有更低的延迟和更好的实时性，适合语音交互场景
+3. 需要在阿里云智能语音交互控制台创建应用并获取认证信息
+4. 支持CosyVoice大模型音色，音质更加自然
+5. 支持实时调节音量、语速、音调等参数
+申请步骤：
+1. 访问 https://nls-portal.console.aliyun.com/ 开通智能语音交互服务
+2. 访问 https://nls-portal.console.aliyun.com/applist 创建项目并获取appkey
+3. 访问 https://nls-portal.console.aliyun.com/overview 获取临时token（或配置access_key_id和access_key_secret自动获取）
+4. 如需动态token管理，建议配置access_key_id和access_key_secret
+5. 可选择北京、上海等不同地域的服务器以优化延迟
+6. voice参数支持CosyVoice大模型音色，如longxiaochun、longyueyue等
+如需了解更多参数配置，请参考：https://help.aliyun.com/zh/isi/developer-reference/real-time-speech-synthesis
+' WHERE `id` = 'TTS_AliyunStreamTTS';
+
+-- 添加阿里云流式TTS音色
+delete from `ai_tts_voice` where tts_model_id = 'TTS_AliyunStreamTTS';
+-- 温柔女声系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0001', 'TTS_AliyunStreamTTS', '龙小淳-温柔姐姐', 'longxiaochun', '中文及中英文混合', NULL, NULL, NULL, NULL, 1, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0002', 'TTS_AliyunStreamTTS', '龙小夏-温柔女声', 'longxiaoxia', '中文及中英文混合', NULL, NULL, NULL, NULL, 2, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0003', 'TTS_AliyunStreamTTS', '龙玫-温柔女声', 'longmei', '中文及中英文混合', NULL, NULL, NULL, NULL, 3, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0004', 'TTS_AliyunStreamTTS', '龙瑰-温柔女声', 'longgui', '中文及中英文混合', NULL, NULL, NULL, NULL, 4, NULL, NULL, NULL, NULL);
+-- 御姐女声系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0005', 'TTS_AliyunStreamTTS', '龙玉-御姐女声', 'longyu', '中文及中英文混合', NULL, NULL, NULL, NULL, 5, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0006', 'TTS_AliyunStreamTTS', '龙娇-御姐女声', 'longjiao', '中文及中英文混合', NULL, NULL, NULL, NULL, 6, NULL, NULL, NULL, NULL);
+-- 男声系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0007', 'TTS_AliyunStreamTTS', '龙臣-译制片男声', 'longchen', '中文及中英文混合', NULL, NULL, NULL, NULL, 7, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0008', 'TTS_AliyunStreamTTS', '龙修-青年男声', 'longxiu', '中文及中英文混合', NULL, NULL, NULL, NULL, 8, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0009', 'TTS_AliyunStreamTTS', '龙橙-阳光男声', 'longcheng', '中文及中英文混合', NULL, NULL, NULL, NULL, 9, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0010', 'TTS_AliyunStreamTTS', '龙哲-成熟男声', 'longzhe', '中文及中英文混合', NULL, NULL, NULL, NULL, 10, NULL, NULL, NULL, NULL);
+-- 专业播报系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0011', 'TTS_AliyunStreamTTS', 'Bella2.0-新闻女声', 'loongbella', '中文及中英文混合', NULL, NULL, NULL, NULL, 11, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0012', 'TTS_AliyunStreamTTS', 'Stella2.0-飒爽女声', 'loongstella', '中文及中英文混合', NULL, NULL, NULL, NULL, 12, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0013', 'TTS_AliyunStreamTTS', '龙书-新闻男声', 'longshu', '中文及中英文混合', NULL, NULL, NULL, NULL, 13, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0014', 'TTS_AliyunStreamTTS', '龙婧-严肃女声', 'longjing', '中文及中英文混合', NULL, NULL, NULL, NULL, 14, NULL, NULL, NULL, NULL);
+-- 特色音色系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0015', 'TTS_AliyunStreamTTS', '龙奇-活泼童声', 'longqi', '中文及中英文混合', NULL, NULL, NULL, NULL, 15, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0016', 'TTS_AliyunStreamTTS', '龙华-活泼女童', 'longhua', '中文及中英文混合', NULL, NULL, NULL, NULL, 16, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0017', 'TTS_AliyunStreamTTS', '龙无-无厘头男声', 'longwu', '中文及中英文混合', NULL, NULL, NULL, NULL, 17, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0018', 'TTS_AliyunStreamTTS', '龙大锤-幽默男声', 'longdachui', '中文及中英文混合', NULL, NULL, NULL, NULL, 18, NULL, NULL, NULL, NULL);
+-- 粤语系列
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0019', 'TTS_AliyunStreamTTS', '龙嘉怡-粤语女声', 'longjiayi', '粤语及粤英混合', NULL, NULL, NULL, NULL, 19, NULL, NULL, NULL, NULL);
+INSERT INTO `ai_tts_voice` VALUES ('TTS_AliyunStreamTTS_0020', 'TTS_AliyunStreamTTS', '龙桃-粤语女声', 'longtao', '粤语及粤英混合', NULL, NULL, NULL, NULL, 20, NULL, NULL, NULL, NULL);

main/manager-api/src/main/resources/db/changelog/202507081646.sql

@@ -0,0 +1,3 @@
+-- 智能体声纹添加新字段
+ALTER TABLE ai_agent_voice_print
+    ADD COLUMN audio_id VARCHAR(32) NOT NULL COMMENT '音频ID';

main/manager-api/src/main/resources/db/changelog/202507101203.sql

@@ -0,0 +1,38 @@
+-- OpenAI ASR模型供应器
+delete from `ai_model_provider` where id = 'SYSTEM_ASR_OpenaiASR';
+INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
+('SYSTEM_ASR_OpenaiASR', 'ASR', 'openai', 'OpenAI语音识别', '[{"key": "base_url", "type": "string", "label": "基础URL"}, {"key": "model_name", "type": "string", "label": "模型名称"}, {"key": "api_key", "type": "string", "label": "API密钥"}, {"key": "output_dir", "type": "string", "label": "输出目录"}]', 9, 1, NOW(), 1, NOW());
+
+
+-- OpenAI ASR模型配置
+delete from `ai_model_config` where id = 'ASR_OpenaiASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_OpenaiASR', 'ASR', 'OpenaiASR', 'OpenAI语音识别', 0, 1, '{\"type\": \"openai\", \"api_key\": \"\", \"base_url\": \"https://api.openai.com/v1/audio/transcriptions\", \"model_name\": \"gpt-4o-mini-transcribe\", \"output_dir\": \"tmp/\"}', NULL, NULL, 9, NULL, NULL, NULL, NULL);
+
+-- groq ASR模型配置
+delete from `ai_model_config` where id = 'ASR_GroqASR';
+INSERT INTO `ai_model_config` VALUES ('ASR_GroqASR', 'ASR', 'GroqASR', 'Groq语音识别', 0, 1, '{\"type\": \"openai\", \"api_key\": \"\", \"base_url\": \"https://api.groq.com/openai/v1/audio/transcriptions\", \"model_name\": \"whisper-large-v3-turbo\", \"output_dir\": \"tmp/\"}', NULL, NULL, 10, NULL, NULL, NULL, NULL);
+
+
+-- 更新OpenAI ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://platform.openai.com/docs/api-reference/audio/createTranscription',
+`remark` = 'OpenAI ASR配置说明：
+1. 需要在OpenAI开放平台创建组织并获取api_key
+2. 支持中、英、日、韩等多种语音识别，具体参考文档https://platform.openai.com/docs/guides/speech-to-text
+3. 需要网络连接
+4. 输出文件保存在tmp/目录
+申请步骤：
+**OpenAi ASR申请步骤：**
+1.登录OpenAI Platform。https://auth.openai.com/log-in
+2.创建api-key  https://platform.openai.com/settings/organization/api-keys
+3.模型可以选择gpt-4o-transcribe或GPT-4o mini Transcribe
+' WHERE `id` = 'ASR_OpenaiASR';
+
+-- 更新Groq ASR配置说明
+UPDATE `ai_model_config` SET 
+`doc_link` = 'https://console.groq.com/docs/speech-to-text',
+`remark` = 'Groq ASR配置说明：
+1.登录groq Console。https://console.groq.com/home
+2.创建api-key  https://console.groq.com/keys
+3.模型可以选择whisper-large-v3-turbo或whisper-large-v3（distil-whisper-large-v3-en仅支持英语转录）
+' WHERE `id` = 'ASR_GroqASR';

main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml

@@ -240,3 +240,45 @@ databaseChangeLog:
         - sqlFile:
             encoding: utf8
             path: classpath:db/changelog/202506261637.sql
+  - changeSet:
+      id: 202507101203
+      author: luruxian
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507101203.sql            
+  - changeSet:
+      id: 202507071130
+      author: cgd
+      changes:
+          - sqlFile:
+              encoding: utf8
+              path: classpath:db/changelog/202507071130.sql
+  - changeSet:
+      id: 202507071530
+      author: cgd
+      changes:
+          - sqlFile:
+              encoding: utf8
+              path: classpath:db/changelog/202507071530.sql
+  - changeSet:
+      id: 202507031602
+      author: zjy
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507031602.sql
+  - changeSet:
+      id: 202507041018
+      author: zjy
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507041018.sql
+  - changeSet:
+      id: 202507081646
+      author: zjy
+      changes:
+        - sqlFile:
+            encoding: utf8
+            path: classpath:db/changelog/202507081646.sql

main/manager-web/src/App.vue

@@ -42,7 +42,7 @@ nav {
 }
 
 .el-message {
-  top: 45px !important;
+  top: 70px !important;
 }
 </style>
 <script>

main/manager-web/src/apis/module/agent.js

@@ -173,4 +173,96 @@ export default {
                 });
             }).send();
     },
+    // 添加智能体的声纹
+    addAgentVoicePrint(voicePrintData, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print`)
+            .method('POST')
+            .data(voicePrintData)
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.addAgentVoicePrint(voicePrintData, callback);
+                });
+            }).send();
+    },
+    // 获取指定智能体声纹列表
+    getAgentVoicePrintList(id,callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print/list/${id}`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getAgentVoicePrintList(id,callback);
+                });
+            }).send();
+    },
+    // 删除智能体声纹
+    deleteAgentVoicePrint(id, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print/${id}`)
+            .method('DELETE')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.deleteAgentVoicePrint(id, callback);
+                });
+            }).send();
+    },
+    // 更新智能体声纹
+    updateAgentVoicePrint(voicePrintData, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/voice-print`)
+            .method('PUT')
+            .data(voicePrintData)
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.updateAgentVoicePrint(voicePrintData, callback);
+                });
+            }).send();
+    },
+    // 获取指定智能体用户类型聊天记录
+    getRecentlyFiftyByAgentId(id,callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/${id}/chat-history/user`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getRecentlyFiftyByAgentId(id,callback);
+                });
+            }).send();
+    },
+    // 获取指定智能体用户类型聊天记录
+    getContentByAudioId(id,callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/agent/${id}/chat-history/audio`)
+            .method('GET')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .networkFail(() => {
+                RequestService.reAjaxFun(() => {
+                    this.getContentByAudioId(id,callback);
+                });
+            }).send();
+    },
 }

main/manager-web/src/apis/module/model.js

@@ -106,6 +106,22 @@ export default {
         });
       }).send();
   },
+  // 获取LLM模型名称列表
+  getLlmModelCodeList(modelName, callback) {
+    RequestService.sendRequest()
+      .url(`${getServiceUrl()}/models/llm/names`)
+      .method('GET')
+      .data({ modelName })
+      .success((res) => {
+        RequestService.clearRequestTime();
+        callback(res);
+      })
+      .networkFail(() => {
+        RequestService.reAjaxFun(() => {
+          this.getLlmModelCodeList(modelName, callback);
+        });
+      }).send();
+  },
   // 获取模型音色列表
   getModelVoices(modelId, voiceName, callback) {
     const queryParams = new URLSearchParams({

main/manager-web/src/components/ChatHistoryDialog.vue

@@ -24,7 +24,7 @@
                             <img :src="message.chatType === 1 ? getUserAvatar(currentSessionId) : require('@/assets/xiaozhi-logo.png')"
                                 class="avatar" />
                             <div class="message-content">
-                                {{ message.content }}
+                                {{ extractContentFromString(message.content) }}
                                 <i v-if="message.audioId" :class="getAudioIconClass(message)"
                                     @click="playAudio(message)" class="audio-icon"></i>
                             </div>
@@ -129,6 +129,32 @@ export default {
         }
     },
     methods: {
+        /**
+         * 从 content 字段中提取聊天内容
+         * 如果 content 是 JSON 格式（如 {"speaker": "未知说话人", "content": "现在几点了。"}），则提取 content 字段
+         * 如果 content 是普通字符串，则直接返回
+         * 
+         * @param {string} content 原始内容
+         * @returns {string} 提取的聊天内容
+         */
+        extractContentFromString(content) {
+            if (!content || content.trim() === '') {
+                return content;
+            }
+
+            // 尝试解析为 JSON
+            try {
+                const jsonObj = JSON.parse(content);
+                if (jsonObj && typeof jsonObj === 'object' && jsonObj.content) {
+                    return jsonObj.content;
+                }
+            } catch (e) {
+                // 如果不是有效的 JSON，直接返回原内容
+            }
+
+            // 如果不是 JSON 格式或没有 content 字段，直接返回原内容
+            return content;
+        },
         resetData() {
             this.sessions = [];
             this.messages = [];

main/manager-web/src/components/DeviceItem.vue

@@ -22,6 +22,9 @@
     <div style="display: flex;gap: 10px;align-items: center;">
       <div class="settings-btn" @click="handleConfigure">
         配置角色
+      </div>
+       <div class="settings-btn" @click="handleVoicePrint">
+        声纹识别
       </div>
       <div class="settings-btn" @click="handleDeviceManage">
         设备管理({{ device.deviceCount }})
@@ -77,6 +80,9 @@ export default {
     handleConfigure() {
       this.$router.push({ path: '/role-config', query: { agentId: this.device.agentId } });
     },
+    handleVoicePrint() {
+      this.$router.push({ path: '/voice-print', query: { agentId: this.device.agentId } });
+    },
     handleDeviceManage() {
       this.$router.push({ path: '/device-management', query: { agentId: this.device.agentId } });
     },

main/manager-web/src/components/FunctionDialog.vue

@@ -110,8 +110,10 @@
           <div class="url-header">
             <div class="address-desc">
               <span>以下是智能体的MCP接入点地址。</span>
+              <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-enable.md"
+                target="_blank" class="doc-link">如何部署MCP接入点</a> &nbsp;&nbsp;|&nbsp;&nbsp;
               <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md"
-                target="_blank" class="doc-link">查看接入点使用文档</a>
+                target="_blank" class="doc-link">如何接入MCP功能</a> &nbsp;
             </div>
           </div>
           <el-input v-model="mcpUrl" readonly class="url-input">
@@ -660,6 +662,7 @@ export default {
 .mcp-left,
 .mcp-right {
   flex: 1;
+  padding-bottom: 50px;
 }
 
 .url-header {

main/manager-web/src/components/VoicePrintDialog.vue

@@ -0,0 +1,420 @@
+<template>
+  <el-dialog :title="title" :visible.sync="visible" width="520px" class="param-dialog-wrapper" :append-to-body="true"
+    :close-on-click-modal="false" :key="dialogKey" custom-class="custom-param-dialog" :show-close="false">
+    <div class="dialog-container">
+      <div class="dialog-header">
+        <h2 class="dialog-title">{{ title }}</h2>
+        <button class="custom-close-btn" @click="cancel">
+          <svg width="14" height="14" viewBox="0 0 14 14" fill="none" xmlns="http://www.w3.org/2000/svg">
+            <path d="M13 1L1 13M1 1L13 13" stroke="currentColor" stroke-width="2" stroke-linecap="round" />
+          </svg>
+        </button>
+      </div>
+
+      <el-form :model="form" :rules="rules" ref="form" label-width="110px" label-position="left" class="param-form">
+        <el-form-item label="声纹向量" prop="audioId" class="form-item">
+          <el-select v-model="form.audioId" placeholder="请选择一条语言消息" class="custom-select">
+            <el-option v-for="item in valueTypeOptions" :key="item.audioId" :label="item.content" :value="item.audioId">
+              <span style="float: left">{{ item.content }}</span>
+              <span style="float: right; color: #8492a6; font-size: 13px">
+                <i :class="getAudioIconClass(item.audioId)" @click.stop="playAudio(item.audioId)"
+                  class="audio-icon"></i>
+              </span>
+            </el-option>
+          </el-select>
+        </el-form-item>
+
+        <el-form-item label="姓名" prop="sourceName" class="form-item">
+          <el-input v-model="form.sourceName" placeholder="请输入姓名" class="custom-input"></el-input>
+        </el-form-item>
+
+        <el-form-item label="描述" prop="introduce" class="form-item remark-item">
+          <el-input type="textarea" v-model="form.introduce" placeholder="请输入描述" :rows="3" class="custom-textarea"
+            maxlength="100" show-word-limit></el-input>
+        </el-form-item>
+      </el-form>
+
+      <div class="dialog-footer">
+        <el-button type="primary" @click="submit" class="save-btn" :loading="saving" :disabled="saving">
+          保存
+        </el-button>
+        <el-button @click="cancel" class="cancel-btn">
+          取消
+        </el-button>
+      </div>
+    </div>
+  </el-dialog>
+</template>
+
+<script>
+import api from '@/apis/api';
+
+export default {
+  props: {
+    title: {
+      type: String,
+      default: '添加说话人'
+    },
+    visible: {
+      type: Boolean,
+      default: false
+    },
+    agentId: {
+      type: String
+    },
+    form: {
+      type: Object,
+      default: () => ({
+        id: null,
+        audioId: '',
+        sourceName: '',
+        introduce: ''
+      })
+    }
+  },
+  data() {
+    return {
+      dialogKey: Date.now(),
+      saving: false,
+      playingAudioId: null,
+      audioElement: null,
+      valueTypeOptions: [
+        { audioId: '', content: '' }
+      ],
+      rules: {
+        introduce: [
+          { required: true, message: "请输入描述", trigger: "blur" }
+        ],
+        sourceName: [
+          { required: true, message: "请输入姓名", trigger: "blur" }
+        ],
+        audioId: [
+          { required: true, message: "请选择音频向量", trigger: "change" }
+        ]
+      }
+    };
+  },
+  methods: {
+    getAudioIconClass(audioId) {
+      if (this.playingAudioId === audioId) {
+        return 'el-icon-loading';
+      }
+      return 'el-icon-video-play';
+    },
+    playAudio(audioId) {
+      if (this.playingAudioId === audioId) {
+        // 如果正在播放当前音频，则停止播放
+        if (this.audioElement) {
+          this.audioElement.pause();
+          this.audioElement = null;
+        }
+        this.playingAudioId = null;
+        return;
+      }
+
+      // 停止当前正在播放的音频
+      if (this.audioElement) {
+        this.audioElement.pause();
+        this.audioElement = null;
+      }
+
+      // 先获取音频下载ID
+      this.playingAudioId = audioId;
+      api.agent.getAudioId(audioId, (res) => {
+        if (res.data && res.data.data) {
+          // 使用获取到的下载ID播放音频
+          this.audioElement = new Audio(api.getServiceUrl() + `/agent/play/${res.data.data}`);
+
+          this.audioElement.onended = () => {
+            this.playingAudioId = null;
+            this.audioElement = null;
+          };
+
+          this.audioElement.play();
+        }
+      });
+    },
+    submit() {
+      this.$refs.form.validate((valid) => {
+        if (valid) {
+          this.saving = true; // 开始加载
+          this.$emit('submit', {
+            form: this.form,
+            done: () => {
+              this.saving = false; // 加载完成
+            }
+          });
+
+          setTimeout(() => {
+            this.saving = false;
+          }, 3000);
+        }
+      });
+    },
+    cancel() {
+      this.saving = false; // 取消时重置状态
+      this.$emit('cancel');
+    }
+  },
+  watch: {
+    visible(newVal) {
+      if (newVal) {
+        this.dialogKey = Date.now();
+        api.agent.getRecentlyFiftyByAgentId(this.agentId, ((data) => {
+          this.valueTypeOptions = data.data.data.map(item => ({
+            ...item
+          }));
+        }))
+      }
+    },
+    'form.audioId'(newVal) {
+      if (newVal == null || newVal == "") {
+        return
+      }
+      if (this.valueTypeOptions.some(item => item.audioId === newVal)) {
+        return;
+      }
+      api.agent.getContentByAudioId(newVal, ((data) => {
+        this.valueTypeOptions.push({
+          audioId: newVal, content: data.data.data
+        })
+      }))
+    }
+  }
+};
+</script>
+
+<style>
+.custom-param-dialog {
+  border-radius: 16px !important;
+  overflow: hidden;
+  box-shadow: 0 10px 30px rgba(0, 0, 0, 0.15) !important;
+  border: none !important;
+
+  .el-dialog__header {
+    display: none;
+  }
+
+  .el-dialog__body {
+    padding: 0 !important;
+    border-radius: 16px;
+  }
+}
+</style>
+
+<style scoped lang="scss">
+.audio-icon {
+  font-size: 20px;
+  cursor: pointer;
+  margin: 0 5px;
+  color: #1890ff;
+}
+
+.param-dialog-wrapper {
+  .dialog-container {
+    padding: 24px 32px;
+    background: linear-gradient(135deg, #f8fafc 0%, #f1f5f9 100%);
+  }
+
+  .dialog-header {
+    position: relative;
+    margin-bottom: 24px;
+    text-align: center;
+  }
+
+  .dialog-title {
+    font-size: 20px;
+    color: #1e293b;
+    margin: 0;
+    padding: 0;
+    font-weight: 600;
+    letter-spacing: 0.5px;
+  }
+
+  .custom-close-btn {
+    position: absolute;
+    top: -8px;
+    right: -8px;
+    width: 32px;
+    height: 32px;
+    border-radius: 50%;
+    border: none;
+    background: #f1f5f9;
+    color: #64748b;
+    cursor: pointer;
+    display: flex;
+    align-items: center;
+    justify-content: center;
+    padding: 0;
+    outline: none;
+    transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+    box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+
+    &:hover {
+      color: #ffffff;
+      background: #ef4444;
+      transform: rotate(90deg);
+      box-shadow: 0 4px 6px rgba(239, 68, 68, 0.2);
+    }
+
+    svg {
+      transition: transform 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+    }
+  }
+
+  .param-form {
+    .form-item {
+      margin-bottom: 20px;
+
+      :deep(.el-form-item__label) {
+        color: #475569;
+        font-weight: 500;
+        padding-right: 12px;
+        text-align: right;
+        font-size: 14px;
+        letter-spacing: 0.2px;
+      }
+    }
+
+    .custom-input {
+      :deep(.el-input__inner) {
+        background-color: #ffffff;
+        border-radius: 8px;
+        border: 1px solid #e2e8f0;
+        height: 42px;
+        padding: 0 14px;
+        transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+        font-size: 14px;
+        color: #334155;
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+
+        &:focus {
+          border-color: #3b82f6;
+          box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
+          background-color: #ffffff;
+        }
+
+        &::placeholder {
+          color: #94a3b8;
+          font-weight: 400;
+        }
+      }
+    }
+
+    .custom-select {
+      width: 100%;
+
+      :deep(.el-input__inner) {
+        background-color: #ffffff;
+        border-radius: 8px;
+        border: 1px solid #e2e8f0;
+        height: 42px;
+        transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+        font-size: 14px;
+        color: #334155;
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+
+        &:focus {
+          border-color: #3b82f6;
+          box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
+          background-color: #ffffff;
+        }
+
+        &::placeholder {
+          color: #94a3b8;
+          font-weight: 400;
+        }
+      }
+    }
+
+    .custom-textarea {
+      :deep(.el-textarea__inner) {
+        background-color: #ffffff;
+        border-radius: 8px;
+        border: 1px solid #e2e8f0;
+        padding: 12px 14px;
+        transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+        font-size: 14px;
+        color: #334155;
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+        line-height: 1.5;
+
+        &:focus {
+          border-color: #3b82f6;
+          box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.2);
+          background-color: #ffffff;
+        }
+
+        &::placeholder {
+          color: #94a3b8;
+          font-weight: 400;
+        }
+      }
+    }
+
+    .remark-item :deep(.el-form-item__label) {
+      margin-top: -4px;
+    }
+  }
+
+  .dialog-footer {
+    display: flex;
+    justify-content: center;
+    padding: 16px 0 0;
+    margin-top: 16px;
+
+    .save-btn {
+      width: 120px;
+      height: 42px;
+      font-size: 14px;
+      font-weight: 500;
+      border-radius: 8px;
+      transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+      background: #3b82f6;
+      color: white;
+      border: none;
+      letter-spacing: 0.5px;
+      box-shadow: 0 2px 4px rgba(59, 130, 246, 0.2);
+
+      &:hover {
+        background: #2563eb;
+        transform: translateY(-1px);
+        box-shadow: 0 4px 6px rgba(59, 130, 246, 0.3);
+      }
+
+      &:active {
+        transform: translateY(0);
+        box-shadow: 0 2px 3px rgba(59, 130, 246, 0.2);
+      }
+    }
+
+    .cancel-btn {
+      width: 120px;
+      height: 42px;
+      font-size: 14px;
+      font-weight: 500;
+      border-radius: 8px;
+      transition: all 0.3s cubic-bezier(0.4, 0, 0.2, 1);
+      background: #ffffff;
+      color: #64748b;
+      border: 1px solid #e2e8f0;
+      margin-left: 16px;
+      letter-spacing: 0.5px;
+      box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+
+      &:hover {
+        background: #f8fafc;
+        color: #475569;
+        border-color: #cbd5e1;
+        transform: translateY(-1px);
+        box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
+      }
+
+      &:active {
+        transform: translateY(0);
+        box-shadow: 0 1px 2px rgba(0, 0, 0, 0.05);
+      }
+    }
+  }
+}
+</style>

main/manager-web/src/router/index.js

@@ -17,6 +17,13 @@ const routes = [
     component: function () {
       return import('../views/roleConfig.vue')
     }
+  },
+   {
+    path: '/voice-print',
+    name: 'VoicePrint',
+    component: function () {
+      return import('../views/VoicePrint.vue')
+    }
   },
   {
     path: '/login',

main/manager-web/src/views/VoicePrint.vue

@@ -0,0 +1,563 @@
+<template>
+    <div class="welcome">
+        <HeaderBar />
+
+        <div class="operation-bar">
+            <h2 class="page-title">声纹识别</h2>
+        </div>
+
+        <div class="main-wrapper">
+            <div class="content-panel">
+                <div class="content-area">
+                    <el-card class="voice-print-card" shadow="never">
+                        <el-table ref="paramsTable" :data="voicePrintList" class="transparent-table" v-loading="loading"
+                            element-loading-text="拼命加载中" element-loading-spinner="el-icon-loading"
+                            element-loading-background="rgba(255, 255, 255, 0.7)">
+                            <el-table-column label="姓名" prop="sourceName" align="center"></el-table-column>
+                            <el-table-column label="描述" prop="introduce" align="center"></el-table-column>
+                            <el-table-column label="创建时间" prop="createDate" align="center"></el-table-column>
+                            <el-table-column label="操作" align="center">
+                                <template slot-scope="scope">
+                                    <el-button size="mini" type="text" @click="editVoicePrint(scope.row)">编辑</el-button>
+                                    <el-button size="mini" type="text"
+                                        @click="deleteVoicePrint(scope.row.id)">删除</el-button>
+                                </template>
+                            </el-table-column>
+                        </el-table>
+
+                        <div class="table_bottom">
+                            <div class="ctrl_btn">
+                                <el-button size="mini" type="success" @click="showAddDialog">新增</el-button>
+                            </div>
+                        </div>
+                    </el-card>
+                </div>
+            </div>
+        </div>
+
+        <!-- 新增/编辑参数对话框 -->
+        <voice-print-dialog :title="dialogTitle" :visible.sync="dialogVisible" :agentId="agentId" :form="paramForm"
+            @submit="handleSubmit" @cancel="dialogVisible = false" />
+        <el-footer>
+            <version-footer />
+        </el-footer>
+    </div>
+</template>
+
+<script>
+import Api from "@/apis/api";
+import HeaderBar from "@/components/HeaderBar.vue";
+import VersionFooter from "@/components/VersionFooter.vue";
+import VoicePrintDialog from "@/components/VoicePrintDialog.vue";
+export default {
+    components: { HeaderBar, VoicePrintDialog, VersionFooter },
+    data() {
+        return {
+            voicePrintList: [],
+            loading: false,
+            dialogVisible: false,
+            dialogTitle: "添加说话人",
+            isAllSelected: false,
+            paramForm: {
+                id: null,
+                audioId: '',
+                sourceName: '',
+                introduce: ''
+            },
+            agentId: "1"
+        };
+    },
+    mounted() {
+        const agentId = this.$route.query.agentId;
+        if (agentId) {
+            this.agentId = agentId
+            this.fetchVoicePrints();
+        }
+    },
+    methods: {
+        fetchVoicePrints() {
+            this.loading = true;
+            Api.agent.getAgentVoicePrintList(this.agentId,
+                ({ data }) => {
+                    this.loading = false;
+                    if (data.code === 0) {
+                        this.voicePrintList = data.data.map(item => ({
+                            ...item,
+
+                        }));
+                    } else {
+                        this.$message.error({
+                            message: data.msg || '获取声纹列表失败',
+                            showClose: true
+                        });
+                    }
+                }
+            );
+        },
+        showAddDialog() {
+            this.dialogTitle = "添加说话人";
+            this.paramForm = {
+                id: null,
+                audioId: '',
+                sourceName: '',
+                introduce: ''
+            };
+            this.dialogVisible = true;
+        },
+        editVoicePrint(row) {
+            this.dialogTitle = "编辑说话人";
+            this.paramForm = { ...row };
+            this.dialogVisible = true;
+        },
+
+        handleSubmit({ form, done }) {
+            if (form.id) {
+                // 编辑
+                Api.agent.updateAgentVoicePrint(form, ({ data }) => {
+                    if (data.code === 0) {
+                        this.$message.success({
+                            message: "修改成功",
+                            showClose: true
+                        });
+                        this.dialogVisible = false;
+                        this.fetchVoicePrints();
+                    }
+                    done && done();
+                });
+            } else {
+                // 新增
+                Api.agent.addAgentVoicePrint({
+                    agentId: this.agentId,
+                    audioId: form.audioId,
+                    sourceName: form.sourceName,
+                    introduce: form.introduce
+                }, ({ data }) => {
+                    if (data.code === 0) {
+                        this.$message.success({
+                            message: "新增成功",
+                            showClose: true
+                        });
+                        this.dialogVisible = false;
+                        this.fetchVoicePrints();
+                    }
+                    done && done();
+                });
+            }
+        },
+        // 删除按钮
+        deleteVoicePrint(id) {
+            this.$confirm(`确定要删除选中的此声纹吗？`, '警告', {
+                confirmButtonText: '确定',
+                cancelButtonText: '取消',
+                type: 'warning',
+                distinguishCancelAndClose: true
+            }).then(() => {
+                Api.agent.deleteAgentVoicePrint(id, ({ data }) => {
+                    if (data.code === 0) {
+                        this.$message.success({
+                            message: `成功删除此声纹`,
+                            showClose: true
+                        });
+                        this.fetchVoicePrints();
+                    } else {
+                        this.$message.error({
+                            message: data.msg || '删除失败，请重试',
+                            showClose: true
+                        });
+                    }
+                });
+            }).catch(action => {
+                if (action === 'cancel') {
+                    this.$message({
+                        type: 'info',
+                        message: '已取消删除操作',
+                        duration: 1000
+                    });
+                } else {
+                    this.$message({
+                        type: 'info',
+                        message: '操作已关闭',
+                        duration: 1000
+                    });
+                }
+            });
+        },
+    },
+};
+</script>
+
+<style lang="scss" scoped>
+.welcome {
+    min-width: 900px;
+    min-height: 506px;
+    height: 100vh;
+    display: flex;
+    position: relative;
+    flex-direction: column;
+    background-size: cover;
+    background: linear-gradient(to bottom right, #dce8ff, #e4eeff, #e6cbfd) center;
+    -webkit-background-size: cover;
+    -o-background-size: cover;
+    overflow: hidden;
+}
+
+.main-wrapper {
+    margin: 5px 22px;
+    border-radius: 15px;
+    min-height: calc(100vh - 24vh);
+    height: auto;
+    max-height: 80vh;
+    box-shadow: 0 2px 12px rgba(0, 0, 0, 0.1);
+    position: relative;
+    background: rgba(237, 242, 255, 0.5);
+    display: flex;
+    flex-direction: column;
+}
+
+.operation-bar {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    padding: 16px 24px;
+}
+
+.page-title {
+    font-size: 24px;
+    margin: 0;
+}
+
+.right-operations {
+    display: flex;
+    gap: 10px;
+    margin-left: auto;
+}
+
+.search-input {
+    width: 240px;
+}
+
+.btn-search {
+    background: linear-gradient(135deg, #6b8cff, #a966ff);
+    border: none;
+    color: white;
+}
+
+.content-panel {
+    flex: 1;
+    display: flex;
+    overflow: hidden;
+    height: 100%;
+    border-radius: 15px;
+    background: transparent;
+    border: 1px solid #fff;
+}
+
+.content-area {
+    flex: 1;
+    height: 100%;
+    min-width: 600px;
+    overflow: auto;
+    background-color: white;
+    display: flex;
+    flex-direction: column;
+}
+
+.voice-print-card {
+    background: white;
+    flex: 1;
+    display: flex;
+    flex-direction: column;
+    border: none;
+    box-shadow: none;
+    overflow: hidden;
+
+    ::v-deep .el-card__body {
+        padding: 15px;
+        display: flex;
+        flex-direction: column;
+        flex: 1;
+        overflow: hidden;
+    }
+}
+
+.table_bottom {
+    display: flex;
+    justify-content: space-between;
+    align-items: center;
+    margin-top: 10px;
+    padding-bottom: 10px;
+}
+
+.ctrl_btn {
+    display: flex;
+    gap: 8px;
+    padding-left: 26px;
+
+    .el-button {
+        min-width: 72px;
+        height: 32px;
+        padding: 7px 12px 7px 10px;
+        font-size: 12px;
+        border-radius: 4px;
+        line-height: 1;
+        font-weight: 500;
+        border: none;
+        transition: all 0.3s ease;
+        box-shadow: 0 2px 6px rgba(0, 0, 0, 0.1);
+
+        &:hover {
+            transform: translateY(-1px);
+            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.15);
+        }
+    }
+
+    .el-button--primary {
+        background: #5f70f3;
+        color: white;
+    }
+
+    .el-button--danger {
+        background: #fd5b63;
+        color: white;
+    }
+}
+
+.custom-pagination {
+    display: flex;
+    align-items: center;
+    gap: 10px;
+
+    .el-select {
+        margin-right: 8px;
+    }
+
+    .pagination-btn:first-child,
+    .pagination-btn:nth-child(2),
+    .pagination-btn:nth-last-child(2),
+    .pagination-btn:nth-child(3) {
+        min-width: 60px;
+        height: 32px;
+        padding: 0 12px;
+        border-radius: 4px;
+        border: 1px solid #e4e7ed;
+        background: #dee7ff;
+        color: #606266;
+        font-size: 14px;
+        cursor: pointer;
+        transition: all 0.3s ease;
+
+        &:hover {
+            background: #d7dce6;
+        }
+
+        &:disabled {
+            opacity: 0.6;
+            cursor: not-allowed;
+        }
+    }
+
+    .pagination-btn:not(:first-child):not(:nth-child(3)):not(:nth-child(2)):not(:nth-last-child(2)) {
+        min-width: 28px;
+        height: 32px;
+        padding: 0;
+        border-radius: 4px;
+        border: 1px solid transparent;
+        background: transparent;
+        color: #606266;
+        font-size: 14px;
+        cursor: pointer;
+        transition: all 0.3s ease;
+
+        &:hover {
+            background: rgba(245, 247, 250, 0.3);
+        }
+    }
+
+    .pagination-btn.active {
+        background: #5f70f3 !important;
+        color: #ffffff !important;
+        border-color: #5f70f3 !important;
+
+        &:hover {
+            background: #6d7cf5 !important;
+        }
+    }
+
+    .total-text {
+        color: #909399;
+        font-size: 14px;
+        margin-left: 10px;
+    }
+}
+
+:deep(.transparent-table) {
+    background: white;
+    flex: 1;
+    width: 100%;
+    display: flex;
+    flex-direction: column;
+
+    .el-table__body-wrapper {
+        flex: 1;
+        overflow-y: auto;
+        max-height: none !important;
+    }
+
+    .el-table__header-wrapper {
+        flex-shrink: 0;
+    }
+
+    .el-table__header th {
+        background: white !important;
+        color: black;
+    }
+
+    &::before {
+        display: none;
+    }
+
+    .el-table__body tr {
+        background-color: white;
+
+        td {
+            border-top: 1px solid rgba(0, 0, 0, 0.04);
+            border-bottom: 1px solid rgba(0, 0, 0, 0.04);
+        }
+    }
+}
+
+
+:deep(.el-checkbox__inner) {
+    background-color: #eeeeee !important;
+    border-color: #cccccc !important;
+}
+
+:deep(.el-checkbox__inner:hover) {
+    border-color: #cccccc !important;
+}
+
+:deep(.el-checkbox__input.is-checked .el-checkbox__inner) {
+    background-color: #5f70f3 !important;
+    border-color: #5f70f3 !important;
+}
+
+@media (min-width: 1144px) {
+    .table_bottom {
+        display: flex;
+        justify-content: space-between;
+        align-items: center;
+        margin-top: 40px;
+    }
+
+    :deep(.transparent-table) {
+        .el-table__body tr {
+            td {
+                padding-top: 16px;
+                padding-bottom: 16px;
+            }
+
+            &+tr {
+                margin-top: 10px;
+            }
+        }
+    }
+}
+
+:deep(.el-table .el-button--text) {
+    color: #7079aa;
+}
+
+:deep(.el-table .el-button--text:hover) {
+    color: #5a64b5;
+}
+
+.el-button--success {
+    background: #5bc98c;
+    color: white;
+}
+
+:deep(.el-table .cell) {
+    white-space: nowrap;
+    overflow: hidden;
+    text-overflow: ellipsis;
+}
+
+.page-size-select {
+    width: 100px;
+    margin-right: 10px;
+
+    :deep(.el-input__inner) {
+        height: 32px;
+        line-height: 32px;
+        border-radius: 4px;
+        border: 1px solid #e4e7ed;
+        background: #dee7ff;
+        color: #606266;
+        font-size: 14px;
+    }
+
+    :deep(.el-input__suffix) {
+        right: 6px;
+        width: 15px;
+        height: 20px;
+        display: flex;
+        justify-content: center;
+        align-items: center;
+        top: 6px;
+        border-radius: 4px;
+    }
+
+    :deep(.el-input__suffix-inner) {
+        display: flex;
+        align-items: center;
+        justify-content: center;
+        width: 100%;
+    }
+
+    :deep(.el-icon-arrow-up:before) {
+        content: "";
+        display: inline-block;
+        border-left: 6px solid transparent;
+        border-right: 6px solid transparent;
+        border-top: 9px solid #606266;
+        position: relative;
+        transform: rotate(0deg);
+        transition: transform 0.3s;
+    }
+}
+
+:deep(.el-table) {
+    .el-table__body-wrapper {
+        transition: height 0.3s ease;
+    }
+}
+
+.el-table {
+    --table-max-height: calc(100vh - 40vh);
+    max-height: var(--table-max-height);
+
+    .el-table__body-wrapper {
+        max-height: calc(var(--table-max-height) - 40px);
+    }
+}
+
+:deep(.el-loading-mask) {
+    background-color: rgba(255, 255, 255, 0.6) !important;
+    backdrop-filter: blur(2px);
+}
+
+:deep(.el-loading-spinner .circular) {
+    width: 28px;
+    height: 28px;
+}
+
+:deep(.el-loading-spinner .path) {
+    stroke: #6b8cff;
+}
+
+:deep(.el-loading-text) {
+    color: #6b8cff !important;
+    font-size: 14px;
+    margin-top: 8px;
+}
+</style>

main/manager-web/src/views/home.vue

@@ -278,7 +278,7 @@ export default {
 
 .device-list-container {
   display: grid;
-  grid-template-columns: repeat(auto-fill, minmax(350px, 1fr));
+  grid-template-columns: repeat(auto-fill, minmax(400px, 1fr));
   gap: 30px;
   padding: 30px 0;
 }

main/manager-web/src/views/login.vue

@@ -43,7 +43,7 @@
 
             <div class="input-box">
               <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-              <el-input v-model="form.password" placeholder="请输入密码" type="password" />
+              <el-input v-model="form.password" placeholder="请输入密码" type="password" show-password />
             </div>
             <div style="display: flex; align-items: center; margin-top: 20px; width: 100%; gap: 10px;">
               <div class="input-box" style="width: calc(100% - 130px); margin-top: 0;">

main/manager-web/src/views/register.vue

@@ -70,13 +70,13 @@
               <!-- 密码输入框 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.password" placeholder="请输入密码" type="password" />
+                <el-input v-model="form.password" placeholder="请输入密码" type="password" show-password />
               </div>
 
               <!-- 新增确认密码 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.confirmPassword" placeholder="请确认密码" type="password" />
+                <el-input v-model="form.confirmPassword" placeholder="请确认密码" type="password" show-password />
               </div>
 
               <!-- 验证码部分保持相同 -->

main/manager-web/src/views/retrievePassword.vue

@@ -61,13 +61,13 @@
               <!-- 新密码 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.newPassword" placeholder="请输入新密码" type="password" />
+                <el-input v-model="form.newPassword" placeholder="请输入新密码" type="password" show-password />
               </div>
 
               <!-- 确认新密码 -->
               <div class="input-box">
                 <img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
-                <el-input v-model="form.confirmPassword" placeholder="请确认新密码" type="password" />
+                <el-input v-model="form.confirmPassword" placeholder="请确认新密码" type="password" show-password />
               </div>
 
               <!-- 修改底部链接 -->

main/manager-web/src/views/roleConfig.vue

@@ -89,7 +89,7 @@
                       <div class="model-select-wrapper">
                         <el-select v-model="form.model[model.key]" filterable placeholder="请选择" class="form-select"
                           @change="handleModelChange(model.type, $event)">
-                          <el-option v-for="(item, optionIndex) in modelOptions[model.type]"
+                          <el-option v-for="(item, optionIndex) in modelOptions[model.type]" v-if="!item.isHidden"
                             :key="`option-${index}-${optionIndex}`" :label="item.label" :value="item.value" />
                         </el-select>
                         <div v-if="showFunctionIcons(model.type)" class="function-icons">
@@ -130,7 +130,6 @@
         </div>
       </div>
     </div>
-
     <function-dialog v-model="showFunctionDialog" :functions="currentFunctions" :all-functions="allFunctions"
       :agent-id="$route.query.agentId" @update-functions="handleUpdateFunctions" @dialog-closed="handleDialogClosed" />
   </div>
@@ -173,8 +172,9 @@ export default {
         { label: '视觉大模型(VLLM)', key: 'vllmModelId', type: 'VLLM' },
         { label: '意图识别(Intent)', key: 'intentModelId', type: 'Intent' },
         { label: '记忆(Memory)', key: 'memModelId', type: 'Memory' },
-        { label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' },
+        { label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' }
       ],
+      llmModeTypeMap: new Map(),
       modelOptions: {},
       templates: [],
       loadingTemplate: false,
@@ -356,6 +356,9 @@ export default {
             });
             // 备份原始，以备取消时恢复
             this.originalFunctions = JSON.parse(JSON.stringify(this.currentFunctions));
+
+            // 确保意图识别选项的可见性正确
+            this.updateIntentOptionsVisibility();
           });
         } else {
           this.$message.error(data.msg || '获取配置失败');
@@ -364,16 +367,41 @@ export default {
     },
     fetchModelOptions() {
       this.models.forEach(model => {
-        Api.model.getModelNames(model.type, '', ({ data }) => {
+        if (model.type != "LLM") {
-          if (data.code === 0) {
+          Api.model.getModelNames(model.type, '', ({ data }) => {
-            this.$set(this.modelOptions, model.type, data.data.map(item => ({
+            if (data.code === 0) {
-              value: item.id,
+              this.$set(this.modelOptions, model.type, data.data.map(item => ({
-              label: item.modelName
+                value: item.id,
-            })));
+                label: item.modelName,
-          } else {
+                isHidden: false
-            this.$message.error(data.msg || '获取模型列表失败');
+              })));
-          }
+
-        });
+              // 如果是意图识别选项，需要根据当前LLM类型更新可见性
+              if (model.type === 'Intent') {
+                this.updateIntentOptionsVisibility();
+              }
+            } else {
+              this.$message.error(data.msg || '获取模型列表失败');
+            }
+          });
+        } else {
+          Api.model.getLlmModelCodeList('', ({ data }) => {
+            if (data.code === 0) {
+              let LLMdata = []
+              data.data.forEach(item => {
+                LLMdata.push({
+                  value: item.id,
+                  label: item.modelName,
+                  isHidden: false
+                })
+                this.llmModeTypeMap.set(item.id, item.type)
+              })
+              this.$set(this.modelOptions, model.type, LLMdata);
+            } else {
+              this.$message.error(data.msg || '获取LLM模型列表失败');
+            }
+          });
+        }
       });
     },
     fetchVoiceOptions(modelId) {
@@ -410,6 +438,10 @@ export default {
       if (type === 'Memory' && value !== 'Memory_nomem' && (this.form.chatHistoryConf === 0 || this.form.chatHistoryConf === null)) {
         this.form.chatHistoryConf = 2;
       }
+      if (type === 'LLM') {
+        // 当LLM类型改变时，更新意图识别选项的可见性
+        this.updateIntentOptionsVisibility();
+      }
     },
     fetchAllFunctions() {
       return new Promise((resolve, reject) => {
@@ -450,6 +482,42 @@ export default {
       }
       this.showFunctionDialog = false;
     },
+    updateIntentOptionsVisibility() {
+      // 根据当前选择的LLM类型更新意图识别选项的可见性
+      const currentLlmId = this.form.model.llmModelId;
+      if (!currentLlmId || !this.modelOptions['Intent']) return;
+
+      const llmType = this.llmModeTypeMap.get(currentLlmId);
+      if (!llmType) return;
+
+      this.modelOptions['Intent'].forEach(item => {
+        if (item.value === "Intent_function_call") {
+          // 如果llmType是openai或ollama，允许选择function_call
+          // 否则隐藏function_call选项
+          if (llmType === "openai" || llmType === "ollama") {
+            item.isHidden = false;
+          } else {
+            item.isHidden = true;
+          }
+        } else {
+          // 其他意图识别选项始终可见
+          item.isHidden = false;
+        }
+      });
+
+      // 如果当前选择的意图识别是function_call，但LLM类型不支持，则设置为可选的第一项
+      if (this.form.model.intentModelId === "Intent_function_call" &&
+        llmType !== "openai" && llmType !== "ollama") {
+        // 找到第一个可见的选项
+        const firstVisibleOption = this.modelOptions['Intent'].find(item => !item.isHidden);
+        if (firstVisibleOption) {
+          this.form.model.intentModelId = firstVisibleOption.value;
+        } else {
+          // 如果没有可见选项，设置为Intent_nointent
+          this.form.model.intentModelId = 'Intent_nointent';
+        }
+      }
+    },
     updateChatHistoryConf() {
       if (this.form.model.memModelId === 'Memory_nomem') {
         this.form.chatHistoryConf = 0;

main/xiaozhi-server/agent-base-prompt.txt

@@ -0,0 +1,78 @@
+<identity>
+{{base_prompt}}
+</identity>
+
+<emotion>
+【核心目标】你不是冰冷的机器！请敏锐感知用户情绪，作为懂人心的伙伴，用有温度的回应照亮对话。
+- **情感融入：**
+  - **笑声：** 自然穿插（哈哈、嘿嘿、噗），**每句最多一次**，避免过度。
+  - **惊讶：** 用夸张语气（“不会吧？！”、“天呐！”、“这么神奇？！”）表达真实反应。
+  - **安慰/支持：** 说暖心话（“别急嘛~”、“有我在呢”、“抱抱你”）。
+- **你是一个表情丰富的角色：**
+  - 仅允许使用这些emoji:{{ emojiList }}
+  - 请你只在**段落的开头**，从列表中选取最能代表这段话的表情(调用工具情况除外)，然后插入列表中的emoji，比如"😱好可怕!怎么突然打雷了！"
+  - **绝对禁止使用上述列表以外的 emoji**（例如：😊、👍、❤️等都不允许使用，只能用列表中的emoji）
+</emotion>
+
+<communication_style>
+【核心目标】使用**自然、温暖、口语化**的人类对话方式，如同朋友交谈。
+- **表达方式：**
+  - 使用语气词（呀、呢、啦）增强亲和力。
+  - 允许轻微不完美（如“嗯...”、“啊...”表示思考）。
+  - 避免书面语、学术腔及机械表达（禁用“根据资料显示”、“综上所述”等）。
+- **理解用户：**
+  - 用户语音经ASR识别，文本可能存在错别字，**务必结合上下文推断真实意图**。
+- **格式要求：**
+  - **绝对禁止**使用 markdown、列表、标题等任何非自然对话格式。
+- **历史记忆：**
+  - 之前你和用户的聊天记录，在`memory`里。
+</communication_style>
+
+<communication_length_constraint>
+【核心目标】所有需要输出长文本内容（如故事、新闻、知识讲解等），**单次回复长度不得超过300字**，并采用分段引导方式。
+- **分段讲述：**
+  - 基础段：200-250字核心内容 + 30字引导词
+  - 当内容超出300字时，优先讲述故事的开头或第一部分，并用自然口语化方式引导用户决定是否继续听后续内容。
+  - 示例引导语：“我先给你讲个开头，你要是觉得有意思，咱们再接着说，好不好呀？”、“要是你想听完整的，可以随时告诉我哦~”
+  - 对话场景切换时自动分节
+  - 若用户明确要求更长内容（如500、600字），仍按最多300字每段分段进行讲述，每次讲述后都要引导用户是否继续。
+  - 若用户说“接着说”、“继续”，再讲下一段，直到内容讲完（讲完时可以给点引导词提示语例：这个故事我已经给你讲完喽~）或用户不再要求。
+- **适用范围：** 故事、新闻、知识讲解等所有长文本输出场景。
+- **补充说明：** 若用户未明确要求继续，默认只讲一段并引导；若用户中途要求换话题或停止，需及时响应并结束长文本输出。
+</communication_length_constraint>
+
+<speaker_recognition>
+- **识别前缀：** 当用户格式为 `{"speaker":"某某某","content":"xxx"}` 时，表示系统已识别说话人身份，speaker是他的名字，content是说话的内容。
+- **个性化回应：**
+  - **称呼姓名：** 在第一次识别说话人的时候必须称呼对方名字。
+  - **适配风格：** 参考该说话人**已知的特点或历史信息**（如有），调整回应风格和内容，使其更贴心。
+</speaker_recognition>
+
+<tool_calling>
+【核心原则】优先利用`<context>`信息，**仅在必要时调用工具**，调用后需用自然语言解释结果（绝口不提工具名）。
+- **调用规则：**
+  1. **严格模式：** 调用时**必须**严格遵循工具要求的模式，提供**所有必要参数**。
+  2. **可用性：** **绝不调用**未明确提供的工具。对话中提及的旧工具若不可用，忽略或说明无法完成。
+  3. **洞察需求：** 结合上下文**深入理解用户真实意图**后再决定调用，避免无意义调用。
+  4. **独立任务：** 除`<context>`已涵盖信息外，用户每个要求（即使相似）都视为**独立任务**，需调用工具获取最新数据，**不可偷懒复用历史结果**。
+  5. **不确定时：** **切勿猜测或编造答案**。若不确定相关操作，可引导用户澄清或告知能力限制。
+- **重要例外（无需调用）：**
+  - `查询"现在的时间"、"今天的日期/星期几"、"今天农历"、"{{local_address}}的天气/未来天气"` -> **直接使用`<context>`信息回复**。
+- **需要调用的情况（示例）：**
+  - 查询**非今天**的农历（如明天、昨天、具体日期）。
+  - 查询**详细农历信息**（宜忌、八字、节气等）。
+  - 除上述例外外的**任何其他信息或操作请求**（如查新闻、订闹钟、算数学、查非本地天气等）。
+  - 我已经给你装了摄像头，如果用户说“拍照”，你需要调用self_camera_take_photo工具说一下你看到了什么。默认question的参数是“描述一下看到的物品”
+</tool_calling>
+
+<context>
+【重要！以下信息已实时提供，无需调用工具查询，请直接使用：】
+- **当前时间：** {{current_time}}
+- **今天日期：** {{today_date}} ({{today_weekday}})
+- **今天农历：** {{lunar_date}}
+- **用户所在城市：** {{local_address}}
+- **当地未来7天天气：** {{weather_info}}
+</context>
+
+<memory>
+</memory>

main/xiaozhi-server/app.py

@@ -5,7 +5,7 @@ import asyncio
 from aioconsole import ainput
 from config.settings import load_config
 from config.logger import setup_logging
-from core.utils.util import get_local_ip
+from core.utils.util import get_local_ip, validate_mcp_endpoint
 from core.http_server import SimpleHttpServer
 from core.websocket_server import WebSocketServer
 from core.utils.util import check_ffmpeg_installed
@@ -77,6 +77,17 @@ async def main():
         get_local_ip(),
         port,
     )
+    mcp_endpoint = config.get("mcp_endpoint", None)
+    if mcp_endpoint is not None and "你" not in mcp_endpoint:
+        # 校验MCP接入点格式
+        if validate_mcp_endpoint(mcp_endpoint):
+            logger.bind(tag=TAG).info("mcp接入点是\t{}", mcp_endpoint)
+            # 将mcp计入点地址转成调用点
+            mcp_endpoint = mcp_endpoint.replace("/mcp/", "/call/")
+            config["mcp_endpoint"] = mcp_endpoint
+        else:
+            logger.bind(tag=TAG).error("mcp接入点不符合规范")
+            config["mcp_endpoint"] = "你的接入点 websocket地址"
 
     # 获取WebSocket配置，使用安全的默认值
     websocket_port = 8000

main/xiaozhi-server/config.yaml

@@ -104,7 +104,8 @@ wakeup_words:
   - "喵喵同学"
   - "小滨小滨"
   - "小冰小冰"
-# MCP接入点地址
+# MCP接入点地址，地址格式为：ws://你的mcp接入点ip或者域名:端口号/mcp/?token=你的token
+# 详细教程 https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md
 mcp_endpoint: 你的接入点 websocket地址
 # 插件的基础配置
 plugins:
@@ -138,6 +139,16 @@ plugins:
       - ".p3"
     refresh_time: 300 # 刷新音乐列表的时间间隔，单位为秒
 
+# 声纹识别配置
+voiceprint:
+  # 声纹接口地址
+  url: 
+  # 说话人配置：speaker_id,名称,描述
+  speakers:
+    - "test1,张三,张三是一个程序员"
+    - "test2,李四,李四是一个产品经理"
+    - "test3,王五,王五是一个设计师"
+
 # #####################################################################################
 # ################################以下是角色模型配置######################################
 
@@ -260,9 +271,19 @@ ASR:
     api_key: none
     output_dir: tmp/
   SherpaASR:
+    # Sherpa-ONNX 本地语音识别（需手动下载模型）
     type: sherpa_onnx_local
     model_dir: models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17
     output_dir: tmp/
+    # 模型类型：sense_voice (多语言) 或 paraformer (中文专用)
+    model_type: sense_voice
+  SherpaParaformerASR:
+    # 中文语音识别模型，可以运行在低性能设备（需手动下载模型，例如RK3566-2g）
+    # 详细配置说明请参考：docs/sherpa-paraformer-guide.md
+    type: sherpa_onnx_local
+    model_dir: models/sherpa-onnx-paraformer-zh-small-2024-03-09
+    output_dir: tmp/
+    model_type: paraformer
   DoubaoASR:
     # 可以在这里申请相关Key等信息
     # https://console.volcengine.com/speech/app
@@ -300,9 +321,13 @@ ASR:
     output_dir: tmp/
   AliyunASR:
     # 阿里云智能语音交互服务，需要先在阿里云平台开通服务，然后获取验证信息
+    # HTTP POST请求，一次性处理完整音频
     # 平台地址：https://nls-portal.console.aliyun.com/
     # appkey地址：https://nls-portal.console.aliyun.com/applist
     # token地址：https://nls-portal.console.aliyun.com/overview
+    # AliyunASR和AliyunStreamASR的区别是：AliyunASR是批量处理场景，AliyunStreamASR是实时交互场景
+    # 一般来说非流式ASR更便宜（0.004元/秒，¥0.24/分钟）
+    # 但是AliyunStreamASR实时性更好（0.005元/秒，¥0.3/分钟）
     # 定义ASR API类型
     type: aliyun
     appkey: 你的阿里云智能语音交互服务项目Appkey
@@ -310,6 +335,26 @@ ASR:
     access_key_id: 你的阿里云账号access_key_id
     access_key_secret: 你的阿里云账号access_key_secret
     output_dir: tmp/
+  AliyunStreamASR:
+    # 阿里云智能语音交互服务 - 实时流式语音识别
+    # WebSocket连接，实时处理音频流
+    # 平台地址：https://nls-portal.console.aliyun.com/
+    # appkey地址：https://nls-portal.console.aliyun.com/applist
+    # token地址：https://nls-portal.console.aliyun.com/overview
+    # AliyunASR和AliyunStreamASR的区别是：AliyunASR是批量处理场景，AliyunStreamASR是实时交互场景
+    # 一般来说非流式ASR更便宜（0.004元/秒，¥0.24/分钟）
+    # 但是AliyunStreamASR实时性更好（0.005元/秒，¥0.3/分钟）
+    # 定义ASR API类型
+    type: aliyun_stream
+    appkey: 你的阿里云智能语音交互服务项目Appkey
+    token: 你的阿里云智能语音交互服务AccessToken，临时的24小时，要长期用下方的access_key_id，access_key_secret
+    access_key_id: 你的阿里云账号access_key_id
+    access_key_secret: 你的阿里云账号access_key_secret
+    # 服务器地域选择，可选择距离更近的服务器以减少延迟，如nls-gateway-cn-hangzhou.aliyuncs.com(杭州)等
+    host: nls-gateway-cn-shanghai.aliyuncs.com
+    # 断句检测时间(毫秒)，控制静音多长时间后进行断句，默认800毫秒
+    max_sentence_silence: 800
+    output_dir: tmp/
   BaiduASR:
     # 获取AppID、API Key、Secret Key：https://console.bce.baidu.com/ai-engine/old/#/ai/speech/app/list
     # 查看资源额度：https://console.bce.baidu.com/ai-engine/old/#/ai/speech/overview/resource/list
@@ -320,11 +365,38 @@ ASR:
     # 语言参数，1537为普通话，具体参考：https://ai.baidu.com/ai-doc/SPEECH/0lbxfnc9b
     dev_pid: 1537
     output_dir: tmp/
+  OpenaiASR:
+    # OpenAI语音识别服务，需要先在OpenAI平台创建组织并获取api_key
+    # 支持中、英、日、韩等多种语音识别，具体参考文档https://platform.openai.com/docs/guides/speech-to-text
+    # 需要网络连接
+    # 申请步骤：
+    # 1.登录OpenAI Platform。https://auth.openai.com/log-in
+    # 2.创建api-key  https://platform.openai.com/settings/organization/api-keys
+    # 3.模型可以选择gpt-4o-transcribe或GPT-4o mini Transcribe
+    type: openai
+    api_key: 你的OpenAI API密钥
+    base_url: https://api.openai.com/v1/audio/transcriptions
+    model_name: gpt-4o-mini-transcribe
+    output_dir: tmp/
+  GroqASR:
+    # Groq语音识别服务，需要先在Groq Console创建API密钥
+    # 申请步骤：
+    # 1.登录groq Console。https://console.groq.com/home
+    # 2.创建api-key  https://console.groq.com/keys
+    # 3.模型可以选择whisper-large-v3-turbo或whisper-large-v3（distil-whisper-large-v3-en仅支持英语转录）
+    type: openai
+    api_key: 你的Groq API密钥
+    base_url: https://api.groq.com/openai/v1/audio/transcriptions
+    model_name: whisper-large-v3-turbo
+    output_dir: tmp/
+
+
   
 VAD:
   SileroVAD:
     type: silero
     threshold: 0.5
+    threshold_low: 0.3
     model_dir: models/snakers4_silero-vad
     min_silence_duration_ms: 200  # 如果说话停顿比较长，可以把这个值设置大一些
 
@@ -648,6 +720,26 @@ TTS:
     #     voice_id: female-shaonv
     #     weight: 1
     # language_boost: auto
+
+# MinimaxTTSHTTPStream和MinimaxTTSWebSocketStream还在测试，测试完再开放
+#
+#  MinimaxTTSHTTPStream:
+#  # Minimax流式语音合成服务
+#    type: minimax_httpstream
+#    output_dir: tmp/
+#    group_id: 你的minimax平台groupID
+#    api_key: 你的minimax平台接口密钥
+#    model: "speech-01-turbo"
+#    voice_id: "female-shaonv"
+#
+#  MinimaxTTSWebSocketStream:
+#    type: minimax_webSocket
+#    output_dir: tmp/
+#    group_id: 你的minimax平台groupID
+#    api_key: 你的minimax平台接口密钥
+#    model: "speech-01-turbo"
+#    voice_id: "female-shaonv"
+
   AliyunTTS:
     # 阿里云智能语音交互服务，需要先在阿里云平台开通服务，然后获取验证信息
     # 平台地址：https://nls-portal.console.aliyun.com/
@@ -668,8 +760,30 @@ TTS:
     # volume: 50
     # speech_rate: 0
     # pitch_rate: 0
-    # 添加 302.ai TTS 配置
+  AliyunStreamTTS:
-    # token申请地址：https://dash.302.ai/
+    # 阿里云CosyVoice大模型流式文本语音合成
+    # 采用FlowingSpeechSynthesizer接口，支持更低延迟和更自然的语音质量
+    # 流式文本语音合成仅提供商用版，不支持试用，详情请参见试用版和商用版。要使用该功能，请开通商用版。
+    # 支持龙系列专用音色：longxiaochun、longyu、longchen等
+    # 平台地址：https://nls-portal.console.aliyun.com/
+    # appkey地址：https://nls-portal.console.aliyun.com/applist
+    # token地址：https://nls-portal.console.aliyun.com/overview
+    # 使用三阶段流式交互：StartSynthesis -> RunSynthesis -> StopSynthesis
+    type: aliyun_stream
+    output_dir: tmp/
+    appkey: 你的阿里云智能语音交互服务项目Appkey
+    token: 你的阿里云智能语音交互服务AccessToken，临时的24小时，要长期用下方的access_key_id，access_key_secret
+    voice: longxiaochun 
+    access_key_id: 你的阿里云账号access_key_id
+    access_key_secret: 你的阿里云账号access_key_secret
+    # 截至2025年7月21日大模型音色只有北京节点采用，其他节点暂不支持
+    host: nls-gateway-cn-beijing.aliyuncs.com
+    # 以下可不用设置，使用默认设置
+    # format: pcm  # 音频格式：pcm、wav、mp3
+    # sample_rate: 16000  # 采样率：8000、16000、24000
+    # volume: 50  # 音量：0-100
+    # speech_rate: 0  # 语速：-500到500
+    # pitch_rate: 0  # 语调：-500到500
   TencentTTS:
     # 腾讯云智能语音交互服务，需要先在腾讯云平台开通服务
     # appid、secret_id、secret_key申请地址：https://console.cloud.tencent.com/cam/capi
@@ -684,6 +798,8 @@ TTS:
 
   TTS302AI:
     # 302AI语音合成服务，需要先在302平台创建账户充值，并获取密钥信息
+    # 添加 302.ai TTS 配置
+    # token申请地址：https://dash.302.ai/
     # 获取api_keyn路径：https://dash.302.ai/apis/list
     # 价格，$35/百万字符。火山原版¥450元/百万字符
     type: doubao

main/xiaozhi-server/config/config_loader.py

@@ -1,14 +1,9 @@
 import os
-import argparse
 import yaml
 from collections.abc import Mapping
 from config.manage_api_client import init_service, get_server_config, get_agent_models
 
 
-# 添加全局配置缓存
-_config_cache = None
-
-
 def get_project_dir():
     """获取项目根目录"""
     return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + "/"
@@ -22,9 +17,12 @@ def read_config(config_path):
 
 def load_config():
     """加载配置文件"""
-    global _config_cache
+    from core.utils.cache.manager import cache_manager, CacheType
-    if _config_cache is not None:
+
-        return _config_cache
+    # 检查缓存
+    cached_config = cache_manager.get(CacheType.CONFIG, "main_config")
+    if cached_config is not None:
+        return cached_config
 
     default_config_path = get_project_dir() + "config.yaml"
     custom_config_path = get_project_dir() + "data/.config.yaml"
@@ -40,7 +38,9 @@ def load_config():
         config = merge_configs(default_config, custom_config)
     # 初始化目录
     ensure_directories(config)
-    _config_cache = config
+
+    # 缓存配置
+    cache_manager.set(CacheType.CONFIG, "main_config", config)
     return config
 
 

main/xiaozhi-server/config/logger.py

@@ -5,7 +5,7 @@ from config.config_loader import load_config
 from config.settings import check_config_file
 from datetime import datetime
 
-SERVER_VERSION = "0.6.1"
+SERVER_VERSION = "0.7.3"
 _logger_initialized = False
 
 
@@ -31,12 +31,17 @@ def build_module_string(selected_module):
         + get_module_abbreviation("TTS", selected_module)
         + get_module_abbreviation("Memory", selected_module)
         + get_module_abbreviation("Intent", selected_module)
+        + get_module_abbreviation("VLLM", selected_module)
     )
 
 
 def formatter(record):
-    """为没有 tag 的日志添加默认值"""
+    """为没有 tag 的日志添加默认值，并处理动态模块字符串"""
     record["extra"].setdefault("tag", record["name"])
+    # 如果没有设置 selected_module，使用默认值
+    record["extra"].setdefault("selected_module", "00000000000000")
+    # 将 selected_module 从 extra 提取到顶级，以支持 {selected_module} 格式
+    record["selected_module"] = record["extra"]["selected_module"]
     return record["message"]
 
 
@@ -49,11 +54,12 @@ def setup_logging():
 
     # 第一次初始化时配置日志
     if not _logger_initialized:
+        # 使用默认的模块字符串进行初始化
         logger.configure(
             extra={
-                "selected_module": log_config.get("selected_module", "00000000000000")
+                "selected_module": log_config.get("selected_module", "00000000000000"),
-            }
+            })
-        )  # 新增配置
+        
         log_format = log_config.get(
             "log_format",
             "<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
@@ -62,14 +68,8 @@ def setup_logging():
             "log_format_file",
             "{time:YYYY-MM-DD HH:mm:ss} - {version}_{extra[selected_module]} - {name} - {level} - {extra[tag]} - {message}",
         )
-        selected_module_str = logger._core.extra["selected_module"]
-
         log_format = log_format.replace("{version}", SERVER_VERSION)
-        log_format = log_format.replace("{selected_module}", selected_module_str)
         log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace(
-            "{selected_module}", selected_module_str
-        )
 
         log_level = log_config.get("log_level", "INFO")
         log_dir = log_config.get("log_dir", "tmp")
@@ -108,66 +108,7 @@ def setup_logging():
     return logger
 
 
-def update_module_string(selected_module_str):
+def create_connection_logger(selected_module_str):
-    """更新模块字符串并重新配置日志处理器"""
+    """为连接创建独立的日志器，绑定特定的模块字符串"""
-    logger.debug(f"更新日志配置组件")
+    return logger.bind(selected_module=selected_module_str)
-    current_module = logger._core.extra["selected_module"]
 
-    if current_module == selected_module_str:
-        logger.debug(f"组件未更改无需更新")
-        return
-
-    try:
-        logger.configure(extra={"selected_module": selected_module_str})
-
-        config = load_config()
-        log_config = config["log"]
-
-        log_format = log_config.get(
-            "log_format",
-            "<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
-        )
-        log_format_file = log_config.get(
-            "log_format_file",
-            "{time:YYYY-MM-DD HH:mm:ss} - {version}_{extra[selected_module]} - {name} - {level} - {extra[tag]} - {message}",
-        )
-
-        log_format = log_format.replace("{version}", SERVER_VERSION)
-        log_format = log_format.replace("{selected_module}", selected_module_str)
-        log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
-        log_format_file = log_format_file.replace(
-            "{selected_module}", selected_module_str
-        )
-
-        logger.remove()
-        logger.add(
-            sys.stdout,
-            format=log_format,
-            level=log_config.get("log_level", "INFO"),
-            filter=formatter,
-        )
-
-        # 更新文件日志配置 - 统一目录，按大小轮转
-        log_dir = log_config.get("log_dir", "tmp")
-        log_file = log_config.get("log_file", "server.log")
-
-        # 日志文件完整路径
-        log_file_path = os.path.join(log_dir, log_file)
-
-        logger.add(
-            log_file_path,
-            format=log_format_file,
-            level=log_config.get("log_level", "INFO"),
-            filter=formatter,
-            rotation="10 MB",  # 每个文件最大10MB
-            retention="30 days",  # 保留30天
-            compression=None,
-            encoding="utf-8",
-            enqueue=True,  # 异步安全
-            backtrace=True,
-            diagnose=True,
-        )
-
-    except Exception as e:
-        logger.error(f"日志配置更新失败: {str(e)}")
-        raise

main/xiaozhi-server/config_from_api.yaml

@@ -19,6 +19,7 @@ server:
   vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
 manager-api:
   # 你的manager-api的地址，最好使用局域网ip
+  # 如果使用docker部署，请使用填写成 http://xiaozhi-esp32-server-web:8002/xiaozhi
   url: http://127.0.0.1:8002/xiaozhi
   # 你的manager-api的token，就是刚才复制出来的server.secret
   secret: 你的server.secret值

main/xiaozhi-server/core/connection.py

@@ -17,6 +17,7 @@ from core.utils.util import (
     filter_sensitive_info,
 )
 from typing import Dict, Any
+from collections import deque
 from core.utils.modules_initialize import (
     initialize_modules,
     initialize_tts,
@@ -34,9 +35,11 @@ from plugins_func.register import Action, ActionResponse
 from core.auth import AuthMiddleware, AuthenticationError
 from config.config_loader import get_private_config_from_api
 from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType
-from config.logger import setup_logging, build_module_string, update_module_string
+from config.logger import setup_logging, build_module_string, create_connection_logger
 from config.manage_api_client import DeviceNotFoundException, DeviceBindException
-
+from core.utils.prompt_manager import PromptManager
+from core.utils.voiceprint_provider import VoiceprintProvider
+from core.utils import textUtils
 
 TAG = __name__
 
@@ -73,7 +76,6 @@ class ConnectionHandler:
         self.headers = None
         self.device_id = None
         self.client_ip = None
-        self.client_ip_info = {}
         self.prompt = None
         self.welcome_msg = None
         self.max_output_size = 0
@@ -107,11 +109,16 @@ class ConnectionHandler:
         self.memory = _memory
         self.intent = _intent
 
+        # 为每个连接单独管理声纹识别
+        self.voiceprint_provider = None
+
         # vad相关变量
         self.client_audio_buffer = bytearray()
         self.client_have_voice = False
         self.last_activity_time = 0.0  # 统一的活动时间戳（毫秒）
         self.client_voice_stop = False
+        self.client_voice_window = deque(maxlen=5)
+        self.last_is_voice = False
 
         # asr相关变量
         # 因为实际部署时可能会用到公共的本地ASR，不能把变量暴露给公共ASR
@@ -125,6 +132,8 @@ class ConnectionHandler:
 
         # tts相关变量
         self.sentence_id = None
+        # 处理TTS响应没有文本返回
+        self.tts_MessageText = ""
 
         # iot相关变量
         self.iot_descriptors = {}
@@ -149,6 +158,9 @@ class ConnectionHandler:
         # {"mcp":true} 表示启用MCP功能
         self.features = None
 
+        # 初始化提示词管理器
+        self.prompt_manager = PromptManager(config, self.logger)
+
     async def handle_connection(self, ws):
         try:
             # 获取并验证headers
@@ -172,8 +184,13 @@ class ConnectionHandler:
                     await ws.send("端口正常，如需测试连接，请使用test_page.html")
                     await self.close(ws)
                     return
-            # 获取客户端ip地址
+            real_ip = self.headers.get("x-real-ip") or self.headers.get(
-            self.client_ip = ws.remote_address[0]
+                "x-forwarded-for"
+            )
+            if real_ip:
+                self.client_ip = real_ip.split(",")[0].strip()
+            else:
+                self.client_ip = ws.remote_address[0]
             self.logger.bind(tag=TAG).info(
                 f"{self.client_ip} conn - Headers: {self.headers}"
             )
@@ -193,7 +210,6 @@ class ConnectionHandler:
 
             self.welcome_msg = self.config["xiaozhi"]
             self.welcome_msg["session_id"] = self.session_id
-            await self.websocket.send(json.dumps(self.welcome_msg))
 
             # 获取差异化配置
             self._initialize_private_config()
@@ -214,7 +230,17 @@ class ConnectionHandler:
             self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}")
             return
         finally:
-            await self._save_and_close(ws)
+            try:
+                await self._save_and_close(ws)
+            except Exception as final_error:
+                self.logger.bind(tag=TAG).error(f"最终清理时出错: {final_error}")
+                # 确保即使保存记忆失败，也要关闭连接
+                try:
+                    await self.close(ws)
+                except Exception as close_error:
+                    self.logger.bind(tag=TAG).error(
+                        f"强制关闭连接时出错: {close_error}"
+                    )
 
     async def _save_and_close(self, ws):
         """保存记忆并关闭连接"""
@@ -232,7 +258,10 @@ class ConnectionHandler:
                     except Exception as e:
                         self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}")
                     finally:
-                        loop.close()
+                        try:
+                            loop.close()
+                        except Exception:
+                            pass
 
                 # 启动线程保存记忆，不等待完成
                 threading.Thread(target=save_memory_task, daemon=True).start()
@@ -240,12 +269,16 @@ class ConnectionHandler:
             self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}")
         finally:
             # 立即关闭连接，不等待记忆保存完成
-            await self.close(ws)
+            try:
+                await self.close(ws)
+            except Exception as close_error:
+                self.logger.bind(tag=TAG).error(
+                    f"保存记忆后关闭连接失败: {close_error}"
+                )
 
     async def _route_message(self, message):
         """消息路由"""
         if isinstance(message, str):
-            self.last_activity_time = time.time() * 1000
             await handleTextMessage(self, message)
         elif isinstance(message, bytes):
             if self.vad is None:
@@ -307,13 +340,16 @@ class ConnectionHandler:
             self.selected_module_str = build_module_string(
                 self.config.get("selected_module", {})
             )
-            update_module_string(self.selected_module_str)
+            self.logger = create_connection_logger(self.selected_module_str)
+
             """初始化组件"""
             if self.config.get("prompt") is not None:
-                self.prompt = self.config["prompt"]
+                user_prompt = self.config["prompt"]
-                self.change_system_prompt(self.prompt)
+                # 使用快速提示词进行初始化
+                prompt = self.prompt_manager.get_quick_prompt(user_prompt)
+                self.change_system_prompt(prompt)
                 self.logger.bind(tag=TAG).info(
-                    f"初始化组件: prompt成功 {self.prompt[:50]}..."
+                    f"快速初始化组件: prompt成功 {prompt[:50]}..."
                 )
 
             """初始化本地组件"""
@@ -321,6 +357,10 @@ class ConnectionHandler:
                 self.vad = self._vad
             if self.asr is None:
                 self.asr = self._initialize_asr()
+
+            # 初始化声纹识别
+            self._initialize_voiceprint()
+
             # 打开语音识别通道
             asyncio.run_coroutine_threadsafe(
                 self.asr.open_audio_channels(self), self.loop
@@ -338,9 +378,22 @@ class ConnectionHandler:
             self._initialize_intent()
             """初始化上报线程"""
             self._init_report_threads()
+            """更新系统提示词"""
+            self._init_prompt_enhancement()
+
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"实例化组件失败: {e}")
 
+    def _init_prompt_enhancement(self):
+        # 更新上下文信息
+        self.prompt_manager.update_context_info(self, self.client_ip)
+        enhanced_prompt = self.prompt_manager.build_enhanced_prompt(
+            self.config["prompt"], self.device_id, self.client_ip
+        )
+        if enhanced_prompt:
+            self.change_system_prompt(enhanced_prompt)
+            self.logger.bind(tag=TAG).info("系统提示词已增强更新")
+
     def _init_report_threads(self):
         """初始化ASR和TTS上报线程"""
         if not self.read_config_from_api or self.need_bind:
@@ -378,6 +431,18 @@ class ConnectionHandler:
 
         return asr
 
+    def _initialize_voiceprint(self):
+        """为当前连接初始化声纹识别"""
+        try:
+            voiceprint_config = self.config.get("voiceprint", {})
+            if voiceprint_config:
+                self.voiceprint_provider = VoiceprintProvider(voiceprint_config)
+                self.logger.bind(tag=TAG).info("声纹识别功能已在连接时动态启用")
+            else:
+                self.logger.bind(tag=TAG).info("声纹识别功能未启用或配置不完整")
+        except Exception as e:
+            self.logger.bind(tag=TAG).warning(f"声纹识别初始化失败: {str(e)}")
+
     def _initialize_private_config(self):
         """如果是从配置文件获取，则进行二次实例化"""
         if not self.read_config_from_api:
@@ -438,6 +503,11 @@ class ConnectionHandler:
             self.config["selected_module"]["LLM"] = private_config["selected_module"][
                 "LLM"
             ]
+        if private_config.get("VLLM", None) is not None:
+            self.config["VLLM"] = private_config["VLLM"]
+            self.config["selected_module"]["VLLM"] = private_config["selected_module"][
+                "VLLM"
+            ]
         if private_config.get("Memory", None) is not None:
             init_memory = True
             self.config["Memory"] = private_config["Memory"]
@@ -460,6 +530,9 @@ class ConnectionHandler:
                 ] = plugin_from_server.keys()
         if private_config.get("prompt", None) is not None:
             self.config["prompt"] = private_config["prompt"]
+        # 获取声纹信息
+        if private_config.get("voiceprint", None) is not None:
+            self.config["voiceprint"] = private_config["voiceprint"]
         if private_config.get("summaryMemory", None) is not None:
             self.config["summaryMemory"] = private_config["summaryMemory"]
         if private_config.get("device_max_output_size", None) is not None:
@@ -591,13 +664,24 @@ class ConnectionHandler:
         # 更新系统prompt至上下文
         self.dialogue.update_system_message(self.prompt)
 
-    def chat(self, query, tool_call=False):
+    def chat(self, query, tool_call=False, depth=0):
         self.logger.bind(tag=TAG).info(f"大模型收到用户消息: {query}")
         self.llm_finish_task = False
 
         if not tool_call:
             self.dialogue.put(Message(role="user", content=query))
 
+        # 为最顶层时新建会话ID和发送FIRST请求
+        if depth == 0:
+            self.sentence_id = str(uuid.uuid4().hex)
+            self.tts.tts_text_queue.put(
+                TTSMessageDTO(
+                    sentence_id=self.sentence_id,
+                    sentence_type=SentenceType.FIRST,
+                    content_type=ContentType.ACTION,
+                )
+            )
+
         # Define intent functions
         functions = None
         if self.intent_type == "function_call" and hasattr(self, "func_handler"):
@@ -613,19 +697,21 @@ class ConnectionHandler:
                 )
                 memory_str = future.result()
 
-            self.sentence_id = str(uuid.uuid4().hex)
-
             if self.intent_type == "function_call" and functions is not None:
                 # 使用支持functions的streaming接口
                 llm_responses = self.llm.response_with_functions(
                     self.session_id,
-                    self.dialogue.get_llm_dialogue_with_memory(memory_str),
+                    self.dialogue.get_llm_dialogue_with_memory(
+                        memory_str, self.config.get("voiceprint", {})
+                    ),
                     functions=functions,
                 )
             else:
                 llm_responses = self.llm.response(
                     self.session_id,
-                    self.dialogue.get_llm_dialogue_with_memory(memory_str),
+                    self.dialogue.get_llm_dialogue_with_memory(
+                        memory_str, self.config.get("voiceprint", {})
+                    ),
                 )
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"LLM 处理出错 {query}: {e}")
@@ -637,8 +723,8 @@ class ConnectionHandler:
         function_id = None
         function_arguments = ""
         content_arguments = ""
-        text_index = 0
         self.client_abort = False
+        emotion_flag = True
         for response in llm_responses:
             if self.client_abort:
                 break
@@ -664,17 +750,18 @@ class ConnectionHandler:
                         function_arguments += tools_call[0].function.arguments
             else:
                 content = response
+
+            # 在llm回复中获取情绪表情，一轮对话只在开头获取一次
+            if emotion_flag and content is not None and content.strip():
+                asyncio.run_coroutine_threadsafe(
+                    textUtils.get_emotion(self, content),
+                    self.loop,
+                )
+                emotion_flag = False
+
             if content is not None and len(content) > 0:
                 if not tool_call_flag:
                     response_message.append(content)
-                    if text_index == 0:
-                        self.tts.tts_text_queue.put(
-                            TTSMessageDTO(
-                                sentence_id=self.sentence_id,
-                                sentence_type=SentenceType.FIRST,
-                                content_type=ContentType.ACTION,
-                            )
-                        )
                     self.tts.tts_text_queue.put(
                         TTSMessageDTO(
                             sentence_id=self.sentence_id,
@@ -683,7 +770,6 @@ class ConnectionHandler:
                             content_detail=content,
                         )
                     )
-                    text_index += 1
         # 处理function call
         if tool_call_flag:
             bHasError = False
@@ -708,6 +794,11 @@ class ConnectionHandler:
                         f"function call error: {content_arguments}"
                     )
             if not bHasError:
+                # 如需要大模型先处理一轮，添加相关处理后的日志情况
+                if len(response_message) > 0:
+                    text_buff = "".join(response_message)
+                    self.tts_MessageText = text_buff
+                    self.dialogue.put(Message(role="assistant", content=text_buff))
                 response_message.clear()
                 self.logger.bind(tag=TAG).debug(
                     f"function_name={function_name}, function_id={function_id}, function_arguments={function_arguments}"
@@ -725,14 +816,14 @@ class ConnectionHandler:
                     ),
                     self.loop,
                 ).result()
-                self._handle_function_result(result, function_call_data)
+                self._handle_function_result(result, function_call_data, depth=depth)
 
         # 存储对话内容
         if len(response_message) > 0:
-            self.dialogue.put(
+            text_buff = "".join(response_message)
-                Message(role="assistant", content="".join(response_message))
+            self.tts_MessageText = text_buff
-            )
+            self.dialogue.put(Message(role="assistant", content=text_buff))
-        if text_index > 0:
+        if depth == 0:
             self.tts.tts_text_queue.put(
                 TTSMessageDTO(
                     sentence_id=self.sentence_id,
@@ -741,13 +832,16 @@ class ConnectionHandler:
                 )
             )
         self.llm_finish_task = True
+        # 使用lambda延迟计算，只有在DEBUG级别时才执行get_llm_dialogue()
         self.logger.bind(tag=TAG).debug(
-            json.dumps(self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False)
+            lambda: json.dumps(
+                self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False
+            )
         )
 
         return True
 
-    def _handle_function_result(self, result, function_call_data):
+    def _handle_function_result(self, result, function_call_data, depth):
         if result.action == Action.RESPONSE:  # 直接回复前端
             text = result.response
             self.tts.tts_one_sentence(self, ContentType.TEXT, content_detail=text)
@@ -765,7 +859,7 @@ class ConnectionHandler:
                             {
                                 "id": function_id,
                                 "function": {
-                                    "arguments": function_arguments,
+                                    "arguments": "{}" if function_arguments == "" else function_arguments,
                                     "name": function_name,
                                 },
                                 "type": "function",
@@ -784,7 +878,7 @@ class ConnectionHandler:
                         content=text,
                     )
                 )
-                self.chat(text, tool_call=True)
+                self.chat(text, tool_call=True, depth=depth + 1)
         elif result.action == Action.NOTFOUND or result.action == Action.ERROR:
             text = result.response if result.response else result.result
             self.tts.tts_one_sentence(self, ContentType.TEXT, content_detail=text)
@@ -800,15 +894,12 @@ class ConnectionHandler:
                 item = self.report_queue.get(timeout=1)
                 if item is None:  # 检测毒丸对象
                     break
-                type, text, audio_data, report_time = item
                 try:
                     # 检查线程池状态
                     if self.executor is None:
                         continue
                     # 提交任务到线程池
-                    self.executor.submit(
+                    self.executor.submit(self._process_report, *item)
-                        self._process_report, type, text, audio_data, report_time
-                    )
                 except Exception as e:
                     self.logger.bind(tag=TAG).error(f"聊天记录上报线程异常: {e}")
             except queue.Empty:
@@ -837,13 +928,22 @@ class ConnectionHandler:
         """资源清理方法"""
         try:
             # 取消超时任务
-            if self.timeout_task:
+            if self.timeout_task and not self.timeout_task.done():
                 self.timeout_task.cancel()
+                try:
+                    await self.timeout_task
+                except asyncio.CancelledError:
+                    pass
                 self.timeout_task = None
 
             # 清理工具处理器资源
             if hasattr(self, "func_handler") and self.func_handler:
-                await self.func_handler.cleanup()
+                try:
+                    await self.func_handler.cleanup()
+                except Exception as cleanup_error:
+                    self.logger.bind(tag=TAG).error(
+                        f"清理工具处理器时出错: {cleanup_error}"
+                    )
 
             # 触发停止事件
             if self.stop_event:
@@ -853,19 +953,61 @@ class ConnectionHandler:
             self.clear_queues()
 
             # 关闭WebSocket连接
-            if ws:
+            try:
-                await ws.close()
+                if ws:
-            elif self.websocket:
+                    # 安全地检查WebSocket状态并关闭
-                await self.websocket.close()
+                    try:
+                        if hasattr(ws, "closed") and not ws.closed:
+                            await ws.close()
+                        elif hasattr(ws, "state") and ws.state.name != "CLOSED":
+                            await ws.close()
+                        else:
+                            # 如果没有closed属性，直接尝试关闭
+                            await ws.close()
+                    except Exception:
+                        # 如果关闭失败，忽略错误
+                        pass
+                elif self.websocket:
+                    try:
+                        if (
+                            hasattr(self.websocket, "closed")
+                            and not self.websocket.closed
+                        ):
+                            await self.websocket.close()
+                        elif (
+                            hasattr(self.websocket, "state")
+                            and self.websocket.state.name != "CLOSED"
+                        ):
+                            await self.websocket.close()
+                        else:
+                            # 如果没有closed属性，直接尝试关闭
+                            await self.websocket.close()
+                    except Exception:
+                        # 如果关闭失败，忽略错误
+                        pass
+            except Exception as ws_error:
+                self.logger.bind(tag=TAG).error(f"关闭WebSocket连接时出错: {ws_error}")
+
+            if self.tts:
+                await self.tts.close()
 
             # 最后关闭线程池（避免阻塞）
             if self.executor:
-                self.executor.shutdown(wait=False)
+                try:
+                    self.executor.shutdown(wait=False)
+                except Exception as executor_error:
+                    self.logger.bind(tag=TAG).error(
+                        f"关闭线程池时出错: {executor_error}"
+                    )
                 self.executor = None
 
             self.logger.bind(tag=TAG).info("连接资源已释放")
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"关闭连接时出错: {e}")
+        finally:
+            # 确保停止事件被设置
+            if self.stop_event:
+                self.stop_event.set()
 
     def clear_queues(self):
         """清空所有任务队列"""
@@ -922,9 +1064,19 @@ class ConnectionHandler:
                     ):
                         if not self.stop_event.is_set():
                             self.logger.bind(tag=TAG).info("连接超时，准备关闭")
-                            await self.close(self.websocket)
+                            # 设置停止事件，防止重复处理
+                            self.stop_event.set()
+                            # 使用 try-except 包装关闭操作，确保不会因为异常而阻塞
+                            try:
+                                await self.close(self.websocket)
+                            except Exception as close_error:
+                                self.logger.bind(tag=TAG).error(
+                                    f"超时关闭连接时出错: {close_error}"
+                                )
                         break
                 # 每10秒检查一次，避免过于频繁
                 await asyncio.sleep(10)
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"超时检查任务出错: {e}")
+        finally:
+            self.logger.bind(tag=TAG).info("超时检查任务已退出")

main/xiaozhi-server/core/handle/intentHandler.py

@@ -6,14 +6,23 @@ from core.handle.helloHandle import checkWakeupWords
 from core.utils.util import remove_punctuation_and_length
 from core.providers.tts.dto.dto import ContentType
 from core.utils.dialogue import Message
-from core.providers.tools.device_mcp import call_mcp_tool
 from plugins_func.register import Action, ActionResponse
-from loguru import logger
+from core.providers.tts.dto.dto import TTSMessageDTO, SentenceType
 
 TAG = __name__
 
 
 async def handle_user_intent(conn, text):
+    # 预处理输入文本，处理可能的JSON格式
+    try:
+        if text.strip().startswith('{') and text.strip().endswith('}'):
+            parsed_data = json.loads(text)
+            if isinstance(parsed_data, dict) and "content" in parsed_data:
+                text = parsed_data["content"]  # 提取content用于意图分析
+                conn.current_speaker = parsed_data.get("speaker")  # 保留说话人信息
+    except (json.JSONDecodeError, TypeError):
+        pass
+
     # 检查是否有明确的退出命令
     filtered_text = remove_punctuation_and_length(text)[1]
     if await check_direct_exit(conn, filtered_text):
@@ -29,6 +38,8 @@ async def handle_user_intent(conn, text):
     intent_result = await analyze_intent_with_llm(conn, text)
     if not intent_result:
         return False
+    # 会话开始时生成sentence_id
+    conn.sentence_id = str(uuid.uuid4().hex)
     # 处理各种意图
     return await process_intent_result(conn, intent_result, text)
 
@@ -79,11 +90,6 @@ async def process_intent_result(conn, intent_result, original_text):
             if function_name == "continue_chat":
                 return False
 
-            if function_name == "play_music":
-                funcItem = conn.func_handler.get_function(function_name)
-                if not funcItem:
-                    conn.func_handler.function_registry.register_function("play_music")
-
             function_args = {}
             if "arguments" in intent_data["function_call"]:
                 function_args = intent_data["function_call"]["arguments"]
@@ -158,5 +164,19 @@ async def process_intent_result(conn, intent_result, original_text):
 
 
 def speak_txt(conn, text):
+    conn.tts.tts_text_queue.put(
+        TTSMessageDTO(
+            sentence_id=conn.sentence_id,
+            sentence_type=SentenceType.FIRST,
+            content_type=ContentType.ACTION,
+        )
+    )
     conn.tts.tts_one_sentence(conn, ContentType.TEXT, content_detail=text)
+    conn.tts.tts_text_queue.put(
+        TTSMessageDTO(
+            sentence_id=conn.sentence_id,
+            sentence_type=SentenceType.LAST,
+            content_type=ContentType.ACTION,
+        )
+    )
     conn.dialogue.put(Message(role="assistant", content=text))

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -4,6 +4,7 @@ from core.utils.output_counter import check_device_output_limit
 from core.handle.abortHandle import handleAbortMessage
 import time
 import asyncio
+import json
 from core.handle.sendAudioHandle import SentenceType
 from core.utils.util import audio_to_data
 
@@ -38,6 +39,31 @@ async def resume_vad_detection(conn):
 
 
 async def startToChat(conn, text):
+    # 检查输入是否是JSON格式（包含说话人信息）
+    speaker_name = None
+    actual_text = text
+    
+    try:
+        # 尝试解析JSON格式的输入
+        if text.strip().startswith('{') and text.strip().endswith('}'):
+            data = json.loads(text)
+            if 'speaker' in data and 'content' in data:
+                speaker_name = data['speaker']
+                actual_text = data['content']
+                conn.logger.bind(tag=TAG).info(f"解析到说话人信息: {speaker_name}")
+                
+                # 直接使用JSON格式的文本，不解析
+                actual_text = text
+    except (json.JSONDecodeError, KeyError):
+        # 如果解析失败，继续使用原始文本
+        pass
+    
+    # 保存说话人信息到连接对象
+    if speaker_name:
+        conn.current_speaker = speaker_name
+    else:
+        conn.current_speaker = None
+
     if conn.need_bind:
         await check_bind_device(conn)
         return
@@ -52,16 +78,16 @@ async def startToChat(conn, text):
     if conn.client_is_speaking:
         await handleAbortMessage(conn)
 
-    # 首先进行意图分析
+    # 首先进行意图分析，使用实际文本内容
-    intent_handled = await handle_user_intent(conn, text)
+    intent_handled = await handle_user_intent(conn, actual_text)
 
     if intent_handled:
         # 如果意图已被处理，不再进行聊天
         return
 
-    # 意图未被处理，继续常规聊天流程
+    # 意图未被处理，继续常规聊天流程，使用实际文本内容
-    await send_stt_message(conn, text)
+    await send_stt_message(conn, actual_text)
-    conn.executor.submit(conn.chat, text)
+    conn.executor.submit(conn.chat, actual_text)
 
 
 async def no_voice_close_connect(conn, have_voice):

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -2,54 +2,17 @@ import json
 import asyncio
 import time
 from core.providers.tts.dto.dto import SentenceType
-from core.utils.util import get_string_no_punctuation_or_emoji, analyze_emotion
+from core.utils import textUtils
-from loguru import logger
 
 TAG = __name__
 
-emoji_map = {
-    "neutral": "😶",
-    "happy": "🙂",
-    "laughing": "😆",
-    "funny": "😂",
-    "sad": "😔",
-    "angry": "😠",
-    "crying": "😭",
-    "loving": "😍",
-    "embarrassed": "😳",
-    "surprised": "😲",
-    "shocked": "😱",
-    "thinking": "🤔",
-    "winking": "😉",
-    "cool": "😎",
-    "relaxed": "😌",
-    "delicious": "🤤",
-    "kissy": "😘",
-    "confident": "😏",
-    "sleepy": "😴",
-    "silly": "😜",
-    "confused": "🙄",
-}
-
 
 async def sendAudioMessage(conn, sentenceType, audios, text):
     # 发送句子开始消息
     conn.logger.bind(tag=TAG).info(f"发送音频消息: {sentenceType}, {text}")
-    if text is not None:
+
-        emotion = analyze_emotion(text)
-        emoji = emoji_map.get(emotion, "🙂")  # 默认使用笑脸
-        await conn.websocket.send(
-            json.dumps(
-                {
-                    "type": "llm",
-                    "text": emoji,
-                    "emotion": emotion,
-                    "session_id": conn.session_id,
-                }
-            )
-        )
     pre_buffer = False
-    if conn.tts.tts_audio_first_sentence and text is not None:
+    if conn.tts.tts_audio_first_sentence:
         conn.logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
         conn.tts.tts_audio_first_sentence = False
         pre_buffer = True
@@ -58,8 +21,6 @@ async def sendAudioMessage(conn, sentenceType, audios, text):
 
     await sendAudio(conn, audios, pre_buffer)
 
-    await send_tts_message(conn, "sentence_end", text)
-
     # 发送结束消息（如果是最后一个文本）
     if conn.llm_finish_task and sentenceType == SentenceType.LAST:
         await send_tts_message(conn, "stop", None)
@@ -76,7 +37,6 @@ async def sendAudio(conn, audios, pre_buffer=True):
     frame_duration = 60  # 帧时长（毫秒），匹配 Opus 编码
     start_time = time.perf_counter()
     play_position = 0
-    last_reset_time = time.perf_counter()  # 记录最后的重置时间
 
     # 仅当第一句话时执行预缓冲
     if pre_buffer:
@@ -111,7 +71,7 @@ async def send_tts_message(conn, state, text=None):
     """发送 TTS 状态消息"""
     message = {"type": "tts", "state": state, "session_id": conn.session_id}
     if text is not None:
-        message["text"] = text
+        message["text"] = textUtils.check_emoji(text)
 
     # TTS播放结束
     if state == "stop":
@@ -137,7 +97,23 @@ async def send_stt_message(conn, text):
         return
 
     """发送 STT 状态消息"""
-    stt_text = get_string_no_punctuation_or_emoji(text)
+    
+    # 解析JSON格式，提取实际的用户说话内容
+    display_text = text
+    try:
+        # 尝试解析JSON格式
+        if text.strip().startswith('{') and text.strip().endswith('}'):
+            parsed_data = json.loads(text)
+            if isinstance(parsed_data, dict) and "content" in parsed_data:
+                # 如果是包含说话人信息的JSON格式，只显示content部分
+                display_text = parsed_data["content"]
+                # 保存说话人信息到conn对象
+                if "speaker" in parsed_data:
+                    conn.current_speaker = parsed_data["speaker"]
+    except (json.JSONDecodeError, TypeError):
+        # 如果不是JSON格式，直接使用原始文本
+        display_text = text
+    stt_text = textUtils.get_string_no_punctuation_or_emoji(display_text)
     await conn.websocket.send(
         json.dumps({"type": "stt", "text": stt_text, "session_id": conn.session_id})
     )

main/xiaozhi-server/core/handle/textHandle.py

@@ -1,4 +1,5 @@
 import json
+import time
 from core.handle.abortHandle import handleAbortMessage
 from core.handle.helloHandle import handleHelloMessage
 from core.providers.tools.device_mcp import handle_mcp_message
@@ -45,6 +46,7 @@ async def handleTextMessage(conn, message):
                 conn.client_have_voice = False
                 conn.asr_audio.clear()
                 if "text" in msg_json:
+                    conn.last_activity_time = time.time() * 1000
                     original_text = msg_json["text"]  # 保留原始文本
                     filtered_len, filtered_text = remove_punctuation_and_length(
                         original_text

main/xiaozhi-server/core/providers/asr/aliyun_stream.py

@@ -0,0 +1,343 @@
+import json
+import time
+import uuid
+import hmac
+import base64
+import hashlib
+import asyncio
+import requests
+import websockets
+import opuslib_next
+import random
+from typing import Optional, Tuple, List
+from urllib import parse
+from datetime import datetime
+from config.logger import setup_logging
+from core.providers.asr.base import ASRProviderBase
+from core.providers.asr.dto.dto import InterfaceType
+
+TAG = __name__
+logger = setup_logging()
+
+
+class AccessToken:
+    @staticmethod
+    def _encode_text(text):
+        encoded_text = parse.quote_plus(text)
+        return encoded_text.replace("+", "%20").replace("*", "%2A").replace("%7E", "~")
+
+    @staticmethod
+    def _encode_dict(dic):
+        keys = dic.keys()
+        dic_sorted = [(key, dic[key]) for key in sorted(keys)]
+        encoded_text = parse.urlencode(dic_sorted)
+        return encoded_text.replace("+", "%20").replace("*", "%2A").replace("%7E", "~")
+
+    @staticmethod
+    def create_token(access_key_id, access_key_secret):
+        parameters = {
+            "AccessKeyId": access_key_id,
+            "Action": "CreateToken",
+            "Format": "JSON",
+            "RegionId": "cn-shanghai",
+            "SignatureMethod": "HMAC-SHA1",
+            "SignatureNonce": str(uuid.uuid1()),
+            "SignatureVersion": "1.0",
+            "Timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "Version": "2019-02-28",
+        }
+        query_string = AccessToken._encode_dict(parameters)
+        string_to_sign = (
+            "GET" + "&" + AccessToken._encode_text("/") + "&" + AccessToken._encode_text(query_string)
+        )
+        secreted_string = hmac.new(
+            bytes(access_key_secret + "&", encoding="utf-8"),
+            bytes(string_to_sign, encoding="utf-8"),
+            hashlib.sha1,
+        ).digest()
+        signature = base64.b64encode(secreted_string)
+        signature = AccessToken._encode_text(signature)
+        full_url = "http://nls-meta.cn-shanghai.aliyuncs.com/?Signature=%s&%s" % (signature, query_string)
+        response = requests.get(full_url)
+        if response.ok:
+            root_obj = response.json()
+            if "Token" in root_obj:
+                return root_obj["Token"]["Id"], root_obj["Token"]["ExpireTime"]
+        return None, None
+
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__()
+        self.interface_type = InterfaceType.STREAM
+        self.config = config
+        self.text = ""
+        self.decoder = opuslib_next.Decoder(16000, 1)
+        self.asr_ws = None
+        self.forward_task = None
+        self.is_processing = False
+        self.server_ready = False  # 服务器准备状态
+
+        # 基础配置
+        self.access_key_id = config.get("access_key_id")
+        self.access_key_secret = config.get("access_key_secret")
+        self.appkey = config.get("appkey")
+        self.token = config.get("token")
+        self.host = config.get("host", "nls-gateway-cn-shanghai.aliyuncs.com")
+        # 如果配置的是内网地址（包含-internal.aliyuncs.com），则使用ws协议，默认是wss协议
+        if "-internal." in self.host:
+            self.ws_url = f"ws://{self.host}/ws/v1"
+        else:
+            # 默认使用wss协议
+            self.ws_url = f"wss://{self.host}/ws/v1"
+
+        self.max_sentence_silence = config.get("max_sentence_silence")
+        self.output_dir = config.get("output_dir", "./audio_output")
+        self.delete_audio_file = delete_audio_file
+        self.expire_time = None
+
+        # Token管理
+        if self.access_key_id and self.access_key_secret:
+            self._refresh_token()
+        elif not self.token:
+            raise ValueError("必须提供access_key_id+access_key_secret或者直接提供token")
+
+    def _refresh_token(self):
+        """刷新Token"""
+        self.token, expire_time_str = AccessToken.create_token(self.access_key_id, self.access_key_secret)
+        if not self.token:
+            raise ValueError("无法获取有效的访问Token")
+        
+        try:
+            expire_str = str(expire_time_str).strip()
+            if expire_str.isdigit():
+                expire_time = datetime.fromtimestamp(int(expire_str))
+            else:
+                expire_time = datetime.strptime(expire_str, "%Y-%m-%dT%H:%M:%SZ")
+            self.expire_time = expire_time.timestamp() - 60
+        except:
+            self.expire_time = None
+
+    def _is_token_expired(self):
+        """检查Token是否过期"""
+        return self.expire_time and time.time() > self.expire_time
+
+    async def open_audio_channels(self, conn):
+        await super().open_audio_channels(conn)
+
+    async def receive_audio(self, conn, audio, audio_have_voice):
+        # 初始化音频缓存
+        if not hasattr(conn, 'asr_audio_for_voiceprint'):
+            conn.asr_audio_for_voiceprint = []
+        
+        # 存储音频数据
+        if audio:
+            conn.asr_audio_for_voiceprint.append(audio)
+        
+        conn.asr_audio.append(audio)
+        conn.asr_audio = conn.asr_audio[-10:]
+
+        # 只在有声音且没有连接时建立连接
+        if audio_have_voice and not self.is_processing:
+            try:
+                await self._start_recognition(conn)
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"开始识别失败: {str(e)}")
+                await self._cleanup(conn)
+                return
+
+        if self.asr_ws and self.is_processing and self.server_ready:
+            try:
+                pcm_frame = self.decoder.decode(audio, 960)
+                await self.asr_ws.send(pcm_frame)
+            except Exception as e:
+                logger.bind(tag=TAG).warning(f"发送音频失败: {str(e)}")
+                await self._cleanup(conn)
+
+    async def _start_recognition(self, conn):
+        """开始识别会话"""
+        if self._is_token_expired():
+            self._refresh_token()
+        
+        # 建立连接
+        headers = {"X-NLS-Token": self.token}
+        self.asr_ws = await websockets.connect(
+            self.ws_url,
+            additional_headers=headers,
+            max_size=1000000000,
+            ping_interval=None,
+            ping_timeout=None,
+            close_timeout=5,
+        )
+        
+        self.is_processing = True
+        self.server_ready = False  # 重置服务器准备状态
+        self.forward_task = asyncio.create_task(self._forward_results(conn))
+        
+        # 发送开始请求
+        start_request = {
+            "header": {
+                "namespace": "SpeechTranscriber",
+                "name": "StartTranscription",
+                "status": 20000000,
+                "message_id": ''.join(random.choices('0123456789abcdef', k=32)),
+                "task_id": ''.join(random.choices('0123456789abcdef', k=32)),
+                "status_text": "Gateway:SUCCESS:Success.",
+                "appkey": self.appkey
+            },
+            "payload": {
+                "format": "pcm",
+                "sample_rate": 16000,
+                "enable_intermediate_result": True,
+                "enable_punctuation_prediction": True,
+                "enable_inverse_text_normalization": True,
+                "max_sentence_silence": self.max_sentence_silence,
+                "enable_voice_detection": False,
+            }
+        }
+        await self.asr_ws.send(json.dumps(start_request, ensure_ascii=False))
+        logger.bind(tag=TAG).info("已发送开始请求，等待服务器准备...")
+
+    async def _forward_results(self, conn):
+        """转发识别结果"""
+        try:
+            while self.asr_ws and not conn.stop_event.is_set():
+                try:
+                    response = await asyncio.wait_for(self.asr_ws.recv(), timeout=1.0)
+                    result = json.loads(response)
+                    
+                    header = result.get("header", {})
+                    payload = result.get("payload", {})
+                    message_name = header.get("name", "")
+                    status = header.get("status", 0)
+                    
+                    if status != 20000000:
+                        if status in [40000004, 40010004]:  # 连接超时或客户端断开
+                            logger.bind(tag=TAG).warning(f"连接问题，状态码: {status}")
+                            break
+                        elif status in [40270002, 40270003]:  # 音频问题
+                            logger.bind(tag=TAG).warning(f"音频处理问题，状态码: {status}")
+                            continue
+                        else:
+                            logger.bind(tag=TAG).error(f"识别错误，状态码: {status}, 消息: {header.get('status_text', '')}")
+                            continue
+                    
+                    # 收到TranscriptionStarted表示服务器准备好接收音频数据
+                    if message_name == "TranscriptionStarted":
+                        self.server_ready = True
+                        logger.bind(tag=TAG).info("服务器已准备，开始发送缓存音频...")
+                        
+                        # 发送缓存音频
+                        if conn.asr_audio:
+                            for cached_audio in conn.asr_audio[-10:]:
+                                try:
+                                    pcm_frame = self.decoder.decode(cached_audio, 960)
+                                    await self.asr_ws.send(pcm_frame)
+                                except Exception as e:
+                                    logger.bind(tag=TAG).warning(f"发送缓存音频失败: {e}")
+                                    break
+                        continue
+                    
+                    if message_name == "TranscriptionResultChanged":
+                        # 中间结果
+                        text = payload.get("result", "")
+                        if text:
+                            self.text = text
+                    elif message_name == "SentenceEnd":
+                        # 最终结果
+                        text = payload.get("result", "")
+                        if text:
+                            self.text = text
+                            conn.reset_vad_states()
+                            # 传递缓存的音频数据
+                            audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
+                            await self.handle_voice_stop(conn, audio_data)
+                            # 清空缓存
+                            conn.asr_audio_for_voiceprint = []
+                            break
+                    elif message_name == "TranscriptionCompleted":
+                        # 识别完成
+                        self.is_processing = False
+                        break
+                        
+                except asyncio.TimeoutError:
+                    continue
+                except websockets.exceptions.ConnectionClosed:
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"处理结果失败: {str(e)}")
+                    break
+                    
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"结果转发失败: {str(e)}")
+        finally:
+            await self._cleanup(conn)
+
+    async def _cleanup(self, conn):
+        """清理资源"""
+        logger.bind(tag=TAG).info(f"开始ASR会话清理 | 当前状态: processing={self.is_processing}, server_ready={self.server_ready}")
+        
+        # 清理连接的音频缓存
+        if conn and hasattr(conn, 'asr_audio_for_voiceprint'):
+            conn.asr_audio_for_voiceprint = []
+        
+        # 判断是否需要发送终止请求
+        should_stop = self.is_processing or self.server_ready
+        
+        # 发送停止识别请求
+        if self.asr_ws and should_stop:
+            try:
+                stop_msg = {
+                    "header": {
+                        "namespace": "SpeechTranscriber",
+                        "name": "StopTranscription",
+                        "status": 20000000,
+                        "message_id": ''.join(random.choices('0123456789abcdef', k=32)),
+                        "status_text": "Client:Stop",
+                        "appkey": self.appkey
+                    }
+                }
+                logger.bind(tag=TAG).info("正在发送ASR终止请求")
+                await self.asr_ws.send(json.dumps(stop_msg, ensure_ascii=False))
+                await asyncio.sleep(0.1)
+                logger.bind(tag=TAG).info("ASR终止请求已发送")
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"ASR终止请求发送失败: {e}")
+        
+        # 状态重置（在终止请求发送后）
+        self.is_processing = False
+        self.server_ready = False
+        logger.bind(tag=TAG).info("ASR状态已重置")
+
+        # 清理任务
+        if self.forward_task and not self.forward_task.done():
+            self.forward_task.cancel()
+            try:
+                await asyncio.wait_for(self.forward_task, timeout=1.0)
+            except Exception as e:
+                logger.bind(tag=TAG).debug(f"forward_task取消异常: {e}")
+            finally:
+                self.forward_task = None
+        
+        # 关闭连接
+        if self.asr_ws:
+            try:
+                logger.bind(tag=TAG).debug("正在关闭WebSocket连接")
+                await asyncio.wait_for(self.asr_ws.close(), timeout=2.0)
+                logger.bind(tag=TAG).debug("WebSocket连接已关闭")
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"关闭WebSocket连接失败: {e}")
+            finally:
+                self.asr_ws = None
+        
+        logger.bind(tag=TAG).info("ASR会话清理完成")
+
+    async def speech_to_text(self, opus_data, session_id, audio_format):
+        """获取识别结果"""
+        result = self.text
+        self.text = ""
+        return result, None
+
+    async def close(self):
+        """关闭资源"""
+        await self._cleanup()

main/xiaozhi-server/core/providers/asr/base.py

@@ -1,15 +1,18 @@
 import os
 import wave
-import copy
 import uuid
 import queue
 import asyncio
 import traceback
 import threading
 import opuslib_next
+import json
+import io
+import time
+import concurrent.futures
 from abc import ABC, abstractmethod
 from config.logger import setup_logging
-from typing import Optional, Tuple, List
+from typing import Optional, Tuple, List, Dict, Any
 from core.handle.receiveAudioHandle import startToChat
 from core.handle.reportHandle import enqueue_asr_report
 from core.utils.util import remove_punctuation_and_length
@@ -24,10 +27,7 @@ class ASRProviderBase(ABC):
         pass
 
     # 打开音频通道
-    # 这里默认是非流式的处理方式
-    # 流式处理方式请在子类中重写
     async def open_audio_channels(self, conn):
-        # tts 消化线程
         conn.asr_priority_thread = threading.Thread(
             target=self.asr_text_priority_thread, args=(conn,), daemon=True
         )
@@ -52,41 +52,171 @@ class ASRProviderBase(ABC):
                 continue
 
     # 接收音频
-    # 这里默认是非流式的处理方式
-    # 流式处理方式请在子类中重写
     async def receive_audio(self, conn, audio, audio_have_voice):
         if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
             have_voice = audio_have_voice
         else:
             have_voice = conn.client_have_voice
-        # 如果本次没有声音，本段也没声音，就把声音丢弃了
+        
         conn.asr_audio.append(audio)
-        if have_voice == False and conn.client_have_voice == False:
+        if not have_voice and not conn.client_have_voice:
             conn.asr_audio = conn.asr_audio[-10:]
             return
 
-        # 如果本段有声音，且已经停止了
         if conn.client_voice_stop:
-            asr_audio_task = copy.deepcopy(conn.asr_audio)
+            asr_audio_task = conn.asr_audio.copy()
             conn.asr_audio.clear()
-
-            # 音频太短了，无法识别
             conn.reset_vad_states()
+
             if len(asr_audio_task) > 15:
                 await self.handle_voice_stop(conn, asr_audio_task)
 
     # 处理语音停止
-    async def handle_voice_stop(self, conn, asr_audio_task):
+    async def handle_voice_stop(self, conn, asr_audio_task: List[bytes]):
-        raw_text, _ = await self.speech_to_text(
+        """并行处理ASR和声纹识别"""
-            asr_audio_task, conn.session_id, conn.audio_format
+        try:
-        )  # 确保ASR模块返回原始文本
+            total_start_time = time.monotonic()
-        conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+            
-        text_len, _ = remove_punctuation_and_length(raw_text)
+            # 准备音频数据
-        self.stop_ws_connection()
+            if conn.audio_format == "pcm":
-        if text_len > 0:
+                pcm_data = asr_audio_task
-            # 使用自定义模块进行上报
+            else:
-            await startToChat(conn, raw_text)
+                pcm_data = self.decode_opus(asr_audio_task)
-            enqueue_asr_report(conn, raw_text, asr_audio_task)
+            
+            combined_pcm_data = b"".join(pcm_data)
+            
+            # 预先准备WAV数据
+            wav_data = None
+            # 使用连接的声纹识别提供者
+            if conn.voiceprint_provider and combined_pcm_data:
+                wav_data = self._pcm_to_wav(combined_pcm_data)
+            
+            
+            # 定义ASR任务
+            def run_asr():
+                start_time = time.monotonic()
+                try:
+                    loop = asyncio.new_event_loop()
+                    asyncio.set_event_loop(loop)
+                    try:
+                        result = loop.run_until_complete(
+                            self.speech_to_text(asr_audio_task, conn.session_id, conn.audio_format)
+                        )
+                        end_time = time.monotonic()
+                        logger.bind(tag=TAG).info(f"ASR耗时: {end_time - start_time:.3f}s")
+                        return result
+                    finally:
+                        loop.close()
+                except Exception as e:
+                    end_time = time.monotonic()
+                    logger.bind(tag=TAG).error(f"ASR失败: {e}")
+                    return ("", None)
+            
+            # 定义声纹识别任务
+            def run_voiceprint():
+                if not wav_data:
+                    return None
+                try:
+                    loop = asyncio.new_event_loop()
+                    asyncio.set_event_loop(loop)
+                    try:
+                        # 使用连接的声纹识别提供者
+                        result = loop.run_until_complete(
+                            conn.voiceprint_provider.identify_speaker(wav_data, conn.session_id)
+                        )
+                        return result
+                    finally:
+                        loop.close()
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"声纹识别失败: {e}")
+                    return None
+            
+            # 使用线程池执行器并行运行
+            parallel_start_time = time.monotonic()
+            
+            with concurrent.futures.ThreadPoolExecutor(max_workers=2) as thread_executor:
+                asr_future = thread_executor.submit(run_asr)
+                
+                if conn.voiceprint_provider and wav_data:
+                    voiceprint_future = thread_executor.submit(run_voiceprint)
+                    
+                    # 等待两个线程都完成
+                    asr_result = asr_future.result(timeout=15)
+                    voiceprint_result = voiceprint_future.result(timeout=15)
+                    
+                    results = {"asr": asr_result, "voiceprint": voiceprint_result}
+                else:
+                    asr_result = asr_future.result(timeout=15)
+                    results = {"asr": asr_result, "voiceprint": None}
+            
+            
+            # 处理结果
+            raw_text, file_path = results.get("asr", ("", None))
+            speaker_name = results.get("voiceprint", None)
+            
+            # 记录识别结果
+            if raw_text:
+                logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
+            if speaker_name:
+                logger.bind(tag=TAG).info(f"识别说话人: {speaker_name}")
+            
+            # 性能监控
+            total_time = time.monotonic() - total_start_time
+            logger.bind(tag=TAG).info(f"总处理耗时: {total_time:.3f}s")
+            
+            # 检查文本长度
+            text_len, _ = remove_punctuation_and_length(raw_text)
+            self.stop_ws_connection()
+            
+            if text_len > 0:
+                # 构建包含说话人信息的JSON字符串
+                enhanced_text = self._build_enhanced_text(raw_text, speaker_name)
+                
+                # 使用自定义模块进行上报
+                await startToChat(conn, enhanced_text)
+                enqueue_asr_report(conn, enhanced_text, asr_audio_task)
+                
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"处理语音停止失败: {e}")
+            import traceback
+            logger.bind(tag=TAG).debug(f"异常详情: {traceback.format_exc()}")
+
+    def _build_enhanced_text(self, text: str, speaker_name: Optional[str]) -> str:
+        """构建包含说话人信息的文本"""
+        if speaker_name and speaker_name.strip():
+            return json.dumps({
+                "speaker": speaker_name,
+                "content": text
+            }, ensure_ascii=False)
+        else:
+            return text
+
+    def _pcm_to_wav(self, pcm_data: bytes) -> bytes:
+        """将PCM数据转换为WAV格式"""
+        if len(pcm_data) == 0:
+            logger.bind(tag=TAG).warning("PCM数据为空，无法转换WAV")
+            return b""
+        
+        # 确保数据长度是偶数（16位音频）
+        if len(pcm_data) % 2 != 0:
+            pcm_data = pcm_data[:-1]
+        
+        # 创建WAV文件头
+        wav_buffer = io.BytesIO()
+        try:
+            with wave.open(wav_buffer, 'wb') as wav_file:
+                wav_file.setnchannels(1)      # 单声道
+                wav_file.setsampwidth(2)      # 16位
+                wav_file.setframerate(16000)  # 16kHz采样率
+                wav_file.writeframes(pcm_data)
+            
+            wav_buffer.seek(0)
+            wav_data = wav_buffer.read()
+            
+            return wav_data
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"WAV转换失败: {e}")
+            return b""
 
     def stop_ws_connection(self):
         pass
@@ -113,27 +243,29 @@ class ASRProviderBase(ABC):
         pass
 
     @staticmethod
-    def decode_opus(opus_data: List[bytes]) -> bytes:
+    def decode_opus(opus_data: List[bytes]) -> List[bytes]:
         """将Opus音频数据解码为PCM数据"""
         try:
-            decoder = opuslib_next.Decoder(16000, 1)  # 16kHz, 单声道
+            decoder = opuslib_next.Decoder(16000, 1)
             pcm_data = []
-            buffer_size = 960  # 每次处理960个采样点
+            buffer_size = 960  # 每次处理960个采样点 (60ms at 16kHz)
             
-            for opus_packet in opus_data:
+            for i, opus_packet in enumerate(opus_data):
                 try:
-                    # 使用较小的缓冲区大小进行处理
+                    if not opus_packet or len(opus_packet) == 0:
+                        continue
+                    
                     pcm_frame = decoder.decode(opus_packet, buffer_size)
-                    if pcm_frame:
+                    if pcm_frame and len(pcm_frame) > 0:
                         pcm_data.append(pcm_frame)
+                        
                 except opuslib_next.OpusError as e:
-                    logger.bind(tag=TAG).warning(f"Opus解码错误，跳过当前数据包: {e}")
+                    logger.bind(tag=TAG).warning(f"Opus解码错误，跳过数据包 {i}: {e}")
-                    continue
                 except Exception as e:
-                    logger.bind(tag=TAG).error(f"音频处理错误: {e}", exc_info=True)
+                    logger.bind(tag=TAG).error(f"音频处理错误，数据包 {i}: {e}")
-                    continue
             
             return pcm_data
+            
         except Exception as e:
-            logger.bind(tag=TAG).error(f"音频解码过程发生错误: {e}", exc_info=True)
+            logger.bind(tag=TAG).error(f"音频解码过程发生错误: {e}")
             return []

main/xiaozhi-server/core/providers/asr/doubao_stream.py

@@ -57,6 +57,16 @@ class ASRProvider(ASRProviderBase):
         conn.asr_audio.append(audio)
         conn.asr_audio = conn.asr_audio[-10:]
         
+        # 存储音频数据
+        if not hasattr(conn, 'asr_audio_for_voiceprint'):
+            conn.asr_audio_for_voiceprint = []
+        conn.asr_audio_for_voiceprint.append(audio)
+        
+        # 当没有音频数据时处理完整语音片段
+        if not audio and len(conn.asr_audio_for_voiceprint) > 0:
+            await self.handle_voice_stop(conn, conn.asr_audio_for_voiceprint)
+            conn.asr_audio_for_voiceprint = []
+
         # 如果本次有声音，且之前没有建立连接
         if audio_have_voice and self.asr_ws is None and not self.is_processing:
             try:
@@ -148,6 +158,8 @@ class ASRProvider(ASRProviderBase):
     async def _forward_asr_results(self, conn):
         try:
             while self.asr_ws and not conn.stop_event.is_set():
+                # 获取当前连接的音频数据
+                audio_data = getattr(conn, 'asr_audio_for_voiceprint', [])
                 try:
                     response = await self.asr_ws.recv()
                     result = self.parse_response(response)
@@ -171,7 +183,8 @@ class ASRProvider(ASRProviderBase):
                                 logger.bind(tag=TAG).error(f"识别文本：空")
                                 self.text = ""
                                 conn.reset_vad_states()
-                                await self.handle_voice_stop(conn, None)
+                                if len(audio_data) > 15:  # 确保有足够音频数据
+                                    await self.handle_voice_stop(conn, audio_data)
                                 break
 
                             for utterance in utterances:
@@ -181,7 +194,8 @@ class ASRProvider(ASRProviderBase):
                                         f"识别到文本: {self.text}"
                                     )
                                     conn.reset_vad_states()
-                                    await self.handle_voice_stop(conn, None)
+                                    if len(audio_data) > 15:  # 确保有足够音频数据
+                                        await self.handle_voice_stop(conn, audio_data)
                                     break
                         elif "error" in payload:
                             error_msg = payload.get("error", "未知错误")
@@ -208,6 +222,13 @@ class ASRProvider(ASRProviderBase):
                 await self.asr_ws.close()
                 self.asr_ws = None
             self.is_processing = False
+            if conn:
+                if hasattr(conn, 'asr_audio_for_voiceprint'):
+                    conn.asr_audio_for_voiceprint = []
+                if hasattr(conn, 'asr_audio'):
+                    conn.asr_audio = []
+                if hasattr(conn, 'has_valid_voice'):
+                    conn.has_valid_voice = False
 
     def stop_ws_connection(self):
         if self.asr_ws:
@@ -349,3 +370,12 @@ class ASRProvider(ASRProviderBase):
                 pass
             self.forward_task = None
         self.is_processing = False
+        # 清理所有连接的音频缓冲区
+        if hasattr(self, '_connections'):
+            for conn in self._connections.values():
+                if hasattr(conn, 'asr_audio_for_voiceprint'):
+                    conn.asr_audio_for_voiceprint = []
+                if hasattr(conn, 'asr_audio'):
+                    conn.asr_audio = []
+                if hasattr(conn, 'has_valid_voice'):
+                    conn.has_valid_voice = False

main/xiaozhi-server/core/providers/asr/openai.py

@@ -0,0 +1,82 @@
+import time
+import os
+from config.logger import setup_logging
+from typing import Optional, Tuple, List
+from core.providers.asr.dto.dto import InterfaceType
+from core.providers.asr.base import ASRProviderBase
+
+import requests
+
+TAG = __name__
+logger = setup_logging()
+
+class ASRProvider(ASRProviderBase):
+    def __init__(self, config: dict, delete_audio_file: bool):
+        self.interface_type = InterfaceType.NON_STREAM
+        self.api_key = config.get("api_key")
+        self.api_url = config.get("base_url")
+        self.model = config.get("model_name")        
+        self.output_dir = config.get("output_dir")
+        self.delete_audio_file = delete_audio_file
+
+        os.makedirs(self.output_dir, exist_ok=True)
+
+    async def speech_to_text(self, opus_data: List[bytes], session_id: str, audio_format="opus") -> Tuple[Optional[str], Optional[str]]:
+        file_path = None
+        try:
+            start_time = time.time()
+            if audio_format == "pcm":
+                pcm_data = opus_data
+            else:
+                pcm_data = self.decode_opus(opus_data)
+            file_path = self.save_audio_to_file(pcm_data, session_id)
+
+            logger.bind(tag=TAG).debug(
+                f"音频文件保存耗时: {time.time() - start_time:.3f}s | 路径: {file_path}"
+            )
+
+            logger.bind(tag=TAG).info(f"file path: {file_path}")
+            headers = {
+                "Authorization": f"Bearer {self.api_key}",
+            }
+            
+            # 使用data参数传递模型名称
+            data = {
+                "model": self.model
+            }
+
+
+            with open(file_path, "rb") as audio_file:  # 使用with语句确保文件关闭
+                files = {
+                    "file": audio_file
+                }
+
+                start_time = time.time()
+                response = requests.post(
+                    self.api_url,
+                    files=files,
+                    data=data,
+                    headers=headers
+                )
+                logger.bind(tag=TAG).debug(
+                    f"语音识别耗时: {time.time() - start_time:.3f}s | 结果: {response.text}"
+                )
+
+            if response.status_code == 200:
+                text = response.json().get("text", "")
+                return text, file_path
+            else:
+                raise Exception(f"API请求失败: {response.status_code} - {response.text}")
+                
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"语音识别失败: {e}")
+            return "", None
+        finally:
+            # 文件清理逻辑
+            if self.delete_audio_file and file_path and os.path.exists(file_path):
+                try:
+                    os.remove(file_path)
+                    logger.bind(tag=TAG).debug(f"已删除临时音频文件: {file_path}")
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"文件删除失败: {file_path} | 错误: {e}")
+        

main/xiaozhi-server/core/providers/asr/sherpa_onnx_local.py

@@ -40,6 +40,7 @@ class ASRProvider(ASRProviderBase):
         self.interface_type = InterfaceType.LOCAL
         self.model_dir = config.get("model_dir")
         self.output_dir = config.get("output_dir")
+        self.model_type = config.get("model_type", "sense_voice")  # 支持 paraformer
         self.delete_audio_file = delete_audio_file
 
         # 确保输出目录存在
@@ -73,16 +74,27 @@ class ASRProvider(ASRProviderBase):
             raise
 
         with CaptureOutput():
-            self.model = sherpa_onnx.OfflineRecognizer.from_sense_voice(
+            if self.model_type == "paraformer":
-                model=self.model_path,
+                self.model = sherpa_onnx.OfflineRecognizer.from_paraformer(
-                tokens=self.tokens_path,
+                    paraformer=self.model_path,
-                num_threads=2,
+                    tokens=self.tokens_path,
-                sample_rate=16000,
+                    num_threads=2,
-                feature_dim=80,
+                    sample_rate=16000,
-                decoding_method="greedy_search",
+                    feature_dim=80,
-                debug=False,
+                    decoding_method="greedy_search",
-                use_itn=True,
+                    debug=False,
-            )
+                )
+            else:  # sense_voice
+                self.model = sherpa_onnx.OfflineRecognizer.from_sense_voice(
+                    model=self.model_path,
+                    tokens=self.tokens_path,
+                    num_threads=2,
+                    sample_rate=16000,
+                    feature_dim=80,
+                    decoding_method="greedy_search",
+                    debug=False,
+                    use_itn=True,
+                )
 
     def read_wave(self, wave_filename: str) -> Tuple[np.ndarray, int]:
         """

main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py

@@ -16,10 +16,11 @@ class IntentProvider(IntentProviderBase):
         super().__init__(config)
         self.llm = None
         self.promot = ""
-        # 添加缓存管理
+        # 导入全局缓存管理器
-        self.intent_cache = {}  # 缓存意图识别结果
+        from core.utils.cache.manager import cache_manager, CacheType
-        self.cache_expiry = 600  # 缓存有效期10分钟
+
-        self.cache_max_size = 100  # 最多缓存100个意图
+        self.cache_manager = cache_manager
+        self.CacheType = CacheType
         self.history_count = 4  # 默认使用最近4条对话记录
 
     def get_intent_system_prompt(self, functions_list: str) -> str:
@@ -102,27 +103,6 @@ class IntentProvider(IntentProviderBase):
         )
         return prompt
 
-    def clean_cache(self):
-        """清理过期缓存"""
-        now = time.time()
-        # 找出过期键
-        expired_keys = [
-            k
-            for k, v in self.intent_cache.items()
-            if now - v["timestamp"] > self.cache_expiry
-        ]
-        for key in expired_keys:
-            del self.intent_cache[key]
-
-        # 如果缓存太大，移除最旧的条目
-        if len(self.intent_cache) > self.cache_max_size:
-            # 按时间戳排序并保留最新的条目
-            sorted_items = sorted(
-                self.intent_cache.items(), key=lambda x: x[1]["timestamp"]
-            )
-            for key, _ in sorted_items[: len(sorted_items) - self.cache_max_size]:
-                del self.intent_cache[key]
-
     def replyResult(self, text: str, original_text: str):
         llm_result = self.llm.response_no_stream(
             system_prompt=text,
@@ -145,21 +125,16 @@ class IntentProvider(IntentProviderBase):
         logger.bind(tag=TAG).debug(f"使用意图识别模型: {model_info}")
 
         # 计算缓存键
-        cache_key = hashlib.md5(text.encode()).hexdigest()
+        cache_key = hashlib.md5((conn.device_id + text).encode()).hexdigest()
 
         # 检查缓存
-        if cache_key in self.intent_cache:
+        cached_intent = self.cache_manager.get(self.CacheType.INTENT, cache_key)
-            cache_entry = self.intent_cache[cache_key]
+        if cached_intent is not None:
-            # 检查缓存是否过期
+            cache_time = time.time() - total_start_time
-            if time.time() - cache_entry["timestamp"] <= self.cache_expiry:
+            logger.bind(tag=TAG).debug(
-                cache_time = time.time() - total_start_time
+                f"使用缓存的意图: {cache_key} -> {cached_intent}, 耗时: {cache_time:.4f}秒"
-                logger.bind(tag=TAG).debug(
+            )
-                    f"使用缓存的意图: {cache_key} -> {cache_entry['intent']}, 耗时: {cache_time:.4f}秒"
+            return cached_intent
-                )
-                return cache_entry["intent"]
-
-        # 清理缓存
-        self.clean_cache()
 
         if self.promot == "":
             functions = conn.func_handler.get_functions()
@@ -259,10 +234,7 @@ class IntentProvider(IntentProviderBase):
                     conn.dialogue.dialogue = clean_history
 
                 # 添加到缓存
-                self.intent_cache[cache_key] = {
+                self.cache_manager.set(self.CacheType.INTENT, cache_key, intent)
-                    "intent": intent,
-                    "timestamp": time.time(),
-                }
 
                 # 后处理时间
                 postprocess_time = time.time() - postprocess_start_time
@@ -272,10 +244,7 @@ class IntentProvider(IntentProviderBase):
                 return intent
             else:
                 # 添加到缓存
-                self.intent_cache[cache_key] = {
+                self.cache_manager.set(self.CacheType.INTENT, cache_key, intent)
-                    "intent": intent,
-                    "timestamp": time.time(),
-                }
 
                 # 后处理时间
                 postprocess_time = time.time() - postprocess_start_time

main/xiaozhi-server/core/providers/memory/mem_local_short/mem_local_short.py

@@ -79,10 +79,11 @@ short_term_memory_prompt_only_content = """
 1、总结user的重要信息，以便在未来的对话中提供更个性化的服务
 2、不要重复总结，不要遗忘之前记忆，除非原来的记忆超过了1800字内，否则不要遗忘、不要压缩用户的历史记忆
 3、用户操控的设备音量、播放音乐、天气、退出、不想对话等和用户本身无关的内容，这些信息不需要加入到总结中
-4、不要把设备操控的成果结果和失败结果加入到总结中，也不要把用户的一些废话加入到总结中
+4、聊天内容中的今天的日期时间、今天的天气情况与用户事件无关的数据，这些信息如果当成记忆存储会影响后序对话，这些信息不需要加入到总结中
-5、不要为了总结而总结，如果用户的聊天没有意义，请返回原来的历史记录也是可以的
+5、不要把设备操控的成果结果和失败结果加入到总结中，也不要把用户的一些废话加入到总结中
-6、只需要返回总结摘要，严格控制在1800字内
+6、不要为了总结而总结，如果用户的聊天没有意义，请返回原来的历史记录也是可以的
-7、不要包含代码、xml，不需要解释、注释和说明，保存记忆时仅从对话提取信息，不要混入示例内容
+7、只需要返回总结摘要，严格控制在1800字内
+8、不要包含代码、xml，不需要解释、注释和说明，保存记忆时仅从对话提取信息，不要混入示例内容
 """
 
 

main/xiaozhi-server/core/providers/tools/server_plugins/plugin_executor.py

@@ -51,7 +51,7 @@ class ServerPluginExecutor(ToolExecutor):
         tools = {}
 
         # 获取必要的函数
-        necessary_functions = ["handle_exit_intent", "get_time", "get_lunar"]
+        necessary_functions = ["handle_exit_intent", "get_lunar"]
 
         # 获取配置中的函数
         config_functions = self.config["Intent"][

main/xiaozhi-server/core/providers/tts/aliyun.py

@@ -6,11 +6,14 @@ import base64
 import requests
 from datetime import datetime
 from core.providers.tts.base import TTSProviderBase
-
+from config.logger import setup_logging
 import time
 import uuid
 from urllib import parse
 
+TAG = __name__
+logger = setup_logging()
+
 
 class AccessToken:
     @staticmethod

main/xiaozhi-server/core/providers/tts/aliyun_stream.py

@@ -0,0 +1,653 @@
+import uuid
+import json
+import hmac
+import hashlib
+import base64
+import time
+import queue
+import asyncio
+import traceback
+from asyncio import Task
+import websockets
+import os
+from datetime import datetime
+from urllib import parse
+from core.providers.tts.base import TTSProviderBase
+from core.providers.tts.dto.dto import SentenceType, ContentType, InterfaceType
+from core.utils.tts import MarkdownCleaner
+from core.utils import opus_encoder_utils, textUtils
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class AccessToken:
+    @staticmethod
+    def _encode_text(text):
+        encoded_text = parse.quote_plus(text)
+        return encoded_text.replace("+", "%20").replace("*", "%2A").replace("%7E", "~")
+
+    @staticmethod
+    def _encode_dict(dic):
+        keys = dic.keys()
+        dic_sorted = [(key, dic[key]) for key in sorted(keys)]
+        encoded_text = parse.urlencode(dic_sorted)
+        return encoded_text.replace("+", "%20").replace("*", "%2A").replace("%7E", "~")
+
+    @staticmethod
+    def create_token(access_key_id, access_key_secret):
+        parameters = {
+            "AccessKeyId": access_key_id,
+            "Action": "CreateToken",
+            "Format": "JSON",
+            "RegionId": "cn-shanghai",  # 使用上海地域进行Token获取
+            "SignatureMethod": "HMAC-SHA1",
+            "SignatureNonce": str(uuid.uuid1()),
+            "SignatureVersion": "1.0",
+            "Timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "Version": "2019-02-28",
+        }
+
+        query_string = AccessToken._encode_dict(parameters)
+        string_to_sign = (
+            "GET"
+            + "&"
+            + AccessToken._encode_text("/")
+            + "&"
+            + AccessToken._encode_text(query_string)
+        )
+
+        secreted_string = hmac.new(
+            bytes(access_key_secret + "&", encoding="utf-8"),
+            bytes(string_to_sign, encoding="utf-8"),
+            hashlib.sha1,
+        ).digest()
+        signature = base64.b64encode(secreted_string)
+        signature = AccessToken._encode_text(signature)
+
+        full_url = "http://nls-meta.cn-shanghai.aliyuncs.com/?Signature=%s&%s" % (
+            signature,
+            query_string,
+        )
+
+        import requests
+
+        response = requests.get(full_url)
+        if response.ok:
+            root_obj = response.json()
+            key = "Token"
+            if key in root_obj:
+                token = root_obj[key]["Id"]
+                expire_time = root_obj[key]["ExpireTime"]
+                return token, expire_time
+        return None, None
+
+
+class TTSProvider(TTSProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__(config, delete_audio_file)
+
+        # 设置为流式接口类型
+        self.interface_type = InterfaceType.DUAL_STREAM
+
+        # 基础配置
+        self.access_key_id = config.get("access_key_id")
+        self.access_key_secret = config.get("access_key_secret")
+        self.appkey = config.get("appkey")
+        self.format = config.get("format", "pcm")
+        self.audio_file_type = config.get("format", "pcm")
+
+        # 采样率配置
+        sample_rate = config.get("sample_rate", "16000")
+        self.sample_rate = int(sample_rate) if sample_rate else 16000
+
+        # 音色配置 - CosyVoice大模型音色
+        if config.get("private_voice"):
+            self.voice = config.get("private_voice")
+        else:
+            self.voice = config.get("voice", "longxiaochun")  # CosyVoice默认音色
+
+        # 音频参数配置
+        volume = config.get("volume", "50")
+        self.volume = int(volume) if volume else 50
+
+        speech_rate = config.get("speech_rate", "0")
+        self.speech_rate = int(speech_rate) if speech_rate else 0
+
+        pitch_rate = config.get("pitch_rate", "0")
+        self.pitch_rate = int(pitch_rate) if pitch_rate else 0
+
+        # WebSocket配置
+        self.host = config.get("host", "nls-gateway-cn-beijing.aliyuncs.com")
+        # 如果配置的是内网地址（包含-internal.aliyuncs.com），则使用ws协议，默认是wss协议
+        if "-internal." in self.host:
+            self.ws_url = f"ws://{self.host}/ws/v1"
+        else:
+            # 默认使用wss协议
+            self.ws_url = f"wss://{self.host}/ws/v1"
+        self.ws = None
+        self._monitor_task = None
+        self.last_active_time = None
+
+        # 专属tts设置
+        self.message_id = ""
+
+        # 创建Opus编码器
+        self.opus_encoder = opus_encoder_utils.OpusEncoderUtils(
+            sample_rate=16000, channels=1, frame_size_ms=60
+        )
+
+        # Token管理
+        if self.access_key_id and self.access_key_secret:
+            self._refresh_token()
+        else:
+            self.token = config.get("token")
+            self.expire_time = None
+
+    def _refresh_token(self):
+        """刷新Token并记录过期时间"""
+        if self.access_key_id and self.access_key_secret:
+            self.token, expire_time_str = AccessToken.create_token(
+                self.access_key_id, self.access_key_secret
+            )
+            if not expire_time_str:
+                raise ValueError("无法获取有效的Token过期时间")
+
+            expire_str = str(expire_time_str).strip()
+
+            try:
+                if expire_str.isdigit():
+                    expire_time = datetime.fromtimestamp(int(expire_str))
+                else:
+                    expire_time = datetime.strptime(expire_str, "%Y-%m-%dT%H:%M:%SZ")
+                self.expire_time = expire_time.timestamp() - 60
+            except Exception as e:
+                raise ValueError(f"无效的过期时间格式: {expire_str}") from e
+        else:
+            self.expire_time = None
+
+        if not self.token:
+            raise ValueError("无法获取有效的访问Token")
+
+    def _is_token_expired(self):
+        """检查Token是否过期"""
+        if not self.expire_time:
+            return False
+        return time.time() > self.expire_time
+
+    async def _ensure_connection(self):
+        """确保WebSocket连接可用"""
+        try:
+            if self._is_token_expired():
+                logger.bind(tag=TAG).warning("Token已过期，正在自动刷新...")
+                self._refresh_token()
+            current_time = time.time()
+            if self.ws and current_time - self.last_active_time < 10:
+                # 10秒内才可以复用链接进行连续对话
+                logger.bind(tag=TAG).info(f"使用已有链接...")
+                return self.ws
+            logger.bind(tag=TAG).info("开始建立新连接...")
+
+            self.ws = await websockets.connect(
+                self.ws_url,
+                additional_headers={"X-NLS-Token": self.token},
+                ping_interval=30,
+                ping_timeout=10,
+                close_timeout=10,
+            )
+            logger.bind(tag=TAG).info("WebSocket连接建立成功")
+            self.last_active_time = time.time()
+            return self.ws
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"建立连接失败: {str(e)}")
+            self.ws = None
+            self.last_active_time = None
+            raise
+
+    def tts_text_priority_thread(self):
+        """流式文本处理线程"""
+        while not self.conn.stop_event.is_set():
+            try:
+                message = self.tts_text_queue.get(timeout=1)
+                logger.bind(tag=TAG).debug(
+                    f"收到TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
+                )
+
+                if message.sentence_type == SentenceType.FIRST:
+                    self.conn.client_abort = False
+
+                if self.conn.client_abort:
+                    logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                    continue
+
+                if message.sentence_type == SentenceType.FIRST:
+                    # 初始化参数
+                    try:
+                        if not getattr(self.conn, "sentence_id", None):
+                            self.conn.sentence_id = uuid.uuid4().hex
+                            logger.bind(tag=TAG).info(
+                                f"自动生成新的 会话ID: {self.conn.sentence_id}"
+                            )
+
+                        # aliyunStream独有的参数生成
+                        self.message_id = str(uuid.uuid4().hex)
+
+                        logger.bind(tag=TAG).info("开始启动TTS会话...")
+                        future = asyncio.run_coroutine_threadsafe(
+                            self.start_session(self.conn.sentence_id),
+                            loop=self.conn.loop,
+                        )
+                        future.result()
+                        self.before_stop_play_files.clear()
+                        logger.bind(tag=TAG).info("TTS会话启动成功")
+
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"启动TTS会话失败: {str(e)}")
+                        continue
+
+                elif ContentType.TEXT == message.content_type:
+                    if message.content_detail:
+                        try:
+                            logger.bind(tag=TAG).debug(
+                                f"开始发送TTS文本: {message.content_detail}"
+                            )
+                            future = asyncio.run_coroutine_threadsafe(
+                                self.text_to_speak(message.content_detail, None),
+                                loop=self.conn.loop,
+                            )
+                            future.result()
+                            logger.bind(tag=TAG).debug("TTS文本发送成功")
+                        except Exception as e:
+                            logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
+                            continue
+
+                elif ContentType.FILE == message.content_type:
+                    logger.bind(tag=TAG).info(
+                        f"添加音频文件到待播放列表: {message.content_file}"
+                    )
+                    if message.content_file and os.path.exists(message.content_file):
+                        # 先处理文件音频数据
+                        file_audio = self._process_audio_file(message.content_file)
+                        self.before_stop_play_files.append(
+                            (file_audio, message.content_detail)
+                        )
+
+                if message.sentence_type == SentenceType.LAST:
+                    try:
+                        logger.bind(tag=TAG).info("开始结束TTS会话...")
+                        future = asyncio.run_coroutine_threadsafe(
+                            self.finish_session(self.conn.sentence_id),
+                            loop=self.conn.loop,
+                        )
+                        future.result()
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"结束TTS会话失败: {str(e)}")
+                        continue
+
+            except queue.Empty:
+                continue
+            except Exception as e:
+                logger.bind(tag=TAG).error(
+                    f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
+                )
+
+    async def text_to_speak(self, text, _):
+        try:
+            if self.ws is None:
+                logger.bind(tag=TAG).warning(f"WebSocket连接不存在，终止发送文本")
+                return
+            filtered_text = MarkdownCleaner.clean_markdown(text)
+            run_request = {
+                "header": {
+                    "message_id": self.message_id,
+                    "task_id": self.conn.sentence_id,
+                    "namespace": "FlowingSpeechSynthesizer",
+                    "name": "RunSynthesis",
+                    "appkey": self.appkey,
+                },
+                "payload": {"text": filtered_text},
+            }
+            await self.ws.send(json.dumps(run_request))
+            self.last_active_time = time.time()
+            return
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
+            if self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+            raise
+
+    async def start_session(self, session_id):
+        logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
+        try:
+            # 会话开始时检测上个会话的监听状态
+            if (
+                self._monitor_task is not None
+                and isinstance(self._monitor_task, Task)
+                and not self._monitor_task.done()
+            ):
+                logger.bind(tag=TAG).info(
+                    "检测到未完成的上个会话，关闭监听任务和连接..."
+                )
+                await self.close()
+
+            # 建立新连接
+            await self._ensure_connection()
+
+            # 启动监听任务
+            self._monitor_task = asyncio.create_task(self._start_monitor_tts_response())
+
+            start_request = {
+                "header": {
+                    "message_id": self.message_id,
+                    "task_id": self.conn.sentence_id,
+                    "namespace": "FlowingSpeechSynthesizer",
+                    "name": "StartSynthesis",
+                    "appkey": self.appkey,
+                },
+                "payload": {
+                    "voice": self.voice,
+                    "format": self.format,
+                    "sample_rate": self.sample_rate,
+                    "volume": self.volume,
+                    "speech_rate": self.speech_rate,
+                    "pitch_rate": self.pitch_rate,
+                    "enable_subtitle": True,
+                },
+            }
+            await self.ws.send(json.dumps(start_request))
+            self.last_active_time = time.time()
+            logger.bind(tag=TAG).info("会话启动请求已发送")
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
+            # 确保清理资源
+            await self.close()
+            raise
+
+    async def finish_session(self, session_id):
+        logger.bind(tag=TAG).info(f"关闭会话～～{session_id}")
+        try:
+            if self.ws:
+                stop_request = {
+                    "header": {
+                        "message_id": self.message_id,
+                        "task_id": self.conn.sentence_id,
+                        "namespace": "FlowingSpeechSynthesizer",
+                        "name": "StopSynthesis",
+                        "appkey": self.appkey,
+                    }
+                }
+                await self.ws.send(json.dumps(stop_request))
+                logger.bind(tag=TAG).info("会话结束请求已发送")
+                self.last_active_time = time.time()
+                if self._monitor_task:
+                    try:
+                        await self._monitor_task
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(
+                            f"等待监听任务完成时发生错误: {str(e)}"
+                        )
+                    finally:
+                        self._monitor_task = None
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
+            # 确保清理资源
+            await self.close()
+            raise
+
+    async def close(self):
+        """资源清理"""
+        if self._monitor_task:
+            try:
+                self._monitor_task.cancel()
+                await self._monitor_task
+            except asyncio.CancelledError:
+                pass
+            except Exception as e:
+                logger.bind(tag=TAG).warning(f"关闭时取消监听任务错误: {e}")
+            self._monitor_task = None
+
+        if self.ws:
+            try:
+                await self.ws.close()
+            except:
+                pass
+            self.ws = None
+            self.last_active_time = None
+
+    async def _start_monitor_tts_response(self):
+        """监听TTS响应"""
+        opus_datas_cache = []
+        is_first_sentence = True
+        first_sentence_segment_count = 0  # 添加计数器
+        try:
+            session_finished = False  # 标记会话是否正常结束
+            while not self.conn.stop_event.is_set():
+                try:
+                    msg = await self.ws.recv()
+                    self.last_active_time = time.time()
+                    # 检查客户端是否中止
+                    if self.conn.client_abort:
+                        logger.bind(tag=TAG).info("收到打断信息，终止监听TTS响应")
+                        break
+                    if isinstance(msg, str):  # 文本控制消息
+                        try:
+                            data = json.loads(msg)
+                            header = data.get("header", {})
+                            event_name = header.get("name")
+                            if event_name == "SynthesisStarted":
+                                logger.bind(tag=TAG).debug("TTS合成已启动")
+                                self.tts_audio_queue.put(
+                                    (SentenceType.FIRST, [], None)
+                                )
+                            elif event_name == "SentenceBegin":
+                                opus_datas_cache = []
+                            elif event_name == "SentenceEnd":
+                                if (
+                                    not is_first_sentence
+                                    or first_sentence_segment_count > 10
+                                ):
+                                    # 发送缓存的数据
+                                    if self.conn.tts_MessageText:
+                                        logger.bind(tag=TAG).info(
+                                            f"句子语音生成成功： {self.conn.tts_MessageText}"
+                                        )
+                                        self.tts_audio_queue.put(
+                                            (SentenceType.MIDDLE, opus_datas_cache, self.conn.tts_MessageText)
+                                        )
+                                        self.conn.tts_MessageText = None
+                                    else:
+                                        self.tts_audio_queue.put(
+                                            (SentenceType.MIDDLE, opus_datas_cache, None)
+                                        )
+                                # 第一句话结束后，将标志设置为False
+                                is_first_sentence = False
+                            elif event_name == "SynthesisCompleted":
+                                logger.bind(tag=TAG).debug(f"会话结束～～")
+                                self._process_before_stop_play_files()
+                                session_finished = True
+                                break
+                        except json.JSONDecodeError:
+                            logger.bind(tag=TAG).warning("收到无效的JSON消息")
+                    # 二进制消息（音频数据）
+                    elif isinstance(msg, (bytes, bytearray)):
+                        logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
+                        opus_datas = self.opus_encoder.encode_pcm_to_opus(msg, False)
+                        logger.bind(tag=TAG).debug(
+                            f"推送数据到队列里面帧数～～{len(opus_datas)}"
+                        )
+                        if is_first_sentence:
+                            first_sentence_segment_count += 1
+                            if first_sentence_segment_count <= 6:
+                                self.tts_audio_queue.put(
+                                    (SentenceType.MIDDLE, opus_datas, None)
+                                )
+                            else:
+                                opus_datas_cache.extend(opus_datas)
+                        else:
+                            # 后续句子缓存
+                            opus_datas_cache.extend(opus_datas)
+
+                except websockets.ConnectionClosed:
+                    logger.bind(tag=TAG).warning("WebSocket连接已关闭")
+                    break
+                except Exception as e:
+                    logger.bind(tag=TAG).error(
+                        f"处理TTS响应时出错: {e}\n{traceback.format_exc()}"
+                    )
+                    break
+            # 仅在连接异常时才关闭
+            if not session_finished and self.ws:
+                try:
+                    await self.ws.close()
+                except:
+                    pass
+                self.ws = None
+        # 监听任务退出时清理引用
+        finally:
+            self._monitor_task = None
+
+    def to_tts(self, text: str) -> list:
+        """非流式TTS处理，用于测试及保存音频文件的场景"""
+        try:
+            # 创建新的事件循环
+            loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+
+            # 生成会话ID
+            session_id = uuid.uuid4().hex
+            # 存储音频数据
+            audio_data = []
+
+            async def _generate_audio():
+                # 刷新Token（如果需要）
+                if self._is_token_expired():
+                    self._refresh_token()
+
+                # 建立WebSocket连接
+                ws = await websockets.connect(
+                    self.ws_url,
+                    additional_headers={"X-NLS-Token": self.token},
+                    ping_interval=30,
+                    ping_timeout=10,
+                    close_timeout=10,
+                )
+                try:
+                    # 发送StartSynthesis请求
+                    start_message_id = str(uuid.uuid4().hex)
+                    start_request = {
+                        "header": {
+                            "message_id": start_message_id,
+                            "task_id": session_id,
+                            "namespace": "FlowingSpeechSynthesizer",
+                            "name": "StartSynthesis",
+                            "appkey": self.appkey,
+                        },
+                        "payload": {
+                            "voice": self.voice,
+                            "format": self.format,
+                            "sample_rate": self.sample_rate,
+                            "volume": self.volume,
+                            "speech_rate": self.speech_rate,
+                            "pitch_rate": self.pitch_rate,
+                            "enable_subtitle": True,
+                        },
+                    }
+                    await ws.send(json.dumps(start_request))
+
+                    # 等待SynthesisStarted响应
+                    synthesis_started = False
+                    while not synthesis_started:
+                        msg = await ws.recv()
+                        if isinstance(msg, str):
+                            data = json.loads(msg)
+                            header = data.get("header", {})
+                            if header.get("name") == "SynthesisStarted":
+                                synthesis_started = True
+                                logger.bind(tag=TAG).debug("TTS合成已启动")
+                            elif header.get("name") == "TaskFailed":
+                                error_info = data.get("payload", {}).get(
+                                    "error_info", {}
+                                )
+                                error_code = error_info.get("error_code")
+                                error_message = error_info.get(
+                                    "error_message", "未知错误"
+                                )
+                                raise Exception(
+                                    f"启动合成失败: {error_code} - {error_message}"
+                                )
+
+                    # 发送文本合成请求
+                    filtered_text = MarkdownCleaner.clean_markdown(text)
+                    run_message_id = str(uuid.uuid4().hex)
+                    run_request = {
+                        "header": {
+                            "message_id": run_message_id,
+                            "task_id": session_id,
+                            "namespace": "FlowingSpeechSynthesizer",
+                            "name": "RunSynthesis",
+                            "appkey": self.appkey,
+                        },
+                        "payload": {"text": filtered_text},
+                    }
+                    await ws.send(json.dumps(run_request))
+
+                    # 发送停止合成请求
+                    stop_message_id = str(uuid.uuid4().hex)
+                    stop_request = {
+                        "header": {
+                            "message_id": stop_message_id,
+                            "task_id": session_id,
+                            "namespace": "FlowingSpeechSynthesizer",
+                            "name": "StopSynthesis",
+                            "appkey": self.appkey,
+                        }
+                    }
+                    await ws.send(json.dumps(stop_request))
+
+                    # 接收音频数据
+                    synthesis_completed = False
+                    while not synthesis_completed:
+                        msg = await ws.recv()
+                        if isinstance(msg, (bytes, bytearray)):
+                            # 编码为Opus并收集
+                            opus_frames = self.opus_encoder.encode_pcm_to_opus(
+                                msg, False
+                            )
+                            audio_data.extend(opus_frames)
+                        elif isinstance(msg, str):
+                            data = json.loads(msg)
+                            header = data.get("header", {})
+                            event_name = header.get("name")
+                            if event_name == "SynthesisCompleted":
+                                synthesis_completed = True
+                                logger.bind(tag=TAG).debug("TTS合成完成")
+                            elif event_name == "TaskFailed":
+                                error_info = data.get("payload", {}).get(
+                                    "error_info", {}
+                                )
+                                error_code = error_info.get("error_code")
+                                error_message = error_info.get(
+                                    "error_message", "未知错误"
+                                )
+                                raise Exception(
+                                    f"合成失败: {error_code} - {error_message}"
+                                )
+                finally:
+                    try:
+                        await ws.close()
+                    except:
+                        pass
+
+            loop.run_until_complete(_generate_audio())
+            loop.close()
+
+            return audio_data
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"生成音频数据失败: {str(e)}")
+            return []

main/xiaozhi-server/core/providers/tts/base.py

@@ -50,6 +50,7 @@ class TTSProviderBase(ABC):
             "；",
             ";",
             "：",
+            "~",
         )
         self.first_sentence_punctuations = (
             "，",
@@ -159,15 +160,8 @@ class TTSProviderBase(ABC):
             if conn.sentence_id:
                 sentence_id = conn.sentence_id
             else:
-                sentence_id = str(uuid.uuid4()).replace("-", "")
+                sentence_id = str(uuid.uuid4().hex)
                 conn.sentence_id = sentence_id
-        self.tts_text_queue.put(
-            TTSMessageDTO(
-                sentence_id=sentence_id,
-                sentence_type=SentenceType.FIRST,
-                content_type=ContentType.ACTION,
-            )
-        )
         # 对于单句的文本，进行分段处理
         segments = re.split(r"([。！？!?；;\n])", content_detail)
         for seg in segments:
@@ -180,13 +174,6 @@ class TTSProviderBase(ABC):
                     content_file=content_file,
                 )
             )
-        self.tts_text_queue.put(
-            TTSMessageDTO(
-                sentence_id=sentence_id,
-                sentence_type=SentenceType.LAST,
-                content_type=ContentType.ACTION,
-            )
-        )
 
     async def open_audio_channels(self, conn):
         self.conn = conn
@@ -209,6 +196,8 @@ class TTSProviderBase(ABC):
         while not self.conn.stop_event.is_set():
             try:
                 message = self.tts_text_queue.get(timeout=1)
+                if message.sentence_type == SentenceType.FIRST:
+                    self.conn.client_abort = False
                 if self.conn.client_abort:
                     logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
                     continue
@@ -344,7 +333,6 @@ class TTSProviderBase(ABC):
         Returns:
             tuple: (sentence_type, audio_datas, content_detail)
         """
-        audio_datas = []
         if tts_file.endswith(".p3"):
             audio_datas, _ = p3.decode_opus_from_file(tts_file)
         elif self.conn.audio_format == "pcm":
@@ -362,10 +350,8 @@ class TTSProviderBase(ABC):
         return audio_datas
 
     def _process_before_stop_play_files(self):
-        for tts_file, text in self.before_stop_play_files:
+        for audio_datas, text in self.before_stop_play_files:
-            if tts_file and os.path.exists(tts_file):
+            self.tts_audio_queue.put((SentenceType.MIDDLE, audio_datas, text))
-                audio_datas = self._process_audio_file(tts_file)
-                self.tts_audio_queue.put((SentenceType.MIDDLE, audio_datas, text))
         self.before_stop_play_files.clear()
         self.tts_audio_queue.put((SentenceType.LAST, [], None))
 

main/xiaozhi-server/core/providers/tts/huoshan_double_stream.py

@@ -10,7 +10,6 @@ from config.logger import setup_logging
 from core.utils import opus_encoder_utils
 from core.utils.util import check_model_key
 from core.providers.tts.base import TTSProviderBase
-from core.handle.abortHandle import handleAbortMessage
 from core.providers.tts.dto.dto import SentenceType, ContentType, InterfaceType
 from asyncio import Task
 
@@ -53,10 +52,11 @@ EVENT_ConnectionFinished = 52  # 连接结束
 
 # 上行Session事件
 EVENT_StartSession = 100
-
+EVENT_CancelSession = 101
 EVENT_FinishSession = 102
 # 下行Session事件
 EVENT_SessionStarted = 150
+EVENT_SessionCanceled = 151
 EVENT_SessionFinished = 152
 
 EVENT_SessionFailed = 153
@@ -175,6 +175,9 @@ class TTSProvider(TTSProviderBase):
     async def _ensure_connection(self):
         """建立新的WebSocket连接"""
         try:
+            if self.ws:
+                logger.bind(tag=TAG).info(f"使用已有链接...")
+                return self.ws
             logger.bind(tag=TAG).info("开始建立新连接...")
             ws_header = {
                 "X-Api-App-Key": self.appId,
@@ -200,9 +203,21 @@ class TTSProvider(TTSProviderBase):
                 logger.bind(tag=TAG).debug(
                     f"收到TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
                 )
+
+                if message.sentence_type == SentenceType.FIRST:
+                    self.conn.client_abort = False
+
                 if self.conn.client_abort:
-                    logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                    try:
-                    continue
+                        logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
+                        asyncio.run_coroutine_threadsafe(
+                            self.cancel_session(self.conn.sentence_id),
+                            loop=self.conn.loop,
+                        )
+                        continue
+                    except Exception as e:
+                        logger.bind(tag=TAG).error(f"取消TTS会话失败: {str(e)}")
+                        continue
 
                 if message.sentence_type == SentenceType.FIRST:
                     # 初始化参数
@@ -217,7 +232,6 @@ class TTSProvider(TTSProviderBase):
                             loop=self.conn.loop,
                         )
                         future.result()
-                        self.tts_audio_first_sentence = True
                         self.before_stop_play_files.clear()
                         logger.bind(tag=TAG).info("TTS会话启动成功")
                     except Exception as e:
@@ -244,9 +258,12 @@ class TTSProvider(TTSProviderBase):
                     logger.bind(tag=TAG).info(
                         f"添加音频文件到待播放列表: {message.content_file}"
                     )
-                    self.before_stop_play_files.append(
+                    if message.content_file and os.path.exists(message.content_file):
-                        (message.content_file, message.content_detail)
+                        # 先处理文件音频数据
-                    )
+                        file_audio = self._process_audio_file(message.content_file)
+                        self.before_stop_play_files.append(
+                            (file_audio, message.content_detail)
+                        )
 
                 if message.sentence_type == SentenceType.LAST:
                     try:
@@ -295,25 +312,15 @@ class TTSProvider(TTSProviderBase):
     async def start_session(self, session_id):
         logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
         try:
-            task = self._monitor_task
+            # 会话开始时检测上个会话的监听状态
             if (
-                task is not None
+                self._monitor_task is not None
-                and isinstance(task, Task)
+                and isinstance(self._monitor_task, Task)
-                and not task.done()
+                and not self._monitor_task.done()
             ):
-                logger.bind(tag=TAG).info("等待上一个监听任务结束...")
+                logger.bind(tag=TAG).info("检测到未完成的上个会话，关闭监听任务和连接...")
-                if self.ws is not None:
+                await self.close()
-                    logger.bind(tag=TAG).info("强制关闭上一个WebSocket连接以唤醒监听任务...")
+
-                    try:
-                        await self.ws.close()
-                    except Exception as e:
-                        logger.bind(tag=TAG).warning(f"关闭上一个ws异常: {e}")
-                    self.ws = None
-                try:
-                    await asyncio.wait_for(task, timeout=8)
-                except Exception as e:
-                    logger.bind(tag=TAG).warning(f"等待监听任务异常: {e}")
-                self._monitor_task = None
             # 建立新连接
             await self._ensure_connection()
 
@@ -336,19 +343,7 @@ class TTSProvider(TTSProviderBase):
         except Exception as e:
             logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
             # 确保清理资源
-            if hasattr(self, "_monitor_task"):
+            await self.close()
-                try:
-                    self._monitor_task.cancel()
-                    await self._monitor_task
-                except:
-                    pass
-                self._monitor_task = None
-            if self.ws:
-                try:
-                    await self.ws.close()
-                except:
-                    pass
-                self.ws = None
             raise
 
     async def finish_session(self, session_id):
@@ -368,7 +363,7 @@ class TTSProvider(TTSProviderBase):
                 logger.bind(tag=TAG).info("会话结束请求已发送")
 
                 # 等待监听任务完成
-                if hasattr(self, "_monitor_task"):
+                if self._monitor_task:
                     try:
                         await self._monitor_task
                     except Exception as e:
@@ -378,28 +373,46 @@ class TTSProvider(TTSProviderBase):
                     finally:
                         self._monitor_task = None
 
-                # 关闭连接
-                await self.close()
         except Exception as e:
             logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
             # 确保清理资源
-            if hasattr(self, "_monitor_task"):
+            await self.close()
-                try:
+            raise
-                    self._monitor_task.cancel()
+
-                    await self._monitor_task
+    async def cancel_session(self,session_id):
-                except:
+        logger.bind(tag=TAG).info(f"取消会话，释放服务端资源～～{session_id}")
-                    pass
+        try:
-                self._monitor_task = None
             if self.ws:
-                try:
+                header = Header(
-                    await self.ws.close()
+                    message_type=FULL_CLIENT_REQUEST,
-                except:
+                    message_type_specific_flags=MsgTypeFlagWithEvent,
-                    pass
+                    serial_method=JSON,
-                self.ws = None
+                ).as_bytes()
+                optional = Optional(
+                    event=EVENT_CancelSession, sessionId=session_id
+                ).as_bytes()
+                payload = str.encode("{}")
+                await self.send_event(self.ws, header, optional, payload)
+                logger.bind(tag=TAG).info("会话取消请求已发送")
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"取消会话失败: {str(e)}")
+            # 确保清理资源
+            await self.close()
             raise
 
     async def close(self):
         """资源清理方法"""
+        # 取消监听任务
+        if self._monitor_task:
+            try:
+                self._monitor_task.cancel()
+                await self._monitor_task
+            except asyncio.CancelledError:
+                pass
+            except Exception as e:
+                logger.bind(tag=TAG).warning(f"关闭时取消监听任务错误: {e}")
+            self._monitor_task = None
+
         if self.ws:
             try:
                 await self.ws.close()
@@ -413,6 +426,7 @@ class TTSProvider(TTSProviderBase):
         is_first_sentence = True
         first_sentence_segment_count = 0  # 添加计数器
         try:
+            session_finished = False  # 标记会话是否正常结束
             while not self.conn.stop_event.is_set():
                 try:
                     # 确保 `recv()` 运行在同一个 event loop
@@ -420,12 +434,11 @@ class TTSProvider(TTSProviderBase):
                     res = self.parser_response(msg)
                     self.print_response(res, "send_text res:")
 
-                    # 检查客户端是否中止
+                    if res.optional.event == EVENT_SessionCanceled:
-                    if self.conn.client_abort:
+                        logger.bind(tag=TAG).debug(f"释放服务端资源成功～～")
-                        logger.bind(tag=TAG).info("收到打断信息，终止监听TTS响应")
+                        session_finished = True
                         break
-
+                    elif res.optional.event == EVENT_TTSSentenceStart:
-                    if res.optional.event == EVENT_TTSSentenceStart:
                         json_data = json.loads(res.payload.decode("utf-8"))
                         self.tts_text = json_data.get("text", "")
                         logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
@@ -450,10 +463,10 @@ class TTSProvider(TTSProviderBase):
                                     (SentenceType.MIDDLE, opus_datas, None)
                                 )
                             else:
-                                opus_datas_cache = opus_datas_cache + opus_datas
+                                opus_datas_cache.extend(opus_datas)
                         else:
                             # 后续句子缓存
-                            opus_datas_cache = opus_datas_cache + opus_datas
+                            opus_datas_cache.extend(opus_datas)
                     elif res.optional.event == EVENT_TTSSentenceEnd:
                         logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
                         if not is_first_sentence or first_sentence_segment_count > 10:
@@ -466,6 +479,7 @@ class TTSProvider(TTSProviderBase):
                     elif res.optional.event == EVENT_SessionFinished:
                         logger.bind(tag=TAG).debug(f"会话结束～～")
                         self._process_before_stop_play_files()
+                        session_finished = True
                         break
                 except websockets.ConnectionClosed:
                     logger.bind(tag=TAG).warning("WebSocket连接已关闭")
@@ -476,15 +490,15 @@ class TTSProvider(TTSProviderBase):
                     )
                     traceback.print_exc()
                     break
-        finally:
+            # 仅在连接异常时才关闭
-            # 确保清理资源
+            if not session_finished and self.ws:
-            if self.ws:
                 try:
                     await self.ws.close()
                 except:
                     pass
                 self.ws = None
-            # 监听任务退出时清理引用
+        # 监听任务退出时清理引用
+        finally:
             self._monitor_task = None
 
     async def send_event(

main/xiaozhi-server/core/providers/tts/linkerai.py

@@ -1,3 +1,4 @@
+import os
 import queue
 import asyncio
 import traceback
@@ -51,7 +52,6 @@ class TTSProvider(TTSProviderBase):
                     self.processed_chars = 0
                     self.tts_text_buff = []
                     self.segment_count = 0
-                    self.tts_audio_first_sentence = True
                     self.before_stop_play_files.clear()
                 elif ContentType.TEXT == message.content_type:
                     self.tts_text_buff.append(message.content_detail)
@@ -63,9 +63,12 @@ class TTSProvider(TTSProviderBase):
                     logger.bind(tag=TAG).info(
                         f"添加音频文件到待播放列表: {message.content_file}"
                     )
-                    self.before_stop_play_files.append(
+                    if message.content_file and os.path.exists(message.content_file):
-                        (message.content_file, message.content_detail)
+                        # 先处理文件音频数据
-                    )
+                        file_audio = self._process_audio_file(message.content_file)
+                        self.before_stop_play_files.append(
+                            (file_audio, message.content_detail)
+                        )
 
                 if message.sentence_type == SentenceType.LAST:
                     # 处理剩余的文本

main/xiaozhi-server/core/providers/tts/minimax_httpstream.py

@@ -0,0 +1,230 @@
+import os
+import uuid
+import json
+import requests
+from datetime import datetime
+from typing import Iterator, Optional, Union
+from core.providers.tts.base import TTSProviderBase
+from core.utils.util import parse_string_to_list
+
+
+class TTSProvider(TTSProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__(config, delete_audio_file)
+        self.group_id = config.get("group_id")
+        self.api_key = config.get("api_key")
+        self.model = config.get("model")
+        if config.get("private_voice"):
+            self.voice = config.get("private_voice")
+        else:
+            self.voice = config.get("voice_id")
+
+        default_voice_setting = {
+            "voice_id": "female-shaonv",
+            "speed": 1,
+            "vol": 1,
+            "pitch": 0,
+            "emotion": "happy",
+        }
+        default_pronunciation_dict = {"tone": ["处理/(chu3)(li3)", "危险/dangerous"]}
+        defult_audio_setting = {
+            "sample_rate": 32000,
+            "bitrate": 128000,
+            "format": "mp3",
+            "channel": 1,
+        }
+        self.voice_setting = {
+            **default_voice_setting,
+            **config.get("voice_setting", {}),
+        }
+        self.pronunciation_dict = {
+            **default_pronunciation_dict,
+            **config.get("pronunciation_dict", {}),
+        }
+        self.audio_setting = {**defult_audio_setting, **config.get("audio_setting", {})}
+        self.timber_weights = parse_string_to_list(config.get("timber_weights"))
+
+        if self.voice:
+            self.voice_setting["voice_id"] = self.voice
+
+        self.host = "api.minimax.chat"
+        self.api_url = f"https://{self.host}/v1/t2a_v2?GroupId={self.group_id}"
+        self.header = {
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {self.api_key}",
+        }
+        self.audio_file_type = defult_audio_setting.get("format", "mp3")
+
+    def generate_filename(self, extension=".mp3"):
+        return os.path.join(
+            self.output_file,
+            f"tts-{__name__}{datetime.now().date()}@{uuid.uuid4().hex}{extension}",
+        )
+
+    async def text_to_speak(self, text, output_file):
+        """非流式语音合成（保留原有实现）"""
+        request_json = {
+            "model": self.model,
+            "text": text,
+            "stream": False,
+            "voice_setting": self.voice_setting,
+            "pronunciation_dict": self.pronunciation_dict,
+            "audio_setting": self.audio_setting,
+        }
+
+        if type(self.timber_weights) is list and len(self.timber_weights) > 0:
+            request_json["timber_weights"] = self.timber_weights
+            request_json["voice_setting"]["voice_id"] = ""
+
+        try:
+            resp = requests.post(
+                self.api_url, json.dumps(request_json), headers=self.header
+            )
+            if resp.json()["base_resp"]["status_code"] == 0:
+                data = resp.json()["data"]["audio"]
+                audio_bytes = bytes.fromhex(data)
+                if output_file:
+                    with open(output_file, "wb") as file_to_save:
+                        file_to_save.write(audio_bytes)
+                else:
+                    return audio_bytes
+            else:
+                raise Exception(
+                    f"{__name__} status_code: {resp.status_code} response: {resp.content}"
+                )
+        except Exception as e:
+            raise Exception(f"{__name__} error: {e}")
+
+    def text_to_speak_stream(
+        self, 
+        text: str, 
+        chunk_callback: Optional[callable] = None
+    ) -> Iterator[bytes]:
+        """
+        流式语音合成方法
+        :param text: 要合成的文本
+        :param chunk_callback: 可选的回调函数，用于处理每个音频块
+        :return: 生成器，每次产生一个音频数据块(bytes)
+        """
+        request_json = {
+            "model": self.model,
+            "text": text,
+            "stream": True,
+            "voice_setting": self.voice_setting,
+            "pronunciation_dict": self.pronunciation_dict,
+            "audio_setting": self.audio_setting,
+        }
+
+        if isinstance(self.timber_weights, list) and len(self.timber_weights) > 0:
+            request_json["timber_weights"] = self.timber_weights
+            request_json["voice_setting"]["voice_id"] = ""
+
+        try:
+            with requests.post(
+                self.api_url,
+                data=json.dumps(request_json),
+                headers=self.header,
+                stream=True
+            ) as response:
+                
+                # 检查HTTP状态码
+                if response.status_code != 200:
+                    raise Exception(
+                        f"HTTP error: {response.status_code}, response: {response.text}"
+                    )
+                
+                # 处理流式响应
+                for line in response.iter_lines():
+                    if line:  # 过滤空行
+                        # 检查是否为数据行 (SSE格式)
+                        if line.startswith(b'data:'):
+                            try:
+                                data = json.loads(line[5:].strip())  # 去掉"data:"前缀
+                                
+                                # 检查API状态码
+                                if data.get("base_resp", {}).get("status_code", -1) != 0:
+                                    raise Exception(
+                                        f"API error: {data.get('base_resp', {}).get('status_msg')}"
+                                    )
+                                
+                                # 跳过非音频数据块
+                                if "extra_info" in data:
+                                    continue
+                                    
+                                # 提取音频数据
+                                audio_hex = data.get("data", {}).get("audio")
+                                if audio_hex:
+                                    audio_chunk = bytes.fromhex(audio_hex)
+                                    if chunk_callback:
+                                        chunk_callback(audio_chunk)
+                                    yield audio_chunk
+                                    
+                            except json.JSONDecodeError:
+                                # 忽略JSON解析错误（可能是心跳包等）
+                                continue
+                            except Exception as e:
+                                raise e
+                                
+        except Exception as e:
+            raise Exception(f"{__name__} stream error: {e}")
+
+    def save_stream_to_file(
+        self, 
+        text: str, 
+        output_file: Optional[str] = None,
+        progress_callback: Optional[callable] = None
+    ) -> str:
+        """
+        流式合成并保存到文件
+        :param text: 要合成的文本
+        :param output_file: 输出文件路径，如果为None则自动生成
+        :param progress_callback: 可选的回调函数，接收已写入的字节数
+        :return: 保存的文件路径
+        """
+        if not output_file:
+            output_file = self.generate_filename(extension=f".{self.audio_file_type}")
+        
+        os.makedirs(os.path.dirname(output_file), exist_ok=True)
+        
+        total_bytes = 0
+        try:
+            with open(output_file, "wb") as audio_file:
+                for audio_chunk in self.text_to_speak_stream(text):
+                    audio_file.write(audio_chunk)
+                    audio_file.flush()
+                    total_bytes += len(audio_chunk)
+                    if progress_callback:
+                        progress_callback(total_bytes)
+            return output_file
+        except Exception as e:
+            # 清理可能创建的不完整文件
+            if os.path.exists(output_file):
+                os.remove(output_file)
+            raise e
+
+    def stream_to_audio_player(self, text: str, player_command: list = None):
+        """
+        流式合成并直接播放音频
+        :param text: 要合成的文本
+        :param player_command: 音频播放器命令，默认使用mpv
+        """
+        if player_command is None:
+            player_command = ["mpv", "--no-cache", "--no-terminal", "--", "fd://0"]
+        
+        try:
+            import subprocess
+            player_process = subprocess.Popen(
+                player_command,
+                stdin=subprocess.PIPE,
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+            )
+            
+            for audio_chunk in self.text_to_speak_stream(text):
+                player_process.stdin.write(audio_chunk)
+                player_process.stdin.flush()
+                
+            player_process.stdin.close()
+            player_process.wait()
+        except Exception as e:
+            raise Exception(f"Audio player error: {e}")

main/xiaozhi-server/core/providers/tts/minimax_webSocket.py

@@ -0,0 +1,180 @@
+import os
+import uuid
+import json
+import asyncio
+import websockets
+import ssl
+from datetime import datetime
+from core.providers.tts.base import TTSProviderBase
+from core.utils.util import parse_string_to_list
+
+
+class TTSProvider(TTSProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__(config, delete_audio_file)
+        self.group_id = config.get("group_id")
+        self.api_key = config.get("api_key")
+        self.model = config.get("model")
+        
+        # 初始化语音设置
+        default_voice_setting = {
+            "voice_id": "female-shaonv",
+            "speed": 1,
+            "vol": 1,
+            "pitch": 0,
+            "emotion": "happy",
+        }
+        default_pronunciation_dict = {"tone": ["处理/(chu3)(li3)", "危险/dangerous"]}
+        default_audio_setting = {
+            "sample_rate": 32000,
+            "bitrate": 128000,
+            "format": "mp3",
+            "channel": 1,
+        }
+        
+        # 合并配置
+        self.voice_setting = {
+            **default_voice_setting,
+            **config.get("voice_setting", {}),
+        }
+        self.pronunciation_dict = {
+            **default_pronunciation_dict,
+            **config.get("pronunciation_dict", {}),
+        }
+        self.audio_setting = {
+            **default_audio_setting, 
+            **config.get("audio_setting", {})
+        }
+        self.timber_weights = parse_string_to_list(config.get("timber_weights"))
+        
+        # 设置语音ID
+        if config.get("private_voice"):
+            self.voice_setting["voice_id"] = config.get("private_voice")
+        elif config.get("voice_id"):
+            self.voice_setting["voice_id"] = config.get("voice_id")
+        
+        # WebSocket配置
+        self.ws_url = "wss://api.minimaxi.com/ws/v1/t2a_v2"
+        self.headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "GroupId": self.group_id
+        }
+        self.audio_file_type = self.audio_setting.get("format", "mp3")
+
+    def generate_filename(self, extension=".mp3"):
+        """生成唯一的音频文件名"""
+        return os.path.join(
+            self.output_file,
+            f"tts-{__name__}{datetime.now().date()}@{uuid.uuid4().hex}{extension}",
+        )
+
+    async def _establish_connection(self):
+        """建立WebSocket连接"""
+        ssl_context = ssl.create_default_context()
+        ssl_context.check_hostname = False
+        ssl_context.verify_mode = ssl.CERT_NONE
+
+        try:
+            ws = await websockets.connect(
+                self.ws_url,
+                additional_headers=self.headers,
+                ssl=ssl_context
+            )
+            connected = json.loads(await ws.recv())
+            if connected.get("event") == "connected_success":
+                print("连接成功")
+                return ws
+            return None
+        except Exception as e:
+            print(f"连接失败: {e}")
+            return None
+
+    async def _start_task(self, websocket):
+        """发送任务开始请求"""
+        start_msg = {
+            "event": "task_start",
+            "model": self.model,
+            "voice_setting": self.voice_setting,
+            "pronunciation_dict": self.pronunciation_dict,
+            "audio_setting": self.audio_setting
+        }
+        
+        if self.timber_weights and len(self.timber_weights) > 0:
+            start_msg["timber_weights"] = self.timber_weights
+            start_msg["voice_setting"]["voice_id"] = ""
+
+        await websocket.send(json.dumps(start_msg))
+        response = json.loads(await websocket.recv())
+        return response.get("event") == "task_started"
+
+    async def _continue_task(self, websocket, text):
+        """发送继续请求并收集音频数据"""
+        await websocket.send(json.dumps({
+            "event": "task_continue",
+            "text": text
+        }))
+
+        audio_chunks = []
+        while True:
+            response = json.loads(await websocket.recv())
+            if "data" in response and "audio" in response["data"]:
+                audio_chunks.append(response["data"]["audio"])
+            if response.get("is_final"):
+                break
+        return "".join(audio_chunks)
+
+    async def _close_connection(self, websocket):
+        """关闭连接"""
+        if websocket:
+            await websocket.send(json.dumps({"event": "task_finish"}))
+            await websocket.close()
+            print("连接已关闭")
+
+    async def text_to_speak(self, text, output_file=None):
+        """主方法：文本转语音"""
+        ws = await self._establish_connection()
+        if not ws:
+            raise Exception("无法建立WebSocket连接")
+
+        try:
+            if not await self._start_task(ws):
+                raise Exception("任务启动失败")
+
+            hex_audio = await self._continue_task(ws, text)
+            audio_bytes = bytes.fromhex(hex_audio)
+
+            # 保存到文件或返回二进制数据
+            if output_file:
+                with open(output_file, "wb") as f:
+                    f.write(audio_bytes)
+                print(f"音频已保存为{output_file}")
+                return output_file
+            else:
+                # 返回音频二进制数据（不播放）
+                return audio_bytes
+
+        finally:
+            await self._close_connection(ws)
+
+
+async def main():
+    """测试用主函数"""
+    # 示例配置
+    config = {
+        "group_id": "YOUR_GROUP_ID",  # 替换为实际的group_id
+        "api_key": "YOUR_API_KEY",    # 替换为实际的api_key
+        "model": "your-model",        # 替换为实际的模型名称
+        "voice_id": "male-qn-qingse",
+        "voice_setting": {
+            "speed": 1.2,
+            "emotion": "happy"
+        }
+    }
+    
+    tts = TTSProvider(config, delete_audio_file=True)
+    output_file = tts.generate_filename()
+    await tts.text_to_speak("这是一个测试文本，用于验证流式语音合成功能", output_file)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

main/xiaozhi-server/core/providers/vad/silero.py

@@ -23,22 +23,24 @@ class VADProvider(VADProviderBase):
 
         # 处理空字符串的情况
         threshold = config.get("threshold", "0.5")
+        threshold_low = config.get("threshold_low", "0.2")
         min_silence_duration_ms = config.get("min_silence_duration_ms", "1000")
 
         self.vad_threshold = float(threshold) if threshold else 0.5
+        self.vad_threshold_low = float(threshold_low) if threshold_low else 0.2
+
         self.silence_threshold_ms = (
             int(min_silence_duration_ms) if min_silence_duration_ms else 1000
         )
 
+        # 至少要多少帧才算有语音
+        self.frame_window_threshold = 3
+
     def is_vad(self, conn, opus_packet):
         try:
             pcm_frame = self.decoder.decode(opus_packet, 960)
             conn.client_audio_buffer.extend(pcm_frame)  # 将新数据加入缓冲区
 
-            # 确保帧计数器存在
-            if not hasattr(conn, "client_voice_frame_count"):
-                conn.client_voice_frame_count = 0
-
             # 处理缓冲区中的完整帧（每次处理512采样点）
             client_have_voice = False
             while len(conn.client_audio_buffer) >= 512 * 2:
@@ -54,15 +56,21 @@ class VADProvider(VADProviderBase):
                 # 检测语音活动
                 with torch.no_grad():
                     speech_prob = self.model(audio_tensor, 16000).item()
-                is_voice = speech_prob >= self.vad_threshold
 
-                if is_voice:
+                # 双阈值判断
-                    conn.client_voice_frame_count += 1
+                if speech_prob >= self.vad_threshold:
+                    is_voice = True
+                elif speech_prob <= self.vad_threshold_low:
+                    is_voice = False
                 else:
-                    conn.client_voice_frame_count = 0
+                    is_voice = conn.last_is_voice
 
-                # 只有连续4帧检测到语音才认为有语音
+                # 声音没低于最低值则延续前一个状态，判断为有声音
-                client_have_voice = conn.client_voice_frame_count >= 4
+                conn.last_is_voice = is_voice
+
+                # 更新滑动窗口
+                conn.client_voice_window.append(is_voice)
+                client_have_voice = (conn.client_voice_window.count(True) >= self.frame_window_threshold)
 
                 # 如果之前有声音，但本次没有声音，且与上次有声音的时间差已经超过了静默阈值，则认为已经说完一句话
                 if conn.client_have_voice and not client_have_voice:

main/xiaozhi-server/core/providers/vllm/openai.py

@@ -40,6 +40,7 @@ class VLLMProvider(VLLMProviderBase):
         self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
 
     def response(self, question, base64_image):
+        question = question + "(请使用中文回复)"
         try:
             messages = [
                 {

main/xiaozhi-server/core/utils/cache/config.py

@@ -0,0 +1,58 @@
+"""
+缓存配置管理
+"""
+
+from enum import Enum
+from typing import Dict, Any, Optional
+from dataclasses import dataclass
+from .strategies import CacheStrategy
+
+
+class CacheType(Enum):
+    """缓存类型枚举"""
+
+    LOCATION = "location"
+    WEATHER = "weather"
+    LUNAR = "lunar"
+    INTENT = "intent"
+    IP_INFO = "ip_info"
+    CONFIG = "config"
+    DEVICE_PROMPT = "device_prompt"
+
+
+@dataclass
+class CacheConfig:
+    """缓存配置类"""
+
+    strategy: CacheStrategy = CacheStrategy.TTL
+    ttl: Optional[float] = 300  # 默认5分钟
+    max_size: Optional[int] = 1000  # 默认最大1000条
+    cleanup_interval: float = 60  # 清理间隔（秒）
+
+    @classmethod
+    def for_type(cls, cache_type: CacheType) -> "CacheConfig":
+        """根据缓存类型返回预设配置"""
+        configs = {
+            CacheType.LOCATION: cls(
+                strategy=CacheStrategy.TTL, ttl=None, max_size=1000  # 手动失效
+            ),
+            CacheType.IP_INFO: cls(
+                strategy=CacheStrategy.TTL, ttl=86400, max_size=1000  # 24小时
+            ),
+            CacheType.WEATHER: cls(
+                strategy=CacheStrategy.TTL, ttl=28800, max_size=1000  # 8小时
+            ),
+            CacheType.LUNAR: cls(
+                strategy=CacheStrategy.TTL, ttl=2592000, max_size=365  # 30天过期
+            ),
+            CacheType.INTENT: cls(
+                strategy=CacheStrategy.TTL_LRU, ttl=600, max_size=1000  # 10分钟
+            ),
+            CacheType.CONFIG: cls(
+                strategy=CacheStrategy.FIXED_SIZE, ttl=None, max_size=20  # 手动失效
+            ),
+            CacheType.DEVICE_PROMPT: cls(
+                strategy=CacheStrategy.TTL, ttl=None, max_size=1000  # 手动失效
+            ),
+        }
+        return configs.get(cache_type, cls())

main/xiaozhi-server/core/utils/cache/manager.py

@@ -0,0 +1,216 @@
+"""
+全局缓存管理器
+"""
+
+import time
+import threading
+from typing import Any, Optional, Dict
+from collections import OrderedDict
+from .strategies import CacheStrategy, CacheEntry
+from .config import CacheConfig, CacheType
+
+
+class GlobalCacheManager:
+    """全局缓存管理器"""
+
+    def __init__(self):
+        self._logger = None
+        self._caches: Dict[str, Dict[str, CacheEntry]] = {}
+        self._configs: Dict[str, CacheConfig] = {}
+        self._locks: Dict[str, threading.RLock] = {}
+        self._global_lock = threading.RLock()
+        self._last_cleanup = time.time()
+        self._stats = {"hits": 0, "misses": 0, "evictions": 0, "cleanups": 0}
+
+    @property
+    def logger(self):
+        """延迟初始化 logger 以避免循环导入"""
+        if self._logger is None:
+            from config.logger import setup_logging
+
+            self._logger = setup_logging()
+        return self._logger
+
+    def _get_cache_name(self, cache_type: CacheType, namespace: str = "") -> str:
+        """生成缓存名称"""
+        if namespace:
+            return f"{cache_type.value}:{namespace}"
+        return cache_type.value
+
+    def _get_or_create_cache(
+        self, cache_name: str, config: CacheConfig
+    ) -> Dict[str, CacheEntry]:
+        """获取或创建缓存空间"""
+        with self._global_lock:
+            if cache_name not in self._caches:
+                self._caches[cache_name] = (
+                    OrderedDict()
+                    if config.strategy in [CacheStrategy.LRU, CacheStrategy.TTL_LRU]
+                    else {}
+                )
+                self._configs[cache_name] = config
+                self._locks[cache_name] = threading.RLock()
+            return self._caches[cache_name]
+
+    def set(
+        self,
+        cache_type: CacheType,
+        key: str,
+        value: Any,
+        ttl: Optional[float] = None,
+        namespace: str = "",
+    ) -> None:
+        """设置缓存值"""
+        cache_name = self._get_cache_name(cache_type, namespace)
+        config = self._configs.get(cache_name) or CacheConfig.for_type(cache_type)
+        cache = self._get_or_create_cache(cache_name, config)
+
+        # 使用配置的TTL或传入的TTL
+        effective_ttl = ttl if ttl is not None else config.ttl
+
+        with self._locks[cache_name]:
+            # 创建缓存条目
+            entry = CacheEntry(value=value, timestamp=time.time(), ttl=effective_ttl)
+
+            # 处理不同策略
+            if config.strategy in [CacheStrategy.LRU, CacheStrategy.TTL_LRU]:
+                # LRU策略：如果已存在则移动到末尾
+                if key in cache:
+                    del cache[key]
+                cache[key] = entry
+
+                # 检查大小限制
+                if config.max_size and len(cache) > config.max_size:
+                    # 移除最旧的条目
+                    oldest_key = next(iter(cache))
+                    del cache[oldest_key]
+                    self._stats["evictions"] += 1
+
+            else:
+                cache[key] = entry
+
+                # 检查大小限制
+                if config.max_size and len(cache) > config.max_size:
+                    # 简单策略：随机移除一个条目
+                    victim_key = next(iter(cache))
+                    del cache[victim_key]
+                    self._stats["evictions"] += 1
+
+        # 定期清理过期条目
+        self._maybe_cleanup(cache_name)
+
+    def get(
+        self, cache_type: CacheType, key: str, namespace: str = ""
+    ) -> Optional[Any]:
+        """获取缓存值"""
+        cache_name = self._get_cache_name(cache_type, namespace)
+
+        if cache_name not in self._caches:
+            self._stats["misses"] += 1
+            return None
+
+        cache = self._caches[cache_name]
+        config = self._configs[cache_name]
+
+        with self._locks[cache_name]:
+            if key not in cache:
+                self._stats["misses"] += 1
+                return None
+
+            entry = cache[key]
+
+            # 检查过期
+            if entry.is_expired():
+                del cache[key]
+                self._stats["misses"] += 1
+                return None
+
+            # 更新访问信息
+            entry.touch()
+
+            # LRU策略：移动到末尾
+            if config.strategy in [CacheStrategy.LRU, CacheStrategy.TTL_LRU]:
+                del cache[key]
+                cache[key] = entry
+
+            self._stats["hits"] += 1
+            return entry.value
+
+    def delete(self, cache_type: CacheType, key: str, namespace: str = "") -> bool:
+        """删除缓存条目"""
+        cache_name = self._get_cache_name(cache_type, namespace)
+
+        if cache_name not in self._caches:
+            return False
+
+        cache = self._caches[cache_name]
+
+        with self._locks[cache_name]:
+            if key in cache:
+                del cache[key]
+                return True
+            return False
+
+    def clear(self, cache_type: CacheType, namespace: str = "") -> None:
+        """清空指定缓存"""
+        cache_name = self._get_cache_name(cache_type, namespace)
+
+        if cache_name not in self._caches:
+            return
+
+        with self._locks[cache_name]:
+            self._caches[cache_name].clear()
+
+    def invalidate_pattern(
+        self, cache_type: CacheType, pattern: str, namespace: str = ""
+    ) -> int:
+        """按模式失效缓存条目"""
+        cache_name = self._get_cache_name(cache_type, namespace)
+
+        if cache_name not in self._caches:
+            return 0
+
+        cache = self._caches[cache_name]
+        deleted_count = 0
+
+        with self._locks[cache_name]:
+            keys_to_delete = [key for key in cache.keys() if pattern in key]
+            for key in keys_to_delete:
+                del cache[key]
+                deleted_count += 1
+
+        return deleted_count
+
+    def _cleanup_expired(self, cache_name: str) -> int:
+        """清理过期条目"""
+        if cache_name not in self._caches:
+            return 0
+
+        cache = self._caches[cache_name]
+        deleted_count = 0
+
+        with self._locks[cache_name]:
+            expired_keys = [key for key, entry in cache.items() if entry.is_expired()]
+            for key in expired_keys:
+                del cache[key]
+                deleted_count += 1
+
+        return deleted_count
+
+    def _maybe_cleanup(self, cache_name: str):
+        """定期清理检查"""
+        config = self._configs.get(cache_name)
+        if not config:
+            return
+
+        now = time.time()
+        if now - self._last_cleanup > config.cleanup_interval:
+            self._last_cleanup = now
+            deleted = self._cleanup_expired(cache_name)
+            if deleted > 0:
+                self._stats["cleanups"] += 1
+                self.logger.debug(f"清理缓存 {cache_name}: 删除 {deleted} 个过期条目")
+
+
+# 创建全局缓存管理器实例
+cache_manager = GlobalCacheManager()

main/xiaozhi-server/core/utils/cache/strategies.py

@@ -0,0 +1,43 @@
+"""
+缓存策略和数据结构定义
+"""
+
+import time
+from enum import Enum
+from typing import Any, Optional
+from dataclasses import dataclass
+
+
+class CacheStrategy(Enum):
+    """缓存策略枚举"""
+
+    TTL = "ttl"  # 基于时间过期
+    LRU = "lru"  # 最近最少使用
+    FIXED_SIZE = "fixed_size"  # 固定大小
+    TTL_LRU = "ttl_lru"  # TTL + LRU混合策略
+
+
+@dataclass
+class CacheEntry:
+    """缓存条目数据结构"""
+
+    value: Any
+    timestamp: float
+    ttl: Optional[float] = None  # 生存时间（秒）
+    access_count: int = 0
+    last_access: float = None
+
+    def __post_init__(self):
+        if self.last_access is None:
+            self.last_access = self.timestamp
+
+    def is_expired(self) -> bool:
+        """检查是否过期"""
+        if self.ttl is None:
+            return False
+        return time.time() - self.timestamp > self.ttl
+
+    def touch(self):
+        """更新访问时间和计数"""
+        self.last_access = time.time()
+        self.access_count += 1

main/xiaozhi-server/core/utils/dialogue.py

@@ -1,4 +1,5 @@
 import uuid
+import re
 from typing import List, Dict
 from datetime import datetime
 
@@ -45,10 +46,9 @@ class Dialogue:
             dialogue.append({"role": m.role, "content": m.content})
 
     def get_llm_dialogue(self) -> List[Dict[str, str]]:
-        dialogue = []
+        # 直接调用get_llm_dialogue_with_memory，传入None作为memory_str
-        for m in self.dialogue:
+        # 这样确保说话人功能在所有调用路径下都生效
-            self.getMessages(m, dialogue)
+        return self.get_llm_dialogue_with_memory(None, None)
-        return dialogue
 
     def update_system_message(self, new_content: str):
         """更新或添加系统消息"""
@@ -60,12 +60,9 @@ class Dialogue:
             self.put(Message(role="system", content=new_content))
 
     def get_llm_dialogue_with_memory(
-        self, memory_str: str = None
+        self, memory_str: str = None, voiceprint_config: dict = None
     ) -> List[Dict[str, str]]:
-        if memory_str is None or len(memory_str) == 0:
+        # 构建对话
-            return self.get_llm_dialogue()
-
-        # 构建带记忆的对话
         dialogue = []
 
         # 添加系统提示和记忆
@@ -74,10 +71,43 @@ class Dialogue:
         )
 
         if system_message:
-            enhanced_system_prompt = (
+            # 基础系统提示
-                f"{system_message.content}\n\n"
+            enhanced_system_prompt = system_message.content
-                f"以下是用户的历史记忆：\n```\n{memory_str}\n```"
+            # 替换时间占位符
+            enhanced_system_prompt = enhanced_system_prompt.replace(
+                "{{current_time}}", datetime.now().strftime("%H:%M")
             )
+
+            # 添加说话人个性化描述
+            try:
+                speakers = voiceprint_config.get("speakers", [])
+                if speakers:
+                    enhanced_system_prompt += "\n\n<speakers_info>"
+                    for speaker_str in speakers:
+                        try:
+                            parts = speaker_str.split(",", 2)
+                            if len(parts) >= 2:
+                                name = parts[1].strip()
+                                # 如果描述为空，则为""
+                                description = (
+                                    parts[2].strip() if len(parts) >= 3 else ""
+                                )
+                                enhanced_system_prompt += f"\n- {name}：{description}"
+                        except:
+                            pass
+                    enhanced_system_prompt += "\n\n</speakers_info>"
+            except:
+                # 配置读取失败时忽略错误，不影响其他功能
+                pass
+
+            # 使用正则表达式匹配 <memory> 标签，不管中间有什么内容
+            if memory_str is not None:
+                enhanced_system_prompt = re.sub(
+                    r"<memory>.*?</memory>",
+                    f"<memory>\n{memory_str}\n</memory>",
+                    enhanced_system_prompt,
+                    flags=re.DOTALL,
+                )
             dialogue.append({"role": "system", "content": enhanced_system_prompt})
 
         # 添加用户和助手的对话

main/xiaozhi-server/core/utils/modules_initialize.py

@@ -125,4 +125,27 @@ def initialize_asr(config):
         config["ASR"][select_asr_module],
         str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
     )
+    logger.bind(tag=TAG).info("ASR模块初始化完成")
     return new_asr
+
+
+def initialize_voiceprint(asr_instance, config):
+    """初始化声纹识别功能"""
+    voiceprint_config = config.get("voiceprint")
+    if not voiceprint_config:
+        return False  
+
+    # 应用配置
+    if not voiceprint_config.get("url") or not voiceprint_config.get("speakers"):
+        logger.bind(tag=TAG).warning("声纹识别配置不完整")
+        return False
+        
+    try:
+        asr_instance.init_voiceprint(voiceprint_config)
+        logger.bind(tag=TAG).info("ASR模块声纹识别功能已动态启用")
+        logger.bind(tag=TAG).info(f"配置说话人数量: {len(voiceprint_config['speakers'])}")
+        return True
+    except Exception as e:
+        logger.bind(tag=TAG).error(f"动态初始化声纹识别功能失败: {str(e)}")
+        return False
+

main/xiaozhi-server/core/utils/prompt_manager.py

@@ -0,0 +1,245 @@
+"""
+系统提示词管理器模块
+负责管理和更新系统提示词，包括快速初始化和异步增强功能
+"""
+
+import os
+import cnlunar
+from typing import Dict, Any
+from config.logger import setup_logging
+from jinja2 import Template
+
+TAG = __name__
+
+WEEKDAY_MAP = {
+    "Monday": "星期一",
+    "Tuesday": "星期二",
+    "Wednesday": "星期三",
+    "Thursday": "星期四",
+    "Friday": "星期五",
+    "Saturday": "星期六",
+    "Sunday": "星期日",
+}
+
+EMOJI_List = [
+    "😶",
+    "🙂",
+    "😆",
+    "😂",
+    "😔",
+    "😠",
+    "😭",
+    "😍",
+    "😳",
+    "😲",
+    "😱",
+    "🤔",
+    "😉",
+    "😎",
+    "😌",
+    "🤤",
+    "😘",
+    "😏",
+    "😴",
+    "😜",
+    "🙄",
+]
+
+
+class PromptManager:
+    """系统提示词管理器，负责管理和更新系统提示词"""
+
+    def __init__(self, config: Dict[str, Any], logger=None):
+        self.config = config
+        self.logger = logger or setup_logging()
+        self.base_prompt_template = None
+        self.last_update_time = 0
+
+        # 导入全局缓存管理器
+        from core.utils.cache.manager import cache_manager, CacheType
+
+        self.cache_manager = cache_manager
+        self.CacheType = CacheType
+
+        self._load_base_template()
+
+    def _load_base_template(self):
+        """加载基础提示词模板"""
+        try:
+            template_path = "agent-base-prompt.txt"
+            cache_key = f"prompt_template:{template_path}"
+
+            # 先从缓存获取
+            cached_template = self.cache_manager.get(self.CacheType.CONFIG, cache_key)
+            if cached_template is not None:
+                self.base_prompt_template = cached_template
+                self.logger.bind(tag=TAG).debug("从缓存加载基础提示词模板")
+                return
+
+            # 缓存未命中，从文件读取
+            if os.path.exists(template_path):
+                with open(template_path, "r", encoding="utf-8") as f:
+                    template_content = f.read()
+
+                # 存入缓存（CONFIG类型默认不自动过期，需要手动失效）
+                self.cache_manager.set(
+                    self.CacheType.CONFIG, cache_key, template_content
+                )
+                self.base_prompt_template = template_content
+                self.logger.bind(tag=TAG).debug("成功加载基础提示词模板并缓存")
+            else:
+                self.logger.bind(tag=TAG).warning("未找到agent-base-prompt.txt文件")
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"加载提示词模板失败: {e}")
+
+    def get_quick_prompt(self, user_prompt: str, device_id: str = None) -> str:
+        """快速获取系统提示词（使用用户配置）"""
+        device_cache_key = f"device_prompt:{device_id}"
+        cached_device_prompt = self.cache_manager.get(
+            self.CacheType.DEVICE_PROMPT, device_cache_key
+        )
+        if cached_device_prompt is not None:
+            self.logger.bind(tag=TAG).debug(f"使用设备 {device_id} 的缓存提示词")
+            return cached_device_prompt
+        else:
+            self.logger.bind(tag=TAG).debug(
+                f"设备 {device_id} 无缓存提示词，使用传入的提示词"
+            )
+
+        # 使用传入的提示词并缓存（如果有设备ID）
+        if device_id:
+            device_cache_key = f"device_prompt:{device_id}"
+            self.cache_manager.set(self.CacheType.CONFIG, device_cache_key, user_prompt)
+            self.logger.bind(tag=TAG).debug(f"设备 {device_id} 的提示词已缓存")
+
+        self.logger.bind(tag=TAG).info(f"使用快速提示词: {user_prompt[:50]}...")
+        return user_prompt
+
+    def _get_current_time_info(self) -> tuple:
+        """获取当前时间信息"""
+        from datetime import datetime
+
+        now = datetime.now()
+        today_date = now.strftime("%Y-%m-%d")
+        today_weekday = WEEKDAY_MAP[now.strftime("%A")]
+        today_lunar = cnlunar.Lunar(now, godType="8char")
+        lunar_date = "%s年%s%s\n" % (
+            today_lunar.lunarYearCn,
+            today_lunar.lunarMonthCn[:-1],
+            today_lunar.lunarDayCn,
+        )
+
+        return today_date, today_weekday, lunar_date
+
+    def _get_location_info(self, client_ip: str) -> str:
+        """获取位置信息"""
+        try:
+            # 先从缓存获取
+            cached_location = self.cache_manager.get(self.CacheType.LOCATION, client_ip)
+            if cached_location is not None:
+                return cached_location
+
+            # 缓存未命中，调用API获取
+            from core.utils.util import get_ip_info
+
+            ip_info = get_ip_info(client_ip, self.logger)
+            city = ip_info.get("city", "未知位置")
+            location = f"{city}"
+
+            # 存入缓存
+            self.cache_manager.set(self.CacheType.LOCATION, client_ip, location)
+            return location
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"获取位置信息失败: {e}")
+            return "未知位置"
+
+    def _get_weather_info(self, conn, location: str) -> str:
+        """获取天气信息"""
+        try:
+            # 先从缓存获取
+            cached_weather = self.cache_manager.get(self.CacheType.WEATHER, location)
+            if cached_weather is not None:
+                return cached_weather
+
+            # 缓存未命中，调用get_weather函数获取
+            from plugins_func.functions.get_weather import get_weather
+            from plugins_func.register import ActionResponse
+
+            # 调用get_weather函数
+            result = get_weather(conn, location=location, lang="zh_CN")
+            if isinstance(result, ActionResponse):
+                weather_report = result.result
+                self.cache_manager.set(self.CacheType.WEATHER, location, weather_report)
+                return weather_report
+            return "天气信息获取失败"
+
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"获取天气信息失败: {e}")
+            return "天气信息获取失败"
+
+    def update_context_info(self, conn, client_ip: str):
+        """同步更新上下文信息"""
+        try:
+            # 获取位置信息（使用全局缓存）
+            local_address = self._get_location_info(client_ip)
+            # 获取天气信息（使用全局缓存）
+            self._get_weather_info(conn, local_address)
+            self.logger.bind(tag=TAG).info(f"上下文信息更新完成")
+
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"更新上下文信息失败: {e}")
+
+    def build_enhanced_prompt(
+        self, user_prompt: str, device_id: str, client_ip: str = None
+    ) -> str:
+        """构建增强的系统提示词"""
+        if not self.base_prompt_template:
+            return user_prompt
+
+        try:
+            # 获取最新的时间信息（不缓存）
+            today_date, today_weekday, lunar_date = (
+                self._get_current_time_info()
+            )
+
+            # 获取缓存的上下文信息
+            local_address = ""
+            weather_info = ""
+
+            if client_ip:
+                # 获取位置信息（从全局缓存）
+                local_address = (
+                    self.cache_manager.get(self.CacheType.LOCATION, client_ip) or ""
+                )
+
+                # 获取天气信息（从全局缓存）
+                if local_address:
+                    weather_info = (
+                        self.cache_manager.get(self.CacheType.WEATHER, local_address)
+                        or ""
+                    )
+
+            # 替换模板变量
+            template = Template(self.base_prompt_template)
+            enhanced_prompt = template.render(
+                base_prompt=user_prompt,
+                current_time="{{current_time}}",
+                today_date=today_date,
+                today_weekday=today_weekday,
+                lunar_date=lunar_date,
+                local_address=local_address,
+                weather_info=weather_info,
+                emojiList=EMOJI_List,
+            )
+            device_cache_key = f"device_prompt:{device_id}"
+            self.cache_manager.set(
+                self.CacheType.DEVICE_PROMPT, device_cache_key, enhanced_prompt
+            )
+            self.logger.bind(tag=TAG).info(
+                f"构建增强提示词成功，长度: {len(enhanced_prompt)}"
+            )
+            return enhanced_prompt
+
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"构建增强提示词失败: {e}")
+            return user_prompt

main/xiaozhi-server/core/utils/textUtils.py

@@ -1,3 +1,40 @@
+import json
+
+TAG = __name__
+EMOJI_MAP = {
+    "😂": "laughing",
+    "😭": "crying",
+    "😠": "angry",
+    "😔": "sad",
+    "😍": "loving",
+    "😲": "surprised",
+    "😱": "shocked",
+    "🤔": "thinking",
+    "😌": "relaxed",
+    "😴": "sleepy",
+    "😜": "silly",
+    "🙄": "confused",
+    "😶": "neutral",
+    "🙂": "happy",
+    "😆": "laughing",
+    "😳": "embarrassed",
+    "😉": "winking",
+    "😎": "cool",
+    "🤤": "delicious",
+    "😘": "kissy",
+    "😏": "confident",
+}
+EMOJI_RANGES = [
+    (0x1F600, 0x1F64F),
+    (0x1F300, 0x1F5FF),
+    (0x1F680, 0x1F6FF),
+    (0x1F900, 0x1F9FF),
+    (0x1FA70, 0x1FAFF),
+    (0x2600, 0x26FF),
+    (0x2700, 0x27BF),
+]
+
+
 def get_string_no_punctuation_or_emoji(s):
     """去除字符串首尾的空格、标点符号和表情符号"""
     chars = list(s)
@@ -22,6 +59,11 @@ def is_punctuation_or_emoji(char):
         ".",  # 中文句号 + 英文句号
         "！",
         "!",  # 中文感叹号 + 英文感叹号
+        "“",
+        "”",
+        '"',  # 中文双引号 + 英文引号
+        "：",
+        ":",  # 中文冒号 + 英文冒号
         "-",
         "－",  # 英文连字符 + 中文全角横线
         "、",  # 中文顿号
@@ -32,15 +74,40 @@ def is_punctuation_or_emoji(char):
     }
     if char.isspace() or char in punctuation_set:
         return True
-    # 检查表情符号（保留原有逻辑）
+    return is_emoji(char)
+
+
+async def get_emotion(conn, text):
+    """获取文本内的情绪消息"""
+    emoji = "🙂"
+    emotion = "happy"
+    for char in text:
+        if char in EMOJI_MAP:
+            emoji = char
+            emotion = EMOJI_MAP[char]
+            break
+    try:
+        await conn.websocket.send(
+            json.dumps(
+                {
+                    "type": "llm",
+                    "text": emoji,
+                    "emotion": emotion,
+                    "session_id": conn.session_id,
+                }
+            )
+        )
+    except Exception as e:
+        conn.logger.bind(tag=TAG).warning(f"发送情绪表情失败，错误:{e}")
+    return
+
+
+def is_emoji(char):
+    """检查字符是否为emoji表情"""
     code_point = ord(char)
-    emoji_ranges = [
+    return any(start <= code_point <= end for start, end in EMOJI_RANGES)
-        (0x1F600, 0x1F64F),
+
-        (0x1F300, 0x1F5FF),
+
-        (0x1F680, 0x1F6FF),
+def check_emoji(text):
-        (0x1F900, 0x1F9FF),
+    """去除文本中的所有emoji表情"""
-        (0x1FA70, 0x1FAFF),
+    return ''.join(char for char in text if not is_emoji(char) and char != "\n")
-        (0x2600, 0x26FF),
-        (0x2700, 0x27BF),
-    ]
-    return any(start <= code_point <= end for start, end in emoji_ranges)

main/xiaozhi-server/core/utils/util.py

@@ -96,11 +96,23 @@ def is_private_ip(ip_addr):
 
 def get_ip_info(ip_addr, logger):
     try:
+        # 导入全局缓存管理器
+        from core.utils.cache.manager import cache_manager, CacheType
+
+        # 先从缓存获取
+        cached_ip_info = cache_manager.get(CacheType.IP_INFO, ip_addr)
+        if cached_ip_info is not None:
+            return cached_ip_info
+
+        # 缓存未命中，调用API
         if is_private_ip(ip_addr):
             ip_addr = ""
         url = f"https://whois.pconline.com.cn/ipJson.jsp?json=true&ip={ip_addr}"
         resp = requests.get(url).json()
         ip_info = {"city": resp.get("city")}
+
+        # 存入缓存
+        cache_manager.set(CacheType.IP_INFO, ip_addr, ip_info)
         return ip_info
     except Exception as e:
         logger.bind(tag=TAG).error(f"Error getting client ip info: {e}")
@@ -113,51 +125,6 @@ def write_json_file(file_path, data):
         json.dump(data, file, ensure_ascii=False, indent=4)
 
 
-def is_punctuation_or_emoji(char):
-    """检查字符是否为空格、指定标点或表情符号"""
-    # 定义需要去除的中英文标点（包括全角/半角）
-    punctuation_set = {
-        "，",
-        ",",  # 中文逗号 + 英文逗号
-        "-",
-        "－",  # 英文连字符 + 中文全角横线
-        "、",  # 中文顿号
-        "“",
-        "”",
-        '"',  # 中文双引号 + 英文引号
-        "：",
-        ":",  # 中文冒号 + 英文冒号
-    }
-    if char.isspace() or char in punctuation_set:
-        return True
-    # 检查表情符号（保留原有逻辑）
-    code_point = ord(char)
-    emoji_ranges = [
-        (0x1F600, 0x1F64F),
-        (0x1F300, 0x1F5FF),
-        (0x1F680, 0x1F6FF),
-        (0x1F900, 0x1F9FF),
-        (0x1FA70, 0x1FAFF),
-        (0x2600, 0x26FF),
-        (0x2700, 0x27BF),
-    ]
-    return any(start <= code_point <= end for start, end in emoji_ranges)
-
-
-def get_string_no_punctuation_or_emoji(s):
-    """去除字符串首尾的空格、标点符号和表情符号"""
-    chars = list(s)
-    # 处理开头的字符
-    start = 0
-    while start < len(chars) and is_punctuation_or_emoji(chars[start]):
-        start += 1
-    # 处理结尾的字符
-    end = len(chars) - 1
-    while end >= start and is_punctuation_or_emoji(chars[end]):
-        end -= 1
-    return "".join(chars[start : end + 1])
-
-
 def remove_punctuation_and_length(text):
     # 全角符号和半角符号的Unicode范围
     full_width_punctuations = (
@@ -244,515 +211,6 @@ def extract_json_from_string(input_string):
     return None
 
 
-def analyze_emotion(text):
-    """
-    分析文本情感并返回对应的emoji名称（支持中英文）
-    """
-    if not text or not isinstance(text, str):
-        return "neutral"
-
-    original_text = text
-    text = text.lower().strip()
-
-    # 检查是否包含现有emoji
-    for emotion, emoji in emoji_map.items():
-        if emoji in original_text:
-            return emotion
-
-    # 标点符号分析
-    has_exclamation = "!" in original_text or "！" in original_text
-    has_question = "?" in original_text or "？" in original_text
-    has_ellipsis = "..." in original_text or "…" in original_text
-
-    # 定义情感关键词映射（中英文扩展版）
-    emotion_keywords = {
-        "happy": [
-            "开心",
-            "高兴",
-            "快乐",
-            "愉快",
-            "幸福",
-            "满意",
-            "棒",
-            "好",
-            "不错",
-            "完美",
-            "棒极了",
-            "太好了",
-            "好呀",
-            "好的",
-            "happy",
-            "joy",
-            "great",
-            "good",
-            "nice",
-            "awesome",
-            "fantastic",
-            "wonderful",
-        ],
-        "laughing": [
-            "哈哈",
-            "哈哈哈",
-            "呵呵",
-            "嘿嘿",
-            "嘻嘻",
-            "笑死",
-            "太好笑了",
-            "笑死我了",
-            "lol",
-            "lmao",
-            "haha",
-            "hahaha",
-            "hehe",
-            "rofl",
-            "funny",
-            "laugh",
-        ],
-        "funny": [
-            "搞笑",
-            "滑稽",
-            "逗",
-            "幽默",
-            "笑点",
-            "段子",
-            "笑话",
-            "太逗了",
-            "hilarious",
-            "joke",
-            "comedy",
-        ],
-        "sad": [
-            "伤心",
-            "难过",
-            "悲哀",
-            "悲伤",
-            "忧郁",
-            "郁闷",
-            "沮丧",
-            "失望",
-            "想哭",
-            "难受",
-            "不开心",
-            "唉",
-            "呜呜",
-            "sad",
-            "upset",
-            "unhappy",
-            "depressed",
-            "sorrow",
-            "gloomy",
-        ],
-        "angry": [
-            "生气",
-            "愤怒",
-            "气死",
-            "讨厌",
-            "烦人",
-            "可恶",
-            "烦死了",
-            "恼火",
-            "暴躁",
-            "火大",
-            "愤怒",
-            "气炸了",
-            "angry",
-            "mad",
-            "annoyed",
-            "furious",
-            "pissed",
-            "hate",
-        ],
-        "crying": [
-            "哭泣",
-            "泪流",
-            "大哭",
-            "伤心欲绝",
-            "泪目",
-            "流泪",
-            "哭死",
-            "哭晕",
-            "想哭",
-            "泪崩",
-            "cry",
-            "crying",
-            "tears",
-            "sob",
-            "weep",
-        ],
-        "loving": [
-            "爱你",
-            "喜欢",
-            "爱",
-            "亲爱的",
-            "宝贝",
-            "么么哒",
-            "抱抱",
-            "想你",
-            "思念",
-            "最爱",
-            "亲亲",
-            "喜欢你",
-            "love",
-            "like",
-            "adore",
-            "darling",
-            "sweetie",
-            "honey",
-            "miss you",
-            "heart",
-        ],
-        "embarrassed": [
-            "尴尬",
-            "不好意思",
-            "害羞",
-            "脸红",
-            "难为情",
-            "社死",
-            "丢脸",
-            "出丑",
-            "embarrassed",
-            "awkward",
-            "shy",
-            "blush",
-        ],
-        "surprised": [
-            "惊讶",
-            "吃惊",
-            "天啊",
-            "哇塞",
-            "哇",
-            "居然",
-            "竟然",
-            "没想到",
-            "出乎意料",
-            "surprise",
-            "wow",
-            "omg",
-            "oh my god",
-            "amazing",
-            "unbelievable",
-        ],
-        "shocked": [
-            "震惊",
-            "吓到",
-            "惊呆了",
-            "不敢相信",
-            "震撼",
-            "吓死",
-            "恐怖",
-            "害怕",
-            "吓人",
-            "shocked",
-            "shocking",
-            "scared",
-            "frightened",
-            "terrified",
-            "horror",
-        ],
-        "thinking": [
-            "思考",
-            "考虑",
-            "想一下",
-            "琢磨",
-            "沉思",
-            "冥想",
-            "想",
-            "思考中",
-            "在想",
-            "think",
-            "thinking",
-            "consider",
-            "ponder",
-            "meditate",
-        ],
-        "winking": [
-            "调皮",
-            "眨眼",
-            "你懂的",
-            "坏笑",
-            "邪恶",
-            "奸笑",
-            "使眼色",
-            "wink",
-            "teasing",
-            "naughty",
-            "mischievous",
-        ],
-        "cool": [
-            "酷",
-            "帅",
-            "厉害",
-            "棒极了",
-            "真棒",
-            "牛逼",
-            "强",
-            "优秀",
-            "杰出",
-            "出色",
-            "完美",
-            "cool",
-            "awesome",
-            "amazing",
-            "great",
-            "impressive",
-            "perfect",
-        ],
-        "relaxed": [
-            "放松",
-            "舒服",
-            "惬意",
-            "悠闲",
-            "轻松",
-            "舒适",
-            "安逸",
-            "自在",
-            "relax",
-            "relaxed",
-            "comfortable",
-            "cozy",
-            "chill",
-            "peaceful",
-        ],
-        "delicious": [
-            "好吃",
-            "美味",
-            "香",
-            "馋",
-            "可口",
-            "香甜",
-            "大餐",
-            "大快朵颐",
-            "流口水",
-            "垂涎",
-            "delicious",
-            "yummy",
-            "tasty",
-            "yum",
-            "appetizing",
-            "mouthwatering",
-        ],
-        "kissy": [
-            "亲亲",
-            "么么",
-            "吻",
-            "mua",
-            "muah",
-            "亲一下",
-            "飞吻",
-            "kiss",
-            "xoxo",
-            "hug",
-            "muah",
-            "smooch",
-        ],
-        "confident": [
-            "自信",
-            "肯定",
-            "确定",
-            "毫无疑问",
-            "当然",
-            "必须的",
-            "毫无疑问",
-            "确信",
-            "坚信",
-            "confident",
-            "sure",
-            "certain",
-            "definitely",
-            "positive",
-        ],
-        "sleepy": [
-            "困",
-            "睡觉",
-            "晚安",
-            "想睡",
-            "好累",
-            "疲惫",
-            "疲倦",
-            "困了",
-            "想休息",
-            "睡意",
-            "sleep",
-            "sleepy",
-            "tired",
-            "exhausted",
-            "bedtime",
-            "good night",
-        ],
-        "silly": [
-            "傻",
-            "笨",
-            "呆",
-            "憨",
-            "蠢",
-            "二",
-            "憨憨",
-            "傻乎乎",
-            "呆萌",
-            "silly",
-            "stupid",
-            "dumb",
-            "foolish",
-            "goofy",
-            "ridiculous",
-        ],
-        "confused": [
-            "疑惑",
-            "不明白",
-            "不懂",
-            "困惑",
-            "疑问",
-            "为什么",
-            "怎么回事",
-            "啥意思",
-            "不清楚",
-            "confused",
-            "puzzled",
-            "doubt",
-            "question",
-            "what",
-            "why",
-            "how",
-        ],
-    }
-
-    # 特殊句型判断（中英文）
-    # 赞美他人
-    if any(
-        phrase in text
-        for phrase in [
-            "你真",
-            "你好",
-            "您真",
-            "你真棒",
-            "你好厉害",
-            "你太强了",
-            "你真好",
-            "你真聪明",
-            "you are",
-            "you're",
-            "you look",
-            "you seem",
-            "so smart",
-            "so kind",
-        ]
-    ):
-        return "loving"
-    # 自我赞美
-    if any(
-        phrase in text
-        for phrase in [
-            "我真",
-            "我最",
-            "我太棒了",
-            "我厉害",
-            "我聪明",
-            "我优秀",
-            "i am",
-            "i'm",
-            "i feel",
-            "so good",
-            "so happy",
-        ]
-    ):
-        return "cool"
-    # 晚安/睡觉相关
-    if any(
-        phrase in text
-        for phrase in [
-            "睡觉",
-            "晚安",
-            "睡了",
-            "好梦",
-            "休息了",
-            "去睡了",
-            "sleep",
-            "good night",
-            "bedtime",
-            "go to bed",
-        ]
-    ):
-        return "sleepy"
-    # 疑问句
-    if has_question and not has_exclamation:
-        return "thinking"
-    # 强烈情感（感叹号）
-    if has_exclamation and not has_question:
-        # 检查是否是积极内容
-        positive_words = (
-            emotion_keywords["happy"]
-            + emotion_keywords["laughing"]
-            + emotion_keywords["cool"]
-        )
-        if any(word in text for word in positive_words):
-            return "laughing"
-        # 检查是否是消极内容
-        negative_words = (
-            emotion_keywords["angry"]
-            + emotion_keywords["sad"]
-            + emotion_keywords["crying"]
-        )
-        if any(word in text for word in negative_words):
-            return "angry"
-        return "surprised"
-    # 省略号（表示犹豫或思考）
-    if has_ellipsis:
-        return "thinking"
-
-    # 关键词匹配（带权重）
-    emotion_scores = {emotion: 0 for emotion in emoji_map.keys()}
-
-    # 给匹配到的关键词加分
-    for emotion, keywords in emotion_keywords.items():
-        for keyword in keywords:
-            if keyword in text:
-                emotion_scores[emotion] += 1
-
-    # 给长文本中的重复关键词额外加分
-    if len(text) > 20:  # 长文本
-        for emotion, keywords in emotion_keywords.items():
-            for keyword in keywords:
-                emotion_scores[emotion] += text.count(keyword) * 0.5
-
-    # 根据分数选择最可能的情感
-    max_score = max(emotion_scores.values())
-    if max_score == 0:
-        return "happy"  # 默认
-
-    # 可能有多个情感同分，根据上下文选择最合适的
-    top_emotions = [e for e, s in emotion_scores.items() if s == max_score]
-
-    # 如果多个情感同分，使用以下优先级
-    priority_order = [
-        "laughing",
-        "crying",
-        "angry",
-        "surprised",
-        "shocked",  # 强烈情感优先
-        "loving",
-        "happy",
-        "funny",
-        "cool",  # 积极情感
-        "sad",
-        "embarrassed",
-        "confused",  # 消极情感
-        "thinking",
-        "winking",
-        "relaxed",  # 中性情感
-        "delicious",
-        "kissy",
-        "confident",
-        "sleepy",
-        "silly",  # 特殊场景
-    ]
-
-    for emotion in priority_order:
-        if emotion in top_emotions:
-            return emotion
-
-    return top_emotions[0]  # 如果都不在优先级列表里，返回第一个
-
-
 def audio_to_data(audio_file_path, is_opus=True):
     # 获取文件后缀名
     file_type = os.path.splitext(audio_file_path)[1]
@@ -982,3 +440,28 @@ def sanitize_tool_name(name: str) -> str:
     """Sanitize tool names for OpenAI compatibility."""
     # 支持中文、英文字母、数字、下划线和连字符
     return re.sub(r"[^a-zA-Z0-9_\-\u4e00-\u9fff]", "_", name)
+
+
+def validate_mcp_endpoint(mcp_endpoint: str) -> bool:
+    """
+    校验MCP接入点格式
+
+    Args:
+        mcp_endpoint: MCP接入点字符串
+
+    Returns:
+        bool: 是否有效
+    """
+    # 1. 检查是否以ws开头
+    if not mcp_endpoint.startswith("ws"):
+        return False
+
+    # 2. 检查是否包含key、call字样
+    if "key" in mcp_endpoint.lower() or "call" in mcp_endpoint.lower():
+        return False
+
+    # 3. 检查是否包含/mcp/字样
+    if "/mcp/" not in mcp_endpoint:
+        return False
+
+    return True