Fix long audio bug (#158 )

* update:异步生成音频 * update:优化LLM断句 --------- Co-authored-by: hrz <1710360675@qq.com>
update:启动时检查配置文件 (#155 )
2026-07-23 23:53:55 +08:00 · 2025-03-01 01:54:55 +08:00 · 2025-02-28 19:17:36 +08:00 · 2025-02-26 16:02:59 +08:00 · 2025-02-26 15:10:12 +08:00 · 2025-02-26 01:33:05 +08:00
26 changed files with 866 additions and 769 deletions
@@ -1,4 +1,15 @@
-![图片](docs/images/banner.png)
+[![SVG Banners](https://svg-banners.vercel.app/api?type=origin&text1=你好😃，小智📟&text2=开源小智ESP-32后端服务&width=830&height=210)](https://github.com/xinnan-tech/xiaozhi-esp32-server)
+<p align="center">
+  <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors">
+    <img alt="GitHub Contributors" src="https://img.shields.io/github/contributors/xinnan-tech/xiaozhi-esp32-server" />
+  </a>
+  <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">
+    <img alt="Issues" src="https://img.shields.io/github/issues/xinnan-tech/xiaozhi-esp32-server?color=0088ff" />
+  </a>
+  <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/pulls">
+    <img alt="GitHub pull requests" src="https://img.shields.io/github/issues-pr/xinnan-tech/xiaozhi-esp32-server?color=0088ff" />
+  </a>
+</p>

 # 小智 ESP-32 后端服务(xiaozhi-esp32-server)

@@ -58,28 +69,13 @@
    </tr>
 </table>

-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
 ---

 ## 系统要求与部署前提 🖥️

 - **硬件**：一套兼容 `xiaozhi-esp32`
  的硬件设备（具体型号请参考 [此处](https://rcnv1t9vps13.feishu.cn/wiki/DdgIw4BUgivWDPkhMj1cGIYCnRf)）。
- **服务器**：至少 4 核 CPU、8G 内存的电脑或服务器。
+- **服务器**：至少 4 核 CPU、8G 内存的电脑。
 - **固件编译**：请将后端服务的接口地址更新至 `xiaozhi-esp32` 项目中，再重新编译固件并烧录到设备上。

 ---
@@ -178,17 +174,26 @@ server:

 ### 一、[部署文档](./docs/Deployment.md)

-本项目支持以下三种部署方式，您可根据实际需求选择：
+本项目支持以下三种部署方式，您可根据实际需求选择。
+
+本项目的文档主要是`文字版本`的教程，如果你想要`视频版本`
+的教程，您可以学习一下[这个大佬的手把手教程](https://www.bilibili.com/video/BV1gePuejEvT)。
+
+如果你能把`文字版本的教程`和`视频版本的教程`结合起来一起看，可以让你更快上手。

 1. [Docker 快速部署](./docs/Deployment.md)
-   适合快速体验，不需过多环境配置。缺点是，拉取镜像有点慢。
+
+适合快速体验的普通用户，不需过多环境配置。缺点是，拉取镜像有点慢。

 2. [借助 Docker 环境运行部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E5%80%9F%E5%8A%A9docker%E7%8E%AF%E5%A2%83%E8%BF%90%E8%A1%8C%E9%83%A8%E7%BD%B2)
-   适用于已安装 Docker 且希望对代码进行自定义修改的用户。
+
+适用于已安装 Docker 且希望对代码进行自定义修改的软件工程师。

 3. [本地源码运行](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%89%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C)
-   适合熟悉 Conda 环境或希望从零搭建运行环境的用户。  
-   对于对响应速度要求较高的场景，推荐使用本地源码运行方式以降低额外开销。
+
+适合熟悉`Conda` 环境或希望从零搭建运行环境的用户。
+
+对于对响应速度要求较高的场景，推荐使用本地源码运行方式以降低额外开销。

 ### 二、[固件编译](./docs/firmware-build.md)

@@ -304,7 +309,6 @@ TTS 性能排行:
 ## 鸣谢 🙏

 - 本项目受 [百聆语音对话机器人](https://github.com/wwbin2017/bailing) 启发，并在其基础上实现。
- 感谢 [腾讯云](https://cloud.tencent.com/) 提供免费 Docker 镜像空间。
 - 感谢 [十方融海](https://www.tenclass.com/) 对小智通讯协议提供的详尽文档支持。

 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
@@ -1,427 +1,291 @@
-![图片](docs/images/banner.png)

-# Xiaozhi ESP-32 Back-end Service (xiaozhi-esp32-server)
+[![SVG Banners](https://svg-banners.vercel.app/api?type=origin&text1=Hello😃,%20XiaoZhi📟&text2=Open%20Source%20XiaoZhi%20ESP-32%20Backend%20Service&width=830&height=210)](https://github.com/xinnan-tech/xiaozhi-esp32-server)
+<p align="center">
+  <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/graphs/contributors">
+    <img alt="GitHub Contributors" src="https://img.shields.io/github/contributors/xinnan-tech/xiaozhi-esp32-server" />
+  </a>
+  <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">
+    <img alt="Issues" src="https://img.shields.io/github/issues/xinnan-tech/xiaozhi-esp32-server?color=0088ff" />
+  </a>
+  <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/pulls">
+    <img alt="GitHub pull requests" src="https://img.shields.io/github/issues-pr/xinnan-tech/xiaozhi-esp32-server?color=0088ff" />
+  </a>
+</p>

-（[中文](README.md) | English）
+# XiaoZhi ESP-32 Backend Service (xiaozhi-esp32-server)

-This project provides backend services for the open-source smart hardware
-project [xiaozhi-esp32](https://github.com/78/xiaozhi-esp32)。Implemented in Python following
-the[Xiaozhi Communication Protocol](https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh)。
+([中文](README.md) | English)

-## Target Audience
+This project provides the backend service for the open source smart hardware project [xiaozhi-esp32](https://github.com/78/xiaozhi-esp32). It is implemented in `Python` based on the [XiaoZhi Communication Protocol](https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh).

-This project requires compatible esp32 hardware devices. If you have purchased esp32 hardware, successfully connected to
-Xiage's deployed backend, and wish to independently set up the `xiaozhi-esp32` backend service, this project is for
-you.
+---

-To see a demo, watch this video:
+## Target Audience 👥

-<a href="https://www.bilibili.com/video/BV1FMFyejExX">
- <picture>
-   <img alt="小智esp32连接自己的后台模型" src="docs/images/demo.png" />
- </picture>
-</a>
+This project is designed to be used in conjunction with ESP32 hardware devices. If you have already purchased an ESP32 device, successfully connected to the backend service deployed by XieGe, and now wish to set up your own `xiaozhi-esp32` backend service, then this project is perfect for you.

-To fully experience this project, follow these steps:
+Want to see it in action? Check out the videos 🎥

- Prepare hardware compatible with the `xiaozhi-esp32` project. For supported
-  models, [click here](https://rcnv1t9vps13.feishu.cn/wiki/DdgIw4BUgivWDPkhMj1cGIYCnRf).
- Use a computer/server with at least 4-core CPU and 8GB RAM to run this project. After deployment, you'll see the
-  service endpoint address in the console.
- Download the `xiaozhi-esp32` project, replace the default `endpoint address` with your own, compile, and flash the
-  firmware to your device.
- Start the device and check your server console logs to verify successful connection.
+<table>
+  <tr>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
+        <picture>
+          <img alt="XiaoZhi ESP32 connecting to a custom backend model" src="docs/images/demo1.png" />
+        </picture>
+      </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1CDKWemEU6" target="_blank">
+        <picture>
+          <img alt="Custom Voice" src="docs/images/demo2.png" />
+        </picture>
+      </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
+        <picture>
+          <img alt="Conversing in Cantonese" src="docs/images/demo3.png" />
+        </picture>
+      </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/av114036381327149" target="_blank">
+        <picture>
+          <img alt="Control Home Appliances" src="docs/images/demo5.png" />
+        </picture>
+      </a>
+    </td>
+    <td>
+      <a href="https://www.bilibili.com/video/BV1kgA2eYEQ9" target="_blank">
+        <picture>
+          <img alt="Lowest Cost Configuration" src="docs/images/demo4.png" />
+        </picture>
+      </a>
+    </td>
+    <td>
+    </td>
+  </tr>
+</table>

-## Warning
+---

-This project has been established for a short time and has not passed the network security assessment, so please do not
-use it in the production environment.
+## System Requirements and Deployment Prerequisites 🖥️

-## Feature List
+- **Hardware**: A set of devices compatible with `xiaozhi-esp32` (for specific models, please refer to [this link](https://rcnv1t9vps13.feishu.cn/wiki/DdgIw4BUgivWDPkhMj1cGIYCnRf)).
+- **Server**: A computer with at least a 4-core CPU and 8GB of memory.
+- **Firmware Compilation**: Please update the backend service API endpoint in the `xiaozhi-esp32` project, then recompile the firmware and flash it to your device.

-## Implemented
+---

- `xiaozhi-esp32` WebSocket communication protocol
- Supports wake-word initiated dialogue, manual dialogue, and real-time interruption of dialogue.
- Support for 5 languages: Mandarin, Cantonese, English, Japanese, Korean (FunASR - default)
- Flexible LLM switching (openai:ChatGLM - default, Aliyun, DeepSeek; dify:Dify)
- Flexible TTS switching (EdgeTTS - default, ByteDance Doubao TTS)
+## Warning ⚠️

-## In Progress
+This project is relatively new and has not yet undergone network security evaluations. **Do not use it in a production environment.**

- Sleep mode after inactivity
- Dialogue memory
- Change the mood mode
+If you deploy this project on a public network for learning purposes, be sure to enable protection in the configuration file `config.yaml`:

-## Supported Services
-
-| Type | Service    |  Usage   | Pricing Model	 | Notes                                                                      |
-|:-----|:-----------|:--------:|:---------------|:---------------------------------------------------------------------------|
-| LLM  | Aliyun     | openai API call | Token-based    | [Apply for API Key](https://bailian.console.aliyun.com/?apiKey=1#/api-key) |
-| LLM  | DeepSeek   | openai API call | Token-based    | [Apply for API Key](https://platform.deepseek.com/)                        |
-| LLM  | Bigmodel   | openai API call | Free           | [Create API Key](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)         |
-| LLM  | Dify       | dify API call | Token-based    | Self-hosted                                                                |
-| TTS  | HuoshanTTS | API call | Token-based    | [Create API Key](https://console.volcengine.com/speech/service/8)          |
-| TTS  | EdgeTTS    | API call | Free           |                                                                            |
-| VAD  | SileroVAD  |  Local   | Free           |                                                                            |
-| ASR  | FunASR     |  Local   | Free           |                                                                            |
-
-In fact, any LLM that supports OpenAI API calls can be integrated and used.
-
-# Deployment
-
-This project supports rapid deployment of docker and local source code operation. If you want to have a quick
-experience, it is recommended to use docker to deploy. If you want to have an in-depth understanding of this project, it
-is recommended to run the local source code.
-
-## Method 1: Quick deployment of docker
-
-The docker image has supported the CPU of x86 architecture and arm64 architecture, and supports running on Chinese
-operating systems.
-
-1. Install docker
-
-If your computer has not installed docker, you can follow the tutorial here to install
-it:[Install docker](https://www.runoob.com/docker/ubuntu-docker-install.html)
-
-2. Create a directory
-
-After installation, you need to find a directory for the configuration file for this project. Let's call it the
-`project directory` for the time being. This directory is preferably a newly created empty directory.
-
-3. Download the configuration file
-
-Open with a browser[This link](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/config.yaml)。
-
-On the right side of the page, find the button named `RAW`, next to the `RAW` button, find the download icon, click the
-Download button, and download the `config.yaml` file. Download the file to your `project directory`.
-
-4. Configure Project
-
-Modify the `config.yaml` file to configure the various parameters required for this project. The default LLM uses
-`ChatGLMLLM`, you need to configure the key to start.
-The default TTS uses `EdgeTTS`. This does not require configuration. If you need to replace it with`Doubao TTS`, you
-need to
-configure the key.
-
-Configuration description: This is the default component of each function, such as LLM default to use the `ChatGLMLLM`
-model. If you need to switch the model, it is the corresponding name.
-
-The default configuration of this project is only the lowest operating cost configuration（`glm-4-flash`and`EdgeTTS`are
-free），If you need to be better and faster, you need to combine the use of the deployment environment to switch the use
-of each component。
-
-```
-selected_module:
-  ASR: FunASR
-  VAD: SileroVAD
-  LLM: ChatGLMLLM
-  TTS: EdgeTTS
+```yaml
+server:
+  auth:
+    # Enable protection
+    enabled: true  
 ```

-For example, to modify the components used by the `LLM`, it depends on which `LLM` API interfaces are supported by this project. Currently, the supported ones are `openai` and `dify`. We welcome validation and support for more LLM platforms' interfaces.
-When using it, change the `selected_module` to the corresponding name of the following LLM configurations:
-
-```
-LLM:
-  AliLLM:
-    type: openai
-    ...
-  DeepSeekLLM:
-    type: openai
-    ...
-  ChatGLMLLM:
-    type: openai
-    ...
-  DifyLLM:
-    type: openai
-    ...
-```
+Once protection is enabled, you will need to validate the machine's token or MAC address based on your actual situation. Please refer to the configuration documentation for details.

-Some services, for example, if you use the TTS` of the `dify` and` bean bags, you need a key, remember to add the
-configuration file!
+---

-5. Execute the docker command
-
-Open the command line tool, `cd` enter your `project directory`, and execute the following command
+## Feature List ✨

-```
-#If you are Linux, execute
-ls
-#If you are Windows, execute
-dir
-```
+### Implemented ✅

-If you can see the `config.yaml` file, you have indeed entered the `project directory`, and then execute the following
-command:
-
-```
-docker run -d --name xiaozhi-esp32-server --restart always --security-opt seccomp:unconfined -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-esp32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
-```
-
-If executed for the first time, it may take several minutes, and you have to be patient to wait for it to complete the
-pull. After normal pulling is completed, you can execute the following command on the command line to see if the service
-is started successfully.
-
-```
-docker ps
-```
-
-If you can see `xiaozhi-server`, it means that the service starts successfully. Then you can further execute the
-following command to view the service log
-
-```
-docker logs -f xiaozhi-esp32-server
-```
-
-If you can see, similar to the following logs, it is a sign that the service of this project is successfully launched.
-
-```
-2025-xx-xx xx:51:59,492 - core.server - INFO - Server is running at ws://xx.xx.xx.xxx:8000
-2025-xx-xx xx:51:59,516 - websockets.server - INFO - server listening on 0.0.0.0:8000
-```
-
-Next, you can start `compiling esp32 firmware`. Please go down and turn to the relevant chapter on
-`compiling esp32 firmware`. So since you are deploying with docker, you have to check the IP of your native computer by
-yourself.
-Normally, assuming your ip is `192.168.1.25`, then your interface address is: `ws://192.168.1.25:8000`. This information
-is very useful, and it is required to `compile esp32 firmware` later.
-
-## Method 2 : Local Source Code Deployment
-
-### 1.Install Prerequisites
-
-This project uses 'conda' to manage dependencies, and after installation, start executing the following commands:
-
-```
-conda remove -n xiaozhi-esp32-server --all -y
-conda create -n xiaozhi-esp32-server python=3.10 -y
-conda activate xiaozhi-esp32-server
-```
-
-After executing the above command, if your computer is Windows or Mac, execute the following statement:
-
-```
-conda activate xiaozhi-esp32-server
-conda install conda-forge::libopus
-conda install conda-forge::ffmpeg
-```
-
-If your computer is ubuntu, execute the following statement:
-
-```
-apt-get install libopus0 ffmpeg 
-```
-
-### 2.Install Dependencies
-
-```
-# Clone the project
-cd xiaozhi-esp32-server
-conda activate xiaozhi-esp32-server
-pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
-pip install -r requirements.txt
-```
-
-### 3.Download ASR Model
-
-Download [SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt) to
-`model/SenseVoiceSmall`.
-
-By default, the `SenseVoiceSmall` model is used to convert voice to text. Because the model is large, it needs to be
-downloaded independently. After downloading, place the `model.pt` file in the `model/SenseVoiceSmall` directory. Choose
-any of the following two download routes.
-
- Line 1: Download Ali Magic
-  Tower[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
- Line 2: Baidu Netdisk download[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna)
-  提取码: `qvna`
+- **Communication Protocol**  
+  Based on the `xiaozhi-esp32` protocol, data exchange is implemented via WebSocket.
+- **Dialogue Interaction**  
+  Supports wake-up dialogues, manual conversations, and real-time interruptions. Automatically enters sleep mode after long periods of inactivity.
+- **Multilingual Recognition**  
+  Supports Mandarin, Cantonese, English, Japanese, and Korean (default using FunASR).
+- **LLM Module**  
+  Allows flexible switching of LLM modules. The default is ChatGLMLLM, with options to use AliLLM, DeepSeek, Ollama, and others.
+- **TTS Module**  
+  Supports multiple TTS interfaces including EdgeTTS (default) and Volcano Engine Doubao TTS to meet speech synthesis requirements.

-### 4.Configure Project
-
-Modify the `config.yaml` file to configure the various parameters required for this project. The default LLM uses
-`ChatGLMLLM`, you need to configure the key to start.
-The default TTS uses `EdgeTTS`. This does not require configuration. If you need to replace it with`Doubao TTS`, you
-need to
-configure the key.
-
-Configuration description: This is the default component of each function, such as LLM default to use the `ChatGLMLLM`
-model. If you need to switch the model, it is the corresponding name.
-
-The default configuration of this project is only the lowest operating cost configuration（`glm-4-flash`and`EdgeTTS`are
-free），If you need to be better and faster, you need to combine the use of the deployment environment to switch the use
-of each component。
+### In Development 🚧

-```
-selected_module:
-  ASR: FunASR
-  VAD: SileroVAD
-  LLM: ChatGLMLLM
-  TTS: EdgeTTS
-```
-
-For example, to modify the components used by the `LLM`, it depends on which `LLM` API interfaces are supported by this project. Currently, the supported ones are `openai` and `dify`. We welcome validation and support for more LLM platforms' interfaces.
-When using it, change the `selected_module` to the corresponding name of the following LLM configurations:
-
-```
-LLM:
-  AliLLM:
-    type: openai
-    ...
-  DeepSeekLLM:
-    type: openai
-    ...
-  ChatGLMLLM:
-    type: openai
-    ...
-  DifyLLM:
-    type: openai
-    ...
-```
-
-Some services, for example, if you use the TTS` of the `dify` and` bean bags, you need a key, remember to add the
-configuration file!
-
-### 5.Run the Project
+- Conversation Memory Feature
+- Multiple Mood Modes
+- Smart Control Panel Web UI

-Run the Project
+![Screenshot](docs/images/webui.png)

-```
-# Make sure to execute in the root directory of this project
-conda activate xiaozhi-esp32-server
-python app.py
-```
+---

-You'll see the WebSocket endpoint in logs:
+## Supported Platforms/Components 📋

-```
-2025-xx-xx xx:51:59,492 - core.server - INFO - Server is running at ws://192.168.1.25:8000
-2025-xx-xx xx:51:59,516 - websockets.server - INFO - server listening on 0.0.0.0:8000
-```
+### LLM

-Among them, the `ws://192.168.1.25:8000` is the interface address provided by this project. Of course, your own machine
-is different from mine. Remember to find your own address.
+| Type |         Platform Name         |         Usage Method          |   Pricing Model   |                                 Remarks                                  |
+|:----:|:-----------------------------:|:-----------------------------:|:-----------------:|:-------------------------------------------------------------------------:|
+| LLM  |   AliLLM (阿里百炼)           | OpenAI API call               | Token consumption | [Click to apply for API key](https://bailian.console.aliyun.com/?apiKey=1#/api-key) |
+| LLM  | DeepSeekLLM (深度求索)        | OpenAI API call               | Token consumption |          [Click to apply for API key](https://platform.deepseek.com/)         |
+| LLM  | ChatGLMLLM (智谱)             | OpenAI API call               | Free              | Although free, you still need to [click to apply for an API key](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) |
+| LLM  | OllamaLLM                   | Ollama API call               | Free/Custom       | Requires pre-downloading the model (`ollama pull`); service URL: `http://localhost:11434` |
+| LLM  | DifyLLM                     | Dify API call                 | Token consumption | For local deployment. Note that prompt configuration must be set in the Dify console. |
+| LLM  | GeminiLLM                   | Gemini API call               | Free              | [Click to apply for API key](https://aistudio.google.com/apikey) |
+| LLM  | CozeLLM                     | Coze API call                 | Token consumption | Requires providing bot_id, user_id, and personal token. |
+| LLM  | Home Assistant              | Home Assistant voice assistant API call | Free       | Requires providing a Home Assistant token. |

-# Compile ESP32 Firmware
+In fact, any LLM that supports OpenAI API calls can be integrated.

-1. Download `xiaozhi-esp32` project, configure the project environment according to this
-   tutorial [" Windows builds ESP IDF 5.3.2 Development Environment and Compiles Xiaozhi "](https://icnynnzcwou8.feishu.cn/wiki/JEYDwTTALi5s2zkGlFGcDiRknXf)
-   Cure
+---

-2. Open the `xiaozhi-esp32/main/kconfig.projbuild` file, find the content of the` websocket_url` `default`, change the
-   ` wss: // api.tenclass.net` to your own address, such as
+### TTS

-Before modification:
+| Type |             Platform Name              | Usage Method |   Pricing Model   |                                       Remarks                                        |
+|:----:|:--------------------------------------:|:------------:|:-----------------:|:--------------------------------------------------------------------------------------:|
+| TTS  |               EdgeTTS                  | API call     | Free              | Default TTS based on Microsoft's speech synthesis technology.                        |
+| TTS  | DoubaoTTS (火山引擎豆包 TTS)             | API call     | Token consumption | [Click to create an API key](https://console.volcengine.com/speech/service/8); it is recommended to use the paid version for higher concurrency. |
+| TTS  | CosyVoiceSiliconflow                   | API call     | Token consumption | Requires application for the Siliconflow API key; output format is WAV.                |
+| TTS  | CozeCnTTS                              | API call     | Token consumption | Requires providing a Coze API key; output format is WAV.                               |
+| TTS  | FishSpeech                             | API call     | Free/Custom       | Starts a local TTS service; see the configuration file for startup instructions.       |
+| TTS  | GPT_SOVITS_V2                          | API call     | Free/Custom       | Starts a local TTS service, suitable for personalized speech synthesis scenarios.      |

-```
-config WEBSOCKET_URL
-    depends on CONNECTION_TYPE_WEBSOCKET
-    string "Websocket URL"
-    default "wss://api.tenclass.net/xiaozhi/v1/"
-    help
-        Communication with the server through websocket after wake up.
-```
+---

-After modification (example):
+### VAD

-```
-config WEBSOCKET_URL
-    depends on CONNECTION_TYPE_WEBSOCKET
-    string "Websocket URL"
-    default "ws://192.168.1.25:8000/xiaozhi/v1/"
-    help
-        Communication with the server through websocket after wake up.
-```
+| Type |    Platform Name    | Usage Method | Pricing Model | Remarks |
+|:----:|:-------------------:|:------------:|:-------------:|:-------:|
+| VAD  | SileroVAD           | Local        | Free          |         |

-3. Configure build settings:
+---

-```
-# The terminal command line enters the root directory of xiaozhi-esp32
-cd xiaozhi-esp32
-# For example, the board I use is ESP32S3, so the compile target is ESP32S3. If your board is other models, please replace it with the corresponding model
-idf.py set-target esp32s3
-# Enter the menu configuration
-idf.py menuconfig
-```
+### ASR

-![图片](docs/images/build_setting01.png)
+| Type |    Platform Name    | Usage Method | Pricing Model | Remarks |
+|:----:|:-------------------:|:------------:|:-------------:|:-------:|
+| ASR  | FunASR              | Local        | Free          |         |
+| ASR  | DoubaoASR           | API call     | Paid          |         |

-After entering the menu configuration, then enter `xiaozhi assistant`, set the` connection_type` to `websocket`
-Go back to the main menu, then enter `xiaozhi assistant`, set the `BOARD_TYPE` of your board
-Save exit and return to the terminal command line.
+---

-![图片](docs/images/build_setting02.png)
+## Usage 🚀

-4. Build and package:
+### 1. [Deployment Documentation](./docs/Deployment.md)

-```
-idf.py build
-cd scripts
-python release.py
-```
+This project supports three deployment methods. Choose the one that best fits your needs.

-After the compilation is successful, the firmware file `merged-binary.bin` is generated in the` build` directory in the
-project root directory.
-This `merged-binary.bin` is the firmware file that will be recorded on the hardware.
+The documentation provided here is a **written tutorial**. If you prefer a **video tutorial**, you can refer to [this expert's hands-on guide](https://www.bilibili.com/video/BV1gePuejEvT).

-6. Flash
-   Connect the ESP32 device to the computer, use the Chrome browser, and open the following URL
+Combining both the written and video tutorials can help you get started more quickly.

-```
-https://espressif.github.io/esp-launchpad/
-```
+1. [Docker Quick Deployment](./docs/Deployment.md)  
+   Suitable for general users who want a quick experience without extensive environment configuration. The only downside is that pulling the image can be a bit slow.

-Open this
-tutorial, [Flash Tools/Web -side Burning Folding Step (No IDF Development Environment)](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS).
-Turn to: `Method 2: ESP-LAUNCHPAD browser web-end burning`, start from
-`3. Burning firmware/download to the development board`, follow the tutorial operation.
+2. [Deployment Using Docker Environment](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E5%80%9F%E5%8A%A9docker%E7%8E%AF%E5%A2%83%E8%BF%90%E8%A1%8C%E9%83%A8%E7%BD%B2)  
+   Ideal for software engineers who already have Docker installed and wish to customize the code.

-# FAQ
+3. [Running from Local Source Code](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%89%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C)  
+   Suitable for users familiar with the `Conda` environment or those who wish to build the runtime environment from scratch.

-## 1、TTS often fails, often overtime
+For scenarios requiring higher response speeds, running from the local source code is recommended to reduce additional overhead.

-Suggestion: If the `Edgetts` is slow or often fails, you can replace it with a bean bag TTS` with a volcanic engine. If
-both are slow, the network environment may need to be optimized.
+### 2. [Firmware Compilation](./docs/firmware-build.md)

-## 2、Big model reply is a bit slow
+Click [here](./docs/firmware-build.md) for a detailed guide on firmware compilation.

-Suggestions: Both big models and TTS are dependent interfaces. If the network environment is not good, you can consider
-changing the local model. Or try to switch different interface models.
+After successful compilation and network connection, wake up XiaoZhi using the wake-up word and monitor the server console for output.

-## 3、Why is my ChatGLMLLM replying to a bit? Obviously it is Xiaozhi, but treats me as Xiaozhi.
+---

-Suggestion: You can modify the prompts in the configuration file first. You can also replace the free `GLM-4-FLASH` to
-the model of other toll versions of `ChatGlm`.
+## Frequently Asked Questions ❓

-## 4、I want to control the operation of electric lights, air conditioners, remote switching and other operations through Xiaozhi.
+### 1. TTS often fails and times out ⏰

-Suggestion: In the configuration file, set the `LLM` to`DifyLLM`, and then arrange the smart application by the
-`Dify`.
+**Suggestion:**  
+If `EdgeTTS` frequently fails, please first check whether you are using a proxy (VPN). If so, try disabling the proxy and try again. If you are using Volcano Engine Doubao TTS and it often fails, it is recommended to use the paid version since the trial only supports 2 concurrent requests.

-## 5、I said very slowly, I paused, Xiaozhi always grabbed me, what to do.
+### 2. I want to control lights, air conditioners, remote power on/off, etc. with XiaoZhi 💡

-Suggestion: In the configuration file, find this section, change the `min_silence_duration_ms` value, such as change to
-` 1000`.
+**Suggestion:**  
+Set the `LLM` to `HomeAssistant` in the configuration file and use the `HomeAssistant` API to perform the relevant controls.

-```
+### 3. I speak slowly, and XiaoZhi always interrupts during pauses 🗣️
+
+**Suggestion:**  
+Locate the following section in the configuration file and increase the value of `min_silence_duration_ms` (for example, change it to `1000`):
+
+```yaml
 VAD:
  SileroVAD:
    threshold: 0.5
    model_dir: models/snakers4_silero-vad
-    min_silence_duration_ms: 700  # 如果说话停顿比较长，可以把这个值设置大一些
+    min_silence_duration_ms: 700  # If your pauses are longer, increase this value
 ```

-## 6、For more questions, contact us to feedback
+### 4. Why does XiaoZhi recognize a lot of Korean, Japanese, and English in what I say? 🇰🇷

-![图片](docs/images/wechat.jpg)
+**Suggestion:**  
+Check whether the `model.pt` file exists in the `models/SenseVoiceSmall` directory. If it does not, please download it. See [Download ASR Model Files](docs/Deployment.md#模型文件) for details.

-# Acknowledgments
+### 5. Why does the error “TTS task error: file does not exist” occur? 📁

- This project is inspired by the [Bailin Voice Dialogue Robot](https://github.com/wwbin2017/bailing) project, and the
-  basic idea of the project is completed。
- Thanks to [Tencent Cloud] (https://cloud.tencent.com/) for providing free docker space for this project。
- Thanks to [tenclass](https://www.tenclass.com/)Provide adequate documentation support on Xiaozhi Communication
-  Protocol。
+**Suggestion:**  
+Verify that you have correctly installed the `libopus` and `ffmpeg` libraries using `conda`. If not, install them using:
+
+```
+conda install conda-forge::libopus
+conda install conda-forge::ffmpeg
+```
+
+### 6. How can I improve XiaoZhi's dialogue response speed? ⚡
+
+The default configuration of this project is designed to be cost-effective. It is recommended that beginners first use the default free models to ensure that the system runs smoothly, then optimize for faster response times.  
+To improve response speed, you can try replacing individual components. Below are the response time test results for each component (for reference only, not a guarantee):
+
+**LLM Performance Ranking:**
+
+| Module Name  | Average First Token Time | Average Total Response Time |
+|--------------|--------------------------|-----------------------------|
+| AliLLM       | 0.547s                   | 1.485s                      |
+| ChatGLMLLM   | 0.677s                   | 3.057s                      |
+| OllamaLLM    | 0.003s                   | 0.003s                      |
+
+**TTS Performance Ranking:**
+
+| Module Name                | Average Synthesis Time |
+|----------------------------|------------------------|
+| EdgeTTS                    | 1.019s                 |
+| DoubaoTTS                  | 0.503s                 |
+| CosyVoiceSiliconflow       | 3.732s                 |
+
+**Recommended Configuration Combination (Overall Response Speed):**
+
+| Combination Scheme                | Overall Score | LLM First Token | TTS Synthesis |
+|-----------------------------------|---------------|-----------------|---------------|
+| AliLLM + DoubaoTTS                | 0.539         | 0.547s          | 0.503s        |
+| AliLLM + EdgeTTS                  | 0.642         | 0.547s          | 1.019s        |
+| ChatGLMLLM + DoubaoTTS            | 0.642         | 0.677s          | 0.503s        |
+| ChatGLMLLM + EdgeTTS              | 0.745         | 0.677s          | 1.019s        |
+| AliLLM + CosyVoiceSiliconflow     | 1.184         | 0.547s          | 3.732s        |
+
+**Conclusion 🔍**
+
+_As of February 19, 2025, if my computer were located in Haizhu District, Guangzhou, Guangdong Province, and connected via China Unicom, I would prioritize using:_
+
+- **LLM:** `AliLLM`
+- **TTS:** `DoubaoTTS`
+
+### 7. For more questions, feel free to contact us for feedback 💬
+
+![WeChat QR Code](docs/images/wechat.jpg)
+
+---
+
+## Acknowledgements 🙏
+
+- This project was inspired by the [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it.
+- Many thanks to [Tenclass](https://www.tenclass.com/) for providing detailed documentation support for the XiaoZhi communication protocol.

 <a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
 <picture>
@@ -1,6 +1,6 @@
 import asyncio
 from config.logger import setup_logging
-from config.settings import load_config
+from config.settings import load_config, check_config_file
 from core.websocket_server import WebSocketServer
 from manager.http_server import WebUI
 from aiohttp import web
@@ -8,10 +8,12 @@ from core.utils.util import get_local_ip

 TAG = __name__

+
 async def main():
+    check_config_file()
    logger = setup_logging()
    config = load_config()
-    
+
    # 启动 WebSocket 服务器
    ws_server = WebSocketServer(config)
    ws_task = asyncio.create_task(ws_server.start())
@@ -33,7 +35,7 @@ async def main():
            logger.bind(tag=TAG).info(f"WebUI server is running at http://{local_ip}:{port}")
        except Exception as e:
            logger.bind(tag=TAG).error(f"Failed to start WebUI server: {e}")
-    
+
    try:
        # 等待 WebSocket 服务器运行
        await ws_task
@@ -42,5 +44,6 @@ async def main():
        if webui_runner:
            await webui_runner.cleanup()

+
 if __name__ == "__main__":
    asyncio.run(main())
@@ -41,7 +41,9 @@ manager:
  enabled: false
  ip: 0.0.0.0
  port: 8002
-
+iot:
+  Speaker:
+    volume: 100
 xiaozhi:
  type: hello
  version: 1
@@ -147,6 +149,12 @@ LLM:
    user_id: 你的user_id
    base_url: "https://api.coze.cn/open_api/v2/chat" # 服务地址
    personal_access_token: 你的coze个人令牌
+  LMStudioLLM:
+    # 定义LLM API类型
+    type: openai
+    model_name: deepseek-r1-distill-llama-8b@q4_k_m # 使用的模型名称，需要预先在社区下载
+    url: http://localhost:1234/v1 # LM Studio服务地址
+    api_key: lm-studio # LM Studio服务的固定API Key
  HomeAssistant:
    # 定义LLM API类型
    type: homeassistant
@@ -304,3 +312,19 @@ module_test:
    - "你好，请介绍一下你自己"
    - "What's the weather like today?"
    - "请用100字概括量子计算的基本原理和应用前景"
+
+# 本地音乐播放配置
+music:
+  music_commands:
+    - "来一首歌"
+    - "唱一首歌"
+    - "播放音乐"
+    - "来点音乐"
+    - "背景音乐"
+    - "放首歌"
+    - "播放歌曲"
+    - "来点背景音乐"
+    - "我想听歌"
+    - "我要听歌"
+    - "放点音乐"
+  music_dir: "./music"  # 音乐文件存放路径
@@ -1,22 +1,26 @@
 import os
 import argparse
 from ruamel.yaml import YAML
+from collections.abc import Mapping
 from core.utils.util import read_config, get_project_dir

+default_config_file = "config.yaml"
+

 def get_config_file():
-    default_config_file = "config.yaml"
+    global default_config_file
    # 判断是否存在私有的配置文件
+    config_file = default_config_file
    if os.path.exists(get_project_dir() + "data/." + default_config_file):
-        default_config_file = "data/." + default_config_file
-    return default_config_file
+        config_file = "data/." + default_config_file
+    return config_file


 def load_config():
    """加载配置文件"""
    parser = argparse.ArgumentParser(description="Server configuration")
-    default_config_file = get_config_file()
-    parser.add_argument("--config_path", type=str, default=default_config_file)
+    config_file = get_config_file()
+    parser.add_argument("--config_path", type=str, default=config_file)
    args = parser.parse_args()
    return read_config(args.config_path)

@@ -27,3 +31,54 @@ def update_config(config):
    """将配置保存到YAML文件"""
    with open(get_config_file(), 'w') as f:
        yaml.dump(config, f)
+
+
+def find_missing_keys(new_config, old_config, parent_key=''):
+    """
+    递归查找缺失的配置项
+    返回格式：[缺失配置路径]
+    """
+    missing_keys = []
+
+    if not isinstance(new_config, Mapping):
+        return missing_keys
+
+    for key, value in new_config.items():
+        # 构建当前配置路径
+        full_path = f"{parent_key}.{key}" if parent_key else key
+
+        # 检查键是否存在
+        if key not in old_config:
+            missing_keys.append(full_path)
+            continue
+
+        # 递归检查嵌套字典
+        if isinstance(value, Mapping):
+            sub_missing = find_missing_keys(
+                value,
+                old_config[key],
+                parent_key=full_path
+            )
+            missing_keys.extend(sub_missing)
+
+    return missing_keys
+
+
+def check_config_file():
+    old_config_file = get_config_file()
+    global default_config_file
+    if not old_config_file.startswith('data'):
+        return
+    old_config = read_config(get_project_dir() + old_config_file)
+    new_config = read_config(get_project_dir() + default_config_file)
+    # 查找缺失的配置项
+    missing_keys = find_missing_keys(new_config, old_config)
+
+    if missing_keys:
+        error_msg = "您的配置文件太旧了，缺少了：\n"
+        error_msg += "\n".join(f"- {key}" for key in missing_keys)
+        error_msg += "\n建议您：\n"
+        error_msg += "1、备份data/.config.yaml文件\n"
+        error_msg += "2、将根目录的config.yaml文件复制到data下，重命名为.config.yaml\n"
+        error_msg += "3、将密钥逐个复制到新的配置文件中\n"
+        raise ValueError(error_msg)
@@ -4,25 +4,26 @@ import uuid
 import time
 import queue
 import asyncio
+import traceback
 from config.logger import setup_logging
 import threading
 import websockets
 from typing import Dict, Any
-from collections import deque
-from core.utils.util import is_segment
 from core.utils.dialogue import Message, Dialogue
 from core.handle.textHandle import handleTextMessage
 from core.utils.util import get_string_no_punctuation_or_emoji
 from concurrent.futures import ThreadPoolExecutor, TimeoutError
-from core.handle.audioHandle import handleAudioMessage, sendAudioMessage
+from core.handle.sendAudioHandle import sendAudioMessage
+from core.handle.receiveAudioHandle import handleAudioMessage
 from config.private_config import PrivateConfig
 from core.auth import AuthMiddleware, AuthenticationError
-from core.utils.auth_code_gen import AuthCodeGenerator  # 添加导入
+from core.utils.auth_code_gen import AuthCodeGenerator

 TAG = __name__

+
 class ConnectionHandler:
-    def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts):
+    def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music):
        self.config = config
        self.logger = setup_logging()
        self.auth = AuthMiddleware(config)
@@ -41,8 +42,8 @@ class ConnectionHandler:
        self.loop = asyncio.get_event_loop()
        self.stop_event = threading.Event()
        self.tts_queue = queue.Queue()
+        self.audio_play_queue = queue.Queue()
        self.executor = ThreadPoolExecutor(max_workers=10)
-        self.scheduled_tasks = deque()

        # 依赖的组件
        self.vad = _vad
@@ -72,28 +73,33 @@ class ConnectionHandler:
        self.tts_start_speak_time = None
        self.tts_duration = 0

+        # iot相关变量
+        self.iot_descriptors = {}
+
        self.cmd_exit = self.config["CMD_exit"]
        self.max_cmd_length = 0
        for cmd in self.cmd_exit:
            if len(cmd) > self.max_cmd_length:
                self.max_cmd_length = len(cmd)
-        
+
        self.private_config = None
        self.auth_code_gen = AuthCodeGenerator.get_instance()
        self.is_device_verified = False  # 添加设备验证状态标志
-
+        self.music_handler = _music

    async def handle_connection(self, ws):
        try:
            # 获取并验证headers
            self.headers = dict(ws.request.headers)
-            self.logger.bind(tag=TAG).info(f"New connection request - Headers: {self.headers}")
+            # 获取客户端ip地址
+            client_ip = ws.remote_address[0]
+            self.logger.bind(tag=TAG).info(f"{client_ip} conn - Headers: {self.headers}")

            # 进行认证
            await self.auth.authenticate(self.headers)

            device_id = self.headers.get("device-id", None)
-            
+
            # Load private configuration if device_id is provided
            bUsePrivateConfig = self.config.get("use_private_config", False)
            self.logger.bind(tag=TAG).info(f"bUsePrivateConfig: {bUsePrivateConfig}, device_id: {device_id}")
@@ -104,10 +110,10 @@ class ConnectionHandler:
                    # 判断是否已经绑定
                    owner = self.private_config.get_owner()
                    self.is_device_verified = owner is not None
-                    
+
                    if self.is_device_verified:
-                        await self.private_config.update_last_chat_time() 
-                    
+                        await self.private_config.update_last_chat_time()
+
                    llm, tts = self.private_config.create_private_instances()
                    if all([llm, tts]):
                        self.llm = llm
@@ -131,9 +137,14 @@ class ConnectionHandler:

            await self.loop.run_in_executor(None, self._initialize_components)

-            tts_priority = threading.Thread(target=self._priority_thread, daemon=True)
+            # tts 消化线程
+            tts_priority = threading.Thread(target=self._tts_priority_thread, daemon=True)
            tts_priority.start()

+            # 音频播放 消化线程
+            audio_play_priority = threading.Thread(target=self._audio_play_priority_thread, daemon=True)
+            audio_play_priority.start()
+
            try:
                async for message in self.websocket:
                    await self._route_message(message)
@@ -146,7 +157,8 @@ class ConnectionHandler:
            await ws.close()
            return
        except Exception as e:
-            self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}")
+            stack_trace = traceback.format_exc()
+            self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}")
            await ws.close()
            return

@@ -166,7 +178,7 @@ class ConnectionHandler:
            date_time = time.strftime("%Y-%m-%d %H:%M", time.localtime())
            self.prompt = self.prompt.replace("{date_time}", date_time)
        self.dialogue.put(Message(role="system", content=self.prompt))
-          
+
    async def _check_and_broadcast_auth_code(self):
        """检查设备绑定状态并广播认证码"""
        if not self.private_config.get_owner():
@@ -186,12 +198,10 @@ class ConnectionHandler:
            # 如果不使用私有配置，就不需要验证
            return False
        return not self.is_device_verified
-    
+
    def chat(self, query):
-        # 如果设备未验证，就发送验证码
        if self.isNeedAuth():
            self.llm_finish_task = True
-            # 创建一个新的事件循环来运行异步函数
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
            try:
@@ -199,52 +209,64 @@ class ConnectionHandler:
            finally:
                loop.close()
            return True
-        
+
        self.dialogue.put(Message(role="user", content=query))
        response_message = []
-        start = 0
-        # 提交 LLM 任务
+        processed_chars = 0  # 跟踪已处理的字符位置
        try:
-            start_time = time.time()  # 记录开始时间
+            start_time = time.time()
            llm_responses = self.llm.response(self.session_id, self.dialogue.get_llm_dialogue())
        except Exception as e:
            self.logger.bind(tag=TAG).error(f"LLM 处理出错 {query}: {e}")
            return None
-        # 提交 TTS 任务到线程池
+
        self.llm_finish_task = False
        for content in llm_responses:
            response_message.append(content)
-            # 如果中途被打断，就停止生成
            if self.client_abort:
-                start = len(response_message)
                break

-            end_time = time.time()  # 记录结束时间
-            self.logger.bind(tag=TAG).debug(f"大模型返回时间时间: {end_time - start_time} 秒, 生成token={content}")
-            if is_segment(response_message):
-                segment_text = "".join(response_message[start:])
-                segment_text = get_string_no_punctuation_or_emoji(segment_text)
-                if len(segment_text) > 0:
+            end_time = time.time()
+            self.logger.bind(tag=TAG).debug(f"大模型返回时间: {end_time - start_time} 秒, 生成token={content}")
+
+            # 合并当前全部文本并处理未分割部分
+            full_text = "".join(response_message)
+            current_text = full_text[processed_chars:]  # 从未处理的位置开始
+
+            # 查找最后一个有效标点
+            punctuations = ("。", "？", "！", ".", "?", "!", ";", "；", ":", "：", "，", ",")
+            last_punct_pos = -1
+            for punct in punctuations:
+                pos = current_text.rfind(punct)
+                if pos > last_punct_pos:
+                    last_punct_pos = pos
+
+            # 找到分割点则处理
+            if last_punct_pos != -1:
+                segment_text_raw = current_text[:last_punct_pos + 1]
+                segment_text = get_string_no_punctuation_or_emoji(segment_text_raw)
+                if segment_text:
                    self.recode_first_last_text(segment_text)
                    future = self.executor.submit(self.speak_and_play, segment_text)
                    self.tts_queue.put(future)
-                    start = len(response_message)
+                    processed_chars += len(segment_text_raw)  # 更新已处理字符位置

-        # 处理剩余的响应
-        if start < len(response_message):
-            segment_text = "".join(response_message[start:])
-            if len(segment_text) > 0:
+        # 处理最后剩余的文本
+        full_text = "".join(response_message)
+        remaining_text = full_text[processed_chars:]
+        if remaining_text:
+            segment_text = get_string_no_punctuation_or_emoji(remaining_text)
+            if segment_text:
                self.recode_first_last_text(segment_text)
                future = self.executor.submit(self.speak_and_play, segment_text)
                self.tts_queue.put(future)

        self.llm_finish_task = True
-        # 更新对话
        self.dialogue.put(Message(role="assistant", content="".join(response_message)))
        self.logger.bind(tag=TAG).debug(json.dumps(self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False))
        return True

-    def _priority_thread(self):
+    def _tts_priority_thread(self):
        while not self.stop_event.is_set():
            text = None
            try:
@@ -266,7 +288,6 @@ class ConnectionHandler:
                    else:
                        self.logger.bind(tag=TAG).error(f"TTS文件不存在: {tts_file}")
                        opus_datas = []
-                        duration = 0
                except TimeoutError:
                    self.logger.bind(tag=TAG).error("TTS 任务超时")
                    continue
@@ -275,9 +296,7 @@ class ConnectionHandler:
                    continue
                if not self.client_abort:
                    # 如果没有中途打断就发送语音
-                    asyncio.run_coroutine_threadsafe(
-                        sendAudioMessage(self, opus_datas, duration, text), self.loop
-                    )
+                    self.audio_play_queue.put((opus_datas, text))
                if self.tts.delete_audio_file and os.path.exists(tts_file):
                    os.remove(tts_file)
            except Exception as e:
@@ -289,6 +308,16 @@ class ConnectionHandler:
                )
                self.logger.bind(tag=TAG).error(f"tts_priority priority_thread: {text}{e}")

+    def _audio_play_priority_thread(self):
+        while not self.stop_event.is_set():
+            text = None
+            try:
+                opus_datas, text = self.audio_play_queue.get()
+                future = asyncio.run_coroutine_threadsafe(sendAudioMessage(self, opus_datas, text), self.loop)
+                future.result()
+            except Exception as e:
+                self.logger.bind(tag=TAG).error(f"audio_play_priority priority_thread: {text}{e}")
+
    def speak_and_play(self, text):
        if text is None or len(text) <= 0:
            self.logger.bind(tag=TAG).info(f"无需tts转换，query为空，{text}")
@@ -316,6 +345,8 @@ class ConnectionHandler:

    async def close(self):
        """资源清理方法"""
+
+        # 清理其他资源
        self.stop_event.set()
        self.executor.shutdown(wait=False)
        if self.websocket:
@@ -328,9 +359,3 @@ class ConnectionHandler:
        self.client_have_voice_last_time = 0
        self.client_voice_stop = False
        self.logger.bind(tag=TAG).debug("VAD states reset.")
-
-    def stop_all_tasks(self):
-        while self.scheduled_tasks:
-            task = self.scheduled_tasks.popleft()
-            task.cancel()
-        self.scheduled_tasks.clear()
@@ -1,4 +1,5 @@
 import json
+import queue
 from config.logger import setup_logging

 TAG = __name__
@@ -9,8 +10,6 @@ async def handleAbortMessage(conn):
    logger.bind(tag=TAG).info("Abort message received")
    # 设置成打断状态，会自动打断llm、tts任务
    conn.client_abort = True
-    # 打断屏显任务
-    conn.stop_all_tasks()
    # 打断客户端说话状态
    await conn.websocket.send(json.dumps({"type": "tts", "state": "stop", "session_id": conn.session_id}))
    conn.clearSpeakStatus()
@@ -1,167 +0,0 @@
-from config.logger import setup_logging
-import json
-import asyncio
-import time
-from core.utils.util import remove_punctuation_and_length, get_string_no_punctuation_or_emoji
-
-TAG = __name__
-logger = setup_logging()
-
-
-async def handleAudioMessage(conn, audio):
-    if not conn.asr_server_receive:
-        logger.bind(tag=TAG).debug(f"前期数据处理中，暂停接收")
-        return
-    if conn.client_listen_mode == "auto":
-        have_voice = conn.vad.is_vad(conn, audio)
-    else:
-        have_voice = conn.client_have_voice
-
-    # 如果本次没有声音，本段也没声音，就把声音丢弃了
-    if have_voice == False and conn.client_have_voice == False:
-        await no_voice_close_connect(conn)
-        conn.asr_audio.clear()
-        return
-    conn.client_no_voice_last_time = 0.0
-    conn.asr_audio.append(audio)
-    # 如果本段有声音，且已经停止了
-    if conn.client_voice_stop:
-        conn.client_abort = False
-        conn.asr_server_receive = False
-        # 音频太短了，无法识别
-        if len(conn.asr_audio) < 3:
-            conn.asr_server_receive = True
-        else:
-            text, file_path = await conn.asr.speech_to_text(conn.asr_audio, conn.session_id)
-            logger.bind(tag=TAG).info(f"识别文本: {text}")
-            text_len, text_without_punctuation = remove_punctuation_and_length(text)
-            if text_len <= conn.max_cmd_length and await handleCMDMessage(conn, text_without_punctuation):
-                return
-            if text_len > 0:
-                await startToChat(conn, text)
-            else:
-                conn.asr_server_receive = True
-        conn.asr_audio.clear()
-        conn.reset_vad_states()
-
-async def handleCMDMessage(conn, text):
-    cmd_exit = conn.cmd_exit
-    for cmd in cmd_exit:
-        if text == cmd:
-            logger.bind(tag=TAG).info("识别到明确的退出命令".format(text))
-            await finishToChat(conn)
-            return True
-    return False
-
-
-async def finishToChat(conn):
-    await conn.close()
-
-
-async def isLLMWantToFinish(conn):
-    first_text = conn.tts_first_text
-    last_text = conn.tts_last_text
-    _, last_text_without_punctuation = remove_punctuation_and_length(last_text)
-    if "再见" in last_text_without_punctuation or "拜拜" in last_text_without_punctuation:
-        return True
-    _, first_text_without_punctuation = remove_punctuation_and_length(first_text)
-    if "再见" in first_text_without_punctuation or "拜拜" in first_text_without_punctuation:
-        return True
-    return False
-
-
-async def startToChat(conn, text):
-    # 异步发送 stt 信息
-    stt_task = asyncio.create_task(
-        schedule_with_interrupt(0, send_stt_message(conn, text))
-    )
-    conn.scheduled_tasks.append(stt_task)
-    conn.executor.submit(conn.chat, text)
-
-
-async def sendAudioMessage(conn, audios, duration, text):
-    base_delay = conn.tts_duration
-
-    # 发送 tts.start
-    if text == conn.tts_first_text:
-        logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
-        conn.tts_start_speak_time = time.time()
-
-    # 发送 sentence_start（每个音频文件之前发送一次）
-    sentence_task = asyncio.create_task(
-        schedule_with_interrupt(base_delay, send_tts_message(conn, "sentence_start", text))
-    )
-    conn.scheduled_tasks.append(sentence_task)
-
-    conn.tts_duration += duration
-
-    # 发送音频数据
-    for idx, opus_packet in enumerate(audios):
-        await conn.websocket.send(opus_packet)
-
-    if conn.llm_finish_task and text == conn.tts_last_text:
-        stop_duration = conn.tts_duration - (time.time() - conn.tts_start_speak_time)
-        stop_task = asyncio.create_task(
-            schedule_with_interrupt(stop_duration, send_tts_message(conn, 'stop'))
-        )
-        conn.scheduled_tasks.append(stop_task)
-        if await isLLMWantToFinish(conn):
-            finish_task = asyncio.create_task(
-                schedule_with_interrupt(stop_duration, finishToChat(conn))
-            )
-            conn.scheduled_tasks.append(finish_task)
-
-
-async def send_tts_message(conn, state, text=None):
-    """发送 TTS 状态消息"""
-    message = {
-        "type": "tts",
-        "state": state,
-        "session_id": conn.session_id
-    }
-    if text is not None:
-        message["text"] = text
-
-    await conn.websocket.send(json.dumps(message))
-    if state == "stop":
-        conn.clearSpeakStatus()
-
-
-async def send_stt_message(conn, text):
-    """发送 STT 状态消息"""
-    stt_text = get_string_no_punctuation_or_emoji(text)
-    await conn.websocket.send(json.dumps({
-        "type": "stt",
-        "text": stt_text,
-        "session_id": conn.session_id}
-    ))
-    await conn.websocket.send(
-        json.dumps({
-            "type": "llm",
-            "text": "😊",
-            "emotion": "happy",
-            "session_id": conn.session_id}
-        ))
-    await send_tts_message(conn, "start")
-
-
-async def schedule_with_interrupt(delay, coro):
-    """可中断的延迟调度"""
-    try:
-        await asyncio.sleep(delay)
-        await coro
-    except asyncio.CancelledError:
-        pass
-
-
-async def no_voice_close_connect(conn):
-    if conn.client_no_voice_last_time == 0.0:
-        conn.client_no_voice_last_time = time.time() * 1000
-    else:
-        no_voice_time = time.time() * 1000 - conn.client_no_voice_last_time
-        close_connection_no_voice_time = conn.config.get("close_connection_no_voice_time", 120)
-        if no_voice_time > 1000 * close_connection_no_voice_time:
-            conn.client_abort = False
-            conn.asr_server_receive = False
-            prompt = "时间过得真快，我都好久没说话了。请你用十个字左右话跟我告别，以“再见”或“拜拜拜”为结尾"
-            await startToChat(conn, prompt)
@@ -0,0 +1,152 @@
+import json
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class IotDescriptor:
+    """
+    A class to represent an IoT descriptor.
+    Attributes:
+    ----------
+    name : str
+        The name of the IoT descriptor.
+    description : str
+        A brief description of the IoT descriptor.
+    properties : dict
+        A dictionary containing properties of the IoT descriptor.
+    methods : dict
+        A dictionary containing methods of the IoT descriptor.
+    -------
+    """
+
+    def __init__(self, name, description, properties, methods):
+        self.name = name
+        self.description = description
+        self.properties = []
+        self.methods = []
+
+        # 根据描述创建属性
+        for key, value in properties.items():
+            # "volume":{"description":"当前音量 值","type":"number"}
+            """
+            等价于
+            {
+                'name': 名字,
+                'description': 描述,
+                'value': 0
+            }
+            """
+            # setattr(self, key, {}) # 创建一个空字典, 名字是属性名
+            property_item = globals()[key] = {}  # 创建一个空字典, 名字是属性名
+            property_item['name'] = key
+            property_item["description"] = value["description"]
+            if value["type"] == "number":
+                property_item["value"] = 0
+            elif value["type"] == "boolean":
+                property_item["value"] = False
+            else:
+                property_item["value"] = ""
+            self.properties.append(property_item)
+
+        # 根据描述创建方法
+        for key, value in methods.items():
+            # "SetVolume": {"description":"设置音量","parameters":{"volume":{"description":"0到100之间的整数","type":"number"}}}
+            """
+            等价于
+            SetVolume = {
+                `description`: 描述,
+                `volume`: {
+                    `description`: 描述,
+                    `value`: 0
+                }
+            }
+            """
+            # setattr(self, key, {}) # 创建一个空字典, 名字是方法名
+            method = globals()[key] = {}  # 创建一个空字典, 名字是方法名
+            method["description"] = value["description"]
+            method['name'] = key
+            for k, v in value["parameters"].items():
+                # 不同的参数解析
+                method[k] = {}
+                method[k]["description"] = v["description"]
+                if v["type"] == "number":
+                    method[k]["value"] = 0
+                elif v["type"] == "boolean":
+                    method[k]["value"] = False
+                else:
+                    method[k]["value"] = ""
+
+            self.methods.append(method)
+
+
+async def handleIotDescriptors(conn, descriptors):
+    """
+    处理物联网描述
+    示例: [{
+        "name":"Speaker",
+        "description":"当前 AI 机器人的扬声器",
+        "properties":{
+            "volume":{"description":"当前音量 值","type":"number"}  可以有boolean, number, string三种类型
+        },
+        "methods":{
+            "SetVolume":{
+                "description":"设置音量","parameters":{"volume":{"description":"0到100之间的整数","type":"number"}}
+            }
+        }
+    }]
+    descriptors: 描述列表
+    """
+    for descriptor in descriptors:
+        iot_descriptor = IotDescriptor(descriptor["name"], descriptor["description"], descriptor["properties"],
+                                       descriptor["methods"])
+        conn.iot_descriptors[descriptor["name"]] = iot_descriptor
+
+    # 暂时从配置文件中设置音量，后期通过意图识别控制音量
+    default_iot_volume = 100
+    if "iot" in conn.config:
+        default_iot_volume = conn.config["iot"]["Speaker"]["volume"]
+    logger.bind(tag=TAG).info(f"服务端设置音量为{default_iot_volume}")
+    await send_iot_conn(conn, "Speaker", "SetVolume", {"volume": default_iot_volume})
+
+
+async def send_iot_conn(conn, name, method_name, parameters):
+    """
+    发送物联网指令
+    name: 设备名称 "Speaker"
+    method: 方法 "SetVolume"
+    parameters: 参数, 是一个字典 {"volume": 100}
+    发送示例:
+    {
+        "type": "iot",
+        "commands": [
+            {
+                "name" :  "Speaker",
+                "method": "SetVolume",
+                "parameters": {
+                    "volume": 100
+                    }
+            }
+        ]
+    }
+    """
+
+    for key, value in conn.iot_descriptors.items():
+        if key == name:
+            # 找到了设备
+            for method in value.methods:
+                # 找到了方法
+                if method["name"] == method_name:
+                    await conn.websocket.send(json.dumps({
+                        "type": "iot",
+                        "commands": [
+                            {
+                                "name": name,
+                                "method": method_name,
+                                "parameters": parameters
+                            }
+                        ]
+                    }))
+                    return
+    logger.bind(tag=TAG).error(f"未找到方法{method_name}")
@@ -0,0 +1,110 @@
+from config.logger import setup_logging
+import os
+import random
+import difflib
+import re
+import traceback
+from core.handle.sendAudioHandle import send_stt_message
+
+TAG = __name__
+logger = setup_logging()
+
+
+def _extract_song_name(text):
+    """从用户输入中提取歌名"""
+    for keyword in ["听", "播放", "放", "唱"]:
+        if keyword in text:
+            parts = text.split(keyword)
+            if len(parts) > 1:
+                return parts[1].strip()
+    return None
+
+
+def _find_best_match(potential_song, music_files):
+    """查找最匹配的歌曲"""
+    best_match = None
+    highest_ratio = 0
+
+    for music_file in music_files:
+        song_name = os.path.splitext(music_file)[0]
+        ratio = difflib.SequenceMatcher(None, potential_song, song_name).ratio()
+        if ratio > highest_ratio and ratio > 0.4:
+            highest_ratio = ratio
+            best_match = music_file
+    return best_match
+
+
+class MusicHandler:
+    def __init__(self, config):
+        self.config = config
+        self.music_related_keywords = []
+
+        if "music" in self.config:
+            self.music_config = self.config["music"]
+            self.music_dir = os.path.abspath(
+                self.music_config.get("music_dir", "./music")  # 默认路径修改
+            )
+            self.music_related_keywords = self.music_config.get("music_commands", [])
+        else:
+            self.music_dir = os.path.abspath("./music")
+            self.music_related_keywords = ["来一首歌", "唱一首歌", "播放音乐", "来点音乐", "背景音乐", "放首歌",
+                                           "播放歌曲", "来点背景音乐", "我想听歌", "我要听歌", "放点音乐"]
+
+    async def handle_music_command(self, conn, text):
+        """处理音乐播放指令"""
+        clean_text = re.sub(r'[^\w\s]', '', text).strip()
+        logger.bind(tag=TAG).debug(f"检查是否是音乐命令: {clean_text}")
+
+        # 尝试匹配具体歌名
+        if os.path.exists(self.music_dir):
+            music_files = [f for f in os.listdir(self.music_dir) if f.endswith('.mp3')]
+            logger.bind(tag=TAG).debug(f"找到的音乐文件: {music_files}")
+
+            potential_song = _extract_song_name(clean_text)
+            if potential_song:
+                best_match = _find_best_match(potential_song, music_files)
+                if best_match:
+                    logger.bind(tag=TAG).info(f"找到最匹配的歌曲: {best_match}")
+                    await self.play_local_music(conn, specific_file=best_match)
+                    return True
+
+        # 检查是否是通用播放音乐命令
+        if any(cmd in clean_text for cmd in self.music_related_keywords):
+            await self.play_local_music(conn)
+            return True
+
+        return False
+
+    async def play_local_music(self, conn, specific_file=None):
+        """播放本地音乐文件"""
+        try:
+            if not os.path.exists(self.music_dir):
+                logger.bind(tag=TAG).error(f"音乐目录不存在: {self.music_dir}")
+                return
+
+            # 确保路径正确性
+            if specific_file:
+                music_path = os.path.join(self.music_dir, specific_file)
+                if not os.path.exists(music_path):
+                    logger.bind(tag=TAG).error(f"指定的音乐文件不存在: {music_path}")
+                    return
+                selected_music = specific_file
+            else:
+                music_files = [f for f in os.listdir(self.music_dir) if f.endswith('.mp3')]
+                if not music_files:
+                    logger.bind(tag=TAG).error("未找到MP3音乐文件")
+                    return
+                selected_music = random.choice(music_files)
+                music_path = os.path.join(self.music_dir, selected_music)
+            text = f"正在播放{selected_music}"
+            await send_stt_message(conn, text)
+            conn.tts_first_text = selected_music
+            conn.tts_last_text = selected_music
+            conn.llm_finish_task = True
+            opus_packets, duration = conn.tts.wav_to_opus_data(music_path)
+
+            conn.audio_play_queue.put((opus_packets, selected_music))
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"播放音乐失败: {str(e)}")
+            logger.bind(tag=TAG).error(f"详细错误: {traceback.format_exc()}")
@@ -0,0 +1,77 @@
+from config.logger import setup_logging
+import time
+from core.utils.util import remove_punctuation_and_length
+from core.handle.sendAudioHandle import send_stt_message
+
+TAG = __name__
+logger = setup_logging()
+
+
+async def handleAudioMessage(conn, audio):
+    if not conn.asr_server_receive:
+        logger.bind(tag=TAG).debug(f"前期数据处理中，暂停接收")
+        return
+    if conn.client_listen_mode == "auto":
+        have_voice = conn.vad.is_vad(conn, audio)
+    else:
+        have_voice = conn.client_have_voice
+
+    # 如果本次没有声音，本段也没声音，就把声音丢弃了
+    if have_voice == False and conn.client_have_voice == False:
+        await no_voice_close_connect(conn)
+        conn.asr_audio.clear()
+        return
+    conn.client_no_voice_last_time = 0.0
+    conn.asr_audio.append(audio)
+    # 如果本段有声音，且已经停止了
+    if conn.client_voice_stop:
+        conn.client_abort = False
+        conn.asr_server_receive = False
+        # 音频太短了，无法识别
+        if len(conn.asr_audio) < 3:
+            conn.asr_server_receive = True
+        else:
+            text, file_path = await conn.asr.speech_to_text(conn.asr_audio, conn.session_id)
+            logger.bind(tag=TAG).info(f"识别文本: {text}")
+            text_len, text_without_punctuation = remove_punctuation_and_length(text)
+            if await conn.music_handler.handle_music_command(conn, text_without_punctuation):
+                conn.asr_server_receive = True
+                conn.asr_audio.clear()
+                return
+            if text_len <= conn.max_cmd_length and await handleCMDMessage(conn, text_without_punctuation):
+                return
+            if text_len > 0:
+                await startToChat(conn, text)
+            else:
+                conn.asr_server_receive = True
+        conn.asr_audio.clear()
+        conn.reset_vad_states()
+
+
+async def handleCMDMessage(conn, text):
+    cmd_exit = conn.cmd_exit
+    for cmd in cmd_exit:
+        if text == cmd:
+            logger.bind(tag=TAG).info("识别到明确的退出命令".format(text))
+            await conn.close()
+            return True
+    return False
+
+
+async def startToChat(conn, text):
+    # 异步发送 stt 信息
+    await send_stt_message(conn, text)
+    conn.executor.submit(conn.chat, text)
+
+
+async def no_voice_close_connect(conn):
+    if conn.client_no_voice_last_time == 0.0:
+        conn.client_no_voice_last_time = time.time() * 1000
+    else:
+        no_voice_time = time.time() * 1000 - conn.client_no_voice_last_time
+        close_connection_no_voice_time = conn.config.get("close_connection_no_voice_time", 120)
+        if no_voice_time > 1000 * close_connection_no_voice_time:
+            conn.client_abort = False
+            conn.asr_server_receive = False
+            prompt = "时间过得真快，我都好久没说话了。请你用十个字左右话跟我告别，以“再见”或“拜拜”为结尾"
+            await startToChat(conn, prompt)
@@ -0,0 +1,76 @@
+from config.logger import setup_logging
+import json
+import asyncio
+import time
+from core.utils.util import remove_punctuation_and_length, get_string_no_punctuation_or_emoji
+
+TAG = __name__
+logger = setup_logging()
+
+
+async def isLLMWantToFinish(last_text):
+    _, last_text_without_punctuation = remove_punctuation_and_length(last_text)
+    if "再见" in last_text_without_punctuation or "拜拜" in last_text_without_punctuation:
+        return True
+    return False
+
+
+async def sendAudioMessage(conn, audios, text):
+    # 发送 tts.start
+    if text == conn.tts_first_text:
+        logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
+        conn.tts_start_speak_time = time.time()
+    await send_tts_message(conn, "sentence_start", text)
+
+    # 发送音频数据
+    frame_duration = 60  # 初始帧持续时间（毫秒）
+    start_time = time.time()  # 记录开始时间
+    for idx, opus_packet in enumerate(audios):
+        if conn.client_abort:
+            return
+        # 计算当前包的预期发送时间
+        expected_time = start_time + idx * (frame_duration / 1000)
+        current_time = time.time()
+        # 如果未到预期时间则等待差值
+        if current_time < expected_time:
+            await asyncio.sleep(expected_time - current_time)
+        # 发送音频包
+        await conn.websocket.send(opus_packet)
+
+    if conn.llm_finish_task and text == conn.tts_last_text:
+        await send_tts_message(conn, 'stop')
+        if await isLLMWantToFinish(text):
+            await conn.close()
+
+
+async def send_tts_message(conn, state, text=None):
+    """发送 TTS 状态消息"""
+    message = {
+        "type": "tts",
+        "state": state,
+        "session_id": conn.session_id
+    }
+    if text is not None:
+        message["text"] = text
+
+    await conn.websocket.send(json.dumps(message))
+    if state == "stop":
+        conn.clearSpeakStatus()
+
+
+async def send_stt_message(conn, text):
+    """发送 STT 状态消息"""
+    stt_text = get_string_no_punctuation_or_emoji(text)
+    await conn.websocket.send(json.dumps({
+        "type": "stt",
+        "text": stt_text,
+        "session_id": conn.session_id}
+    ))
+    await conn.websocket.send(
+        json.dumps({
+            "type": "llm",
+            "text": "😊",
+            "emotion": "happy",
+            "session_id": conn.session_id}
+        ))
+    await send_tts_message(conn, "start")
@@ -2,7 +2,8 @@ from config.logger import setup_logging
 import json
 from core.handle.abortHandle import handleAbortMessage
 from core.handle.helloHandle import handleHelloMessage
-from core.handle.audioHandle import startToChat
+from core.handle.receiveAudioHandle import startToChat
+from core.handle.iotHandle import handleIotDescriptors

 TAG = __name__
 logger = setup_logging()
@@ -36,5 +37,8 @@ async def handleTextMessage(conn, message):
                conn.asr_audio.clear()
                if "text" in msg_json:
                    await startToChat(conn, msg_json["text"])
+        elif msg_json["type"] == "iot":
+            if "descriptors" in msg_json:
+                await handleIotDescriptors(conn, msg_json["descriptors"])
    except json.JSONDecodeError:
        await conn.websocket.send(message)
@@ -25,12 +25,25 @@ class LLMProvider(LLMProviderBase):
                messages=dialogue,
                stream=True
            )
+            
+            is_active = True
            for chunk in responses:
-                # 检查是否存在有效的choice且content不为空
-                if chunk.choices and len(chunk.choices) > 0:
-                    delta = chunk.choices[0].delta
-                    content = getattr(delta, 'content', '')
-                    if content:  # 仅在content非空时生成
+                try:
+                    # 检查是否存在有效的choice且content不为空
+                    delta = chunk.choices[0].delta if getattr(chunk, 'choices', None) else None
+                    content = delta.content if hasattr(delta, 'content') else ''
+                except IndexError:
+                    content = ''
+                if content:
+                    # 处理标签跨多个chunk的情况
+                    if '<think>' in content:
+                        is_active = False
+                        content = content.split('<think>')[0]
+                    if '</think>' in content:
+                        is_active = True
+                        content = content.split('</think>')[-1]
+                    if is_active:
                        yield content
+
        except Exception as e:
            logger.bind(tag=TAG).error(f"Error in response generation: {e}")
@@ -1,7 +1,5 @@
 import os
 import sys
-import asyncio
-from typing import List, Dict, Any

 # 添加项目根目录到Python路径
 current_dir = os.path.dirname(os.path.abspath(__file__))
@@ -10,10 +8,6 @@ sys.path.insert(0, project_root)

 from config.logger import setup_logging
 import importlib
-from datetime import datetime
-from core.utils.util import is_segment
-from core.utils.util import get_string_no_punctuation_or_emoji
-from core.utils.util import read_config, get_project_dir

 logger = setup_logging()

@@ -27,117 +21,3 @@ def create_instance(class_name, *args, **kwargs):
        return sys.modules[lib_name].LLMProvider(*args, **kwargs)

    raise ValueError(f"不支持的LLM类型: {class_name}，请检查该配置的type是否设置正确")
-
-
-async def test_single_model(llm_name: str, llm_config: Dict[str, Any], test_prompt: str, config: Dict[str, Any]) -> Dict[str, Any]:
-    """异步测试单个模型"""
-    try:
-        # 获取实际的LLM类型
-        llm_type = llm_config["type"] if "type" in llm_config else llm_name
-        llm = create_instance(llm_type, llm_config)
-        
-        # 开始测试
-        dialogue = []
-        dialogue.append({"role": "system", "content": config.get("prompt")})
-        dialogue.append({"role": "user", "content": test_prompt})
-        
-        start_time = datetime.now()
-        llm_responses = llm.response("test", dialogue)
-        response_message = []
-        first_response_time = None
-        total_response_time = None
-        start = 0
-        full_response = ""
-
-        for content in llm_responses:
-            response_message.append(content)
-            full_response += content
-
-            if is_segment(response_message):
-                segment_text = "".join(response_message[start:])
-                segment_text = get_string_no_punctuation_or_emoji(segment_text)
-                if len(segment_text) > 0:
-                    if first_response_time is None:
-                        first_response_time = (datetime.now() - start_time).total_seconds()
-                    start = len(response_message)
-
-        total_response_time = (datetime.now() - start_time).total_seconds()
-        
-        return {
-            "name": llm_name,
-            "type": llm_type,
-            "first_response_time": first_response_time,
-            "total_response_time": total_response_time,
-            "response_length": len(full_response),
-            "status": "成功",
-            "response": full_response
-        }
-
-    except Exception as e:
-        print(f"测试 {llm_name} 时发生错误: {str(e)}")
-        return {
-            "name": llm_name,
-            "type": llm_config.get("type", llm_name),
-            "first_response_time": None,
-            "total_response_time": None,
-            "response_length": 0,
-            "status": f"失败 - {str(e)}",
-            "response": ""
-        }
-
-
-async def main():
-    """
-    LLM模型响应速度测试和排行（异步版本）
-    """
-    config = read_config(get_project_dir() + "config.yaml")
-    test_prompt = "你好小智"
-    
-    print("开始并发测试所有模型...")
-    
-    # 创建所有模型的测试任务
-    tasks = []
-    for llm_name, llm_config in config["LLM"].items():
-        task = asyncio.create_task(test_single_model(llm_name, llm_config, test_prompt, config))
-        tasks.append(task)
-    
-    # 等待所有测试完成
-    test_results = await asyncio.gather(*tasks)
-    
-    # 打印测试结果排行榜
-    print("\n========= LLM模型性能测试排行榜 =========")
-    print("测试提示词:", test_prompt)
-    
-    # 过滤出成功的结果，并确保数值有效
-    successful_results = [r for r in test_results if r["status"] == "成功" and r["first_response_time"] is not None]
-    
-    if successful_results:
-        print("\n1. 首次响应时间排行:")
-        sorted_by_first = sorted(successful_results, key=lambda x: x["first_response_time"])
-        for i, result in enumerate(sorted_by_first, 1):
-            print(f"{i}. {result['name']}({result['type']}) - {result['first_response_time']:.2f}秒")
-            print(f"   响应内容: {result['response'][:50]}...")  # 只显示前50个字符
-
-        print("\n2. 总响应时间排行:")
-        sorted_by_total = sorted(successful_results, key=lambda x: x["total_response_time"] or float('inf'))
-        for i, result in enumerate(sorted_by_total, 1):
-            if result["total_response_time"] is not None:
-                print(f"{i}. {result['name']}({result['type']}) - {result['total_response_time']:.2f}秒")
-
-        print("\n3. 响应长度比较:")
-        sorted_by_length = sorted(successful_results, key=lambda x: x["response_length"], reverse=True)
-        for i, result in enumerate(sorted_by_length, 1):
-            print(f"{i}. {result['name']}({result['type']}) - {result['response_length']}字符")
-    else:
-        print("\n没有成功完成测试的模型。")
-
-    if len(test_results) != len(successful_results):
-        print("\n测试失败的模型:")
-        failed_results = [r for r in test_results if r["status"] != "成功" or r["first_response_time"] is None]
-        for result in failed_results:
-            print(f"- {result['name']}({result['type']}): {result['status']}")
-
-
-if __name__ == "__main__":
-    # 运行异步主函数
-    asyncio.run(main())
@@ -2,8 +2,6 @@ import os
 import sys
 from config.logger import setup_logging
 import importlib
-from datetime import datetime
-from core.utils.util import read_config, get_project_dir

 logger = setup_logging()

@@ -16,26 +14,4 @@ def create_instance(class_name, *args, **kwargs):
            sys.modules[lib_name] = importlib.import_module(f'{lib_name}')
        return sys.modules[lib_name].TTSProvider(*args, **kwargs)

-    raise ValueError(f"不支持的TTS类型: {class_name}，请检查该配置的type是否设置正确")
-
-
-if __name__ == "__main__":
-    """
-      响应速度测试
-    """
-    config = read_config(get_project_dir() + "config.yaml")
-    tts = create_instance(
-        config["selected_module"]["TTS"]
-        if not 'type' in config["TTS"][config["selected_module"]["TTS"]]
-        else
-        config["TTS"][config["selected_module"]["TTS"]]["type"],
-        config["TTS"][config["selected_module"]["TTS"]],
-        config["delete_audio"]
-    )
-    tts.output_file = get_project_dir() + tts.output_file
-    start = datetime.now()
-    file_path = tts.to_tts("你好，测试,我是人工智能小智")
-    print("语音合成耗时：" + str(datetime.now() - start))
-    start = datetime.now()
-    tts.wav_to_opus_data(file_path)
-    print("语音opus耗时：" + str(datetime.now() - start))
+    raise ValueError(f"不支持的TTS类型: {class_name}，请检查该配置的type是否设置正确")
@@ -34,13 +34,6 @@ def write_json_file(file_path, data):
        json.dump(data, file, ensure_ascii=False, indent=4)


-def is_segment(tokens):
-    if tokens[-1] in (",", ".", "?", "，", "。", "？", "！", "!", ";", "；", ":", "："):
-        return True
-    else:
-        return False
-
-
 def is_punctuation_or_emoji(char):
    """检查字符是否为空格、指定标点或表情符号"""
    # 定义需要去除的中英文标点（包括全角/半角）
@@ -2,6 +2,7 @@ import asyncio
 import websockets
 from config.logger import setup_logging
 from core.connection import ConnectionHandler
+from core.handle.musicHandler import MusicHandler
 from core.utils.util import get_local_ip
 from core.utils import asr, vad, llm, tts

@@ -12,7 +13,8 @@ class WebSocketServer:
    def __init__(self, config: dict):
        self.config = config
        self.logger = setup_logging()
-        self._vad, self._asr, self._llm, self._tts = self._create_processing_instances()
+        self._vad, self._asr, self._llm, self._tts, self._music = self._create_processing_instances()
+        self.active_connections = set()  # 添加全局连接记录

    def _create_processing_instances(self):
        """创建处理模块实例"""
@@ -43,7 +45,8 @@ class WebSocketServer:
                self.config["TTS"][self.config["selected_module"]["TTS"]]["type"],
                self.config["TTS"][self.config["selected_module"]["TTS"]],
                self.config["delete_audio"]
-            )
+            ),
+            MusicHandler(self.config)
        )

    async def start(self):
@@ -62,5 +65,10 @@ class WebSocketServer:

    async def _handle_connection(self, websocket):
        """处理新连接，每次创建独立的ConnectionHandler"""
-        handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts)
-        await handler.handle_connection(websocket)
+        # 创建ConnectionHandler时传入当前server实例
+        handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music)
+        self.active_connections.add(handler)
+        try:
+            await handler.handle_connection(websocket)
+        finally:
+            self.active_connections.discard(handler)
@@ -1,3 +1,4 @@
+version: '3'
 services:
  xiaozhi-esp32-server:
    image: ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:latest
@@ -117,7 +117,7 @@ docker run -it --name xiaozhi-env --restart always --security-opt seccomp:unconf
  -p 8000:8000 \
  -p 8002:8002 \
  -v ./:/app \
-  ccr.ccs.tencentyun.com/kalicyh/poetry:v3.10_latest
+  kalicyh/poetry:v3.10_xiaozhi
 ```

 然后就和正常开发一样了
@@ -16,4 +16,4 @@ aiohttp_cors==0.7.0
 ormsgpack==1.7.0
 ruamel.yaml==0.18.10
 loguru==0.7.3
-requests>=2.0.0
+requests==2.32.3
Author	SHA1	Message	Date
欣南科技 GitHub hrz	b94842c312	Fix long audio bug (#158 ) * update:异步生成音频 * update:优化LLM断句 --------- Co-authored-by: hrz <1710360675@qq.com>	2025-03-01 01:54:55 +08:00
欣南科技 GitHub hrz	2c88a36f32	update:启动时检查配置文件 (#155 ) Co-authored-by: hrz <1710360675@qq.com>	2025-02-28 19:17:36 +08:00
欣南科技 GitHub hrz	242f4d87e7	Qq group (#146 ) * update:qq group * update:更新英文版说明 --------- Co-authored-by: hrz <1710360675@qq.com>	2025-02-26 16:02:59 +08:00
欣南科技 GitHub hrz	2255aacfd0	update:qq group (#145 ) Co-authored-by: hrz <1710360675@qq.com>	2025-02-26 15:10:12 +08:00
欣南科技 GitHub linqingping Chris hrz	87cb0b484d	Music play (#142 ) * Update util.py解决返回元组报错 (#103) 解决返回元组报错 * 增加播放本地音乐功能 (#105) * Please enter the commit message for your changes. Lines starting with '#' will be ignored, and an empty message aborts the commit. On branch Music-playback Your branch is up to date with 'origin/Music-playback'. Changes to be committed: modified: core/handle/audioHandle.py new file: "music/\346\234\210\344\272\256\344\273\243\350\241\250\346\210\221\347\232\204\345\277\203_\351\202\223\344\270\275\345\220\233.mp3" new file: "music/\350\270\217\345\261\261\346\262\263.mp3" * Please enter the commit message for your changes. Lines starting with '#' will be ignored, and an empty message aborts the commit. On branch Music-playback Your branch is up to date with 'origin/Music-playback'. Changes to be committed: modified: config.yaml modified: core/connection.py modified: core/handle/abortHandle.py modified: core/handle/audioHandle.py new file: core/handle/musicHandler.py modified: core/handle/textHandle.py modified: core/websocket_server.py new file: "music/\344\270\200\345\277\265\345\215\203\345\271\264_\345\233\275\351\243\216\347\211\210.mp3" new file: "music/\344\270\255\347\247\213\346\234\210.mp3" deleted: "music/\346\234\210\344\272\256\344\273\243\350\241\250\346\210\221\347\232\204\345\277\203_\351\202\223\344\270\275\345\220\233.mp3" deleted: "music/\350\270\217\345\261\261\346\262\263.mp3" * 增加播放本地音乐功能 * 增加播放本地音乐功能 * 让音乐配置变的优雅 On branch Music-playback Your branch is up to date with 'origin/Music-playback'. Changes to be committed: modified: config.yaml modified: core/handle/musicHandler.py renamed: "music/\344\270\200\345\277\265\345\215\203\345\271\264_\345\233\275\351\243\216\347\211\210.mp3" -> "music/mp3/\344\270\200\345\277\265\345\215\203\345\271\264_\345\233\275\351\243\216\347\211\210.mp3" renamed: "music/\344\270\255\347\247\213\346\234\210.mp3" -> "music/mp3/\344\270\255\347\247\213\346\234\210.mp3" renamed: "music/\345\273\211\346\263\242\350\200\201\347\237\243\357\274\214\345\260\232\350\203\275\351\245\255\345\220\246.mp3" -> "music/mp3/\345\273\211\346\263\242\350\200\201\347\237\243\357\274\214\345\260\232\350\203\275\351\245\255\345\220\246.mp3" new file: music/music_config.yaml --------- Co-authored-by: 欣南科技 <huangrongzhuang@xin-nan.com> * update:优化代码 * update:流控调试 * update:优化音乐播放 * update:优化音乐配置初始化 --------- Co-authored-by: linqingping <linqingping@users.noreply.github.com> Co-authored-by: Chris <119588753+Chris-websketch@users.noreply.github.com> Co-authored-by: hrz <1710360675@qq.com>	2025-02-26 01:33:05 +08:00
欣南科技 GitHub hrz	73803f49fc	update:wechat group (#141 ) Co-authored-by: hrz <1710360675@qq.com>	2025-02-25 23:51:57 +08:00
欣南科技 GitHub hrz	5fd87752d5	fixed：更新基础docker环境地址 (#139 ) Co-authored-by: hrz <1710360675@qq.com>	2025-02-25 21:11:06 +08:00
欣南科技 GitHub Jiao Haoyang hrz	94a90c4124	Iot (#129 ) * 2025-2-17-iot配置消息处理, 以及控制iot命令发送 (#40) * update:优化 * update:增加视频教程 * update:iot兼容性又问题，撤回 * fixed:iot boolean 参数接收bug --------- Co-authored-by: Jiao Haoyang <108573524+XuSenfeng@users.noreply.github.com> Co-authored-by: hrz <1710360675@qq.com>	2025-02-24 22:04:48 +08:00
欣南科技 GitHub Jiao Haoyang hrz	c3b0347155	Iot (#128 ) * 2025-2-17-iot配置消息处理, 以及控制iot命令发送 (#40) * update:优化 * update:增加视频教程 * update:iot兼容性又问题，撤回 --------- Co-authored-by: Jiao Haoyang <108573524+XuSenfeng@users.noreply.github.com> Co-authored-by: hrz <1710360675@qq.com>	2025-02-24 21:20:42 +08:00
欣南科技 GitHub Jiao Haoyang hrz	600f712750	Iot (#127 ) * 2025-2-17-iot配置消息处理, 以及控制iot命令发送 (#40) * update:优化 * update:增加视频教程 --------- Co-authored-by: Jiao Haoyang <108573524+XuSenfeng@users.noreply.github.com> Co-authored-by: hrz <1710360675@qq.com>	2025-02-24 18:18:08 +08:00
欣南科技 GitHub Jiao Haoyang hrz	9e76869c36	Iot (#126 ) * 2025-2-17-iot配置消息处理, 以及控制iot命令发送 (#40) * update:优化 --------- Co-authored-by: Jiao Haoyang <108573524+XuSenfeng@users.noreply.github.com> Co-authored-by: hrz <1710360675@qq.com>	2025-02-24 18:06:13 +08:00
kalicyh GitHub Sinyo hrz	cb540736ab	跳过使用Open ai 接口时，DeepSeek-R1 模型的深度思考内容 (#54 ) (#95 ) * 跳过使用Open ai 接口时，DeepSeek-R1 模型的深度思考内容 (#54) * 跳过 DeepSeek-R1 模型的深度思考内容 * 跳过 DeepSeek-R1 模型的深度思考内容 * 新增LM Studio本地大模型API接口 * 优化代码，遇到Bad Case安全处理 * update:优化 --------- Co-authored-by: Sinyo <38577585+SinyoWong@users.noreply.github.com> Co-authored-by: hrz <1710360675@qq.com>	2025-02-24 16:16:06 +08:00
欣南科技 GitHub hrz	66cd5bb4a5	fixed:修复低版本docker-compose模版参数 (#124 ) Co-authored-by: hrz <1710360675@qq.com>	2025-02-24 14:56:39 +08:00
欣南科技 GitHub hrz	1226e8be47	Fix opuslib (#122 ) * fixed:ModuleNotFoundError: No module named 'opuslib' * update:格式化文档 --------- Co-authored-by: hrz <1710360675@qq.com>	2025-02-23 20:14:46 +08:00