mirror of
https://github.com/xinnan-tech/xiaozhi-esp32-server.git
synced 2026-07-22 07:03:53 +08:00
Merge branch 'main' into py_linkerai_single
This commit is contained in:
hrz
@@ -121,6 +121,33 @@
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="实时打断" src="docs/images/demo10.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="拍照识物品" src="docs/images/demo12.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="多指令任务" src="docs/images/demo11.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
</td>
|
||||
<td>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
@@ -141,11 +168,10 @@
|
||||
本项目提供两种部署方式，请根据您的具体需求选择：
|
||||
|
||||
#### 🚀 部署方式选择
|
||||
|
||||
| 部署方式 | 特点 | 适用场景 | Docker部署文档 | 源码部署文档 | 视频教程 |
|
||||
| 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **最简化安装** | 智能对话、IOT功能，数据存储在配置文件 | 低配置环境，无需数据库 | [Docker只运行Server](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [本地源码只运行Server](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| - |
|
||||
| **全模块安装** | 智能对话、IOT、OTA、智控台，数据存储在数据库 | 完整功能体验 |[Docker运行全模块](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [本地源码运行全模块](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [本地源码视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
| **最简化安装** | 智能对话、IOT、MCP、视觉感知，数据存储在配置文件 | 低配置环境，无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G，如果全API，要2核2G | - |
|
||||
| **全模块安装** | 智能对话、IOT、MCP、视觉感知、OTA、智控台，数据存储在数据库 | 完整功能体验 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G，如果全API，要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
|
||||
|
||||
|
||||
> 💡 提示：以下是按最新代码部署后的测试平台，有需要可烧录测试，并发为6个，每天会清空数据
|
||||
@@ -158,22 +184,49 @@ OTA接口地址: https://2662r3426b.vicp.fun/xiaozhi/ota/
|
||||
Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
```
|
||||
|
||||
#### 🚩 配置说明和推荐
|
||||
> [!Note]
|
||||
> 本项目默认的配置是`入门全免费`设置，如果想效果更优，推荐使用`流式配置`。
|
||||
>
|
||||
> 本项目自`0.5.2`版本，已支持使用流式配置，相比`0.5`版本以前，响应速度提升约`2.5秒`
|
||||
|
||||
| 模块名称 | 入门全免费设置 | 流式配置 |
|
||||
|:---:|:---:|:---:|
|
||||
| ASR(语音识别) | FunASR(本地) | 👍DoubaoStreamASR(火山流式语音识别) |
|
||||
| LLM(大模型) | ChatGLMLLM(智谱glm-4-flash) | 👍DoubaoLLM(火山doubao-1-5-pro-32k-250115) |
|
||||
| VLLM(视觉大模型) | ChatGLMVLLM(智谱glm-4v-flash) | 👍QwenVLVLLM(千问qwen2.5-vl-3b-instructh) |
|
||||
| TTS(语音合成) | EdgeTTS(微软语音) | 👍HuoshanDoubleStreamTTS(火山双流式语音合成) |
|
||||
| Intent(意图识别) | function_call(函数调用) | ✅function_call(函数调用) |
|
||||
| Memory(记忆功能) | mem_local_short(本地短期记忆） | ✅mem_local_short（本地短期记忆） |
|
||||
|
||||
#### 🔧 测试工具
|
||||
本项目提供以下测试工具，帮助您验证系统和选择合适的模型：
|
||||
|
||||
| 工具名称 | 位置 | 使用方法 | 功能说明 |
|
||||
|:---:|:---|:---:|:---:|
|
||||
| 音频交互测试工具 | main》xiaozhi-server》test/》est_page.html | 使用谷歌浏览器直接打开 | 测试音频播放和接收功能，验证Python端音频处理是否正常 |
|
||||
| 模型响应测试工具1 | main》xiaozhi-server》performance_tester.py | 执行 `python performance_tester.py` | 测试ASR(语音识别)、LLM(大模型)、TTS(语音合成)三个核心模块的响应速度 |
|
||||
| 模型响应测试工具2 | main》xiaozhi-server》performance_tester_vllm.py | 执行 `python performance_tester_vllm.py` | 测试VLLM(视觉模型)的响应速度 |
|
||||
|
||||
> 💡 提示：测试模型速度时，只会测试配置了密钥的模型。
|
||||
|
||||
---
|
||||
## 功能清单 ✨
|
||||
|
||||
### 已实现 ✅
|
||||
|
||||
| 功能模块 | 描述 |
|
||||
|---------|------|
|
||||
| 通信协议 | 基于 `xiaozhi-esp32` 协议，通过 WebSocket 实现数据交互 |
|
||||
| 对话交互 | 支持唤醒对话、手动对话及实时打断。长时间无对话时自动休眠 |
|
||||
| 意图识别 | 支持使用LLM意图识别、function call函数调用，减少硬编码意图判断 |
|
||||
| 多语言识别 | 支持国语、粤语、英语、日语、韩语（默认使用 FunASR） |
|
||||
| LLM 模块 | 支持灵活切换 LLM 模块，默认使用 ChatGLMLLM，也可选用阿里百炼、DeepSeek、Ollama 等接口 |
|
||||
| TTS 模块 | 支持 EdgeTTS（默认）、火山引擎豆包 TTS 等多种 TTS 接口，满足语音合成需求 |
|
||||
| 记忆功能 | 支持超长记忆、本地总结记忆、无记忆三种模式，满足不同场景需求 |
|
||||
| IOT功能 | 支持管理注册设备IOT功能，支持基于对话上下文语境下的智能物联网控制 |
|
||||
| 智控台 | 提供Web管理界面，支持智能体管理、用户管理、系统配置等功能，方便管理员和用户进行管理 |
|
||||
|:---:|:---|
|
||||
| 核心服务架构 | 基于WebSocket和HTTP服务器，提供完整的控制台管理和认证系统 |
|
||||
| 语音交互系统 | 支持流式ASR(语音识别)、流式TTS(语音合成)、VAD(语音活动检测)，支持多语言识别和语音处理 |
|
||||
| 智能对话系统 | 支持多种LLM(大语言模型)，实现智能对话 |
|
||||
| 视觉感知系统 | 支持多种VLLM(视觉大模型)，实现多模态交互 |
|
||||
| 意图识别系统 | 支持LLM意图识别、Function Call函数调用，提供插件化意图处理机制 |
|
||||
| 记忆系统 | 支持本地短期记忆、mem0ai接口记忆，具备记忆总结功能 |
|
||||
| IOT/MCP控制协议 | 支持设备注册管理、智能控制接口，同时支持IOT、MCP控制协议 |
|
||||
| 管理后台 | 提供Web管理界面，支持用户管理、系统配置和设备管理 |
|
||||
| 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
|
||||
| 部署支持 | 支持Docker部署和本地部署，提供完整的配置文件管理 |
|
||||
| 插件系统 | 支持功能插件扩展、自定义插件开发和插件热加载 |
|
||||
|
||||
### 正在开发 🚧
|
||||
|
||||
@@ -210,6 +263,16 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
|
||||
---
|
||||
|
||||
### VLLM 视觉模型
|
||||
|
||||
| 使用方式 | 支持平台 | 免费平台 |
|
||||
|:---:|:---:|:---:|
|
||||
| openai 接口调用 | 阿里百炼、智谱ChatGLMVLLM | 智谱ChatGLMVLLM |
|
||||
|
||||
实际上，任何支持 openai 接口调用的 VLLM 均可接入使用。
|
||||
|
||||
---
|
||||
|
||||
### TTS 语音合成
|
||||
|
||||
| 使用方式 | 支持平台 | 免费平台 |
|
||||
|
||||
+92
-54
@@ -6,14 +6,14 @@
|
||||
This project provides backend services for the open-source smart hardware project
|
||||
<a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
|
||||
Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
|
||||
Helping you quickly set up your Xiaozhi server
|
||||
Helps you quickly set up your Xiaozhi server
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="./README.md">中文</a>
|
||||
· <a href="./docs/FAQ.md">FAQ</a>
|
||||
· <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
|
||||
· <a href="./README_ed.md#deployment-documentation">Deployment Guide</a>
|
||||
· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
|
||||
· <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
|
||||
</p>
|
||||
<p align="center">
|
||||
@@ -50,7 +50,7 @@ Want to see it in action? Check out these videos 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
|
||||
<picture>
|
||||
<img alt="Xiaozhi esp32 connecting to custom backend model" src="docs/images/demo1.png" />
|
||||
<img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
@@ -64,14 +64,14 @@ Want to see it in action? Check out these videos 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
|
||||
<picture>
|
||||
<img alt="Cantonese communication" src="docs/images/demo3.png" />
|
||||
<img alt="Using Cantonese" src="docs/images/demo3.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
|
||||
<picture>
|
||||
<img alt="Home appliance control" src="docs/images/demo5.png" />
|
||||
<img alt="Control home appliances" src="docs/images/demo5.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
@@ -94,7 +94,7 @@ Want to see it in action? Check out these videos 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
|
||||
<picture>
|
||||
<img alt="Music playback" src="docs/images/demo7.png" />
|
||||
<img alt="Play music" src="docs/images/demo7.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
@@ -108,7 +108,7 @@ Want to see it in action? Check out these videos 🎥
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
|
||||
<picture>
|
||||
<img alt="IOT device control" src="docs/images/demo9.png" />
|
||||
<img alt="IOT command control" src="docs/images/demo9.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
@@ -120,6 +120,33 @@ Want to see it in action? Check out these videos 🎥
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
|
||||
<picture>
|
||||
<img alt="Real-time interruption" src="docs/images/demo10.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
|
||||
<picture>
|
||||
<img alt="Photo recognition" src="docs/images/demo12.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
|
||||
<picture>
|
||||
<img alt="Multi-command tasks" src="docs/images/demo11.png" />
|
||||
</picture>
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
</td>
|
||||
<td>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
---
|
||||
@@ -137,91 +164,102 @@ It is recommended that users prioritize service providers with relevant business
|
||||
|
||||
|
||||
|
||||
This project offers two deployment methods. Please choose based on your specific needs:
|
||||
This project provides two deployment methods. Please choose according to your specific needs:
|
||||
|
||||
#### 🚀 Deployment Method Selection
|
||||
| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial |
|
||||
|---------|------|---------|---------|---------|---------|
|
||||
| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - |
|
||||
| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) |
|
||||
|
||||
| Deployment Method | Features | Use Case | Docker Deployment Guide | Source Code Deployment Guide |
|
||||
|---------|------|---------|---------|---------|
|
||||
| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database required | [Docker Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [Local Source Code Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)|
|
||||
| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience |[Docker Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [Local Source Code Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) |
|
||||
|
||||
> 💡 Note: Below is the test platform deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data cleared daily
|
||||
> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
|
||||
|
||||
```
|
||||
Control Panel: https://2662r3426b.vicp.fun
|
||||
Control Panel Address: https://2662r3426b.vicp.fun
|
||||
|
||||
Service Test Tool: https://2662r3426b.vicp.fun/test/
|
||||
OTA Interface: https://2662r3426b.vicp.fun/xiaozhi/ota/
|
||||
Websocket Interface: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
|
||||
Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
|
||||
```
|
||||
|
||||
#### 🚩 Configuration Description and Recommendations
|
||||
> [!Note]
|
||||
> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
|
||||
>
|
||||
> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
|
||||
|
||||
| Module Name | Entry Level Free Settings | Full Streaming Configuration |
|
||||
|---------|---------|------|
|
||||
| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
|
||||
| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
|
||||
| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
|
||||
| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
|
||||
| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
|
||||
| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
|
||||
|
||||
---
|
||||
## Feature List ✨
|
||||
|
||||
### Implemented ✅
|
||||
|
||||
| Feature Module | Description |
|
||||
|---------|------|
|
||||
| Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
|
||||
| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of inactivity |
|
||||
| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
|
||||
| Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
|
||||
| Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
|
||||
| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, also supports Ali Bailing, DeepSeek, Ollama, etc. |
|
||||
| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces for speech synthesis |
|
||||
| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes for different scenarios |
|
||||
| IOT Function | Supports managing registered device IOT functionality, intelligent IoT control based on dialogue context |
|
||||
| Control Panel | Provides web management interface, supports agent management, user management, system configuration, etc. |
|
||||
| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
|
||||
| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
|
||||
| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
|
||||
| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
|
||||
| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
|
||||
|
||||
### In Development 🚧
|
||||
|
||||
To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
|
||||
|
||||
If you're a software developer, here's an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
|
||||
If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
|
||||
|
||||
---
|
||||
|
||||
## Product Ecosystem 👬
|
||||
Xiaozhi is an ecosystem. When using this product, you might want to check out other excellent projects in this ecosystem:
|
||||
Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
|
||||
|
||||
| Project Name | Project Link | Description |
|
||||
| Project Name | Project Address | Project Description |
|
||||
|:---------------------|:--------|:--------|
|
||||
| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue |
|
||||
| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | A Python-based AI client that allows you to experience Xiaozhi AI functionality through code without physical hardware |
|
||||
| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | A Java-based open-source project providing complete backend service solutions |
|
||||
| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
|
||||
| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
|
||||
| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
|
||||
|
||||
---
|
||||
|
||||
## Supported Platforms/Components 📋
|
||||
## Supported Platforms/Components List 📋
|
||||
|
||||
### LLM Language Models
|
||||
|
||||
| Usage Method | Supported Platforms | Free Platforms |
|
||||
|:---:|:---:|:---:|
|
||||
| openai API | Ali Bailing, Volcano Engine Doubao, DeepSeek, ChatGLM, Gemini | ChatGLM, Gemini |
|
||||
| ollama API | Ollama | - |
|
||||
| dify API | Dify | - |
|
||||
| fastgpt API | Fastgpt | - |
|
||||
| coze API | Coze | - |
|
||||
| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
|
||||
| ollama interface call | Ollama | - |
|
||||
| dify interface call | Dify | - |
|
||||
| fastgpt interface call | Fastgpt | - |
|
||||
| coze interface call | Coze | - |
|
||||
|
||||
Actually, any LLM supporting openai API calls can be integrated.
|
||||
|
||||
---
|
||||
In fact, any LLM that supports openai interface calls can be integrated and used.
|
||||
|
||||
### TTS Speech Synthesis
|
||||
|
||||
| Usage Method | Supported Platforms | Free Platforms |
|
||||
|:---:|:---:|:---:|
|
||||
| API Calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Aliyun TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
|
||||
| API Call | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Alibaba Cloud TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
|
||||
| Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
|
||||
|
||||
---
|
||||
|
||||
### VAD Voice Activity Detection
|
||||
|
||||
| Type | Platform Name | Usage Method | Pricing | Notes |
|
||||
| Type | Platform Name | Usage Method | Pricing Model | Notes |
|
||||
|:---:|:---------:|:----:|:----:|:--:|
|
||||
| VAD | SileroVAD | Local Use | Free | |
|
||||
| VAD | SileroVAD | Local Usage | Free | |
|
||||
|
||||
---
|
||||
|
||||
@@ -229,26 +267,26 @@ Actually, any LLM supporting openai API calls can be integrated.
|
||||
|
||||
| Usage Method | Supported Platforms | Free Platforms |
|
||||
|:---:|:---:|:---:|
|
||||
| Local Use | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| API Calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
|
||||
| Local Usage | FunASR, SherpaASR | FunASR, SherpaASR |
|
||||
| API Call | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
|
||||
|
||||
---
|
||||
|
||||
### Memory Storage
|
||||
|
||||
| Type | Platform Name | Usage Method | Pricing | Notes |
|
||||
| Type | Platform Name | Usage Method | Pricing Model | Notes |
|
||||
|:------:|:---------------:|:----:|:---------:|:--:|
|
||||
| Memory | mem0ai | API Calls | 1000 calls/month quota | |
|
||||
| Memory | mem0ai | API Call | 1000 calls/month quota | |
|
||||
| Memory | mem_local_short | Local Summary | Free | |
|
||||
|
||||
---
|
||||
|
||||
### Intent Recognition
|
||||
|
||||
| Type | Platform Name | Usage Method | Pricing | Notes |
|
||||
| Type | Platform Name | Usage Method | Pricing Model | Notes |
|
||||
|:------:|:-------------:|:----:|:-------:|:---------------------:|
|
||||
| Intent | intent_llm | API Calls | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
|
||||
| Intent | function_call | API Calls | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
|
||||
| Intent | intent_llm | API Call | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
|
||||
| Intent | function_call | API Call | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
|
||||
|
||||
---
|
||||
|
||||
@@ -257,10 +295,10 @@ Actually, any LLM supporting openai API calls can be integrated.
|
||||
| Logo | Project/Company | Description |
|
||||
|:---:|:---:|:---|
|
||||
| <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
|
||||
| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for developing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing comprehensive technical documentation support for this project |
|
||||
| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
|
||||
| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering product user experience with their design experience serving over a thousand enterprises |
|
||||
| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
|
||||
| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for establishing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing full-chain technical documentation support for this project |
|
||||
| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing the function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
|
||||
| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering the product user experience with their design experience serving over a thousand enterprises |
|
||||
| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening the visual system of this project, ensuring consistency and extensibility of the overall design style in multi-scenario applications |
|
||||
|
||||
|
||||
<a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
|
||||
@@ -270,4 +308,4 @@ Actually, any LLM supporting openai API calls can be integrated.
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
|
||||
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
|
||||
</picture>
|
||||
</a>
|
||||
</a>
|
||||
|
||||
+5
-1
@@ -108,7 +108,11 @@ VAD:
|
||||
|
||||
参考教程[阿里云短信集成指南](./ali-sms-integration.md)
|
||||
|
||||
### 9、更多问题，可联系我们反馈 💬
|
||||
### 9、如何开启视觉模型实现拍照识物 📷
|
||||
|
||||
参考教程[视觉模型使用指南](./mcp-vision-integration.md)
|
||||
|
||||
### 10、更多问题，可联系我们反馈 💬
|
||||
|
||||
可以在[issues](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues)提交您的问题。
|
||||
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
# Technical Documentation: xiaozhi-esp32-server
|
||||
|
||||
**Table of Contents:**
|
||||
|
||||
1. [Introduction](#1-introduction)
|
||||
2. [Overall Architecture](#2-overall-architecture)
|
||||
3. [Component Deep Dive](#3-component-deep-dive)
|
||||
* [xiaozhi-server (Python AI Engine)](#31-xiaozhi-server-python-ai-engine)
|
||||
* [manager-api (Java Management Backend)](#32-manager-api-java-management-backend)
|
||||
* [manager-web (Vue.js Management Frontend)](#33-manager-web-vuejs-management-frontend)
|
||||
4. [Data Flow and Interaction Mechanisms](#4-data-flow-and-interaction-mechanisms)
|
||||
5. [Key Features Summary](#5-key-features-summary)
|
||||
6. [Deployment and Configuration Overview](#6-deployment-and-configuration-overview)
|
||||
|
||||
---
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
The `xiaozhi-esp32-server` project provides a comprehensive backend system designed to power intelligent voice interactions for ESP32-based smart hardware. Its primary purpose is to enable developers to quickly establish a robust server infrastructure capable of understanding natural language commands, interacting with various AI services (for speech recognition, language understanding, and speech synthesis), managing IoT devices, and offering a web-based interface for system configuration and administration. This project facilitates the creation of customizable voice assistants and smart control systems by integrating multiple cutting-edge technologies into a cohesive and extensible platform.
|
||||
|
||||
---
|
||||
|
||||
## 2. Overall Architecture
|
||||
|
||||
The `xiaozhi-esp32-server` system is architected as a distributed suite of interconnected components, each with a distinct role, ensuring modularity and scalability. The primary components are:
|
||||
|
||||
1. **ESP32 Hardware (Client Device):** This is the physical smart hardware device that the end-user interacts with. It's responsible for:
|
||||
* Capturing user's voice commands.
|
||||
* Sending captured audio to the `xiaozhi-server`.
|
||||
* Receiving synthesized audio responses from `xiaozhi-server` and playing them back.
|
||||
* Potentially controlling other connected peripherals or IoT devices based on commands from `xiaozhi-server`.
|
||||
|
||||
2. **`xiaozhi-server` (Core AI Engine):** This Python-based server is the central brain for voice processing and interaction logic. Its key responsibilities include:
|
||||
* Establishing real-time, bidirectional WebSocket communication with ESP32 devices.
|
||||
* Receiving audio streams and performing Voice Activity Detection (VAD).
|
||||
* Converting speech to text using integrated Automatic Speech Recognition (ASR) services.
|
||||
* Interpreting user intent and generating responses by interacting with Large Language Models (LLMs).
|
||||
* Managing dialogue context and memory.
|
||||
* Converting text responses back to speech using Text-to-Speech (TTS) services.
|
||||
* Executing commands, including IoT device control via a plugin system.
|
||||
* Fetching its operational configuration from the `manager-api`.
|
||||
|
||||
3. **`manager-api` (Management Backend):** A Java Spring Boot application that provides a RESTful API for system administration and configuration. It serves as the backend for the `manager-web` frontend and a configuration source for `xiaozhi-server`. Its functions include:
|
||||
* User authentication and management for the control panel.
|
||||
* Registration and management of ESP32 devices.
|
||||
* Storage and retrieval of system configurations (e.g., selected AI service providers, API keys, device settings) in a MySQL database.
|
||||
* Providing endpoints for `xiaozhi-server` to fetch its configuration.
|
||||
* Managing voice timbre settings, OTA firmware updates, and other system parameters.
|
||||
* Utilizing Redis for caching to enhance performance.
|
||||
|
||||
4. **`manager-web` (Web Control Panel):** A Vue.js Single Page Application (SPA) that provides a graphical user interface for administrators. It allows for:
|
||||
* Easy configuration of `xiaozhi-server`'s AI services and operational parameters.
|
||||
* Management of users, devices, and their respective settings.
|
||||
* Monitoring system status (potentially) and managing other administrative tasks.
|
||||
* Interaction with all backend functionalities exposed by `manager-api`.
|
||||
|
||||
**High-Level Interaction Flow:**
|
||||
|
||||
* The **ESP32** device captures voice and communicates primarily with **`xiaozhi-server`** via WebSockets for all voice-related interactions.
|
||||
* **`xiaozhi-server`** processes the voice data, interacts with various AI cloud services or local models, and sends responses back to the ESP32.
|
||||
* The **`manager-web`** frontend communicates with **`manager-api`** using RESTful HTTP calls to manage and configure the entire system.
|
||||
* **`xiaozhi-server`** also communicates with **`manager-api`** (via REST) to pull its latest configuration, ensuring that changes made in the web panel are reflected in its operation.
|
||||
|
||||
This separation of concerns allows the `xiaozhi-server` to focus on efficient real-time AI processing, while the `manager-api` and `manager-web` provide a robust and user-friendly interface for administration and setup.
|
||||
|
||||
---
|
||||
|
||||
## 3. Component Deep Dive
|
||||
|
||||
### 3.1. `xiaozhi-server` (Python AI Engine)
|
||||
|
||||
The `xiaozhi-server` is the intelligent core of the system, responsible for processing voice interactions, interfacing with AI services, and managing communication with ESP32 devices.
|
||||
|
||||
* **Purpose:**
|
||||
* To provide real-time processing of voice commands from ESP32 devices.
|
||||
* To integrate with various AI services for Speech-to-Text (ASR), Natural Language Understanding (via Large Language Models - LLMs), Text-to-Speech (TTS), Voice Activity Detection (VAD), Intent Recognition, and Memory.
|
||||
* To manage dialogue flow and context with users.
|
||||
* To execute custom functions and control IoT devices based on user commands.
|
||||
* To be dynamically configurable through the `manager-api`.
|
||||
|
||||
* **Core Technologies:**
|
||||
* **Python 3:** The primary programming language.
|
||||
* **Asyncio:** Python's asynchronous programming framework, crucial for handling concurrent WebSocket connections and non-blocking I/O for AI service API calls.
|
||||
* **`websockets` Library:** For WebSocket server implementation.
|
||||
* **HTTP Client (e.g., `aiohttp`, `httpx`):** For asynchronous HTTP requests to `manager-api` and external AI services.
|
||||
* **YAML (PyYAML):** For local configuration file parsing.
|
||||
|
||||
* **Key Implementation Aspects:**
|
||||
|
||||
1. **AI Service Provider Pattern (`core/providers/`):**
|
||||
* **Concept:** A flexible design for integrating AI services. Each service type (ASR, TTS, LLM, etc.) has an abstract base class defining a common interface. Concrete classes implement this interface for specific vendors or local models.
|
||||
* **Benefit:** Allows easy switching of AI service backends via configuration and simplifies adding new service integrations.
|
||||
* **Initialization:** `core/utils/modules_initialize.py` acts as a factory to load and instantiate configured providers.
|
||||
|
||||
2. **WebSocket Communication & Connection Handling (`core/websocket_server.py`, `core/connection.py`):**
|
||||
* **Server Setup:** Manages WebSocket connections from ESP32 devices.
|
||||
* **Connection Isolation:** Each ESP32 client gets a dedicated `ConnectionHandler` instance, isolating its session state and dialogue.
|
||||
* **Dynamic Configuration Updates:** Can fetch updated configurations from `manager-api` and re-initialize AI service modules live, without a full server restart.
|
||||
|
||||
3. **Message Handling & Dialogue Flow (`core/handle/`):**
|
||||
* Employs a modular handler pattern. The `ConnectionHandler` dispatches message processing to specialized modules based on message type or dialogue phase (e.g., `receiveAudioHandle.py` for audio input, `intentHandler.py` for NLU, `functionHandler.py` for plugin execution, `sendAudioHandle.py` for TTS output).
|
||||
|
||||
4. **Plugin System for Extensible Functions (`plugins_func/`):**
|
||||
* **Purpose:** Allows adding custom "skills" (e.g., weather, news, Home Assistant control).
|
||||
* **Mechanism:** Plugins define functions and schemas. The LLM can request execution of these functions (function calling). `loadplugins.py` and `register.py` manage plugin discovery and registration.
|
||||
|
||||
5. **Configuration Management (`config/`):**
|
||||
* Loads settings from a local `config.yaml` and merges them with configurations fetched from `manager-api` (via `manage_api_client.py`), enabling remote dynamic configuration.
|
||||
* `logger.py` sets up structured application logging.
|
||||
* `config/assets/` stores predefined audio files for system notifications.
|
||||
|
||||
6. **Auxiliary HTTP Server (`core/http_server.py`):**
|
||||
* Handles specific HTTP requests, notably for OTA firmware updates (`/xiaozhi/ota/`) and other utility endpoints.
|
||||
|
||||
### 3.2. `manager-api` (Java Management Backend)
|
||||
|
||||
The `manager-api` component is a backend server built using Java and the Spring Boot framework, serving as the administrative hub.
|
||||
|
||||
* **Purpose:**
|
||||
* Provide a secure RESTful API for the `manager-web` frontend.
|
||||
* Act as a centralized configuration provider for `xiaozhi-server`.
|
||||
* Manage persistent data (users, devices, AI configurations, voice timbres, OTA firmware).
|
||||
|
||||
* **Core Technologies:**
|
||||
* **Java 21 & Spring Boot 3:** Core language and framework.
|
||||
* **Spring MVC:** For building REST controllers.
|
||||
* **MyBatis-Plus:** ORM for database interaction with MySQL.
|
||||
* **MySQL:** Relational database.
|
||||
* **Druid:** JDBC connection pool.
|
||||
* **Redis (Spring Data Redis):** For caching.
|
||||
* **Apache Shiro:** Security framework for authentication and authorization.
|
||||
* **Liquibase:** Database schema migration.
|
||||
* **Knife4j:** OpenAPI (Swagger) API documentation.
|
||||
* **Maven:** Build and dependency management.
|
||||
|
||||
* **Key Implementation Aspects:**
|
||||
|
||||
1. **Modular Architecture (`modules/` package):**
|
||||
* Business logic is organized into distinct modules (e.g., `sys` for users/roles, `agent` for assistant configs, `device` for ESP32s, `config` for `xiaozhi-server` settings, `security`, `timbre`, `ota`).
|
||||
* Each module typically follows a layered pattern: Controller, Service, DAO (Mapper), Entity, DTO.
|
||||
|
||||
2. **Layered Architecture:**
|
||||
* **Controller Layer (`@RestController`):** Defines API endpoints, handles HTTP request/response.
|
||||
* **Service Layer (`@Service`):** Contains business logic, transaction management.
|
||||
* **Data Access Layer (MyBatis-Plus Mappers):** Interacts with the MySQL database.
|
||||
|
||||
3. **Common Functionalities (`common/` package):**
|
||||
* Provides shared code: base classes, global configurations (Spring, MyBatis, Redis, Knife4j), custom annotations (e.g., `@LogOperation`), AOP aspects, global exception handling, utility classes, and XSS protection.
|
||||
|
||||
4. **Security (Apache Shiro):**
|
||||
* Manages user authentication and permissions for accessing API endpoints. Configured with Shiro Realms and security filters.
|
||||
|
||||
5. **Database Schema Management (Liquibase):**
|
||||
* Ensures consistent database structure across environments through versioned schema changes.
|
||||
|
||||
### 3.3. `manager-web` (Vue.js Management Frontend)
|
||||
|
||||
The `manager-web` is a Single Page Application (SPA) providing the administrative user interface.
|
||||
|
||||
* **Purpose:**
|
||||
* Offer a web-based control panel for system configuration and management.
|
||||
* Enable administrators to configure `xiaozhi-server`'s AI services, manage users and devices, customize voice timbres, and handle OTA updates.
|
||||
|
||||
* **Core Technologies:**
|
||||
* **Vue.js 2 & Vue CLI:** Core JavaScript framework and build tools.
|
||||
* **Vue Router:** For client-side routing within the SPA.
|
||||
* **Vuex:** For centralized state management.
|
||||
* **Element UI:** UI component library for a consistent look and feel.
|
||||
* **SCSS:** CSS preprocessor.
|
||||
* **HTTP Client (Flyio or Axios):** For API calls to `manager-api`.
|
||||
* **Workbox:** For PWA features (caching, service worker).
|
||||
* **Opus Libraries:** For potential in-browser audio recording/playback.
|
||||
|
||||
* **Key Implementation Aspects:**
|
||||
|
||||
1. **SPA Structure:** Single HTML page with dynamic view updates.
|
||||
2. **Component-Based Architecture:** UI built from reusable Vue components (`.vue` files in `src/views/` for pages and `src/components/` for smaller elements).
|
||||
3. **Client-Side Routing (`src/router/index.js`):** Maps browser URLs to view components, with route guards for authentication.
|
||||
4. **State Management (`src/store/index.js`):** Vuex manages global state (user info, device lists, etc.) via state, getters, mutations, and actions (often involving API calls).
|
||||
5. **API Communication (`src/apis/`):** Modularized API service files make asynchronous calls to `manager-api`.
|
||||
6. **Build Process & PWA Features:** Vue CLI (Webpack) bundles assets. Workbox enables PWA features like caching.
|
||||
7. **Environment Configuration (`.env` files):** Manages settings like the `manager-api` base URL for different environments.
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Flow and Interaction Mechanisms
|
||||
|
||||
The system uses WebSockets for real-time voice interactions and RESTful APIs for management tasks.
|
||||
|
||||
* **Core Voice Interaction (ESP32 <-> `xiaozhi-server` - WebSockets):**
|
||||
* ESP32 connects to `xiaozhi-server` via WebSocket.
|
||||
* Audio is streamed from ESP32 to server.
|
||||
* Server processes audio (VAD, ASR), interacts with LLM (possibly executing plugin functions), synthesizes response via TTS.
|
||||
* Synthesized audio is streamed back to ESP32.
|
||||
* JSON control/status messages are also exchanged.
|
||||
|
||||
* **Management & Configuration (RESTful APIs - HTTP/JSON):**
|
||||
* **`manager-web` -> `manager-api`:** Admin actions in the web UI trigger REST API calls to `manager-api` for managing users, devices, configurations, etc. Shiro secures these endpoints.
|
||||
* **`xiaozhi-server` -> `manager-api`:** `xiaozhi-server` pulls its operational configuration from `manager-api` via REST API calls.
|
||||
|
||||
* **OTA Updates (Conceptual - HTTP & WebSocket):**
|
||||
* Firmware uploaded via `manager-web` to `manager-api`.
|
||||
* `xiaozhi-server` may notify ESP32 of updates via WebSocket.
|
||||
* ESP32 downloads firmware via HTTP from an endpoint (likely on `xiaozhi-server`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Key Features Summary
|
||||
|
||||
* **Modular AI Services:** Pluggable ASR, LLM, TTS, VAD, Intent, Memory.
|
||||
* **Advanced Dialogue:** Real-time interruption, contextual memory, multi-language support.
|
||||
* **Extensible Skills:** Plugin system for custom functions (e.g., IoT, Home Assistant).
|
||||
* **Comprehensive Web Management:** UI for users, devices, AI configs, OTA, timbres.
|
||||
* **Flexible Deployment:** Docker (simplified/full) and source code options.
|
||||
* **Dynamic Remote Configuration:** `xiaozhi-server` updates settings from `manager-api` live.
|
||||
* **Open Source (MIT License).**
|
||||
* **Cost-Effective Options:** "Entry Level Free Settings" available.
|
||||
* **PWA Admin Panel:** Enhanced caching and user experience.
|
||||
* **API Documentation:** Knife4j for `manager-api`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Deployment and Configuration Overview
|
||||
|
||||
* **Deployment:**
|
||||
* **Docker:** Recommended for ease. Options for `xiaozhi-server` only or full stack (all components + databases). `docker-compose.yml` files provided.
|
||||
* **Source Code:** For development or custom setups, requiring manual environment setup for Python, Java/Maven, and Node.js.
|
||||
|
||||
* **Configuration:**
|
||||
* **`xiaozhi-server`:** Uses a local `config.yaml`, but primarily pulls dynamic configurations (AI providers, API keys) from `manager-api` via its `manage_api_client.py`.
|
||||
* **`manager-api`:** Configured via Spring Boot's `application.properties` or `application.yml` (database, Redis, Shiro settings).
|
||||
* **`manager-web`:** Configured via `.env` files (e.g., `manager-api` URL).
|
||||
* The `manager-web` UI is the primary interface for most system configurations in a full deployment.
|
||||
* Predefined profiles like "Entry Level Free Settings" and "Full Streaming Configuration" guide AI service choices.
|
||||
|
||||
---
|
||||
@@ -0,0 +1,450 @@
|
||||
# 技术文档：`xiaozhi-esp32-server`
|
||||
|
||||
**目录：**
|
||||
|
||||
1. [引言](#1-引言)
|
||||
2. [整体架构](#2-整体架构)
|
||||
3. [核心组件深度剖析](#3-核心组件深度剖析)
|
||||
* [3.1. `xiaozhi-server` (核心AI引擎 - Python实现)](#31-xiaozhi-server-核心ai引擎---python实现)
|
||||
* [3.2. `manager-api` (管理后端 - Java Spring Boot实现)](#32-manager-api-管理后端---java-spring-boot实现)
|
||||
* [3.3. `manager-web` (Web管理前端 - Vue.js实现)](#33-manager-web-web管理前端---vuejs实现)
|
||||
4. [数据流与交互机制](#4-数据流与交互机制)
|
||||
5. [核心功能概要](#5-核心功能概要)
|
||||
6. [部署与配置概述](#6-部署与配置概述)
|
||||
|
||||
---
|
||||
|
||||
## 1. 引言
|
||||
|
||||
`xiaozhi-esp32-server` 项目是一个专为基于ESP32的智能硬件提供支持的**综合性后端系统**。其核心目标是使开发人员能够快速构建一个强大的服务器基础设施，该设施不仅能够理解自然语言指令，还能与多种AI服务（用于语音识别、自然语言理解及语音合成）进行高效交互、管理物联网（IoT）设备，并提供一个基于Web的用户界面以进行系统配置和管理。通过将多种尖端技术整合到一个高内聚且可扩展的平台中，本项目旨在简化和加速可定制化语音助手及智能控制系统的开发进程。它不仅仅是一个简单的服务器，更是一个连接硬件、AI能力与用户管理的桥梁。
|
||||
|
||||
---
|
||||
|
||||
## 2. 整体架构
|
||||
|
||||
`xiaozhi-esp32-server` 系统采用了一种**分布式、多组件协作**的架构设计，确保了系统的模块化、可维护性和可扩展性。各个核心组件各司其职，协同工作。主要组件包括：
|
||||
|
||||
1. **ESP32 硬件 (客户端设备):**
|
||||
这是终端用户直接与之交互的物理智能硬件设备。其主要职责包括：
|
||||
* 捕捉用户的语音指令。
|
||||
* 将捕捉到的原始音频数据安全地发送至 `xiaozhi-server` 进行处理。
|
||||
* 接收来自 `xiaozhi-server` 合成的语音回复，并通过扬声器播放给用户。
|
||||
* 根据从 `xiaozhi-server` 收到的指令，控制与之连接的其他外围设备或IoT设备（例如智能灯泡、传感器等）。
|
||||
|
||||
2. **`xiaozhi-server` (核心AI引擎 - Python实现):**
|
||||
这个基于Python的服务器是整个系统的“大脑”，负责处理所有语音相关的逻辑和AI交互。其关键职责细化如下：
|
||||
* 通过WebSocket协议与ESP32设备建立**稳定、低延迟的实时双向通信链路**。
|
||||
* 接收来自ESP32的音频流，并利用语音活动检测（VAD）技术精确切分有效的语音片段。
|
||||
* 集成并调用自动语音识别（ASR）服务（可配置本地或云端），将语音片段转换为文本。
|
||||
* 通过与大型语言模型（LLM）的交互来解析用户意图、生成智能回复，并支持复杂的自然语言理解任务。
|
||||
* 管理多轮对话中的上下文信息和用户记忆，以提供连贯的交互体验。
|
||||
* 调用文本转语音（TTS）服务，将LLM生成的文本回复合成为自然流畅的语音。
|
||||
* 通过一个灵活的**插件系统**执行自定义命令，包括对IoT设备的控制逻辑。
|
||||
* 从 `manager-api` 服务获取其详细的运行时操作配置。
|
||||
|
||||
3. **`manager-api` (管理后端 - Java实现):**
|
||||
这是一个基于Java Spring Boot框架构建的应用程序，它为整个系统的管理和配置提供了一套安全的RESTful API。它不仅是 `manager-web` 控制台的后端支撑，也是 `xiaozhi-server` 的配置数据来源。其核心功能包括：
|
||||
* 为Web控制台提供用户认证（登录、权限验证）和用户账户管理功能。
|
||||
* ESP32设备的注册、信息管理以及设备特定配置的维护。
|
||||
* 在**MySQL数据库**中持久化存储系统配置，例如用户选择的AI服务提供商、API密钥、设备参数、插件设置等。
|
||||
* 提供特定的API端点，供 `xiaozhi-server` 拉取其所需的最新配置。
|
||||
* 管理TTS音色选项、处理OTA（Over-The-Air）固件更新流程及相关元数据。
|
||||
* 利用 **Redis** 作为高速缓存，存储热点数据（如会话信息、频繁访问的配置），以提升API响应速度和系统整体性能。
|
||||
|
||||
4. **`manager-web` (Web控制面板 - Vue.js实现):**
|
||||
这是一个基于Vue.js构建的单页应用（SPA），为系统管理员提供了一个图形化、用户友好的操作界面。其主要能力包括：
|
||||
* 便捷地配置 `xiaozhi-server` 所使用的各项AI服务（如ASR、LLM、TTS的提供商切换、参数调整）。
|
||||
* 管理平台用户账户、角色分配及权限控制。
|
||||
* 管理已注册的ESP32设备及其相关设置。
|
||||
* （潜在功能）监控系统运行状态、查看日志、进行故障排查等。
|
||||
* 与 `manager-api` 提供的所有后端管理功能进行全面的交互。
|
||||
|
||||
**高层交互流程概述:**
|
||||
|
||||
* **语音交互主线:** **ESP32设备**捕捉到用户语音后，通过**WebSocket**将音频数据实时传输给**`xiaozhi-server`**。`xiaozhi-server`完成一系列AI处理（VAD、ASR、LLM交互、TTS）后，再通过WebSocket将合成的语音回复发送回ESP32设备进行播放。所有与语音直接相关的实时交互均在此链路完成。
|
||||
* **管理配置主线:** 管理员通过浏览器访问**`manager-web`**控制台。`manager-web`通过调用**`manager-api`**提供的**RESTful HTTP接口**来执行各种管理操作（如修改配置、管理用户或设备）。数据以JSON格式在两者间传递。
|
||||
* **配置同步:** **`xiaozhi-server`**在启动或特定更新机制触发时，会主动通过HTTP请求从**`manager-api`**拉取其最新的操作配置。这确保了管理员在Web界面上所做的配置更改能够及时有效地应用到核心AI引擎的运行中。
|
||||
|
||||
这种**前后端分离、核心服务与管理服务分离**的架构设计，使得 `xiaozhi-server`能够专注于高效的实时AI处理任务，而 `manager-api` 和 `manager-web` 则共同提供了一个功能强大且易于使用的管理和配置平台。各组件职责清晰，有利于独立开发、测试、部署和扩展。
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心组件深度剖析
|
||||
|
||||
### 3.1. `xiaozhi-server` (核心AI引擎 - Python实现)
|
||||
|
||||
`xiaozhi-server` 作为系统的智能核心，全权负责处理语音交互、对接各类AI服务以及管理与ESP32设备间的通信。其设计目标是实现高效、灵活且可扩展的语音AI处理能力。
|
||||
|
||||
* **核心目标:**
|
||||
* 为ESP32设备提供实时的语音指令处理服务。
|
||||
* 深度集成各类AI服务，包括：自动语音识别 (ASR)、大型语言模型 (LLM) 进行自然语言理解 (NLU)、文本转语音 (TTS)、语音活动检测 (VAD)、意图识别 (Intent Recognition) 及对话记忆 (Memory)。
|
||||
* 精细管理用户与设备间的对话流程及上下文状态。
|
||||
* 基于用户指令，通过插件化机制执行自定义函数及控制物联网 (IoT) 设备。
|
||||
* 支持通过 `manager-api`进行动态配置加载与更新。
|
||||
|
||||
* **核心技术栈:**
|
||||
* **Python 3:** 作为主要编程语言，Python以其丰富的AI/ML生态库和快速开发特性被选用。
|
||||
* **Asyncio:** Python的异步编程框架，是`xiaozhi-server`高性能的关键。它被广泛用于高效处理来自大量ESP32设备的并发WebSocket连接，以及执行与外部AI服务API通信时的非阻塞I/O操作，确保服务器在高并发下的响应能力。
|
||||
* **`websockets` 库:** 提供WebSocket服务器的具体实现，支持与ESP32客户端进行全双工实时通信。
|
||||
* **HTTP客户端 (如 `aiohttp`, `httpx`):** 用于异步执行HTTP请求，主要目的是从`manager-api`获取配置信息，以及与云端AI服务的API进行交互。
|
||||
* **YAML (通常通过 PyYAML 库):** 用于解析本地的 `config.yaml` 配置文件。
|
||||
* **FFmpeg (外部依赖):** 在 `app.py` 启动时会进行检查 (`check_ffmpeg_installed()`)。FFmpeg通常用于音频处理和格式转换，例如，确保音频数据符合特定AI服务的要求或进行内部处理。
|
||||
|
||||
* **关键实现细节:**
|
||||
|
||||
1. **AI服务提供者模式 (Provider Pattern - `core/providers/`):**
|
||||
* **设计思想:** 这是`xiaozhi-server`集成不同AI服务的核心设计模式，极大地增强了系统的灵活性和可扩展性。针对每一种AI服务类型（ASR, TTS, LLM, VAD, Intent, Memory, VLLM），都在其对应子目录下定义了一个抽象基类 (ABC, Abstract Base Class)，例如 `core/providers/asr/base.py`。这个基类规定了该类型服务必须实现的通用接口方法（如ASR的 `async def transcribe(self, audio_chunk: bytes) -> str: pass`）。
|
||||
* **具体实现:** 各种具体的AI服务提供商或本地模型的实现，则以独立的Python类形式存在（例如 `core/providers/asr/fun_local.py` 实现了本地FunASR的逻辑，`core/providers/llm/openai.py` 实现了与OpenAI GPT模型的对接）。这些具体类继承自相应的抽象基类，并实现其定义的接口。部分提供者还使用DTOs (Data Transfer Objects, 存在于各自的 `dto/` 目录) 来结构化与外部服务交换的数据。
|
||||
* **优势:** 使得核心业务逻辑能够以统一的方式调用不同的AI服务，而无需关心其底层具体实现。用户可以通过配置文件轻松切换AI服务后端。添加对新AI服务的支持也变得相对简单，只需实现对应的Provider接口。
|
||||
* **动态加载与初始化:** `core/utils/modules_initialize.py` 脚本扮演了工厂的角色。它在服务器启动时，或在接收到配置更新指令时，会根据配置文件中 `selected_module` 及各项服务的具体provider设置，动态地导入并实例化相应的Provider类。
|
||||
|
||||
2. **WebSocket通信与连接处理 (`app.py`, `core/websocket_server.py`, `core/connection.py`):**
|
||||
* **服务器启动与入口 (`app.py`):**
|
||||
* `app.py` 作为主入口，负责初始化应用环境（如检查FFmpeg、加载配置、设置日志）。
|
||||
* 它会生成或加载一个 `auth_key` (JWT密钥)，用于保护特定的HTTP接口（如视觉分析接口 `/mcp/vision/explain`）。若配置中 `manager-api.secret` 为空，则会生成一个UUID作为 `auth_key`。
|
||||
* 使用 `asyncio.create_task()` 并发启动 `WebSocketServer` (监听如 `ws://0.0.0.0:8000/xiaozhi/v1/`) 和 `SimpleHttpServer` (监听如 `http://0.0.0.0:8003/xiaozhi/ota/`)。
|
||||
* 包含一个 `monitor_stdin()` 协程，用于在某些环境下保持应用存活或处理终端输入。
|
||||
* **WebSocket服务器核心 (`core/websocket_server.py`):**
|
||||
* `WebSocketServer` 类使用 `websockets` 库监听来自ESP32设备的连接请求。
|
||||
* 对于每一个成功的WebSocket连接，它都会创建一个**独立的 `ConnectionHandler` 实例** (推测定义于 `core/connection.py`)。这种每个连接一个处理程序实例的设计模式，是实现多设备状态隔离和并发处理的关键，确保每个设备的对话流程和上下文信息互不干扰。
|
||||
* 该服务器还提供一个 `_http_response` 方法，允许在同一端口上对非WebSocket升级的HTTP GET请求做出简单响应（例如返回 "Server is running"），便于进行健康检查。
|
||||
* **动态配置更新:** `WebSocketServer` 包含一个 `update_config()` 异步方法。此方法使用 `config_lock` (一个 `asyncio.Lock`) 保证配置更新的原子性。它调用 `get_config_from_api()` (可能在 `config_loader.py` 中实现，通过 `manage_api_client.py` 与 `manager-api` 通信) 来获取新的配置。通过 `check_vad_update()` 和 `check_asr_update()` 等辅助函数判断是否需要重新初始化特定的AI模块，避免不必要的开销。更新后的配置会用于重新调用 `initialize_modules()`，从而实现AI服务提供者的热切换。
|
||||
|
||||
3. **消息处理与对话流程控制 (`core/handle/` 和 `ConnectionHandler`):**
|
||||
* `ConnectionHandler` (推测) 作为每个连接的控制中心，负责接收来自ESP32的消息，并根据消息类型或当前对话状态，将其分发给 `core/handle/` 目录下的相应处理模块。这种模块化的处理器设计使得 `ConnectionHandler` 逻辑更清晰，易于扩展。
|
||||
* **主要处理模块及其职责:**
|
||||
* `helloHandle.py`: 处理与ESP32初次连接时的握手协议、设备认证或初始化信息交换。
|
||||
* `receiveAudioHandle.py`: 接收音频流数据，调用VAD Provider进行语音活动检测，并将有效的音频片段传递给ASR Provider进行识别。
|
||||
* `textHandle.py` / `intentHandler.py`: 获取ASR识别出的文本后，与Intent Provider (可能利用LLM进行意图识别) 和LLM Provider交互，以理解用户意图并生成初步回复或决策。
|
||||
* `functionHandler.py`: 当LLM的响应包含执行特定“函数调用”的指令时，此模块负责从插件注册表中查找并执行对应的插件函数。
|
||||
* `sendAudioHandle.py`: 将LLM最终生成的文本回复交给TTS Provider合成语音，并将音频流通过WebSocket发送回ESP32。
|
||||
* `abortHandle.py`: 处理来自ESP32的中断请求，例如停止当前的TTS播报。
|
||||
* `iotHandle.py`, `mcpHandle.py`: 处理与IoT设备控制相关的特定指令或更复杂的模块通信协议 (MCP)。
|
||||
|
||||
4. **插件化功能扩展系统 (`plugins_func/`):**
|
||||
* **设计目的:** 提供一种标准化的方式来扩展语音助手的功能和“技能”，而无需修改核心代码。
|
||||
* **实现机制:**
|
||||
* 各个具体功能以独立的Python脚本形式存在于 `plugins_func/functions/` 目录中（例如 `get_weather.py`, `hass_set_state.py` 用于Home Assistant集成）。
|
||||
* `loadplugins.py` 在服务器启动时负责扫描并加载这些插件模块。
|
||||
* `register.py` (或插件模块内部的特定装饰器/函数) 可能用于定义每个插件函数的元数据，包括：
|
||||
* **函数名称 (Function Name):** LLM调用时使用的标识符。
|
||||
* **功能描述 (Description):** 供LLM理解此函数的作用。
|
||||
* **参数模式 (Parameters Schema):** 通常是一个JSON Schema，详细定义了函数所需的参数、类型、是否必需以及描述。这是LLM能够正确生成函数调用参数的关键。
|
||||
* **执行流程:** 当LLM在其思考过程中决定需要调用某个外部工具或函数来获取信息或执行操作时，它会依据预先提供的函数模式生成一个结构化的“函数调用”请求。`xiaozhi-server`中的`functionHandler.py`捕获此请求，从插件注册表中找到对应的Python函数并执行，然后将执行结果返回给LLM，LLM再基于此结果生成最终给用户的自然语言回复。
|
||||
|
||||
5. **配置管理 (`config/`):**
|
||||
* **加载机制:** `config_loader.py` (通过 `settings.py` 被调用) 负责从根目录的 `config.yaml` 文件加载基础配置。
|
||||
* **远程配置与合并:** 通过 `manage_api_client.py` (使用如`aiohttp`的库与`manager-api`通信) 可以从`manager-api`服务拉取配置。远程配置通常会覆盖本地 `config.yaml` 中的同名设置，从而实现通过Web界面动态调整服务器行为。
|
||||
* **日志系统:** `logger.py` 初始化应用日志系统（可能使用 `loguru` 或对标准 `logging` 模块进行封装，支持通过 `logger.bind(tag=TAG)` 添加标签，便于追踪和过滤）。
|
||||
* **静态资源:** `config/assets/` 目录下存放了用于系统提示音的静态音频文件（如设备绑定提示音 `bind_code.wav`、错误提示音等）。
|
||||
|
||||
6. **辅助HTTP服务 (`core/http_server.py`):**
|
||||
* 与WebSocket服务并行运行一个简单的HTTP服务器，用于处理特定的HTTP请求。最主要的功能是为ESP32设备提供OTA (Over-The-Air) 固件更新的下载服务 (通过 `/xiaozhi/ota/` 端点)。此外，也可能承载其他如 `/mcp/vision/explain` (视觉分析) 等工具性HTTP接口。
|
||||
|
||||
综上所述，`xiaozhi-server` 是一个采用现代Python异步编程模型构建的、高度模块化、配置驱动的AI应用服务器。其精心设计的Provider模式和插件架构赋予了它强大的适应性和扩展性，能够灵活接入不同的AI能力并支持日益增长的功能需求。
|
||||
|
||||
---
|
||||
|
||||
### 3.2. `manager-api` (管理后端 - Java Spring Boot实现)
|
||||
|
||||
`manager-api` 组件是使用Java和Spring Boot框架构建的强大后端服务，作为整个`xiaozhi-esp32-server`生态系统的中央行政管理和配置中枢。
|
||||
|
||||
* **核心目标:**
|
||||
* 为`manager-web`（Vue.js前端）提供一套安全、稳定、符合RESTful规范的API接口，使得管理员能够便捷地管理用户、设备、系统配置及其他相关资源。
|
||||
* 充当`xiaozhi-server`（Python核心AI引擎）的集中化配置数据提供者，允许`xiaozhi-server`实例在启动或运行时获取其最新的操作参数。
|
||||
* 持久化存储关键数据，例如：用户账户信息、设备注册详情、AI服务提供商配置（包括API密钥、选定的服务模型等）、TTS音色参数，以及OTA固件版本信息等。
|
||||
|
||||
* **核心技术栈:**
|
||||
* **Java 21:** 项目采用的JDK版本，确保了对现代Java特性的支持。
|
||||
* **Spring Boot 3:** 作为核心开发框架，极大地简化了独立、生产级别的Spring应用的创建和部署。它提供了自动配置、内嵌Web服务器（默认为Tomcat）、依赖管理等关键功能。
|
||||
* **Spring MVC:** Spring框架中用于构建Web应用和RESTful API的模块。
|
||||
* **MyBatis-Plus:** 一个对MyBatis进行功能增强的ORM（对象关系映射）框架。它简化了数据库操作，提供了强大的CRUD（增删改查）功能、条件构造器、代码生成器等，并能很好地与Spring Boot集成。
|
||||
* **MySQL:** 作为主要的后端关系型数据库，用于存储所有需要持久化的管理数据和配置信息。
|
||||
* **Druid (Alibaba Druid):** 一个功能强大的JDBC连接池实现，提供了丰富的监控功能和优秀的性能，用于高效管理数据库连接。
|
||||
* **Redis (通过 Spring Data Redis):** 一个高性能的内存数据结构存储，常用于实现数据缓存（例如缓存热点配置数据、用户会话信息），以显著提升API的响应速度。
|
||||
* **Apache Shiro:** 一个成熟且易用的Java安全框架，负责处理应用的认证（用户身份验证）和授权（API访问权限控制）需求。
|
||||
* **Liquibase:** 一个用于跟踪、管理和应用数据库 schéma（模式）变更的开源工具。它允许开发者以数据库无关的方式定义和版本化数据库结构变更。
|
||||
* **Knife4j:** 一个集成了Swagger并增强了UI的API文档生成工具，专为Java MVC框架（尤其是Spring Boot）设计。它能生成美观且易于交互的API文档界面（通常通过 `/xiaozhi/doc.html` 访问）。
|
||||
* **Maven:** 用于项目的构建自动化和依赖项管理。
|
||||
* **Lombok:** 一个Java库，通过注解自动生成构造函数、getter/setter、equals/hashCode、toString等样板代码，减少冗余。
|
||||
* **HuTool / Google Guava:** 提供大量实用工具类，简化常见编程任务。
|
||||
* **Aliyun Dysmsapi:** 阿里云短信服务SDK，用于集成发送短信功能（如验证码、通知）。
|
||||
|
||||
* **关键实现细节:**
|
||||
|
||||
1. **模块化项目结构 (`modules/` 包):**
|
||||
* `manager-api` 的核心业务逻辑被清晰地划分到 `src/main/java/xiaozhi/modules/` 目录下的不同模块中。这种按功能领域划分模块的方式（例如 `sys` 负责系统管理，`agent` 负责智能体配置，`device` 负责设备管理，`config` 负责为`xiaozhi-server`提供配置，`security` 负责安全，`timbre` 负责音色管理，`ota` 负责固件升级）极大地提高了代码的可维护性和可扩展性。
|
||||
* **各模块内部结构:** 每个业务模块通常遵循经典的三层架构或其变体：
|
||||
* **Controller (控制层):** 位于 `xiaozhi.modules.[模块名].controller`。
|
||||
* **Service (服务层):** 位于 `xiaozhi.modules.[模块名].service`。
|
||||
* **DAO/Mapper (数据访问层):** 位于 `xiaozhi.modules.[模块名].dao`。
|
||||
* **Entity (实体类):** 位于 `xiaozhi.modules.[模块名].entity`。
|
||||
* **DTO (数据传输对象):** 位于 `xiaozhi.modules.[模块名].dto`。
|
||||
|
||||
2. **分层架构实现:**
|
||||
* **Controller层 (`@RestController`):** 这些类使用Spring MVC注解（如 `@GetMapping`, `@PostMapping` 等）来定义API的端点(endpoints)。它们负责接收HTTP请求，将请求体中的JSON数据反序列化为DTO对象，调用相应的Service层方法处理业务逻辑，最后将Service层的返回结果序列化为JSON并作为HTTP响应返回给客户端。
|
||||
* **Service层 (`@Service`):** 这些类（通常是接口及其实现类的组合）封装了核心的业务规则和操作流程。它们可能会调用一个或多个DAO/Mapper对象来与数据库交互，并常常使用 `@Transactional` 注解来管理数据库事务的原子性。
|
||||
* **Data Access (DAO/Mapper) 层 (MyBatis-Plus Mappers):** 这些是Java接口，继承自MyBatis-Plus提供的 `BaseMapper<Entity>` 接口。MyBatis-Plus会为这些接口自动提供标准的CRUD方法。对于更复杂的数据库查询，开发者可以通过在Mapper接口中定义方法并使用注解（如 `@Select`, `@Update`）或编写对应的XML映射文件来实现。例如，`UserMapper.selectById(userId)` 会被MyBatis-Plus自动实现。
|
||||
* **Entity层 (`@TableName`, `@TableId` 等MyBatis-Plus注解):** 这些POJO（Plain Old Java Objects）类直接映射到数据库中的表结构。Lombok的 `@Data` 注解常用于自动生成getter/setter等。
|
||||
* **DTO层:** 用于在各层之间，特别是Controller层与Service层之间，以及API的请求/响应体中传递数据。使用DTO有助于解耦API接口的数据结构与数据库实体的数据结构，使API更稳定。
|
||||
|
||||
3. **通用功能与配置 (`common/` 包):**
|
||||
* `src/main/java/xiaozhi/common/` 包提供了一系列跨模块共享的通用组件和配置：
|
||||
* **基类:** 如 `BaseDao`, `BaseEntity`, `BaseService`, `CrudService`，为各模块的相应组件提供通用的属性或方法。
|
||||
* **全局配置:** 包括 `MybatisPlusConfig` (MyBatis-Plus的配置，如分页插件、数据权限插件等)、`RedisConfig` (Redis连接及序列化配置)、`SwaggerConfig` (Knife4j的配置)、`AsyncConfig` (异步任务执行器配置)。
|
||||
* **自定义注解:** 例如 `@LogOperation` 用于通过AOP记录操作日志，`@DataFilter` 可能用于实现数据范围过滤。
|
||||
* **AOP切面:** 如 `RedisAspect` 可能用于实现方法级别的缓存逻辑。
|
||||
* **全局异常处理:** `RenExceptionHandler` (使用 `@ControllerAdvice` 注解) 捕获应用中抛出的特定或所有异常 (如自定义的 `RenException`)，并返回统一格式的JSON错误响应给客户端。`ErrorCode` 定义了标准化的错误码。
|
||||
* **工具类:** 提供了日期转换、JSON处理(Jackson)、IP地址获取、HTTP上下文操作、统一结果封装 (`Result` 类)等多种实用工具。
|
||||
* **校验工具:** `ValidatorUtils` 和 `AssertUtils` 用于简化参数校验逻辑。
|
||||
* **XSS防护:** `XssFilter` 等组件用于防止跨站脚本攻击。
|
||||
* **MyBatis-Plus自动填充:** `FieldMetaObjectHandler` 用于在执行插入或更新数据库操作时，自动填充如 `createTime`, `updateTime` 等公共字段。
|
||||
|
||||
4. **安全机制 (Apache Shiro):**
|
||||
* Shiro的配置（通常在 `modules/security/config/` 或 `common/config/` 下）定义了如何进行用户认证和授权。
|
||||
* **Realms (域):** 自定义的Shiro Realm类负责从数据库中查询用户信息（用户名、密码、盐值）进行身份验证，以及获取用户的角色和权限信息用于授权决策。
|
||||
* **Filters (过滤器):** Shiro过滤器链被应用于保护API端点，确保只有经过认证且拥有足够权限的用户才能访问特定资源。
|
||||
* **Session/Token Management:** Shiro管理用户会话。对于RESTful API，可能结合OAuth2或JWT等令牌机制实现无状态认证。
|
||||
|
||||
5. **数据库版本控制 (Liquibase):**
|
||||
* 数据库的表结构、索引、初始数据等变更，都通过Liquibase的 `changelog` 文件（通常是XML格式）进行定义和版本化管理。当应用启动时，Liquibase会自动检查并应用必要的数据库结构更新，确保开发、测试和生产环境数据库结构的一致性。
|
||||
|
||||
`manager-api` 通过这些精心选择的技术和设计模式，构建了一个功能全面、结构清晰、安全可靠且易于维护和扩展的Java后端服务。其模块化的设计特别适合处理具有多种管理功能需求的复杂系统。
|
||||
|
||||
---
|
||||
|
||||
### 3.3. `manager-web` (Web管理前端 - Vue.js实现)
|
||||
|
||||
`manager-web` 组件是一个采用 Vue.js 2 框架构建的单页应用 (SPA - Single Page Application)。它为系统管理员提供了一个功能丰富、交互友好的图形用户界面，用于全面管理和配置 `xiaozhi-esp32-server` 生态系统。
|
||||
|
||||
* **核心目标:**
|
||||
* 提供一个基于Web的集中式控制面板，供管理员进行系统操作与监控。
|
||||
* 实现对 `xiaozhi-server` 中AI服务提供商（ASR、LLM、TTS等）及其相关API密钥或许可配置的便捷管理。
|
||||
* 支持用户账户、角色及权限的精细化管理。
|
||||
* 提供ESP32设备的注册、配置及状态查看功能。
|
||||
* 允许管理员自定义TTS音色、管理OTA固件更新流程、调整系统级参数及字典数据等。
|
||||
* 作为 `manager-api` 所暴露各项功能的图形化交互前端。
|
||||
|
||||
* **核心技术栈:**
|
||||
* **Vue.js 2:** 一个渐进式的JavaScript框架，用于构建用户界面。其核心特性包括声明式渲染、组件化系统、数据绑定等，非常适合构建复杂的SPA。
|
||||
* **Vue CLI (`@vue/cli-service`):** Vue.js的官方命令行工具，用于项目的快速搭建、开发服务器的运行（支持热模块替换HMR）、以及生产环境构建打包（内部集成并配置了Webpack）。
|
||||
* **Vue Router (`vue-router`):** Vue.js官方的路由管理器。它负责在SPA内部实现不同“页面”或视图组件之间的导航切换，而无需重新加载整个HTML页面，提供了流畅的用户体验。
|
||||
* **Vuex (`vuex`):** Vue.js官方的状态管理模式和库。它充当了应用中所有组件的“中央数据存储”，用于管理全局共享状态（例如当前登录用户信息、设备列表、应用配置等），特别适用于大型复杂应用。
|
||||
* **Element UI (`element-ui`):** 一个广受欢迎的基于Vue 2.0的桌面端UI组件库。它提供了大量预先设计和实现的组件（如表单、表格、对话框、导航菜单、按钮、提示等），帮助开发者快速构建出专业且一致的用户界面。
|
||||
* **JavaScript (ES6+):** 前端逻辑实现的主要编程语言，利用其现代特性进行开发。
|
||||
* **SCSS (Sassy CSS):** 一种CSS预处理器，它为CSS增加了变量、嵌套规则、混合(Mixin)、继承等高级特性，使得CSS代码更易于组织、维护和复用。
|
||||
* **HTTP客户端 (Flyio 或 Axios 通过 `vue-axios`):** 用于在浏览器端向 `manager-api` 后端发起异步HTTP（AJAX）请求，以获取数据或提交操作。
|
||||
* **Webpack:** 一个强大的模块打包工具（由Vue CLI在底层管理和配置）。它将项目中的各种资源（JavaScript文件、CSS、图片、字体等）视为模块，并将它们打包成浏览器可识别的静态文件。
|
||||
* **Workbox (通过 `workbox-webpack-plugin`):** Google开发的一个库，用于简化Service Worker的编写和PWA（Progressive Web App - 渐进式Web应用）的实现。它可以帮助生成Service Worker脚本，实现资源缓存、离线访问等功能。
|
||||
* **Opus库 (`opus-decoder`, `opus-recorder`):** 这些音频处理库表明前端可能具备一些直接在浏览器中处理Opus格式音频的能力，例如：用于测试麦克风输入、允许管理员录制自定义音频片段（可能用于TTS音色样本或语音指令测试），或播放在管理界面中预览的Opus编码音频。
|
||||
|
||||
* **关键实现细节:**
|
||||
|
||||
1. **单页应用 (SPA) 结构:**
|
||||
* 整个前端应用加载一个主HTML文件 (`public/index.html`)。后续的所有页面切换和内容更新都在客户端由Vue Router动态完成，无需每次都从服务器请求新的HTML页面。这种模式能提供更快的页面加载速度和更流畅的交互体验。
|
||||
|
||||
2. **组件化架构 (Component-Based Architecture):**
|
||||
* 用户界面由一系列可复用的Vue组件 (`.vue` 单文件组件) 构成，形成一个组件树。这种方式提高了代码的模块化程度、可维护性和复用性。
|
||||
* **`src/main.js`:** 应用的入口JS文件。它负责创建和初始化根Vue实例，注册全局插件（如Vue Router, Vuex, Element UI），并把根Vue实例挂载到 `public/index.html` 中的某个DOM元素上（通常是 `#app`）。
|
||||
* **`src/App.vue`:** 应用的根组件。它通常定义了应用的基础布局结构（如包含导航栏、侧边栏、主内容区），并通过 `<router-view></router-view>` 标签来显示当前路由匹配到的视图组件。
|
||||
* **视图组件 (`src/views/`):** 这些组件代表了应用中的各个“页面”或主要功能区（例如 `Login.vue` 登录页, `DeviceManagement.vue` 设备管理页, `UserManagement.vue` 用户管理页, `ModelConfig.vue` 模型配置页）。它们通常由Vue Router直接映射。
|
||||
* **可复用UI组件 (`src/components/`):** 包含了在不同视图之间共享的、更小粒度的UI组件（例如 `HeaderBar.vue` 顶部导航栏, `AddDeviceDialog.vue` 添加设备对话框, `AudioPlayer.vue` 音频播放器组件）。
|
||||
|
||||
3. **客户端路由 (`src/router/index.js`):**
|
||||
* Vue Router在此文件中进行配置，定义了应用的路由表。每个路由规则将一个特定的URL路径映射到一个视图组件。
|
||||
* 常常包含**导航守卫 (Navigation Guards)**，例如 `beforeEach` 守卫，用于在路由跳转前执行逻辑，如检查用户是否已登录，如果未登录则重定向到登录页面，从而保护需要认证才能访问的页面。
|
||||
|
||||
4. **状态管理 (`src/store/index.js`):**
|
||||
* Vuex被用来构建一个集中的状态管理中心（Store）。这个Store包含了：
|
||||
* **State:** 存储应用级别的共享数据（例如，当前登录用户的详细信息、从API获取的设备列表、系统配置等）。
|
||||
* **Getters:** 类似于Vue组件中的计算属性，用于从State派生出一些状态值，方便组件使用。
|
||||
* **Mutations:** **唯一**可以同步修改State中数据的方法。它们必须是同步函数。
|
||||
* **Actions:** 用于处理异步操作（如API调用）或封装多个Mutation提交。Actions会调用API，获取数据后，通过 `commit` 一个或多个Mutation来更新State。
|
||||
* 例如，用户登录时，一个名为 `login` 的Action可能会被调用，它会向后端API发送登录请求，成功后获取到用户信息和token，然后 `commit` 一个名为 `SET_USER_INFO` 的Mutation来更新State中的用户信息和token。
|
||||
|
||||
5. **API通信 (`src/apis/`):**
|
||||
* 与 `manager-api` 后端的所有HTTP通信逻辑被封装在 `src/apis/` 目录下，通常会按照后端API的模块进行组织（例如 `src/apis/module/agent.js`, `src/apis/module/device.js`）。
|
||||
* 每个模块导出一系列函数，每个函数对应一个具体的API请求。这些函数内部使用配置好的HTTP客户端实例 (例如，在 `src/apis/api.js` 或 `src/apis/httpRequest.js` 中统一配置Axios或Flyio实例，可能包含设置请求基地址、请求/响应拦截器等)。
|
||||
* **拦截器 (Interceptors):** HTTP客户端的请求拦截器常用于在每个请求发送前自动添加认证令牌（如JWT）；响应拦截器则可用于全局处理API错误（如权限不足、服务器错误）或对响应数据进行预处理。
|
||||
|
||||
6. **样式与资源 (`src/styles/`, `src/assets/`):**
|
||||
* `Element UI` 提供了基础的组件样式。
|
||||
* `src/styles/global.scss` 文件用于定义全局共享的SCSS样式、变量、混合(Mixin)等。
|
||||
* Vue单文件组件内部的 `<style scoped>` 标签允许编写只作用于当前组件的局部样式。
|
||||
* `src/assets/` 目录存放图片、字体等静态资源。
|
||||
|
||||
7. **构建与PWA特性:**
|
||||
* Vue CLI通过Webpack将所有代码和资源打包成优化的静态文件，用于生产部署。
|
||||
* `workbox-webpack-plugin` 的使用（体现在 `service-worker.js` 和 `registerServiceWorker.js` 文件）表明项目集成了Service Worker技术。Service Worker可以拦截网络请求，实现前端资源的智能缓存（从而加快后续访问速度），甚至在网络断开时提供一定的离线访问能力，是PWA的核心技术之一。
|
||||
|
||||
8. **环境配置 (`.env`系列文件):**
|
||||
* 项目根目录下的 `.env` (以及 `.env.development`, `.env.production` 等) 文件用于定义环境变量。这些变量（例如 `VUE_APP_API_BASE_URL` 来指定 `manager-api` 的基础URL）可以在应用代码中通过 `process.env.VUE_APP_XXX` 的形式访问，从而允许为不同构建环境（开发、测试、生产）配置不同的参数。
|
||||
|
||||
`manager-web` 通过这些技术的综合运用，构建了一个功能强大、易于维护且用户体验良好的管理界面，为 `xiaozhi-esp32-server` 系统的配置和监控提供了坚实的前端支持。
|
||||
|
||||
---
|
||||
|
||||
## 4. 数据流与交互机制
|
||||
|
||||
`xiaozhi-esp32-server` 系统通过各组件间定义清晰的数据流和交互协议来协同工作。主要的通信方式依赖于针对实时交互优化的WebSocket协议和适用于客户端-服务器请求的RESTful API。
|
||||
|
||||
**4.1.核心语音交互流程 (ESP32设备 <-> `xiaozhi-server`)**
|
||||
|
||||
此流程是实时的，主要通过WebSocket进行低延迟、双向的数据交换。
|
||||
|
||||
* **连接建立与握手:**
|
||||
* ESP32设备作为客户端，主动向`xiaozhi-server`的指定端点（例如 `ws://<服务器IP>:<WebSocket端口>/xiaozhi/v1/`）发起WebSocket连接请求。
|
||||
* `xiaozhi-server` (`core/websocket_server.py`) 接收连接，并为每个成功连接的ESP32设备实例化一个独立的`ConnectionHandler`对象来管理该会话的整个生命周期。
|
||||
* 连接建立后，可能会执行一个初始握手流程（由`core/handle/helloHandle.py`处理），用于交换设备标识、认证信息、协议版本或基本状态。
|
||||
|
||||
* **音频上行传输 (ESP32 -> `xiaozhi-server`):**
|
||||
* 用户对ESP32设备讲话后，设备上的麦克风捕捉原始音频数据（通常是PCM或经过压缩如Opus的格式）。
|
||||
* ESP32将这些音频数据块（chunks）作为WebSocket的**二进制消息 (binary messages)** 实时推送到`xiaozhi-server`对应的`ConnectionHandler`。
|
||||
* 服务器端的`core/handle/receiveAudioHandle.py`模块负责接收、缓冲并处理这些音频数据。
|
||||
|
||||
* **AI核心处理 (在`xiaozhi-server`内部):**
|
||||
* **VAD (语音活动检测):** `receiveAudioHandle.py`利用配置的VAD提供者（如SileroVAD）分析音频流，以准确识别语音的起始和结束点，滤除静默或噪声片段。
|
||||
* **ASR (自动语音识别):** 检测到的有效语音片段被送往配置的ASR提供者（本地如FunASR，或云端服务）。ASR引擎将音频信号转换为文本字符串。
|
||||
* **NLU/LLM (自然语言理解/大型语言模型):** ASR输出的文本，连同从Memory提供者获取的当前对话上下文历史，以及从`plugins_func/`加载的可用函数（工具）的描述模式，一同被传递给配置的LLM提供者。
|
||||
* **函数调用执行 (若LLM决策需要):** 如果LLM分析后认为需要调用外部函数（例如查询天气、控制家电），它会生成一个结构化的函数调用请求。`core/handle/functionHandler.py`接收此请求，查找并执行在`plugins_func/`中定义的相应Python函数，并将函数的执行结果返回给LLM。LLM随后基于此结果生成最终的自然语言回复。
|
||||
* **回复生成:** LLM综合所有信息（用户输入、上下文、函数调用结果等）生成最终的文本回复。
|
||||
* **记忆更新:** 当前轮次的交互（用户问题、LLM回复、可能的功能调用）会被Memory提供者处理，以更新对话历史，供后续交互使用。
|
||||
* **TTS (文本转语音):** LLM生成的最终文本回复被送往配置的TTS提供者，后者将文本合成为语音数据流（例如MP3或WAV格式）。
|
||||
|
||||
* **音频下行响应 (`xiaozhi-server` -> ESP32):**
|
||||
* 由TTS提供者合成的语音数据流，通过`core/handle/sendAudioHandle.py`模块，作为WebSocket的**二进制消息**实时发送回ESP32设备。
|
||||
* ESP32设备接收这些音频数据块并立即通过扬声器播放给用户。
|
||||
|
||||
* **控制与状态消息 (双向):**
|
||||
* 除了音频流，ESP32与`xiaozhi-server`之间也通过WebSocket交换**文本消息 (text messages)**，这些消息通常采用JSON格式封装。
|
||||
* **ESP32 -> Server:** 设备可能发送状态报告（如网络状况、麦克风状态）、错误代码、或特定的控制命令（例如用户按键触发的“停止TTS播报”）。
|
||||
* **Server -> ESP32:** 服务器可能发送控制指令给设备（如“开始监听”、“停止监听”、调整灵敏度、下发特定配置参数）。
|
||||
* `core/handle/abortHandle.py`（处理中断请求）、`core/handle/reportHandle.py`（处理设备报告）等模块负责解析和响应这些控制/状态消息。
|
||||
|
||||
**4.2.管理与配置流程 (`manager-web` <-> `manager-api` <-> `xiaozhi-server`)**
|
||||
|
||||
此流程主要依赖于基于HTTP/HTTPS的RESTful API进行请求-响应式的交互。
|
||||
|
||||
* **管理员UI后端交互 (`manager-web` -> `manager-api`):**
|
||||
* 当管理员在`manager-web`界面执行操作时（例如保存一项配置、添加一个新用户、注册一台ESP32设备）：
|
||||
* Vue.js前端应用 (`manager-web`) 会通过其API封装模块（位于`src/apis/module/`）向`manager-api`的对应REST API端点发起异步HTTP请求（通常是GET, POST, PUT, DELETE）。
|
||||
* 请求体和响应体通常使用JSON格式。
|
||||
* `manager-api`中的`@RestController`类接收这些请求。**Apache Shiro**框架会首先对请求进行认证和授权检查。
|
||||
* 通过验证后，Controller将请求分发给相应的Service层处理业务逻辑。Service层可能会与MySQL数据库（通过MyBatis-Plus）交互，并可能利用Redis进行数据缓存。
|
||||
* 处理完成后，`manager-api`向`manager-web`返回一个JSON格式的HTTP响应。
|
||||
* `manager-web`根据响应结果更新其Vuex状态存储和用户界面显示。
|
||||
|
||||
* **配置同步 (`manager-api` -> `xiaozhi-server`):**
|
||||
* `xiaozhi-server`的运行依赖于从`manager-api`获取的动态配置（例如当前选用的AI服务提供商及其API密钥）。
|
||||
* **拉取机制 (Pull Mechanism):** `xiaozhi-server`内部的`config/manage_api_client.py`模块，在服务器启动时或通过特定更新触发器（例如`WebSocketServer.update_config()`被调用），会向`manager-api`的一个指定端点（例如由`modules/config/controller/`中的某个Controller提供）发起HTTP GET请求。
|
||||
* `manager-api`响应该请求，返回`xiaozhi-server`所需的配置数据（JSON格式）。
|
||||
* `xiaozhi-server`接收到配置后，会更新其内部状态，并可能重新初始化相关的AI服务模块，以使新配置生效。
|
||||
|
||||
* **OTA固件更新流程 (概念性描述):**
|
||||
* 管理员通过`manager-web`界面上传新的ESP32固件包到`manager-api`的特定端点。
|
||||
* `manager-api`将固件文件存储起来，并记录相关元数据（版本号、适用设备型号等）。
|
||||
* 当管理员触发对特定设备的OTA更新时：
|
||||
* `manager-api`可能会通知`xiaozhi-server`（具体通知机制可能是一个轮询检查点，或`xiaozhi-server`暴露一个接收更新通知的API，或者更松耦合的如消息队列）。
|
||||
* `xiaozhi-server`随后可以通过WebSocket向目标ESP32设备发送一条包含固件下载URL的指令消息。
|
||||
* ESP32设备收到指令后，通过HTTP GET请求从该URL下载固件。此URL可能指向`xiaozhi-server`自身运行的`SimpleHttpServer`所服务的路径（如`/xiaozhi/ota/`），或者在某些架构中，也可能直接指向`manager-api`或专用的文件服务器。
|
||||
|
||||
**4.3. 主要协议总结:**
|
||||
|
||||
* **WebSocket:** 被选用于ESP32与`xiaozhi-server`之间的通信链路，因为它非常适合实时、低延迟、双向的数据流传输（尤其是音频），以及异步控制消息的传递。
|
||||
* **RESTful APIs (基于HTTP/HTTPS，通常使用JSON作为数据交换格式):** 这是Web服务间通信的标准方式。用于`manager-web`（客户端）与`manager-api`（服务器）之间的请求-响应交互，也用于`xiaozhi-server`（作为客户端）从`manager-api`（作为服务器）拉取配置信息。其无状态特性、广泛的库支持和易于理解的语义使其成为此类交互的理想选择。
|
||||
|
||||
这种多协议并用的通信策略，确保了系统内不同类型的交互需求都能得到高效和恰当的处理，兼顾了实时性和标准化的请求-响应模式。
|
||||
|
||||
---
|
||||
|
||||
## 5. 核心功能概要
|
||||
|
||||
`xiaozhi-esp32-server` 系统提供了一系列丰富的功能，旨在支持开发者构建先进的语音控制应用：
|
||||
|
||||
1. **全面的语音交互后端:** 提供从语音捕获指导到响应生成和动作执行的端到端解决方案。
|
||||
2. **模块化和可插拔的AI服务:**
|
||||
* 支持广泛的ASR（自动语音识别）、LLM（大型语言模型）、TTS（文本转语音）、VAD（语音活动检测）、意图识别和记忆提供商。
|
||||
* 允许动态选择和配置这些服务（包括基于云的API和本地模型），以平衡成本、性能、隐私和语言需求。
|
||||
3. **高级对话管理:**
|
||||
* 支持自然交互，具有唤醒词启动对话、手动（按键说话式）对话以及对系统响应的实时打断等功能。
|
||||
* 包含上下文记忆，以在多轮对话中保持连贯性。
|
||||
* 在一段时间不活动后具有自动休眠模式。
|
||||
4. **多语言能力:**
|
||||
* 支持多种语言的识别和合成，包括普通话、粤语、英语、日语和韩语（具体取决于所选的ASR/LLM/TTS提供商）。
|
||||
5. **通过插件实现的可扩展功能:**
|
||||
* 强大的插件系统允许开发人员添加自定义“技能”或函数（例如，获取天气、控制智能家居设备、访问新闻）。
|
||||
* 这些函数可以由LLM使用其函数调用能力，根据提供的模式来触发。
|
||||
* 内置对Home Assistant集成的支持。
|
||||
6. **物联网设备控制:**
|
||||
* 设计用于通过语音命令管理和控制智能家居设备及其他物联网硬件，并利用插件系统。
|
||||
7. **基于Web的管理控制台 (`manager-web` & `manager-api`):**
|
||||
* 提供全面的图形界面，用于：
|
||||
* 系统配置（AI服务选择、API密钥、操作参数）。
|
||||
* 基于角色的访问控制的用户管理。
|
||||
* ESP32设备注册和管理。
|
||||
* 语音音色/TTS语音定制。
|
||||
* ESP32设备的OTA（空中下载）固件更新管理。
|
||||
* 系统参数和字典的管理。
|
||||
8. **灵活的部署选项:**
|
||||
* 支持通过Docker容器（用于简化的仅服务器或全栈设置）和直接从源代码部署，以适应各种环境和用户专业知识。
|
||||
9. **动态远程配置:**
|
||||
* `xiaozhi-server`可以从`manager-api`获取其配置，允许实时更新AI提供商和设置，而无需重新启动服务器。
|
||||
10. **开源和社区驱动:**
|
||||
* 根据MIT许可证授权，鼓励透明、协作和社区贡献。
|
||||
11. **经济高效的解决方案:**
|
||||
* 提供“入门全免费设置”路径，利用AI服务的免费套餐或本地模型，使其易于进行实验和个人项目。
|
||||
12. **渐进式Web应用 (PWA) 特性:**
|
||||
* `manager-web`控制面板包含Service Worker集成，以增强缓存和潜在的离线访问能力。
|
||||
13. **详细的API文档:**
|
||||
* `manager-api`通过Knife4j提供OpenAPI (Swagger) 文档，以便清晰理解和测试其RESTful端点。
|
||||
|
||||
这些功能共同使`xiaozhi-esp32-server`成为一个强大、适应性强且用户友好的平台，用于构建复杂的语音交互应用程序。
|
||||
|
||||
---
|
||||
|
||||
## 6. 部署与配置概述
|
||||
|
||||
`xiaozhi-esp32-server`系统在设计上充分考虑了灵活性，提供了多种部署方法和全面的配置选项，以适应不同的使用场景和需求。
|
||||
|
||||
**部署选项:**
|
||||
|
||||
项目可以通过多种方式部署，主要包括使用Docker简化安装过程，或直接从源代码部署以获得更大的控制权和进行开发。
|
||||
|
||||
1. **基于Docker的部署:**
|
||||
* **简化安装 (仅`xiaozhi-server`):** 此选项仅部署核心的基于Python的`xiaozhi-server`。它适用于主要需要语音AI处理能力和IoT控制，而不需要完整Web管理界面和数据库支持功能（如OTA）的用户。在此模式下，配置通常通过本地文件（`config.yaml`）管理，但如果需要，仍可将其指向一个已存在的`manager-api`实例。
|
||||
* **全模块安装 (所有组件):** 此方案部署所有核心组件：`xiaozhi-server`、基于Java的`manager-api`、以及基于Vue.js的`manager-web`，同时还包括所需的数据库服务（MySQL和Redis）。这提供了完整的系统体验，包括用于全面配置和管理的Web控制面板。
|
||||
* 项目为每个服务提供了`Dockerfile`定义，并使用`docker-compose.yml`文件（例如`docker-compose.yml`用于基础版，`docker-compose_all.yml`用于全功能版）来编排和管理多容器的部署。此外，还可能提供一个`docker-setup.sh`脚本来辅助自动化部分Docker环境的搭建工作。
|
||||
|
||||
2. **源代码部署:**
|
||||
* 这种方法需要为每个组件手动设置相应的开发环境：Python环境用于`xiaozhi-server`，Java/Maven环境用于`manager-api`，Node.js/Vue CLI环境用于`manager-web`。
|
||||
* 对于全模块安装，还需要手动安装和配置MySQL及Redis数据库服务。
|
||||
* 这种方式通常用于项目开发、深度定制、调试，或者在对环境有特殊要求的生产场景中。
|
||||
|
||||
**配置管理:**
|
||||
|
||||
配置是定制系统行为的关键，尤其是在选择AI服务提供商和管理API密钥方面。
|
||||
|
||||
1. **`xiaozhi-server` 配置:**
|
||||
* **本地`config.yaml`:** 位于`xiaozhi-server`根目录下的一个主要的YAML格式配置文件。它定义了服务器端口、选定的AI服务提供商（ASR、LLM、TTS、VAD、意图识别、记忆模块等）、它们各自的API密钥或模型路径、插件配置以及日志级别等。
|
||||
* **通过`manager-api`进行远程配置:** `xiaozhi-server`被设计为可以从`manager-api`获取其运行配置。从`manager-api`获取的设置通常会覆盖本地`config.yaml`中的同名设置。这带来了两大好处：
|
||||
* **集中管理:** 所有配置都可以通过`manager-web`界面进行统一管理。
|
||||
* **动态更新:** `xiaozhi-server`可以刷新其配置并重新初始化AI模块，而无需完全重启服务。
|
||||
* `xiaozhi-server`中的`config/config_loader.py`和`config/manage_api_client.py`负责处理配置的加载、合并及从`manager-api`拉取的逻辑。
|
||||
|
||||
2. **`manager-api` 配置:**
|
||||
* 作为一个Spring Boot应用，其配置主要通过位于`src/main/resources`目录下的`application.properties`或`application.yml`文件进行管理。
|
||||
* 关键配置项包括：数据库连接信息（MySQL的URL、用户名、密码）、Redis服务器地址和端口、应用服务端口（默认为8002）、Apache Shiro安全相关的设置，以及任何集成的第三方服务（如阿里云短信）的配置参数。
|
||||
|
||||
3. **`manager-web` 配置:**
|
||||
* Vue.js前端应用的环境特定设置通过项目根目录下的`.env`系列文件（例如`.env`, `.env.development`, `.env.production`）进行管理。
|
||||
* 这里最关键的配置通常是`manager-api`后端的API基础URL地址 (例如 `VUE_APP_API_BASE_URL`)，前端应用将向此地址发送所有API请求。
|
||||
|
||||
4. **预定义的配置方案:**
|
||||
* 项目文档（通常是README）中会推荐一些常见的配置组合，例如：
|
||||
* **“入门全免费设置”:** 该方案旨在利用云AI服务的免费套餐额度或完全免费的本地模型，以最大程度地降低用户的初始使用成本和运营费用。
|
||||
* **“全流式配置”:** 该方案优先考虑系统的响应速度和交互的流畅性，通常会选用支持流式处理的（可能付费的）AI服务。
|
||||
* 这些预定义方案为用户在`xiaozhi-server`中配置AI服务提供商（通过`manager-web`界面或直接修改`config.yaml`）提供了指导。
|
||||
|
||||
在全模块部署的情况下，推荐使用`manager-web`控制面板作为大多数配置任务的主要操作界面，因为它提供了一种用户友好的方式来管理由`manager-api`持久化并最终由`xiaozhi-server`使用的各项设置。
|
||||
|
||||
---
|
||||
@@ -0,0 +1,163 @@
|
||||
# 全模块源码部署自动升级方法
|
||||
|
||||
本教程是方便全模块源码部署的爱好者，如何通过自动命令，自动拉取源码，自动编译，自动启动端口运行。实现最高效率的升级系统。
|
||||
|
||||
本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。
|
||||
|
||||
# 开始条件
|
||||
- 你的电脑/服务器是linux操作系统
|
||||
- 你已经跑通了整个流程
|
||||
- 你喜欢跟进最新功能，但是觉得每次手动部署有点麻烦，期待有一个自动更新的方法
|
||||
|
||||
第二个条件必须满足，因为本教程所涉及的某些文件，JDK、Node.js环境、Conda环境等，是需要你跑通整个流程才有的，如果你没有跑通，当我讲到某个文件的时候，你可能就不知道什么意思。
|
||||
|
||||
# 教程效果
|
||||
- 解决国内不能拉取最新项目源码问题
|
||||
- 自动拉取代码编译前端文件
|
||||
- 自动拉取代码编译java文件，自动杀掉8002端口，自动启动8002端口
|
||||
- 自动拉取python代码，自动杀掉8000端口，自动启动8000端口
|
||||
|
||||
# 第一步 选好你的项目目录
|
||||
|
||||
例如，我规划了我的项目目录是，这是一个新建的空白的目录，如果你不想出错，可以和我一样
|
||||
```
|
||||
/home/system/xiaozhi
|
||||
```
|
||||
|
||||
# 第二步 克隆本项目
|
||||
此刻，先要执行第一句话，拉取源码，这句命令适用于国内网络的服务器和电脑，无需翻墙
|
||||
|
||||
```
|
||||
cd /home/system/xiaozhi
|
||||
git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-server.git
|
||||
```
|
||||
|
||||
执行完后，你的项目目录会多了一个文件夹`xiaozhi-esp32-server`，这个就是项目的源码
|
||||
|
||||
# 第三步 复制基础的文件
|
||||
|
||||
如果你之前已经跑通了整个流程，对funasr的模型文件`xiaozhi-server/models/SenseVoiceSmall/model.pt`和你的私有配置文件`xiaozhi-server/data/.config.yaml`这两个文件不会陌生。
|
||||
|
||||
此刻你需要把`model.pt`文件复制到新的目录去，你可以这样
|
||||
```
|
||||
cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
|
||||
cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
|
||||
```
|
||||
|
||||
# 第四步 建立三个自动编译文件
|
||||
|
||||
## 4.1 自动编译mananger-web模块
|
||||
在`/home/system/xiaozhi/`目录下，创建名字为`update_8001.sh`的文件，内容如下
|
||||
|
||||
```
|
||||
cd /home/system/xiaozhi/xiaozhi-esp32-server
|
||||
git fetch --all
|
||||
git reset --hard
|
||||
git pull origin main
|
||||
|
||||
|
||||
cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web
|
||||
npm install
|
||||
npm run build
|
||||
rm -rf /home/system/xiaozhi/manager-web
|
||||
mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-web/dist /home/system/xiaozhi/manager-web
|
||||
```
|
||||
|
||||
保存好后执行赋权命令
|
||||
```
|
||||
chmod 777 update_8001.sh
|
||||
```
|
||||
执行完后，继续往下
|
||||
|
||||
## 4.2 自动编译运行manager-api模块
|
||||
在`/home/system/xiaozhi/`目录下，创建名字为`update_8002.sh`的文件，内容如下
|
||||
|
||||
```
|
||||
cd /home/system/xiaozhi/xiaozhi-esp32-server
|
||||
git pull origin main
|
||||
|
||||
|
||||
cd /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api
|
||||
rm -rf target
|
||||
mvn clean package -Dmaven.test.skip=true
|
||||
cd /home/system/xiaozhi/
|
||||
|
||||
# 查找占用8002端口的进程号
|
||||
PID=$(sudo netstat -tulnp | grep 8002 | awk '{print $7}' | cut -d'/' -f1)
|
||||
|
||||
rm -rf /home/system/xiaozhi/xiaozhi-esp32-api.jar
|
||||
mv /home/system/xiaozhi/xiaozhi-esp32-server/main/manager-api/target/xiaozhi-esp32-api.jar /home/system/xiaozhi/xiaozhi-esp32-api.jar
|
||||
|
||||
# 检查是否找到进程号
|
||||
if [ -z "$PID" ]; then
|
||||
echo "没有找到占用8002端口的进程"
|
||||
else
|
||||
echo "找到占用8002端口的进程，进程号为: $PID"
|
||||
# 杀掉进程
|
||||
kill -9 $PID
|
||||
kill -9 $PID
|
||||
echo "已杀掉进程 $PID"
|
||||
fi
|
||||
|
||||
nohup java -jar xiaozhi-esp32-api.jar --spring.profiles.active=dev &
|
||||
```
|
||||
|
||||
保存好后执行赋权命令
|
||||
```
|
||||
chmod 777 update_8002.sh
|
||||
```
|
||||
执行完后，继续往下
|
||||
|
||||
## 4.3 自动编译运行Python项目
|
||||
在`/home/system/xiaozhi/`目录下，创建名字为`update_8000.sh`的文件，内容如下
|
||||
|
||||
```
|
||||
cd /home/system/xiaozhi/xiaozhi-esp32-server
|
||||
git pull origin main
|
||||
|
||||
# 查找占用8000端口的进程号
|
||||
PID=$(sudo netstat -tulnp | grep 8000 | awk '{print $7}' | cut -d'/' -f1)
|
||||
|
||||
# 检查是否找到进程号
|
||||
if [ -z "$PID" ]; then
|
||||
echo "没有找到占用8000端口的进程"
|
||||
else
|
||||
echo "找到占用8000端口的进程，进程号为: $PID"
|
||||
# 杀掉进程
|
||||
kill -9 $PID
|
||||
kill -9 $PID
|
||||
echo "已杀掉进程 $PID"
|
||||
fi
|
||||
cd main/xiaozhi-server
|
||||
pip install -r requirements.txt
|
||||
nohup python app.py >/dev/null &
|
||||
```
|
||||
|
||||
保存好后执行赋权命令
|
||||
```
|
||||
chmod 777 update_8000.sh
|
||||
```
|
||||
执行完后，继续往下
|
||||
|
||||
# 日常更新
|
||||
|
||||
以上的脚本都建立好后，日常更新，我们只要依次执行以下命令就可以做到自动更新和启动
|
||||
|
||||
```
|
||||
# 进入pyhton环境
|
||||
conda activate xiaozhi-esp32-server
|
||||
cd /home/system/xiaozhi
|
||||
# 更新并启动Java程序
|
||||
./update_8001.sh
|
||||
# 更新web程序
|
||||
./update_8002.sh
|
||||
# 更新并启动python程序
|
||||
./update_8000.sh
|
||||
# 查看Java日志
|
||||
tail -f nohup.out
|
||||
# 查看Python日志
|
||||
tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server.log
|
||||
```
|
||||
|
||||
# 注意事项
|
||||
测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)
|
||||
@@ -202,7 +202,15 @@ change_role;get_weather;get_news;play_music;hass_get_state;hass_set_state
|
||||
|
||||
#### 2. 配置小智开源服务端MCP配置信息
|
||||
|
||||
切换到小智开源服务端`xiaozhi-esp32-server`的`mcp_server_settings.json`文件内，在`"mcpServers"`的括号内添加以下内容：
|
||||
|
||||
进入`data`目录，找到`.mcp_server_settings.json`文件。
|
||||
|
||||
如果你的`data`目录下没有`.mcp_server_settings.json`文件，
|
||||
- 请把在`xiaozhi-server`文件夹根目录的`mcp_server_settings.json`文件复制到`data`目录下，并重命名为`.mcp_server_settings.json`
|
||||
- 或[下载这个文件](https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/main/xiaozhi-server/mcp_server_settings.json)，下载到`data`目录下，并重命名为`.mcp_server_settings.json`
|
||||
|
||||
|
||||
修改`"mcpServers"`里的这部分的内容：
|
||||
|
||||
```json
|
||||
"Home Assistant": {
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 386 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 518 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 361 KiB |
@@ -0,0 +1,171 @@
|
||||
# 视觉模型使用指南
|
||||
本教程分为两部分：
|
||||
- 第一部分：单模块运行xiaozhi-server开启视觉模型
|
||||
- 第二部分：全模块运行时，如何开启视觉模型
|
||||
|
||||
开启视觉模型前，你需要准备三件事：
|
||||
- 你需要准备一台带摄像头的设备，而且这台设备已经在虾哥仓库里，实现了调用摄像头功能。例如`立创·实战派ESP32-S3开发板`
|
||||
- 你设备固件的版本升级到1.6.6及以上
|
||||
- 你已经成功跑通基础对话模块
|
||||
|
||||
## 单模块运行xiaozhi-server开启视觉模型
|
||||
|
||||
### 第一步确认网络
|
||||
由于视觉模型会默认启动8003端口。
|
||||
|
||||
如果你是docker运行，请确认一下你的`docker-compose.yml`是否放了`8003`端口，如果没有就更新最新的`docker-compose.yml`文件
|
||||
|
||||
如果你是源码运行，确认防火墙是否放行`8003`端口
|
||||
|
||||
### 第二步选择你的视觉模型
|
||||
打开你的`data/.config.yaml`文件，设置你的`selected_module.VLLM`设置为某个视觉模型。目前我们已经支持`openai`类型接口的视觉模型。`ChatGLMVLLM`就是其中一款兼容`openai`的模型。
|
||||
|
||||
```
|
||||
selected_module:
|
||||
VAD: ..
|
||||
ASR: ..
|
||||
LLM: ..
|
||||
VLLM: ChatGLMVLLM
|
||||
TTS: ..
|
||||
Memory: ..
|
||||
Intent: ..
|
||||
```
|
||||
|
||||
假设我们使用`ChatGLMVLLM`作为视觉模型，那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
|
||||
|
||||
在你的配置文件中，增加这个配置，如果已经有了这个配置，就设置好你的api_key。
|
||||
|
||||
```
|
||||
VLLM:
|
||||
ChatGLMVLLM:
|
||||
api_key: 你的api_key
|
||||
```
|
||||
|
||||
### 第三步启动xiaozhi-server服务
|
||||
如果你是源码，就输入命令启动
|
||||
```
|
||||
python app.py
|
||||
```
|
||||
如果你是docker运行，就重启容器
|
||||
```
|
||||
docker restart xiaozhi-esp32-server
|
||||
```
|
||||
|
||||
启动后会输出以下内容的日志。
|
||||
|
||||
```
|
||||
2025-06-01 **** - OTA接口是 http://192.168.4.7:8003/xiaozhi/ota/
|
||||
2025-06-01 **** - 视觉分析接口是 http://192.168.4.7:8003/mcp/vision/explain
|
||||
2025-06-01 **** - Websocket地址是 ws://192.168.4.7:8000/xiaozhi/v1/
|
||||
2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
|
||||
2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
|
||||
2025-06-01 **** - =============================================================
|
||||
```
|
||||
|
||||
启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
|
||||
```
|
||||
curl -i 你的视觉分析接口
|
||||
```
|
||||
|
||||
正常来说会这样显示
|
||||
```
|
||||
MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
|
||||
```
|
||||
|
||||
请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
|
||||
```
|
||||
server:
|
||||
vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
|
||||
```
|
||||
|
||||
为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
|
||||
|
||||
假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
|
||||
|
||||
```
|
||||
server:
|
||||
vision_explain: http://111.111.111.111:8003/mcp/vision/explain
|
||||
```
|
||||
|
||||
如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
|
||||
|
||||
### 第四步 设备唤醒开启
|
||||
|
||||
对设备说“请打开摄像头，说你你看到了什么”
|
||||
|
||||
留意xiaozhi-server的日志输出，看看有没有报错。
|
||||
|
||||
|
||||
## 全模块运行时，如何开启视觉模型
|
||||
|
||||
### 第一步 确认网络
|
||||
由于视觉模型会默认启动8003端口。
|
||||
|
||||
如果你是docker运行，请确认一下你的`docker-compose_all.yml`是否映射了`8003`端口，如果没有就更新最新的`docker-compose_all.yml`文件
|
||||
|
||||
如果你是源码运行，确认防火墙是否放行`8003`端口
|
||||
|
||||
### 第二步 确认你配置文件
|
||||
|
||||
打开你的`data/.config.yaml`文件，确认一下你的配置文件的结构，是否和`data/config_from_api.yaml`一样。如果不一样，或缺少某项，请补齐。
|
||||
|
||||
### 第三步 配置视觉模型密钥
|
||||
|
||||
那我们需要先登录[智谱AI](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)网站，申请密钥。如果你之前已经申请过了密钥，可以复用这个密钥。
|
||||
|
||||
登录`智控台`，顶部菜单点击`模型配置`，在左侧栏点击`视觉打语言模型`，找到`VLLM_ChatGLMVLLM`，点击修改按钮，在弹框中，在`API密钥`输入你密钥，点击保存。
|
||||
|
||||
保存成功后，去到你需要测试的智能体哪里，点击`配置角色`，在打开的内容里，查看`视觉大语言模型(VLLM)`是否选择了刚才的视觉模型。点击保存。
|
||||
|
||||
### 第三步 启动xiaozhi-server模块
|
||||
如果你是源码，就输入命令启动
|
||||
```
|
||||
python app.py
|
||||
```
|
||||
如果你是docker运行，就重启容器
|
||||
```
|
||||
docker restart xiaozhi-esp32-server
|
||||
```
|
||||
|
||||
启动后会输出以下内容的日志。
|
||||
|
||||
```
|
||||
2025-06-01 **** - 视觉分析接口是 http://192.168.4.7:8003/mcp/vision/explain
|
||||
2025-06-01 **** - Websocket地址是 ws://192.168.4.7:8000/xiaozhi/v1/
|
||||
2025-06-01 **** - =======上面的地址是websocket协议地址，请勿用浏览器访问=======
|
||||
2025-06-01 **** - 如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
|
||||
2025-06-01 **** - =============================================================
|
||||
```
|
||||
|
||||
启动后，使用使用浏览器打开日志里`视觉分析接口`连接。看看输出了什么？如果你是linux,没有浏览器，你可以执行这个命令：
|
||||
```
|
||||
curl -i 你的视觉分析接口
|
||||
```
|
||||
|
||||
正常来说会这样显示
|
||||
```
|
||||
MCP Vision 接口运行正常，视觉解释接口地址是：http://xxxx:8003/mcp/vision/explain
|
||||
```
|
||||
|
||||
请注意，如果你是公网部署，或者docker部署，一定要改一下你的`data/.config.yaml`里这个配置
|
||||
```
|
||||
server:
|
||||
vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
|
||||
```
|
||||
|
||||
为什么呢？因为视觉解释接口需要下发到设备，如果你的地址是局域网地址，或者是docker内部地址，设备是无法访问的。
|
||||
|
||||
假设你的公网地址是`111.111.111.111`，那么`vision_explain`应该这么配
|
||||
|
||||
```
|
||||
server:
|
||||
vision_explain: http://111.111.111.111:8003/mcp/vision/explain
|
||||
```
|
||||
|
||||
如果你的MCP Vision 接口运行正常，且你也试着用浏览器访问正常打开下发的`视觉解释接口地址`，请继续下一步
|
||||
|
||||
### 第四步 设备唤醒开启
|
||||
|
||||
对设备说“请打开摄像头，说你你看到了什么”
|
||||
|
||||
留意xiaozhi-server的日志输出，看看有没有报错。
|
||||
@@ -227,7 +227,7 @@ public interface Constant {
|
||||
/**
|
||||
* 版本号
|
||||
*/
|
||||
public static final String VERSION = "0.5.1";
|
||||
public static final String VERSION = "0.5.4";
|
||||
|
||||
/**
|
||||
* 无效固件URL
|
||||
|
||||
@@ -171,6 +171,9 @@ public class AgentController {
|
||||
if (dto.getLlmModelId() != null) {
|
||||
existingEntity.setLlmModelId(dto.getLlmModelId());
|
||||
}
|
||||
if (dto.getVllmModelId() != null) {
|
||||
existingEntity.setVllmModelId(dto.getVllmModelId());
|
||||
}
|
||||
if (dto.getTtsModelId() != null) {
|
||||
existingEntity.setTtsModelId(dto.getTtsModelId());
|
||||
}
|
||||
|
||||
@@ -27,6 +27,9 @@ public class AgentDTO {
|
||||
@Schema(description = "大语言模型名称", example = "llm_model_01")
|
||||
private String llmModelName;
|
||||
|
||||
@Schema(description = "视觉模型名称", example = "vllm_model_01")
|
||||
private String vllmModelName;
|
||||
|
||||
@Schema(description = "记忆模型ID", example = "mem_model_01")
|
||||
private String memModelId;
|
||||
|
||||
|
||||
@@ -30,6 +30,9 @@ public class AgentUpdateDTO implements Serializable {
|
||||
@Schema(description = "大语言模型标识", example = "llm_model_02", required = false)
|
||||
private String llmModelId;
|
||||
|
||||
@Schema(description = "VLLM模型标识", example = "vllm_model_02", required = false)
|
||||
private String vllmModelId;
|
||||
|
||||
@Schema(description = "语音合成模型标识", example = "tts_model_02", required = false)
|
||||
private String ttsModelId;
|
||||
|
||||
|
||||
@@ -36,6 +36,9 @@ public class AgentEntity {
|
||||
@Schema(description = "大语言模型标识")
|
||||
private String llmModelId;
|
||||
|
||||
@Schema(description = "VLLM模型标识")
|
||||
private String vllmModelId;
|
||||
|
||||
@Schema(description = "语音合成模型标识")
|
||||
private String ttsModelId;
|
||||
|
||||
|
||||
@@ -49,6 +49,11 @@ public class AgentTemplateEntity implements Serializable {
|
||||
*/
|
||||
private String llmModelId;
|
||||
|
||||
/**
|
||||
* VLLM模型标识
|
||||
*/
|
||||
private String vllmModelId;
|
||||
|
||||
/**
|
||||
* 语音合成模型标识
|
||||
*/
|
||||
|
||||
+3
@@ -102,6 +102,9 @@ public class AgentServiceImpl extends BaseServiceImpl<AgentDao, AgentEntity> imp
|
||||
// 获取 LLM 模型名称
|
||||
dto.setLlmModelName(modelConfigService.getModelNameById(agent.getLlmModelId()));
|
||||
|
||||
// 获取 VLLM 模型名称
|
||||
dto.setVllmModelName(modelConfigService.getModelNameById(agent.getVllmModelId()));
|
||||
|
||||
// 获取记忆模型名称
|
||||
dto.setMemModelId(agent.getMemModelId());
|
||||
|
||||
|
||||
+5
-2
@@ -72,6 +72,7 @@ public class ConfigServiceImpl implements ConfigService {
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
result,
|
||||
isCache);
|
||||
|
||||
@@ -140,6 +141,7 @@ public class ConfigServiceImpl implements ConfigService {
|
||||
agent.getVadModelId(),
|
||||
agent.getAsrModelId(),
|
||||
agent.getLlmModelId(),
|
||||
agent.getVllmModelId(),
|
||||
agent.getTtsModelId(),
|
||||
agent.getMemModelId(),
|
||||
agent.getIntentModelId(),
|
||||
@@ -241,6 +243,7 @@ public class ConfigServiceImpl implements ConfigService {
|
||||
String vadModelId,
|
||||
String asrModelId,
|
||||
String llmModelId,
|
||||
String vllmModelId,
|
||||
String ttsModelId,
|
||||
String memModelId,
|
||||
String intentModelId,
|
||||
@@ -248,8 +251,8 @@ public class ConfigServiceImpl implements ConfigService {
|
||||
boolean isCache) {
|
||||
Map<String, String> selectedModule = new HashMap<>();
|
||||
|
||||
String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM" };
|
||||
String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId };
|
||||
String[] modelTypes = { "VAD", "ASR", "TTS", "Memory", "Intent", "LLM", "VLLM" };
|
||||
String[] modelIds = { vadModelId, asrModelId, ttsModelId, memModelId, intentModelId, llmModelId, vllmModelId };
|
||||
String intentLLMModelId = null;
|
||||
String memLocalShortLLMModelId = null;
|
||||
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
-- VLLM模型供应器
|
||||
delete from `ai_model_provider` where id = 'SYSTEM_VLLM_openai';
|
||||
INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
|
||||
('SYSTEM_VLLM_openai', 'VLLM', 'openai', 'OpenAI接口', '[{"key":"base_url","label":"基础URL","type":"string"},{"key":"model_name","label":"模型名称","type":"string"},{"key":"api_key","label":"API密钥","type":"string"}]', 9, 1, NOW(), 1, NOW());
|
||||
|
||||
-- VLLM模型配置
|
||||
delete from `ai_model_config` where id = 'VLLM_ChatGLMVLLM';
|
||||
INSERT INTO `ai_model_config` VALUES ('VLLM_ChatGLMVLLM', 'VLLM', 'ChatGLMVLLM', '智谱视觉AI', 1, 1, '{\"type\": \"openai\", \"model_name\": \"glm-4v-flash\", \"base_url\": \"https://open.bigmodel.cn/api/paas/v4/\", \"api_key\": \"你的api_key\"}', NULL, NULL, 1, NULL, NULL, NULL, NULL);
|
||||
|
||||
-- 更新文档
|
||||
UPDATE `ai_model_config` SET
|
||||
`doc_link` = 'https://bigmodel.cn/usercenter/proj-mgmt/apikeys',
|
||||
`remark` = '智谱视觉AI配置说明：
|
||||
1. 访问 https://bigmodel.cn/usercenter/proj-mgmt/apikeys
|
||||
2. 注册并获取API密钥
|
||||
3. 填入配置文件中' WHERE `id` = 'VLLM_ChatGLMVLLM';
|
||||
|
||||
|
||||
-- 添加参数
|
||||
INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (113, 'server.http_port', '8003', 'number', 1, 'http服务的端口，用于启动视觉分析接口');
|
||||
INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) VALUES (114, 'server.vision_explain', 'null', 'string', 1, '视觉分析接口地址，用于下发到设备，多个用;分隔');
|
||||
|
||||
-- 智能体表增加VLLM模型配置
|
||||
ALTER TABLE `ai_agent`
|
||||
ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;
|
||||
|
||||
-- 智能体模版表增加VLLM模型配置
|
||||
ALTER TABLE `ai_agent_template`
|
||||
ADD COLUMN `vllm_model_id` varchar(32) NULL DEFAULT 'VLLM_ChatGLMVLLM' COMMENT '视觉模型标识' AFTER `llm_model_id`;
|
||||
@@ -0,0 +1,45 @@
|
||||
-- VLLM模型供应器
|
||||
delete from `ai_model_provider` where id = 'SYSTEM_ASR_DoubaoStreamASR';
|
||||
INSERT INTO `ai_model_provider` (`id`, `model_type`, `provider_code`, `name`, `fields`, `sort`, `creator`, `create_date`, `updater`, `update_date`) VALUES
|
||||
('SYSTEM_ASR_DoubaoStreamASR', 'ASR', 'doubao_stream', '火山引擎语音识别(流式)', '[{"key":"appid","label":"应用ID","type":"string"},{"key":"access_token","label":"访问令牌","type":"string"},{"key":"cluster","label":"集群","type":"string"},{"key":"boosting_table_name","label":"热词文件名称","type":"string"},{"key":"correct_table_name","label":"替换词文件名称","type":"string"},{"key":"output_dir","label":"输出目录","type":"string"}]', 3, 1, NOW(), 1, NOW());
|
||||
|
||||
|
||||
-- VLLM模型配置
|
||||
delete from `ai_model_config` where id = 'ASR_DoubaoStreamASR';
|
||||
INSERT INTO `ai_model_config` VALUES ('ASR_DoubaoStreamASR', 'ASR', 'DoubaoStreamASR', '豆包语音识别(流式)', 0, 1, '{\"type\": \"doubao_stream\", \"appid\": \"\", \"access_token\": \"\", \"cluster\": \"volcengine_input_common\", \"output_dir\": \"tmp/\"}', NULL, NULL, 3, NULL, NULL, NULL, NULL);
|
||||
|
||||
|
||||
-- 更新豆包ASR配置说明
|
||||
UPDATE `ai_model_config` SET
|
||||
`doc_link` = 'https://console.volcengine.com/speech/app',
|
||||
`remark` = '豆包ASR配置说明：
|
||||
1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
|
||||
2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
|
||||
3. 需要在火山引擎控制台创建应用并获取appid和access_token
|
||||
4. 支持中文语音识别
|
||||
5. 需要网络连接
|
||||
6. 输出文件保存在tmp/目录
|
||||
申请步骤：
|
||||
1. 访问 https://console.volcengine.com/speech/app
|
||||
2. 创建新应用
|
||||
3. 获取appid和access_token
|
||||
4. 填入配置文件中
|
||||
如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
|
||||
' WHERE `id` = 'ASR_DoubaoASR';
|
||||
|
||||
UPDATE `ai_model_config` SET
|
||||
`doc_link` = 'https://console.volcengine.com/speech/app',
|
||||
`remark` = '豆包ASR配置说明：
|
||||
1. 豆包ASR和豆包(流式)ASR的区别是：豆包ASR是按次收费，豆包(流式)ASR是按时收费
|
||||
2. 一般来说按次收费的更便宜，但是豆包(流式)ASR使用了大模型技术，效果更好
|
||||
3. 需要在火山引擎控制台创建应用并获取appid和access_token
|
||||
4. 支持中文语音识别
|
||||
5. 需要网络连接
|
||||
6. 输出文件保存在tmp/目录
|
||||
申请步骤：
|
||||
1. 访问 https://console.volcengine.com/speech/app
|
||||
2. 创建新应用
|
||||
3. 获取appid和access_token
|
||||
4. 填入配置文件中
|
||||
如需设置热词，请参考：https://www.volcengine.com/docs/6561/155738
|
||||
' WHERE `id` = 'ASR_DoubaoStreamASR';
|
||||
@@ -0,0 +1,14 @@
|
||||
-- VLLM模型配置
|
||||
delete from `ai_model_config` where id = 'VLLM_QwenVLVLLM';
|
||||
INSERT INTO `ai_model_config` VALUES ('VLLM_QwenVLVLLM', 'VLLM', 'QwenVLVLLM', '千问视觉模型', 0, 1, '{\"type\": \"openai\", \"model_name\": \"qwen2.5-vl-3b-instruct\", \"base_url\": \"https://dashscope.aliyuncs.com/compatible-mode/v1\", \"api_key\": \"你的api_key\"}', NULL, NULL, 2, NULL, NULL, NULL, NULL);
|
||||
|
||||
-- 更新文档
|
||||
UPDATE `ai_model_config` SET
|
||||
`doc_link` = 'https://bailian.console.aliyun.com/?tab=api#/api/?type=model&url=https%3A%2F%2Fhelp.aliyun.com%2Fdocument_detail%2F2845564.html&renderType=iframe',
|
||||
`remark` = '千问视觉模型配置说明：
|
||||
1. 访问 https://bailian.console.aliyun.com/?tab=model#/api-key
|
||||
2. 注册并获取API密钥
|
||||
3. 填入配置文件中' WHERE `id` = 'VLLM_QwenVLVLLM';
|
||||
|
||||
-- 删除参数，这两个参数已挪至python配置文件
|
||||
delete from `sys_params` where id in (113,114);
|
||||
@@ -169,4 +169,25 @@ databaseChangeLog:
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202505271414.sql
|
||||
path: classpath:db/changelog/202505271414.sql
|
||||
- changeSet:
|
||||
id: 202506010920
|
||||
author: hrz
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202506010920.sql
|
||||
- changeSet:
|
||||
id: 202506031639
|
||||
author: hrz
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202506031639.sql
|
||||
- changeSet:
|
||||
id: 202506032232
|
||||
author: hrz
|
||||
changes:
|
||||
- sqlFile:
|
||||
encoding: utf8
|
||||
path: classpath:db/changelog/202506032232.sql
|
||||
@@ -14,10 +14,10 @@
|
||||
</div>
|
||||
</div>
|
||||
<div class="device-name">
|
||||
设备型号：{{ device.ttsModelName }}
|
||||
语言模型：{{ device.llmModelName }}
|
||||
</div>
|
||||
<div class="device-name">
|
||||
音色模型：{{ device.ttsVoiceName }}
|
||||
音色模型：{{ device.ttsModelName }} ({{ device.ttsVoiceName }})
|
||||
</div>
|
||||
<div style="display: flex;gap: 10px;align-items: center;">
|
||||
<div class="settings-btn" @click="handleConfigure">
|
||||
|
||||
@@ -30,6 +30,9 @@
|
||||
<el-menu-item index="llm">
|
||||
<span class="menu-text">大语言模型</span>
|
||||
</el-menu-item>
|
||||
<el-menu-item index="vllm">
|
||||
<span class="menu-text">视觉大模型</span>
|
||||
</el-menu-item>
|
||||
<el-menu-item index="intent">
|
||||
<span class="menu-text">意图识别</span>
|
||||
</el-menu-item>
|
||||
@@ -173,6 +176,7 @@ export default {
|
||||
vad: '语言活动检测模型(VAD)',
|
||||
asr: '语音识别模型(ASR)',
|
||||
llm: '大语言模型（LLM）',
|
||||
vllm: '视觉大模型（VLLM）',
|
||||
intent: '意图识别模型(Intent)',
|
||||
tts: '语音合成模型(TTS)',
|
||||
memory: '记忆模型(Memory)'
|
||||
|
||||
@@ -64,7 +64,27 @@
|
||||
</el-form-item>
|
||||
</div>
|
||||
<div class="form-column">
|
||||
<el-form-item v-for="(model, index) in models" :key="`model-${index}`" :label="model.label"
|
||||
<div class="model-row">
|
||||
<el-form-item label="语音活动检测(VAD)" class="model-item">
|
||||
<div class="model-select-wrapper">
|
||||
<el-select v-model="form.model.vadModelId" filterable placeholder="请选择" class="form-select"
|
||||
@change="handleModelChange('VAD', $event)">
|
||||
<el-option v-for="(item, optionIndex) in modelOptions['VAD']"
|
||||
:key="`option-vad-${optionIndex}`" :label="item.label" :value="item.value" />
|
||||
</el-select>
|
||||
</div>
|
||||
</el-form-item>
|
||||
<el-form-item label="语音识别(ASR)" class="model-item">
|
||||
<div class="model-select-wrapper">
|
||||
<el-select v-model="form.model.asrModelId" filterable placeholder="请选择" class="form-select"
|
||||
@change="handleModelChange('ASR', $event)">
|
||||
<el-option v-for="(item, optionIndex) in modelOptions['ASR']"
|
||||
:key="`option-asr-${optionIndex}`" :label="item.label" :value="item.value" />
|
||||
</el-select>
|
||||
</div>
|
||||
</el-form-item>
|
||||
</div>
|
||||
<el-form-item v-for="(model, index) in models.slice(2)" :key="`model-${index}`" :label="model.label"
|
||||
class="model-item">
|
||||
<div class="model-select-wrapper">
|
||||
<el-select v-model="form.model[model.key]" filterable placeholder="请选择" class="form-select"
|
||||
@@ -148,6 +168,7 @@ export default {
|
||||
vadModelId: "",
|
||||
asrModelId: "",
|
||||
llmModelId: "",
|
||||
vllmModelId: "",
|
||||
memModelId: "",
|
||||
intentModelId: "",
|
||||
}
|
||||
@@ -156,6 +177,7 @@ export default {
|
||||
{ label: '语音活动检测(VAD)', key: 'vadModelId', type: 'VAD' },
|
||||
{ label: '语音识别(ASR)', key: 'asrModelId', type: 'ASR' },
|
||||
{ label: '大语言模型(LLM)', key: 'llmModelId', type: 'LLM' },
|
||||
{ label: '视觉大模型(VLLM)', key: 'vllmModelId', type: 'VLLM' },
|
||||
{ label: '意图识别(Intent)', key: 'intentModelId', type: 'Intent' },
|
||||
{ label: '记忆(Memory)', key: 'memModelId', type: 'Memory' },
|
||||
{ label: '语音合成(TTS)', key: 'ttsModelId', type: 'TTS' },
|
||||
@@ -189,6 +211,7 @@ export default {
|
||||
asrModelId: this.form.model.asrModelId,
|
||||
vadModelId: this.form.model.vadModelId,
|
||||
llmModelId: this.form.model.llmModelId,
|
||||
vllmModelId: this.form.model.vllmModelId,
|
||||
ttsModelId: this.form.model.ttsModelId,
|
||||
ttsVoiceId: this.form.ttsVoiceId,
|
||||
chatHistoryConf: this.form.chatHistoryConf,
|
||||
@@ -236,6 +259,7 @@ export default {
|
||||
vadModelId: "",
|
||||
asrModelId: "",
|
||||
llmModelId: "",
|
||||
vllmModelId: "",
|
||||
memModelId: "",
|
||||
intentModelId: "",
|
||||
}
|
||||
@@ -289,6 +313,7 @@ export default {
|
||||
vadModelId: templateData.vadModelId || this.form.model.vadModelId,
|
||||
asrModelId: templateData.asrModelId || this.form.model.asrModelId,
|
||||
llmModelId: templateData.llmModelId || this.form.model.llmModelId,
|
||||
vllmModelId: templateData.vllmModelId || this.form.model.vllmModelId,
|
||||
memModelId: templateData.memModelId || this.form.model.memModelId,
|
||||
intentModelId: templateData.intentModelId || this.form.model.intentModelId
|
||||
}
|
||||
@@ -305,6 +330,7 @@ export default {
|
||||
vadModelId: data.data.vadModelId,
|
||||
asrModelId: data.data.asrModelId,
|
||||
llmModelId: data.data.llmModelId,
|
||||
vllmModelId: data.data.vllmModelId,
|
||||
memModelId: data.data.memModelId,
|
||||
intentModelId: data.data.intentModelId
|
||||
}
|
||||
@@ -587,6 +613,25 @@ export default {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.model-row {
|
||||
display: flex;
|
||||
gap: 20px;
|
||||
margin-bottom: 6px;
|
||||
}
|
||||
|
||||
.model-row .model-item {
|
||||
flex: 1;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.model-row .el-form-item__label {
|
||||
font-size: 12px !important;
|
||||
color: #3d4566 !important;
|
||||
font-weight: 400;
|
||||
line-height: 22px;
|
||||
padding-bottom: 2px;
|
||||
}
|
||||
|
||||
.function-icons {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
|
||||
+24
-11
@@ -1,13 +1,14 @@
|
||||
import asyncio
|
||||
import sys
|
||||
import uuid
|
||||
import signal
|
||||
import asyncio
|
||||
from aioconsole import ainput
|
||||
from config.settings import load_config
|
||||
from core.websocket_server import WebSocketServer
|
||||
from core.ota_server import SimpleOtaServer
|
||||
from core.utils.util import check_ffmpeg_installed
|
||||
from config.logger import setup_logging
|
||||
from core.utils.util import get_local_ip
|
||||
from aioconsole import ainput
|
||||
from core.http_server import SimpleHttpServer
|
||||
from core.websocket_server import WebSocketServer
|
||||
from core.utils.util import check_ffmpeg_installed
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
@@ -45,25 +46,37 @@ async def main():
|
||||
check_ffmpeg_installed()
|
||||
config = load_config()
|
||||
|
||||
# 默认使用manager-api的secret作为auth_key
|
||||
# 如果secret为空，则生成随机密钥
|
||||
# auth_key用于jwt认证，比如视觉分析接口的jwt认证
|
||||
auth_key = config.get("manager-api", {}).get("secret", "")
|
||||
if not auth_key or len(auth_key) == 0 or "你" in auth_key:
|
||||
auth_key = str(uuid.uuid4().hex)
|
||||
config["server"]["auth_key"] = auth_key
|
||||
|
||||
# 添加 stdin 监控任务
|
||||
stdin_task = asyncio.create_task(monitor_stdin())
|
||||
|
||||
# 启动 WebSocket 服务器
|
||||
ws_server = WebSocketServer(config)
|
||||
ws_task = asyncio.create_task(ws_server.start())
|
||||
ota_task = None
|
||||
# 启动 Simple http 服务器
|
||||
ota_server = SimpleHttpServer(config)
|
||||
ota_task = asyncio.create_task(ota_server.start())
|
||||
|
||||
read_config_from_api = config.get("read_config_from_api", False)
|
||||
port = int(config["server"].get("http_port", 8003))
|
||||
if not read_config_from_api:
|
||||
# 启动 Simple OTA 服务器
|
||||
ota_server = SimpleOtaServer(config)
|
||||
ota_task = asyncio.create_task(ota_server.start())
|
||||
|
||||
logger.bind(tag=TAG).info(
|
||||
"OTA接口是\t\thttp://{}:{}/xiaozhi/ota/",
|
||||
get_local_ip(),
|
||||
config["server"]["ota_port"],
|
||||
port,
|
||||
)
|
||||
logger.bind(tag=TAG).info(
|
||||
"视觉分析接口是\thttp://{}:{}/mcp/vision/explain",
|
||||
get_local_ip(),
|
||||
port,
|
||||
)
|
||||
|
||||
# 获取WebSocket配置，使用安全的默认值
|
||||
websocket_port = 8000
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
# 然后你想修改覆盖修改什么配置，就修改【.config.yaml】文件，而不是修改【config.yaml】文件
|
||||
# 系统会优先读取【data/.config.yaml】文件的配置，如果【.config.yaml】文件里的配置不存在，系统会自动去读取【config.yaml】文件的配置。
|
||||
# 这样做，可以最简化配置，保护您的密钥安全。
|
||||
# 如果你使用了智控台，那么以下所有配置，都不会生效，请在智控台中修改配置
|
||||
|
||||
# #####################################################################################
|
||||
# #############################以下是服务器基本运行配置####################################
|
||||
@@ -9,13 +10,21 @@ server:
|
||||
# 服务器监听地址和端口(Server listening address and port)
|
||||
ip: 0.0.0.0
|
||||
port: 8000
|
||||
# OTA接口的端口号
|
||||
ota_port: 8002
|
||||
# http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
|
||||
http_port: 8003
|
||||
# 这个websocket配置是指ota接口向设备发送的websocket地址
|
||||
# 如果按默认的写法，ota接口会自动生成websocket地址。这个地址你可以直接用浏览器访问ota接口确认一下
|
||||
# 如果按默认的写法，ota接口会自动生成websocket地址，并输出在启动日志里，这个地址你可以直接用浏览器访问ota接口确认一下
|
||||
# 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
|
||||
# 所以如果你使用docker部署或使用公网部署时，请设置正确的websocket地址
|
||||
# 所以如果你使用docker部署时，将websocket设置成局域网地址
|
||||
# 如果你使用公网部署时，将vwebsocket设置成公网地址
|
||||
websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
|
||||
# 视觉分析接口地址
|
||||
# 向设备发送的视觉分析的接口地址
|
||||
# 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
|
||||
# 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
|
||||
# 所以如果你使用docker部署时，将vision_explain设置成局域网地址
|
||||
# 如果你使用公网部署时，将vision_explain设置成公网地址
|
||||
vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
|
||||
# OTA返回信息时区偏移量
|
||||
timezone_offset: +8
|
||||
# 认证配置
|
||||
@@ -160,6 +169,8 @@ selected_module:
|
||||
ASR: FunASR
|
||||
# 将根据配置名称对应的type调用实际的LLM适配器
|
||||
LLM: ChatGLMLLM
|
||||
# 视觉语言大模型
|
||||
VLLM: ChatGLMVLLM
|
||||
# TTS将根据配置名称对应的type调用实际的TTS适配器
|
||||
TTS: EdgeTTS
|
||||
# 记忆模块，默认不开启记忆；如果想使用超长记忆，推荐使用mem0ai；如果注重隐私，请使用本地的mem_local_short
|
||||
@@ -253,6 +264,8 @@ ASR:
|
||||
DoubaoASR:
|
||||
# 可以在这里申请相关Key等信息
|
||||
# https://console.volcengine.com/speech/app
|
||||
# DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
|
||||
# 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
|
||||
type: doubao
|
||||
appid: 你的火山引擎语音合成服务appid
|
||||
access_token: 你的火山引擎语音合成服务access_token
|
||||
@@ -261,6 +274,20 @@ ASR:
|
||||
boosting_table_name: （选填）你的热词文件名称
|
||||
correct_table_name: （选填）你的替换词文件名称
|
||||
output_dir: tmp/
|
||||
DoubaoStreamASR:
|
||||
# 可以在这里申请相关Key等信息
|
||||
# https://console.volcengine.com/speech/app
|
||||
# DoubaoASR和DoubaoStreamASR的区别是：DoubaoASR是按次收费，DoubaoStreamASR是按时收费
|
||||
# 开通地址https://console.volcengine.com/speech/service/10011
|
||||
# 一般来说按次收费的更便宜，但是DoubaoStreamASR使用了大模型技术，效果更好
|
||||
type: doubao_stream
|
||||
appid: 你的火山引擎语音合成服务appid
|
||||
access_token: 你的火山引擎语音合成服务access_token
|
||||
cluster: volcengine_input_common
|
||||
# 热词、替换词使用流程：https://www.volcengine.com/docs/6561/155738
|
||||
boosting_table_name: （选填）你的热词文件名称
|
||||
correct_table_name: （选填）你的替换词文件名称
|
||||
output_dir: tmp/
|
||||
TencentASR:
|
||||
# token申请地址：https://console.cloud.tencent.com/cam/capi
|
||||
# 免费领取资源：https://console.cloud.tencent.com/asr/resourcebundle
|
||||
@@ -432,6 +459,21 @@ LLM:
|
||||
# Xinference服务地址和模型名称
|
||||
model_name: qwen2.5:3b-AWQ # 使用的小模型名称，用于意图识别
|
||||
base_url: http://localhost:9997 # Xinference服务地址
|
||||
# VLLM配置（视觉语言大模型）
|
||||
VLLM:
|
||||
ChatGLMVLLM:
|
||||
type: openai
|
||||
# glm-4v-flash是智谱免费AI的视觉模型，需要先在智谱AI平台创建API密钥并获取api_key
|
||||
# 可在这里找到你的api key https://bigmodel.cn/usercenter/proj-mgmt/apikeys
|
||||
model_name: glm-4v-flash # 智谱AI的视觉模型
|
||||
url: https://open.bigmodel.cn/api/paas/v4/
|
||||
api_key: 你的api_key
|
||||
QwenVLVLLM:
|
||||
type: openai
|
||||
model_name: qwen2.5-vl-3b-instruct
|
||||
url: https://dashscope.aliyuncs.com/compatible-mode/v1
|
||||
# 可在这里找到你的api key https://bailian.console.aliyun.com/?apiKey=1#/api-key
|
||||
api_key: 你的api_key
|
||||
TTS:
|
||||
# 当前支持的type为edge、doubao，可自行适配
|
||||
EdgeTTS:
|
||||
|
||||
@@ -59,10 +59,14 @@ def get_config_from_api(config):
|
||||
"url": config["manager-api"].get("url", ""),
|
||||
"secret": config["manager-api"].get("secret", ""),
|
||||
}
|
||||
# server的配置以本地为准
|
||||
if config.get("server"):
|
||||
config_data["server"] = {
|
||||
"ip": config["server"].get("ip", ""),
|
||||
"port": config["server"].get("port", ""),
|
||||
"http_port": config["server"].get("http_port", ""),
|
||||
"vision_explain": config["server"].get("vision_explain", ""),
|
||||
"auth_key": config["server"].get("auth_key", ""),
|
||||
}
|
||||
return config_data
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ from loguru import logger
|
||||
from config.config_loader import load_config
|
||||
from config.settings import check_config_file
|
||||
|
||||
SERVER_VERSION = "0.5.1"
|
||||
SERVER_VERSION = "0.5.4"
|
||||
_logger_initialized = False
|
||||
|
||||
|
||||
@@ -48,7 +48,11 @@ def setup_logging():
|
||||
|
||||
# 第一次初始化时配置日志
|
||||
if not _logger_initialized:
|
||||
logger.configure(extra={"selected_module": log_config.get("selected_module", "00000000000000")}) # 新增配置
|
||||
logger.configure(
|
||||
extra={
|
||||
"selected_module": log_config.get("selected_module", "00000000000000")
|
||||
}
|
||||
) # 新增配置
|
||||
log_format = log_config.get(
|
||||
"log_format",
|
||||
"<green>{time:YYMMDD HH:mm:ss}</green>[{version}_{extra[selected_module]}][<light-blue>{extra[tag]}</light-blue>]-<level>{level}</level>-<light-green>{message}</light-green>",
|
||||
@@ -62,7 +66,9 @@ def setup_logging():
|
||||
log_format = log_format.replace("{version}", SERVER_VERSION)
|
||||
log_format = log_format.replace("{selected_module}", selected_module_str)
|
||||
log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
|
||||
log_format_file = log_format_file.replace("{selected_module}", selected_module_str)
|
||||
log_format_file = log_format_file.replace(
|
||||
"{selected_module}", selected_module_str
|
||||
)
|
||||
|
||||
log_level = log_config.get("log_level", "INFO")
|
||||
log_dir = log_config.get("log_dir", "tmp")
|
||||
@@ -116,15 +122,25 @@ def update_module_string(selected_module_str):
|
||||
log_format = log_format.replace("{version}", SERVER_VERSION)
|
||||
log_format = log_format.replace("{selected_module}", selected_module_str)
|
||||
log_format_file = log_format_file.replace("{version}", SERVER_VERSION)
|
||||
log_format_file = log_format_file.replace("{selected_module}", selected_module_str)
|
||||
log_format_file = log_format_file.replace(
|
||||
"{selected_module}", selected_module_str
|
||||
)
|
||||
|
||||
logger.remove()
|
||||
logger.add(sys.stdout, format=log_format, level=log_config.get("log_level", "INFO"), filter=formatter)
|
||||
logger.add(
|
||||
os.path.join(log_config.get("log_dir", "tmp"), log_config.get("log_file", "server.log")),
|
||||
sys.stdout,
|
||||
format=log_format,
|
||||
level=log_config.get("log_level", "INFO"),
|
||||
filter=formatter,
|
||||
)
|
||||
logger.add(
|
||||
os.path.join(
|
||||
log_config.get("log_dir", "tmp"),
|
||||
log_config.get("log_file", "server.log"),
|
||||
),
|
||||
format=log_format_file,
|
||||
level=log_config.get("log_level", "INFO"),
|
||||
filter=formatter
|
||||
filter=formatter,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
|
||||
@@ -8,6 +8,15 @@
|
||||
server:
|
||||
ip: 0.0.0.0
|
||||
port: 8000
|
||||
# http服务的端口，用于视觉分析接口
|
||||
http_port: 8003
|
||||
# 视觉分析接口地址
|
||||
# 向设备发送的视觉分析的接口地址
|
||||
# 如果按下面默认的写法，系统会自动生成视觉识别地址，并输出在启动日志里，这个地址你可以直接用浏览器访问确认一下
|
||||
# 当你使用docker部署或使用公网部署(使用ssl、域名)时，不一定准确
|
||||
# 所以如果你使用docker部署时，将vision_explain设置成局域网地址
|
||||
# 如果你使用公网部署时，将vision_explain设置成公网地址
|
||||
vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
|
||||
manager-api:
|
||||
# 你的manager-api的地址，最好使用局域网ip
|
||||
url: http://127.0.0.1:8002/xiaozhi
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
from aiohttp import web
|
||||
from config.logger import setup_logging
|
||||
|
||||
|
||||
class BaseHandler:
|
||||
def __init__(self, config: dict):
|
||||
self.config = config
|
||||
self.logger = setup_logging()
|
||||
|
||||
def _add_cors_headers(self, response):
|
||||
"""添加CORS头信息"""
|
||||
response.headers["Access-Control-Allow-Headers"] = (
|
||||
"client-id, content-type, device-id"
|
||||
)
|
||||
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
+12
-53
@@ -1,18 +1,15 @@
|
||||
import json
|
||||
import time
|
||||
import asyncio
|
||||
from aiohttp import web
|
||||
from config.logger import setup_logging
|
||||
from core.connection import ConnectionHandler
|
||||
from core.utils.util import get_local_ip, initialize_modules
|
||||
from core.utils.util import get_local_ip
|
||||
from core.api.base_handler import BaseHandler
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
class SimpleOtaServer:
|
||||
class OTAHandler(BaseHandler):
|
||||
def __init__(self, config: dict):
|
||||
self.config = config
|
||||
self.logger = setup_logging()
|
||||
super().__init__(config)
|
||||
|
||||
def _get_websocket_url(self, local_ip: str, port: int) -> str:
|
||||
"""获取websocket地址
|
||||
@@ -25,41 +22,15 @@ class SimpleOtaServer:
|
||||
str: websocket地址
|
||||
"""
|
||||
server_config = self.config["server"]
|
||||
websocket_config = server_config.get("websocket")
|
||||
websocket_config = server_config.get("websocket", "")
|
||||
|
||||
if websocket_config and "你" not in websocket_config:
|
||||
if "你的" not in websocket_config:
|
||||
return websocket_config
|
||||
else:
|
||||
return f"ws://{local_ip}:{port}/xiaozhi/v1/"
|
||||
|
||||
async def start(self):
|
||||
server_config = self.config["server"]
|
||||
host = server_config.get("ip", "0.0.0.0")
|
||||
port = int(server_config.get("ota_port"))
|
||||
|
||||
if port:
|
||||
app = web.Application()
|
||||
# 添加路由
|
||||
app.add_routes(
|
||||
[
|
||||
web.get("/xiaozhi/ota/", self._handle_ota_get_request),
|
||||
web.post("/xiaozhi/ota/", self._handle_ota_request),
|
||||
web.options("/xiaozhi/ota/", self._handle_ota_request),
|
||||
]
|
||||
)
|
||||
|
||||
# 运行服务
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, host, port)
|
||||
await site.start()
|
||||
|
||||
# 保持服务运行
|
||||
while True:
|
||||
await asyncio.sleep(3600) # 每隔 1 小时检查一次
|
||||
|
||||
async def _handle_ota_request(self, request):
|
||||
"""处理 /xiaozhi/ota/ 的 POST 请求"""
|
||||
async def handle_post(self, request):
|
||||
"""处理 OTA POST 请求"""
|
||||
try:
|
||||
data = await request.text()
|
||||
self.logger.bind(tag=TAG).debug(f"OTA请求方法: {request.method}")
|
||||
@@ -75,11 +46,9 @@ class SimpleOtaServer:
|
||||
data_json = json.loads(data)
|
||||
|
||||
server_config = self.config["server"]
|
||||
host = server_config.get("ip", "0.0.0.0")
|
||||
port = int(server_config.get("port", 8000))
|
||||
local_ip = get_local_ip()
|
||||
|
||||
# OTA基础信息
|
||||
return_json = {
|
||||
"server_time": {
|
||||
"timestamp": int(round(time.time() * 1000)),
|
||||
@@ -104,16 +73,11 @@ class SimpleOtaServer:
|
||||
content_type="application/json",
|
||||
)
|
||||
finally:
|
||||
# 添加header，允许跨域访问
|
||||
response.headers["Access-Control-Allow-Headers"] = (
|
||||
"client-id, content-type, device-id"
|
||||
)
|
||||
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
self._add_cors_headers(response)
|
||||
return response
|
||||
|
||||
async def _handle_ota_get_request(self, request):
|
||||
"""处理 /xiaozhi/ota/ 的 GET 请求"""
|
||||
async def handle_get(self, request):
|
||||
"""处理 OTA GET 请求"""
|
||||
try:
|
||||
server_config = self.config["server"]
|
||||
local_ip = get_local_ip()
|
||||
@@ -125,10 +89,5 @@ class SimpleOtaServer:
|
||||
self.logger.bind(tag=TAG).error(f"OTA GET请求异常: {e}")
|
||||
response = web.Response(text="OTA接口异常", content_type="text/plain")
|
||||
finally:
|
||||
# 添加header，允许跨域访问
|
||||
response.headers["Access-Control-Allow-Headers"] = (
|
||||
"client-id, content-type, device-id"
|
||||
)
|
||||
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
self._add_cors_headers(response)
|
||||
return response
|
||||
@@ -0,0 +1,184 @@
|
||||
import json
|
||||
import copy
|
||||
from aiohttp import web
|
||||
from config.logger import setup_logging
|
||||
from core.utils.util import get_vision_url, is_valid_image_file
|
||||
from core.utils.vllm import create_instance
|
||||
from config.config_loader import get_private_config_from_api
|
||||
from core.utils.auth import AuthToken
|
||||
import base64
|
||||
from typing import Tuple, Optional
|
||||
|
||||
TAG = __name__
|
||||
|
||||
# 设置最大文件大小为5MB
|
||||
MAX_FILE_SIZE = 5 * 1024 * 1024
|
||||
|
||||
|
||||
class VisionHandler:
|
||||
def __init__(self, config: dict):
|
||||
self.config = config
|
||||
self.logger = setup_logging()
|
||||
# 初始化认证工具
|
||||
self.auth = AuthToken(config["server"]["auth_key"])
|
||||
|
||||
def _create_error_response(self, message: str) -> dict:
|
||||
"""创建统一的错误响应格式"""
|
||||
return {"success": False, "message": message}
|
||||
|
||||
def _verify_auth_token(self, request) -> Tuple[bool, Optional[str]]:
|
||||
"""验证认证token"""
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
if not auth_header.startswith("Bearer "):
|
||||
return False, None
|
||||
|
||||
token = auth_header[7:] # 移除"Bearer "前缀
|
||||
return self.auth.verify_token(token)
|
||||
|
||||
async def handle_post(self, request):
|
||||
"""处理 MCP Vision POST 请求"""
|
||||
response = None # 初始化response变量
|
||||
try:
|
||||
# 验证token
|
||||
is_valid, token_device_id = self._verify_auth_token(request)
|
||||
if not is_valid:
|
||||
response = web.Response(
|
||||
text=json.dumps(
|
||||
self._create_error_response("无效的认证token或token已过期")
|
||||
),
|
||||
content_type="application/json",
|
||||
status=401,
|
||||
)
|
||||
return response
|
||||
|
||||
# 获取请求头信息
|
||||
device_id = request.headers.get("Device-Id", "")
|
||||
client_id = request.headers.get("Client-Id", "")
|
||||
if device_id != token_device_id:
|
||||
return web.Response(
|
||||
text=json.dumps(self._create_error_response("设备ID与token不匹配")),
|
||||
content_type="application/json",
|
||||
status=401,
|
||||
)
|
||||
# 解析multipart/form-data请求
|
||||
reader = await request.multipart()
|
||||
|
||||
# 读取question字段
|
||||
question_field = await reader.next()
|
||||
if question_field is None:
|
||||
raise ValueError("缺少问题字段")
|
||||
question = await question_field.text()
|
||||
self.logger.bind(tag=TAG).debug(f"Question: {question}")
|
||||
|
||||
# 读取图片文件
|
||||
image_field = await reader.next()
|
||||
if image_field is None:
|
||||
raise ValueError("缺少图片文件")
|
||||
|
||||
# 读取图片数据
|
||||
image_data = await image_field.read()
|
||||
if not image_data:
|
||||
raise ValueError("图片数据为空")
|
||||
|
||||
# 检查文件大小
|
||||
if len(image_data) > MAX_FILE_SIZE:
|
||||
raise ValueError(
|
||||
f"图片大小超过限制，最大允许{MAX_FILE_SIZE/1024/1024}MB"
|
||||
)
|
||||
|
||||
# 检查文件格式
|
||||
if not is_valid_image_file(image_data):
|
||||
raise ValueError(
|
||||
"不支持的文件格式，请上传有效的图片文件（支持JPEG、PNG、GIF、BMP、TIFF、WEBP格式）"
|
||||
)
|
||||
|
||||
# 将图片转换为base64编码
|
||||
image_base64 = base64.b64encode(image_data).decode("utf-8")
|
||||
|
||||
# 如果开启了智控台，则从智控台获取模型配置
|
||||
current_config = copy.deepcopy(self.config)
|
||||
read_config_from_api = current_config.get("read_config_from_api", False)
|
||||
if read_config_from_api:
|
||||
current_config = get_private_config_from_api(
|
||||
current_config,
|
||||
device_id,
|
||||
client_id,
|
||||
)
|
||||
|
||||
select_vllm_module = current_config["selected_module"].get("VLLM")
|
||||
if not select_vllm_module:
|
||||
raise ValueError("您还未设置默认的视觉分析模块")
|
||||
|
||||
vllm_type = (
|
||||
select_vllm_module
|
||||
if "type" not in current_config["VLLM"][select_vllm_module]
|
||||
else current_config["VLLM"][select_vllm_module]["type"]
|
||||
)
|
||||
|
||||
if not vllm_type:
|
||||
raise ValueError(f"无法找到VLLM模块对应的供应器{vllm_type}")
|
||||
|
||||
vllm = create_instance(
|
||||
vllm_type, current_config["VLLM"][select_vllm_module]
|
||||
)
|
||||
|
||||
result = vllm.response(question, image_base64)
|
||||
|
||||
return_json = {
|
||||
"success": True,
|
||||
"result": result,
|
||||
}
|
||||
|
||||
response = web.Response(
|
||||
text=json.dumps(return_json, separators=(",", ":")),
|
||||
content_type="application/json",
|
||||
)
|
||||
except ValueError as e:
|
||||
self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
|
||||
return_json = self._create_error_response(str(e))
|
||||
response = web.Response(
|
||||
text=json.dumps(return_json, separators=(",", ":")),
|
||||
content_type="application/json",
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"MCP Vision POST请求异常: {e}")
|
||||
return_json = self._create_error_response("处理请求时发生错误")
|
||||
response = web.Response(
|
||||
text=json.dumps(return_json, separators=(",", ":")),
|
||||
content_type="application/json",
|
||||
)
|
||||
finally:
|
||||
if response:
|
||||
self._add_cors_headers(response)
|
||||
return response
|
||||
|
||||
async def handle_get(self, request):
|
||||
"""处理 MCP Vision GET 请求"""
|
||||
try:
|
||||
vision_explain = get_vision_url(self.config)
|
||||
if vision_explain and len(vision_explain) > 0 and "null" != vision_explain:
|
||||
message = (
|
||||
f"MCP Vision 接口运行正常，视觉解释接口地址是：{vision_explain}"
|
||||
)
|
||||
else:
|
||||
message = "MCP Vision 接口运行不正常，请打开data目录下的.config.yaml文件，找到【server.vision_explain】，设置好地址"
|
||||
|
||||
response = web.Response(text=message, content_type="text/plain")
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"MCP Vision GET请求异常: {e}")
|
||||
return_json = self._create_error_response("服务器内部错误")
|
||||
response = web.Response(
|
||||
text=json.dumps(return_json, separators=(",", ":")),
|
||||
content_type="application/json",
|
||||
)
|
||||
finally:
|
||||
self._add_cors_headers(response)
|
||||
return response
|
||||
|
||||
def _add_cors_headers(self, response):
|
||||
"""添加CORS头信息"""
|
||||
response.headers["Access-Control-Allow-Headers"] = (
|
||||
"client-id, content-type, device-id"
|
||||
)
|
||||
response.headers["Access-Control-Allow-Credentials"] = "true"
|
||||
response.headers["Access-Control-Allow-Origin"] = "*"
|
||||
@@ -10,27 +10,31 @@ import threading
|
||||
import traceback
|
||||
import subprocess
|
||||
import websockets
|
||||
from core.handle.mcpHandle import call_mcp_tool
|
||||
from core.utils.util import (
|
||||
extract_json_from_string,
|
||||
initialize_modules,
|
||||
check_vad_update,
|
||||
check_asr_update,
|
||||
filter_sensitive_info,
|
||||
initialize_tts,
|
||||
)
|
||||
from typing import Dict, Any
|
||||
from core.mcp.manager import MCPManager
|
||||
from core.utils.modules_initialize import (
|
||||
initialize_modules,
|
||||
initialize_tts,
|
||||
initialize_asr,
|
||||
)
|
||||
from core.handle.reportHandle import report
|
||||
from core.providers.tts.default import DefaultTTS
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from core.utils.dialogue import Message, Dialogue
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
from core.handle.textHandle import handleTextMessage
|
||||
from core.handle.functionHandler import FunctionHandler
|
||||
from plugins_func.loadplugins import auto_import_modules
|
||||
from plugins_func.register import Action, ActionResponse
|
||||
from core.auth import AuthMiddleware, AuthenticationError
|
||||
from config.config_loader import get_private_config_from_api
|
||||
from core.handle.receiveAudioHandle import handleAudioMessage
|
||||
from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType
|
||||
from config.logger import setup_logging, build_module_string, update_module_string
|
||||
from config.manage_api_client import DeviceNotFoundException, DeviceBindException
|
||||
@@ -76,9 +80,11 @@ class ConnectionHandler:
|
||||
self.welcome_msg = None
|
||||
self.max_output_size = 0
|
||||
self.chat_history_conf = 0
|
||||
self.audio_format = "opus"
|
||||
|
||||
# 客户端状态相关
|
||||
self.client_abort = False
|
||||
self.client_is_speaking = False
|
||||
self.client_listen_mode = "auto"
|
||||
|
||||
# 线程任务相关
|
||||
@@ -111,8 +117,10 @@ class ConnectionHandler:
|
||||
self.client_voice_stop = False
|
||||
|
||||
# asr相关变量
|
||||
# 因为实际部署时可能会用到公共的本地ASR，不能把变量暴露给公共ASR
|
||||
# 所以涉及到ASR的变量，需要在这里定义，属于connection的私有变量
|
||||
self.asr_audio = []
|
||||
self.asr_server_receive = True
|
||||
self.asr_audio_queue = queue.Queue()
|
||||
|
||||
# llm相关变量
|
||||
self.llm_finish_task = True
|
||||
@@ -141,7 +149,8 @@ class ConnectionHandler:
|
||||
int(self.config.get("close_connection_no_voice_time", 120)) + 60
|
||||
) # 在原来第一道关闭的基础上加60秒，进行二道关闭
|
||||
|
||||
self.audio_format = "opus"
|
||||
# {"mcp":true} 表示启用MCP功能
|
||||
self.features = None
|
||||
|
||||
async def handle_connection(self, ws):
|
||||
try:
|
||||
@@ -247,7 +256,11 @@ class ConnectionHandler:
|
||||
if isinstance(message, str):
|
||||
await handleTextMessage(self, message)
|
||||
elif isinstance(message, bytes):
|
||||
await handleAudioMessage(self, message)
|
||||
if self.vad is None:
|
||||
return
|
||||
if self.asr is None:
|
||||
return
|
||||
self.asr_audio_queue.put(message)
|
||||
|
||||
async def handle_restart(self, message):
|
||||
"""处理服务器重启请求"""
|
||||
@@ -315,10 +328,14 @@ class ConnectionHandler:
|
||||
if self.vad is None:
|
||||
self.vad = self._vad
|
||||
if self.asr is None:
|
||||
self.asr = self._asr
|
||||
self.asr = self._initialize_asr()
|
||||
# 打开语音识别通道
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self.asr.open_audio_channels(self), self.loop
|
||||
)
|
||||
if self.tts is None:
|
||||
self.tts = self._initialize_tts()
|
||||
# 使用事件循环运行异步方法
|
||||
# 打开语音合成通道
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self.tts.open_audio_channels(self), self.loop
|
||||
)
|
||||
@@ -356,6 +373,19 @@ class ConnectionHandler:
|
||||
|
||||
return tts
|
||||
|
||||
def _initialize_asr(self):
|
||||
"""初始化ASR"""
|
||||
if self._asr.interface_type == InterfaceType.LOCAL:
|
||||
# 如果公共ASR是本地服务，则直接返回
|
||||
# 因为本地一个实例ASR，可以被多个连接共享
|
||||
asr = self._asr
|
||||
else:
|
||||
# 如果公共ASR是远程服务，则初始化一个新实例
|
||||
# 因为远程ASR，涉及到websocket连接和接收线程，需要每个连接一个实例
|
||||
asr = initialize_asr(self.config)
|
||||
|
||||
return asr
|
||||
|
||||
def _initialize_private_config(self):
|
||||
"""如果是从配置文件获取，则进行二次实例化"""
|
||||
if not self.read_config_from_api:
|
||||
@@ -454,6 +484,8 @@ class ConnectionHandler:
|
||||
self.memory = modules["memory"]
|
||||
|
||||
def _initialize_memory(self):
|
||||
if self.memory is None:
|
||||
return
|
||||
"""初始化记忆模块"""
|
||||
self.memory.init_memory(
|
||||
role_id=self.device_id,
|
||||
@@ -494,6 +526,8 @@ class ConnectionHandler:
|
||||
self.logger.bind(tag=TAG).info("使用主LLM作为意图识别模型")
|
||||
|
||||
def _initialize_intent(self):
|
||||
if self.intent is None:
|
||||
return
|
||||
self.intent_type = self.config["Intent"][
|
||||
self.config["selected_module"]["Intent"]
|
||||
]["type"]
|
||||
@@ -558,6 +592,12 @@ class ConnectionHandler:
|
||||
functions = None
|
||||
if self.intent_type == "function_call" and hasattr(self, "func_handler"):
|
||||
functions = self.func_handler.get_functions()
|
||||
if hasattr(self, "mcp_client"):
|
||||
mcp_tools = self.mcp_client.get_available_tools()
|
||||
if mcp_tools is not None and len(mcp_tools) > 0:
|
||||
if functions is None:
|
||||
functions = []
|
||||
functions.extend(mcp_tools)
|
||||
response_message = []
|
||||
|
||||
try:
|
||||
@@ -595,8 +635,10 @@ class ConnectionHandler:
|
||||
function_arguments = ""
|
||||
content_arguments = ""
|
||||
text_index = 0
|
||||
|
||||
self.client_abort = False
|
||||
for response in llm_responses:
|
||||
if self.client_abort:
|
||||
break
|
||||
if functions is not None:
|
||||
content, tools_call = response
|
||||
if "content" in response:
|
||||
@@ -622,9 +664,6 @@ class ConnectionHandler:
|
||||
if content is not None and len(content) > 0:
|
||||
if not tool_call_flag:
|
||||
response_message.append(content)
|
||||
if self.client_abort:
|
||||
break
|
||||
|
||||
if text_index == 0:
|
||||
self.tts.tts_text_queue.put(
|
||||
TTSMessageDTO(
|
||||
@@ -676,9 +715,32 @@ class ConnectionHandler:
|
||||
"arguments": function_arguments,
|
||||
}
|
||||
|
||||
# 处理MCP工具调用
|
||||
# 处理Server端MCP工具调用
|
||||
if self.mcp_manager.is_mcp_tool(function_name):
|
||||
result = self._handle_mcp_tool_call(function_call_data)
|
||||
elif hasattr(self, "mcp_client") and self.mcp_client.has_tool(
|
||||
function_name
|
||||
):
|
||||
# 如果是小智端MCP工具调用
|
||||
self.logger.bind(tag=TAG).debug(
|
||||
f"调用小智端MCP工具: {function_name}, 参数: {function_arguments}"
|
||||
)
|
||||
try:
|
||||
result = asyncio.run_coroutine_threadsafe(
|
||||
call_mcp_tool(
|
||||
self, self.mcp_client, function_name, function_arguments
|
||||
),
|
||||
self.loop,
|
||||
).result()
|
||||
self.logger.bind(tag=TAG).debug(f"MCP工具调用结果: {result}")
|
||||
result = ActionResponse(
|
||||
action=Action.REQLLM, result=result, response=""
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.bind(tag=TAG).error(f"MCP工具调用失败: {e}")
|
||||
result = ActionResponse(
|
||||
action=Action.REQLLM, result="MCP工具调用失败", response=""
|
||||
)
|
||||
else:
|
||||
# 处理系统函数
|
||||
result = self.func_handler.handle_llm_function_call(
|
||||
@@ -801,9 +863,7 @@ class ConnectionHandler:
|
||||
item = self.report_queue.get(timeout=1)
|
||||
if item is None: # 检测毒丸对象
|
||||
break
|
||||
|
||||
type, text, audio_data, report_time = item
|
||||
|
||||
try:
|
||||
# 检查线程池状态
|
||||
if self.executor is None:
|
||||
@@ -833,8 +893,8 @@ class ConnectionHandler:
|
||||
self.report_queue.task_done()
|
||||
|
||||
def clearSpeakStatus(self):
|
||||
self.client_is_speaking = False
|
||||
self.logger.bind(tag=TAG).debug(f"清除服务端讲话状态")
|
||||
self.asr_server_receive = True
|
||||
|
||||
async def close(self, ws=None):
|
||||
"""资源清理方法"""
|
||||
|
||||
@@ -1,6 +1,4 @@
|
||||
import json
|
||||
import queue
|
||||
from config.logger import setup_logging
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ class FunctionHandler:
|
||||
self.function_registry.register_function("plugin_loader")
|
||||
self.function_registry.register_function("get_time")
|
||||
self.function_registry.register_function("get_lunar")
|
||||
self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
|
||||
# self.function_registry.register_function("handle_speaker_volume_or_screen_brightness")
|
||||
|
||||
def register_config_functions(self):
|
||||
"""注册配置中的函数,可以不同客户端使用不同的配置"""
|
||||
|
||||
@@ -7,6 +7,11 @@ import asyncio
|
||||
from core.handle.sendAudioHandle import send_stt_message
|
||||
from core.utils.util import remove_punctuation_and_length
|
||||
from core.providers.tts.dto.dto import ContentType, InterfaceType
|
||||
from core.handle.mcpHandle import (
|
||||
MCPClient,
|
||||
send_mcp_initialize_message,
|
||||
send_mcp_tools_list_request,
|
||||
)
|
||||
|
||||
|
||||
TAG = __name__
|
||||
@@ -28,9 +33,18 @@ async def handleHelloMessage(conn, msg_json):
|
||||
format = audio_params.get("format")
|
||||
conn.logger.bind(tag=TAG).info(f"客户端音频格式: {format}")
|
||||
conn.audio_format = format
|
||||
if conn.asr is not None:
|
||||
conn.asr.set_audio_format(format)
|
||||
conn.welcome_msg["audio_params"] = audio_params
|
||||
features = msg_json.get("features")
|
||||
if features:
|
||||
conn.logger.bind(tag=TAG).info(f"客户端特性: {features}")
|
||||
conn.features = features
|
||||
if features.get("mcp"):
|
||||
conn.logger.bind(tag=TAG).info("客户端支持MCP")
|
||||
conn.mcp_client = MCPClient()
|
||||
# 发送初始化
|
||||
asyncio.create_task(send_mcp_initialize_message(conn))
|
||||
# 发送mcp消息，获取tools列表
|
||||
asyncio.create_task(send_mcp_tools_list_request(conn))
|
||||
|
||||
await conn.websocket.send(json.dumps(conn.welcome_msg))
|
||||
|
||||
@@ -49,6 +63,8 @@ async def checkWakeupWords(conn, text):
|
||||
"""检查是否是唤醒词"""
|
||||
_, filtered_text = remove_punctuation_and_length(text)
|
||||
if filtered_text in conn.config.get("wakeup_words"):
|
||||
# 设置刚刚被唤醒的标志
|
||||
conn.just_woken_up = True
|
||||
await send_stt_message(conn, text)
|
||||
|
||||
file = getWakeupWordFile(WAKEUP_CONFIG["file_name"])
|
||||
@@ -58,6 +74,8 @@ async def checkWakeupWords(conn, text):
|
||||
text_hello = WAKEUP_CONFIG["text"]
|
||||
if not text_hello:
|
||||
text_hello = text
|
||||
if conn.tts is None:
|
||||
return False
|
||||
conn.tts.tts_one_sentence(
|
||||
conn, ContentType.FILE, content_file=file, content_detail=text_hello
|
||||
)
|
||||
@@ -92,7 +110,13 @@ async def wakeupWordsResponse(conn):
|
||||
|
||||
"""唤醒词响应"""
|
||||
wakeup_word = random.choice(WAKEUP_CONFIG["words"])
|
||||
result = conn.llm.response_no_stream(conn.config["prompt"], wakeup_word)
|
||||
question = (
|
||||
"此刻用户正在和你说```"
|
||||
+ wakeup_word
|
||||
+ "```。\n请你根据以上用户的内容，进行简短回复，文字内容控制在15个字以内。\n"
|
||||
+ "请勿对这条内容本身进行任何解释和回应，仅返回对用户的内容的回复。"
|
||||
)
|
||||
result = conn.llm.response_no_stream(conn.config["prompt"], question)
|
||||
if result is None or result == "":
|
||||
return
|
||||
tts_file = await asyncio.to_thread(conn.tts.to_tts, result)
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
from config.logger import setup_logging
|
||||
import json
|
||||
import uuid
|
||||
from core.handle.sendAudioHandle import send_stt_message
|
||||
@@ -97,6 +96,7 @@ async def process_intent_result(conn, intent_result, original_text):
|
||||
}
|
||||
|
||||
await send_stt_message(conn, original_text)
|
||||
conn.client_abort = False
|
||||
|
||||
# 使用executor执行函数调用和结果处理
|
||||
def process_function_call():
|
||||
@@ -146,4 +146,3 @@ async def process_intent_result(conn, intent_result, original_text):
|
||||
def speak_txt(conn, text):
|
||||
conn.tts.tts_one_sentence(conn, ContentType.TEXT, content_detail=text)
|
||||
conn.dialogue.put(Message(role="assistant", content=text))
|
||||
|
||||
|
||||
@@ -0,0 +1,374 @@
|
||||
import json
|
||||
import asyncio
|
||||
from concurrent.futures import Future
|
||||
from core.utils.util import get_vision_url
|
||||
from core.utils.auth import AuthToken
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
class MCPClient:
|
||||
"""MCPClient，用于管理MCP状态和工具"""
|
||||
|
||||
def __init__(self):
|
||||
self.tools = {} # Dictionary for O(1) lookup
|
||||
self.ready = False
|
||||
self.call_results = {} # To store Futures for tool call responses
|
||||
self.next_id = 1
|
||||
self.lock = asyncio.Lock()
|
||||
self._cached_available_tools = None # Cache for get_available_tools
|
||||
|
||||
def has_tool(self, name: str) -> bool:
|
||||
return name in self.tools
|
||||
|
||||
def get_available_tools(self) -> list:
|
||||
# Check if the cache is valid
|
||||
if self._cached_available_tools is not None:
|
||||
return self._cached_available_tools
|
||||
|
||||
# If cache is not valid, regenerate the list
|
||||
result = []
|
||||
for tool_name, tool_data in self.tools.items():
|
||||
function_def = {
|
||||
"name": tool_data["name"],
|
||||
"description": tool_data["description"],
|
||||
"parameters": {
|
||||
"type": tool_data["inputSchema"].get("type", "object"),
|
||||
"properties": tool_data["inputSchema"].get("properties", {}),
|
||||
"required": tool_data["inputSchema"].get("required", []),
|
||||
},
|
||||
}
|
||||
result.append({"type": "function", "function": function_def})
|
||||
|
||||
self._cached_available_tools = result # Store the generated list in cache
|
||||
return result
|
||||
|
||||
async def is_ready(self) -> bool:
|
||||
async with self.lock:
|
||||
return self.ready
|
||||
|
||||
async def set_ready(self, status: bool):
|
||||
async with self.lock:
|
||||
self.ready = status
|
||||
|
||||
async def add_tool(self, tool_data: dict):
|
||||
async with self.lock:
|
||||
self.tools[tool_data["name"]] = tool_data
|
||||
self._cached_available_tools = (
|
||||
None # Invalidate the cache when a tool is added
|
||||
)
|
||||
|
||||
async def get_next_id(self) -> int:
|
||||
async with self.lock:
|
||||
current_id = self.next_id
|
||||
self.next_id += 1
|
||||
return current_id
|
||||
|
||||
async def register_call_result_future(self, id: int, future: Future):
|
||||
async with self.lock:
|
||||
self.call_results[id] = future
|
||||
|
||||
async def resolve_call_result(self, id: int, result: any):
|
||||
async with self.lock:
|
||||
if id in self.call_results:
|
||||
future = self.call_results.pop(id)
|
||||
if not future.done():
|
||||
future.set_result(result)
|
||||
|
||||
async def reject_call_result(self, id: int, exception: Exception):
|
||||
async with self.lock:
|
||||
if id in self.call_results:
|
||||
future = self.call_results.pop(id)
|
||||
if not future.done():
|
||||
future.set_exception(exception)
|
||||
|
||||
async def cleanup_call_result(self, id: int):
|
||||
async with self.lock:
|
||||
if id in self.call_results:
|
||||
self.call_results.pop(id)
|
||||
|
||||
|
||||
async def send_mcp_message(conn, payload: dict):
|
||||
"""Helper to send MCP messages, encapsulating common logic."""
|
||||
if not conn.features.get("mcp"):
|
||||
conn.logger.bind(tag=TAG).warning("客户端不支持MCP，无法发送MCP消息")
|
||||
return
|
||||
|
||||
message = json.dumps({"type": "mcp", "payload": payload})
|
||||
|
||||
try:
|
||||
await conn.websocket.send(message)
|
||||
conn.logger.bind(tag=TAG).info(f"成功发送MCP消息: {message}")
|
||||
except Exception as e:
|
||||
conn.logger.bind(tag=TAG).error(f"发送MCP消息失败: {e}")
|
||||
|
||||
|
||||
async def handle_mcp_message(conn, mcp_client: MCPClient, payload: dict):
|
||||
"""处理MCP消息,包括初始化、工具列表和工具调用响应等"""
|
||||
conn.logger.bind(tag=TAG).info(f"处理MCP消息: {payload}")
|
||||
|
||||
if not isinstance(payload, dict):
|
||||
conn.logger.bind(tag=TAG).error("MCP消息缺少payload字段或格式错误")
|
||||
return
|
||||
|
||||
# Handle result
|
||||
if "result" in payload:
|
||||
result = payload["result"]
|
||||
msg_id = int(payload.get("id", 0))
|
||||
|
||||
# Check for tool call response first
|
||||
if msg_id in mcp_client.call_results:
|
||||
conn.logger.bind(tag=TAG).debug(
|
||||
f"收到工具调用响应，ID: {msg_id}, 结果: {result}"
|
||||
)
|
||||
await mcp_client.resolve_call_result(msg_id, result)
|
||||
return
|
||||
|
||||
if msg_id == 1: # mcpInitializeID
|
||||
conn.logger.bind(tag=TAG).debug("收到MCP初始化响应")
|
||||
server_info = result.get("serverInfo")
|
||||
if isinstance(server_info, dict):
|
||||
name = server_info.get("name")
|
||||
version = server_info.get("version")
|
||||
conn.logger.bind(tag=TAG).info(
|
||||
f"客户端MCP服务器信息: name={name}, version={version}"
|
||||
)
|
||||
await send_mcp_tools_list_request(
|
||||
conn
|
||||
) # After initialization, request tool list
|
||||
return
|
||||
|
||||
elif msg_id == 2: # mcpToolsListID
|
||||
conn.logger.bind(tag=TAG).debug("收到MCP工具列表响应")
|
||||
if isinstance(result, dict) and "tools" in result:
|
||||
tools_data = result["tools"]
|
||||
if not isinstance(tools_data, list):
|
||||
conn.logger.bind(tag=TAG).error("工具列表格式错误")
|
||||
return
|
||||
|
||||
conn.logger.bind(tag=TAG).info(
|
||||
f"客户端设备支持的工具数量: {len(tools_data)}"
|
||||
)
|
||||
|
||||
for i, tool in enumerate(tools_data):
|
||||
if not isinstance(tool, dict):
|
||||
continue
|
||||
|
||||
name = tool.get("name", "")
|
||||
description = tool.get("description", "")
|
||||
input_schema = {"type": "object", "properties": {}, "required": []}
|
||||
|
||||
if "inputSchema" in tool and isinstance(tool["inputSchema"], dict):
|
||||
schema = tool["inputSchema"]
|
||||
input_schema["type"] = schema.get("type", "object")
|
||||
input_schema["properties"] = schema.get("properties", {})
|
||||
input_schema["required"] = [
|
||||
s for s in schema.get("required", []) if isinstance(s, str)
|
||||
]
|
||||
|
||||
new_tool = {
|
||||
"name": name,
|
||||
"description": description,
|
||||
"inputSchema": input_schema,
|
||||
}
|
||||
await mcp_client.add_tool(new_tool)
|
||||
conn.logger.bind(tag=TAG).debug(f"客户端工具 #{i+1}: {name}")
|
||||
|
||||
next_cursor = result.get("nextCursor", "")
|
||||
if next_cursor:
|
||||
conn.logger.bind(tag=TAG).info(
|
||||
f"有更多工具，nextCursor: {next_cursor}"
|
||||
)
|
||||
await send_mcp_tools_list_continue_request(conn, next_cursor)
|
||||
else:
|
||||
await mcp_client.set_ready(True)
|
||||
conn.logger.bind(tag=TAG).info("所有工具已获取，MCP客户端准备就绪")
|
||||
return
|
||||
|
||||
# Handle method calls (requests from the client)
|
||||
elif "method" in payload:
|
||||
method = payload["method"]
|
||||
conn.logger.bind(tag=TAG).info(f"收到MCP客户端请求: {method}")
|
||||
|
||||
elif "error" in payload:
|
||||
error_data = payload["error"]
|
||||
error_msg = error_data.get("message", "未知错误")
|
||||
conn.logger.bind(tag=TAG).error(f"收到MCP错误响应: {error_msg}")
|
||||
|
||||
msg_id = int(payload.get("id", 0))
|
||||
if msg_id in mcp_client.call_results:
|
||||
await mcp_client.reject_call_result(
|
||||
msg_id, Exception(f"MCP错误: {error_msg}")
|
||||
)
|
||||
|
||||
|
||||
# --- Outgoing MCP Messages ---
|
||||
|
||||
|
||||
async def send_mcp_initialize_message(conn):
|
||||
"""发送MCP初始化消息"""
|
||||
|
||||
vision_url = get_vision_url(conn.config)
|
||||
|
||||
# 密钥生成token
|
||||
auth = AuthToken(conn.config["server"]["auth_key"])
|
||||
token = auth.generate_token(conn.headers.get("device-id"))
|
||||
|
||||
vision = {
|
||||
"url": vision_url,
|
||||
"token": token,
|
||||
}
|
||||
|
||||
conn.logger.bind(tag=TAG).info(f"视觉服务信息: {vision}")
|
||||
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1, # mcpInitializeID
|
||||
"method": "initialize",
|
||||
"params": {
|
||||
"protocolVersion": "2024-11-05",
|
||||
"capabilities": {
|
||||
"roots": {"listChanged": True},
|
||||
"sampling": {},
|
||||
"vision": vision,
|
||||
},
|
||||
"clientInfo": {
|
||||
"name": "XiaozhiClient",
|
||||
"version": "1.0.0",
|
||||
},
|
||||
},
|
||||
}
|
||||
conn.logger.bind(tag=TAG).info("发送MCP初始化消息")
|
||||
await send_mcp_message(conn, payload)
|
||||
|
||||
|
||||
async def send_mcp_tools_list_request(conn):
|
||||
"""发送MCP工具列表请求"""
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 2, # mcpToolsListID
|
||||
"method": "tools/list",
|
||||
}
|
||||
conn.logger.bind(tag=TAG).debug("发送MCP工具列表请求")
|
||||
await send_mcp_message(conn, payload)
|
||||
|
||||
|
||||
async def send_mcp_tools_list_continue_request(conn, cursor: str):
|
||||
"""发送带有cursor的MCP工具列表请求"""
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 2, # mcpToolsListID (same ID for continuation)
|
||||
"method": "tools/list",
|
||||
"params": {"cursor": cursor},
|
||||
}
|
||||
conn.logger.bind(tag=TAG).info(f"发送带cursor的MCP工具列表请求: {cursor}")
|
||||
await send_mcp_message(conn, payload)
|
||||
|
||||
|
||||
async def call_mcp_tool(
|
||||
conn, mcp_client: MCPClient, tool_name: str, args: str = "{}", timeout: int = 30
|
||||
):
|
||||
"""
|
||||
调用指定的工具，并等待响应
|
||||
"""
|
||||
if not await mcp_client.is_ready():
|
||||
raise RuntimeError("MCP客户端尚未准备就绪")
|
||||
|
||||
if not mcp_client.has_tool(tool_name):
|
||||
raise ValueError(f"工具 {tool_name} 不存在")
|
||||
|
||||
tool_call_id = await mcp_client.get_next_id()
|
||||
result_future = asyncio.Future()
|
||||
await mcp_client.register_call_result_future(tool_call_id, result_future)
|
||||
|
||||
# 处理参数
|
||||
try:
|
||||
if isinstance(args, str):
|
||||
# 确保字符串是有效的JSON
|
||||
if not args.strip():
|
||||
arguments = {}
|
||||
else:
|
||||
try:
|
||||
# 尝试直接解析
|
||||
arguments = json.loads(args)
|
||||
except json.JSONDecodeError:
|
||||
# 如果解析失败，尝试合并多个JSON对象
|
||||
try:
|
||||
# 使用正则表达式匹配所有JSON对象
|
||||
import re
|
||||
|
||||
json_objects = re.findall(r"\{[^{}]*\}", args)
|
||||
if len(json_objects) > 1:
|
||||
# 合并所有JSON对象
|
||||
merged_dict = {}
|
||||
for json_str in json_objects:
|
||||
try:
|
||||
obj = json.loads(json_str)
|
||||
if isinstance(obj, dict):
|
||||
merged_dict.update(obj)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if merged_dict:
|
||||
arguments = merged_dict
|
||||
else:
|
||||
raise ValueError(f"无法解析任何有效的JSON对象: {args}")
|
||||
else:
|
||||
raise ValueError(f"参数JSON解析失败: {args}")
|
||||
except Exception as e:
|
||||
conn.logger.bind(tag=TAG).error(
|
||||
f"参数JSON解析失败: {str(e)}, 原始参数: {args}"
|
||||
)
|
||||
raise ValueError(f"参数JSON解析失败: {str(e)}")
|
||||
elif isinstance(args, dict):
|
||||
arguments = args
|
||||
else:
|
||||
raise ValueError(f"参数类型错误，期望字符串或字典，实际类型: {type(args)}")
|
||||
|
||||
# 确保参数是字典类型
|
||||
if not isinstance(arguments, dict):
|
||||
raise ValueError(f"参数必须是字典类型，实际类型: {type(arguments)}")
|
||||
|
||||
except Exception as e:
|
||||
if not isinstance(e, ValueError):
|
||||
raise ValueError(f"参数处理失败: {str(e)}")
|
||||
raise e
|
||||
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": tool_call_id,
|
||||
"method": "tools/call",
|
||||
"params": {"name": tool_name, "arguments": arguments},
|
||||
}
|
||||
|
||||
conn.logger.bind(tag=TAG).info(
|
||||
f"发送客户端mcp工具调用请求: {tool_name}，参数: {args}"
|
||||
)
|
||||
await send_mcp_message(conn, payload)
|
||||
|
||||
try:
|
||||
# Wait for response or timeout
|
||||
raw_result = await asyncio.wait_for(result_future, timeout=timeout)
|
||||
conn.logger.bind(tag=TAG).info(
|
||||
f"客户端mcp工具调用 {tool_name} 成功，原始结果: {raw_result}"
|
||||
)
|
||||
|
||||
if isinstance(raw_result, dict):
|
||||
if raw_result.get("isError") is True:
|
||||
error_msg = raw_result.get(
|
||||
"error", "工具调用返回错误，但未提供具体错误信息"
|
||||
)
|
||||
raise RuntimeError(f"工具调用错误: {error_msg}")
|
||||
|
||||
content = raw_result.get("content")
|
||||
if isinstance(content, list) and len(content) > 0:
|
||||
if isinstance(content[0], dict) and "text" in content[0]:
|
||||
# 直接返回文本内容，不进行JSON解析
|
||||
return content[0]["text"]
|
||||
# 如果结果不是预期的格式，将其转换为字符串
|
||||
return str(raw_result)
|
||||
except asyncio.TimeoutError:
|
||||
await mcp_client.cleanup_call_result(tool_call_id)
|
||||
raise TimeoutError("工具调用请求超时")
|
||||
except Exception as e:
|
||||
await mcp_client.cleanup_call_result(tool_call_id)
|
||||
raise e
|
||||
@@ -1,10 +1,9 @@
|
||||
import time
|
||||
import copy
|
||||
from core.utils.util import remove_punctuation_and_length
|
||||
from core.handle.sendAudioHandle import send_stt_message
|
||||
from core.handle.intentHandler import handle_user_intent
|
||||
from core.utils.output_counter import check_device_output_limit
|
||||
from core.handle.reportHandle import enqueue_asr_report
|
||||
from core.handle.abortHandle import handleAbortMessage
|
||||
import time
|
||||
import asyncio
|
||||
from core.handle.sendAudioHandle import SentenceType
|
||||
from core.utils.util import audio_to_data
|
||||
|
||||
@@ -12,47 +11,28 @@ TAG = __name__
|
||||
|
||||
|
||||
async def handleAudioMessage(conn, audio):
|
||||
if conn.vad is None:
|
||||
return
|
||||
if not conn.asr_server_receive:
|
||||
conn.logger.bind(tag=TAG).debug(f"前期数据处理中，暂停接收")
|
||||
return
|
||||
if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
|
||||
have_voice = conn.vad.is_vad(conn, audio)
|
||||
else:
|
||||
have_voice = conn.client_have_voice
|
||||
# 当前片段是否有人说话
|
||||
have_voice = conn.vad.is_vad(conn, audio)
|
||||
|
||||
# 如果本次没有声音，本段也没声音，就把声音丢弃了
|
||||
if have_voice == False and conn.client_have_voice == False:
|
||||
await no_voice_close_connect(conn)
|
||||
conn.asr_audio.append(audio)
|
||||
conn.asr_audio = conn.asr_audio[
|
||||
-10:
|
||||
] # 保留最新的10帧音频内容，解决ASR句首丢字问题
|
||||
return
|
||||
conn.client_no_voice_last_time = 0.0
|
||||
conn.asr_audio.append(audio)
|
||||
# 如果本段有声音，且已经停止了
|
||||
if conn.client_voice_stop:
|
||||
conn.client_abort = False
|
||||
conn.asr_server_receive = False
|
||||
# 音频太短了，无法识别
|
||||
if len(conn.asr_audio) < 15:
|
||||
conn.asr_server_receive = True
|
||||
else:
|
||||
raw_text, _ = await conn.asr.speech_to_text(
|
||||
conn.asr_audio, conn.session_id
|
||||
) # 确保ASR模块返回原始文本
|
||||
conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
|
||||
text_len, _ = remove_punctuation_and_length(raw_text)
|
||||
if text_len > 0:
|
||||
# 使用自定义模块进行上报
|
||||
await startToChat(conn, raw_text)
|
||||
enqueue_asr_report(conn, raw_text, copy.deepcopy(conn.asr_audio))
|
||||
else:
|
||||
conn.asr_server_receive = True
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
# 如果设备刚刚被唤醒，短暂忽略VAD检测
|
||||
if hasattr(conn, "just_woken_up") and conn.just_woken_up:
|
||||
have_voice = False
|
||||
# 设置一个短暂延迟后恢复VAD检测
|
||||
asyncio.create_task(resume_vad_detection(conn))
|
||||
|
||||
if have_voice:
|
||||
if conn.client_is_speaking:
|
||||
await handleAbortMessage(conn)
|
||||
# 设备长时间空闲检测，用于say goodbye
|
||||
await no_voice_close_connect(conn, have_voice)
|
||||
# 接收音频
|
||||
await conn.asr.receive_audio(conn, audio, have_voice)
|
||||
|
||||
|
||||
async def resume_vad_detection(conn):
|
||||
# 等待2秒后恢复VAD检测
|
||||
await asyncio.sleep(2)
|
||||
conn.just_woken_up = False
|
||||
|
||||
|
||||
async def startToChat(conn, text):
|
||||
@@ -67,13 +47,14 @@ async def startToChat(conn, text):
|
||||
):
|
||||
await max_out_size(conn)
|
||||
return
|
||||
if conn.client_is_speaking:
|
||||
await handleAbortMessage(conn)
|
||||
|
||||
# 首先进行意图分析
|
||||
intent_handled = await handle_user_intent(conn, text)
|
||||
|
||||
if intent_handled:
|
||||
# 如果意图已被处理，不再进行聊天
|
||||
conn.asr_server_receive = True
|
||||
return
|
||||
|
||||
# 意图未被处理，继续常规聊天流程
|
||||
@@ -81,7 +62,10 @@ async def startToChat(conn, text):
|
||||
conn.executor.submit(conn.chat, text)
|
||||
|
||||
|
||||
async def no_voice_close_connect(conn):
|
||||
async def no_voice_close_connect(conn, have_voice):
|
||||
if have_voice:
|
||||
conn.client_no_voice_last_time = 0.0
|
||||
return
|
||||
if conn.client_no_voice_last_time == 0.0:
|
||||
conn.client_no_voice_last_time = time.time() * 1000
|
||||
else:
|
||||
@@ -95,7 +79,6 @@ async def no_voice_close_connect(conn):
|
||||
):
|
||||
conn.close_after_chat = True
|
||||
conn.client_abort = False
|
||||
conn.asr_server_receive = False
|
||||
end_prompt = conn.config.get("end_prompt", {})
|
||||
if end_prompt and end_prompt.get("enable", True) is False:
|
||||
conn.logger.bind(tag=TAG).info("结束对话，无需发送结束提示语")
|
||||
@@ -103,7 +86,7 @@ async def no_voice_close_connect(conn):
|
||||
return
|
||||
prompt = end_prompt.get("prompt")
|
||||
if not prompt:
|
||||
prompt = "请你以“时间过得真快”未来头，用富有感情、依依不舍的话来结束这场对话吧。！"
|
||||
prompt = "请你以```时间过得真快```未来头，用富有感情、依依不舍的话来结束这场对话吧。！"
|
||||
await startToChat(conn, prompt)
|
||||
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ TTS上报功能已集成到ConnectionHandler类中。
|
||||
|
||||
具体实现请参考core/connection.py中的相关代码。
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import opuslib_next
|
||||
@@ -145,4 +146,4 @@ def enqueue_asr_report(conn, text, opus_data):
|
||||
f"ASR数据已加入上报队列: {conn.device_id}, 不上报音频"
|
||||
)
|
||||
except Exception as e:
|
||||
conn.logger.bind(tag=TAG).error(f"加入ASR上报队列失败: {text}, {e}")
|
||||
conn.logger.bind(tag=TAG).debug(f"加入ASR上报队列失败: {text}, {e}")
|
||||
|
||||
@@ -62,6 +62,7 @@ async def sendAudioMessage(conn, sentenceType, audios, text):
|
||||
# 发送结束消息（如果是最后一个文本）
|
||||
if conn.llm_finish_task and sentenceType == SentenceType.LAST:
|
||||
await send_tts_message(conn, "stop", None)
|
||||
conn.client_is_speaking = False
|
||||
if conn.close_after_chat:
|
||||
await conn.close()
|
||||
|
||||
@@ -88,7 +89,7 @@ async def sendAudio(conn, audios, pre_buffer=True):
|
||||
# 播放剩余音频帧
|
||||
for opus_packet in remaining_audios:
|
||||
if conn.client_abort:
|
||||
return
|
||||
break
|
||||
|
||||
# 每分钟重置一次计时器
|
||||
if time.perf_counter() - last_reset_time > 60:
|
||||
@@ -131,9 +132,15 @@ async def send_tts_message(conn, state, text=None):
|
||||
|
||||
|
||||
async def send_stt_message(conn, text):
|
||||
end_prompt_str = conn.config.get("end_prompt", {}).get("prompt")
|
||||
if end_prompt_str and end_prompt_str == text:
|
||||
await send_tts_message(conn, "start")
|
||||
return
|
||||
|
||||
"""发送 STT 状态消息"""
|
||||
stt_text = get_string_no_punctuation_or_emoji(text)
|
||||
await conn.websocket.send(
|
||||
json.dumps({"type": "stt", "text": stt_text, "session_id": conn.session_id})
|
||||
)
|
||||
conn.client_is_speaking = True
|
||||
await send_tts_message(conn, "start")
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import json
|
||||
from core.handle.abortHandle import handleAbortMessage
|
||||
from core.handle.helloHandle import handleHelloMessage
|
||||
from core.handle.mcpHandle import handle_mcp_message
|
||||
from core.utils.util import remove_punctuation_and_length, filter_sensitive_info
|
||||
from core.handle.receiveAudioHandle import startToChat, handleAudioMessage
|
||||
from core.handle.sendAudioHandle import send_stt_message, send_tts_message
|
||||
@@ -41,12 +42,13 @@ async def handleTextMessage(conn, message):
|
||||
if len(conn.asr_audio) > 0:
|
||||
await handleAudioMessage(conn, b"")
|
||||
elif msg_json["state"] == "detect":
|
||||
conn.asr_server_receive = False
|
||||
conn.client_have_voice = False
|
||||
conn.asr_audio.clear()
|
||||
if "text" in msg_json:
|
||||
original_text = msg_json["text"] # 保留原始文本
|
||||
filtered_len, filtered_text = remove_punctuation_and_length(original_text)
|
||||
filtered_len, filtered_text = remove_punctuation_and_length(
|
||||
original_text
|
||||
)
|
||||
|
||||
# 识别是否是唤醒词
|
||||
is_wakeup_words = filtered_text in conn.config.get("wakeup_words")
|
||||
@@ -57,7 +59,9 @@ async def handleTextMessage(conn, message):
|
||||
# 如果是唤醒词，且关闭了唤醒词回复，就不用回答
|
||||
await send_stt_message(conn, original_text)
|
||||
await send_tts_message(conn, "stop", None)
|
||||
conn.client_is_speaking = False
|
||||
elif is_wakeup_words:
|
||||
conn.just_woken_up = True
|
||||
# 上报纯文字数据（复用ASR上报功能，但不提供音频数据）
|
||||
enqueue_asr_report(conn, "嘿，你好呀", [])
|
||||
await startToChat(conn, "嘿，你好呀")
|
||||
@@ -72,6 +76,12 @@ async def handleTextMessage(conn, message):
|
||||
asyncio.create_task(handleIotDescriptors(conn, msg_json["descriptors"]))
|
||||
if "states" in msg_json:
|
||||
asyncio.create_task(handleIotStatus(conn, msg_json["states"]))
|
||||
elif msg_json["type"] == "mcp":
|
||||
conn.logger.bind(tag=TAG).info(f"收到mcp消息：{message}")
|
||||
if "payload" in msg_json:
|
||||
asyncio.create_task(
|
||||
handle_mcp_message(conn, conn.mcp_client, msg_json["payload"])
|
||||
)
|
||||
elif msg_json["type"] == "server":
|
||||
# 记录日志时过滤敏感信息
|
||||
conn.logger.bind(tag=TAG).info(
|
||||
@@ -151,5 +161,7 @@ async def handleTextMessage(conn, message):
|
||||
# 重启服务器
|
||||
elif msg_json["action"] == "restart":
|
||||
await conn.handle_restart(msg_json)
|
||||
else:
|
||||
conn.logger.bind(tag=TAG).error(f"收到未知类型消息：{message}")
|
||||
except json.JSONDecodeError:
|
||||
await conn.websocket.send(message)
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
import asyncio
|
||||
from aiohttp import web
|
||||
from config.logger import setup_logging
|
||||
from core.api.ota_handler import OTAHandler
|
||||
from core.api.vision_handler import VisionHandler
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
class SimpleHttpServer:
|
||||
def __init__(self, config: dict):
|
||||
self.config = config
|
||||
self.logger = setup_logging()
|
||||
self.ota_handler = OTAHandler(config)
|
||||
self.vision_handler = VisionHandler(config)
|
||||
|
||||
def _get_websocket_url(self, local_ip: str, port: int) -> str:
|
||||
"""获取websocket地址
|
||||
|
||||
Args:
|
||||
local_ip: 本地IP地址
|
||||
port: 端口号
|
||||
|
||||
Returns:
|
||||
str: websocket地址
|
||||
"""
|
||||
server_config = self.config["server"]
|
||||
websocket_config = server_config.get("websocket")
|
||||
|
||||
if websocket_config and "你" not in websocket_config:
|
||||
return websocket_config
|
||||
else:
|
||||
return f"ws://{local_ip}:{port}/xiaozhi/v1/"
|
||||
|
||||
async def start(self):
|
||||
server_config = self.config["server"]
|
||||
host = server_config.get("ip", "0.0.0.0")
|
||||
port = int(server_config.get("http_port", 8003))
|
||||
|
||||
if port:
|
||||
app = web.Application()
|
||||
|
||||
read_config_from_api = server_config.get("read_config_from_api", False)
|
||||
|
||||
if not read_config_from_api:
|
||||
# 如果没有开启智控台，只是单模块运行，就需要再添加简单OTA接口，用于下发websocket接口
|
||||
app.add_routes(
|
||||
[
|
||||
web.get("/xiaozhi/ota/", self.ota_handler.handle_get),
|
||||
web.post("/xiaozhi/ota/", self.ota_handler.handle_post),
|
||||
web.options("/xiaozhi/ota/", self.ota_handler.handle_post),
|
||||
]
|
||||
)
|
||||
# 添加路由
|
||||
app.add_routes(
|
||||
[
|
||||
web.get("/mcp/vision/explain", self.vision_handler.handle_get),
|
||||
web.post("/mcp/vision/explain", self.vision_handler.handle_post),
|
||||
web.options("/mcp/vision/explain", self.vision_handler.handle_post),
|
||||
]
|
||||
)
|
||||
|
||||
# 运行服务
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, host, port)
|
||||
await site.start()
|
||||
|
||||
# 保持服务运行
|
||||
while True:
|
||||
await asyncio.sleep(3600) # 每隔 1 小时检查一次
|
||||
@@ -92,11 +92,21 @@ class MCPClient:
|
||||
args=self.config.get("args", []),
|
||||
env=env,
|
||||
)
|
||||
stdio_r, stdio_w = await stack.enter_async_context(stdio_client(params))
|
||||
stdio_r, stdio_w = await stack.enter_async_context(
|
||||
stdio_client(params)
|
||||
)
|
||||
read_stream, write_stream = stdio_r, stdio_w
|
||||
# 建立SSEClient
|
||||
elif "url" in self.config:
|
||||
sse_r, sse_w = await stack.enter_async_context(sse_client(self.config["url"]))
|
||||
if "API_ACCESS_TOKEN" in self.config:
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.config['API_ACCESS_TOKEN']}"
|
||||
}
|
||||
else:
|
||||
headers = {}
|
||||
sse_r, sse_w = await stack.enter_async_context(
|
||||
sse_client(self.config["url"], headers=headers)
|
||||
)
|
||||
read_stream, write_stream = sse_r, sse_w
|
||||
|
||||
else:
|
||||
|
||||
@@ -2,9 +2,6 @@ import http.client
|
||||
import json
|
||||
import asyncio
|
||||
from typing import Optional, Tuple, List
|
||||
import opuslib_next
|
||||
import wave
|
||||
import io
|
||||
import os
|
||||
import uuid
|
||||
import hmac
|
||||
@@ -16,6 +13,7 @@ import time
|
||||
from datetime import datetime
|
||||
from config.logger import setup_logging
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
@@ -92,6 +90,7 @@ class AccessToken:
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config: dict, delete_audio_file: bool):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.NON_STREAM
|
||||
"""阿里云ASR初始化"""
|
||||
# 新增空值判断逻辑
|
||||
self.access_key_id = config.get("access_key_id")
|
||||
@@ -165,21 +164,6 @@ class ASRProvider(ASRProviderBase):
|
||||
request += "&enable_voice_detection=false"
|
||||
return request
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1) # 单声道
|
||||
wf.setsampwidth(2) # 16-bit
|
||||
wf.setframerate(self.sample_rate)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
logger.bind(tag=TAG).debug(f"音频文件已保存至: {file_path}")
|
||||
return file_path
|
||||
|
||||
async def _send_request(self, pcm_data: bytes) -> Optional[str]:
|
||||
"""发送请求到阿里云ASR服务"""
|
||||
try:
|
||||
@@ -229,7 +213,7 @@ class ASRProvider(ASRProviderBase):
|
||||
return None
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""将语音数据转换为文本"""
|
||||
if self._is_token_expired():
|
||||
@@ -239,7 +223,7 @@ class ASRProvider(ASRProviderBase):
|
||||
file_path = None
|
||||
try:
|
||||
# 解码Opus为PCM
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -1,18 +1,10 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
import os
|
||||
import uuid
|
||||
from typing import Optional, Tuple, List
|
||||
import wave
|
||||
import opuslib_next
|
||||
|
||||
from aip import AipSpeech
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from config.logger import setup_logging
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
@@ -21,6 +13,7 @@ logger = setup_logging()
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config: dict, delete_audio_file: bool = True):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.NON_STREAM
|
||||
self.app_id = config.get("app_id")
|
||||
self.api_key = config.get("api_key")
|
||||
self.secret_key = config.get("secret_key")
|
||||
@@ -36,22 +29,8 @@ class ASRProvider(ASRProviderBase):
|
||||
# 确保输出目录存在
|
||||
os.makedirs(self.output_dir, exist_ok=True)
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""将语音数据转换为文本"""
|
||||
if not opus_data:
|
||||
@@ -66,7 +45,7 @@ class ASRProvider(ASRProviderBase):
|
||||
return None, file_path
|
||||
|
||||
# 将Opus音频数据解码为PCM
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -1,7 +1,19 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Optional, Tuple, List
|
||||
import os
|
||||
import wave
|
||||
import copy
|
||||
import uuid
|
||||
import queue
|
||||
import asyncio
|
||||
import traceback
|
||||
import threading
|
||||
import opuslib_next
|
||||
from abc import ABC, abstractmethod
|
||||
from config.logger import setup_logging
|
||||
from typing import Optional, Tuple, List
|
||||
from core.handle.receiveAudioHandle import startToChat
|
||||
from core.handle.reportHandle import enqueue_asr_report
|
||||
from core.utils.util import remove_punctuation_and_length
|
||||
from core.handle.receiveAudioHandle import handleAudioMessage
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
@@ -9,24 +21,97 @@ logger = setup_logging()
|
||||
|
||||
class ASRProviderBase(ABC):
|
||||
def __init__(self):
|
||||
self.audio_format = "opus"
|
||||
pass
|
||||
|
||||
# 打开音频通道
|
||||
# 这里默认是非流式的处理方式
|
||||
# 流式处理方式请在子类中重写
|
||||
async def open_audio_channels(self, conn):
|
||||
# tts 消化线程
|
||||
conn.asr_priority_thread = threading.Thread(
|
||||
target=self.asr_text_priority_thread, args=(conn,), daemon=True
|
||||
)
|
||||
conn.asr_priority_thread.start()
|
||||
|
||||
# 有序处理ASR音频
|
||||
def asr_text_priority_thread(self, conn):
|
||||
while not conn.stop_event.is_set():
|
||||
try:
|
||||
message = conn.asr_audio_queue.get(timeout=1)
|
||||
future = asyncio.run_coroutine_threadsafe(
|
||||
handleAudioMessage(conn, message),
|
||||
conn.loop,
|
||||
)
|
||||
future.result()
|
||||
except queue.Empty:
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(
|
||||
f"处理ASR文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
|
||||
)
|
||||
continue
|
||||
|
||||
# 接收音频
|
||||
# 这里默认是非流式的处理方式
|
||||
# 流式处理方式请在子类中重写
|
||||
async def receive_audio(self, conn, audio, audio_have_voice):
|
||||
if conn.client_listen_mode == "auto" or conn.client_listen_mode == "realtime":
|
||||
have_voice = audio_have_voice
|
||||
else:
|
||||
have_voice = conn.client_have_voice
|
||||
# 如果本次没有声音，本段也没声音，就把声音丢弃了
|
||||
conn.asr_audio.append(audio)
|
||||
if have_voice == False and conn.client_have_voice == False:
|
||||
conn.asr_audio = conn.asr_audio[-10:]
|
||||
return
|
||||
|
||||
# 如果本段有声音，且已经停止了
|
||||
if conn.client_voice_stop:
|
||||
asr_audio_task = copy.deepcopy(conn.asr_audio)
|
||||
conn.asr_audio.clear()
|
||||
|
||||
# 音频太短了，无法识别
|
||||
conn.reset_vad_states()
|
||||
if len(asr_audio_task) > 15:
|
||||
await self.handle_voice_stop(conn, asr_audio_task)
|
||||
|
||||
# 处理语音停止
|
||||
async def handle_voice_stop(self, conn, asr_audio_task):
|
||||
raw_text, _ = await self.speech_to_text(
|
||||
asr_audio_task, conn.session_id, conn.audio_format
|
||||
) # 确保ASR模块返回原始文本
|
||||
conn.logger.bind(tag=TAG).info(f"识别文本: {raw_text}")
|
||||
text_len, _ = remove_punctuation_and_length(raw_text)
|
||||
self.stop_ws_connection()
|
||||
if text_len > 0:
|
||||
# 使用自定义模块进行上报
|
||||
await startToChat(conn, raw_text)
|
||||
enqueue_asr_report(conn, raw_text, asr_audio_task)
|
||||
|
||||
def stop_ws_connection(self):
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
pass
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
@abstractmethod
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""将语音数据转换为文本"""
|
||||
pass
|
||||
|
||||
def set_audio_format(self, format: str) -> None:
|
||||
"""设置音频格式"""
|
||||
self.audio_format = format
|
||||
|
||||
@staticmethod
|
||||
def decode_opus(opus_data: List[bytes]) -> bytes:
|
||||
"""将Opus音频数据解码为PCM数据"""
|
||||
|
||||
@@ -1,17 +1,14 @@
|
||||
import time
|
||||
import io
|
||||
import wave
|
||||
import os
|
||||
from typing import Optional, Tuple, List
|
||||
import uuid
|
||||
import websockets
|
||||
import json
|
||||
import gzip
|
||||
|
||||
import opuslib_next
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
|
||||
import websockets
|
||||
from config.logger import setup_logging
|
||||
from typing import Optional, Tuple, List
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
@@ -86,6 +83,7 @@ def parse_response(res):
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config: dict, delete_audio_file: bool):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.NON_STREAM
|
||||
self.appid = config.get("appid")
|
||||
self.cluster = config.get("cluster")
|
||||
self.access_token = config.get("access_token")
|
||||
@@ -102,20 +100,6 @@ class ASRProvider(ASRProviderBase):
|
||||
# 确保输出目录存在
|
||||
os.makedirs(self.output_dir, exist_ok=True)
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
@staticmethod
|
||||
def _generate_header(
|
||||
message_type=CLIENT_FULL_REQUEST, message_type_specific_flags=NO_SEQUENCE
|
||||
@@ -244,14 +228,14 @@ class ASRProvider(ASRProviderBase):
|
||||
yield data[offset:data_len], True
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""将语音数据转换为文本"""
|
||||
|
||||
file_path = None
|
||||
try:
|
||||
# 合并所有opus数据包
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -0,0 +1,344 @@
|
||||
import json
|
||||
import gzip
|
||||
import uuid
|
||||
import asyncio
|
||||
import websockets
|
||||
import opuslib_next
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from config.logger import setup_logging
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
|
||||
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.STREAM
|
||||
self.config = config
|
||||
self.text = ""
|
||||
self.max_retries = 3
|
||||
self.retry_delay = 2
|
||||
self.decoder = opuslib_next.Decoder(16000, 1)
|
||||
self.asr_ws = None
|
||||
self.forward_task = None
|
||||
self.is_processing = False # 添加处理状态标志
|
||||
|
||||
# 配置参数
|
||||
self.appid = str(config.get("appid"))
|
||||
self.cluster = config.get("cluster")
|
||||
self.access_token = config.get("access_token")
|
||||
self.boosting_table_name = config.get("boosting_table_name", "")
|
||||
self.correct_table_name = config.get("correct_table_name", "")
|
||||
self.output_dir = config.get("output_dir", "tmp/")
|
||||
self.delete_audio_file = delete_audio_file
|
||||
|
||||
# 火山引擎ASR配置
|
||||
self.ws_url = "wss://openspeech.bytedance.com/api/v3/sauc/bigmodel"
|
||||
self.uid = config.get("uid", "streaming_asr_service")
|
||||
self.workflow = config.get(
|
||||
"workflow", "audio_in,resample,partition,vad,fe,decode,itn,nlu_punctuate"
|
||||
)
|
||||
self.result_type = config.get("result_type", "single")
|
||||
self.format = config.get("format", "pcm")
|
||||
self.codec = config.get("codec", "pcm")
|
||||
self.rate = config.get("sample_rate", 16000)
|
||||
self.language = config.get("language", "zh-CN")
|
||||
self.bits = config.get("bits", 16)
|
||||
self.channel = config.get("channel", 1)
|
||||
self.auth_method = config.get("auth_method", "token")
|
||||
self.secret = config.get("secret", "access_secret")
|
||||
|
||||
async def open_audio_channels(self, conn):
|
||||
await super().open_audio_channels(conn)
|
||||
|
||||
async def receive_audio(self, conn, audio, audio_have_voice):
|
||||
conn.asr_audio.append(audio)
|
||||
conn.asr_audio = conn.asr_audio[-10:]
|
||||
|
||||
# 如果本次有声音，且之前没有建立连接
|
||||
if audio_have_voice and self.asr_ws is None and not self.is_processing:
|
||||
try:
|
||||
self.is_processing = True
|
||||
# 建立新的WebSocket连接
|
||||
headers = self.token_auth() if self.auth_method == "token" else None
|
||||
logger.bind(tag=TAG).info(f"正在连接ASR服务，headers: {headers}")
|
||||
|
||||
self.asr_ws = await websockets.connect(
|
||||
self.ws_url,
|
||||
additional_headers=headers,
|
||||
max_size=1000000000,
|
||||
ping_interval=None,
|
||||
ping_timeout=None,
|
||||
close_timeout=10,
|
||||
)
|
||||
|
||||
# 发送初始化请求
|
||||
request_params = self.construct_request(str(uuid.uuid4()))
|
||||
try:
|
||||
payload_bytes = str.encode(json.dumps(request_params))
|
||||
payload_bytes = gzip.compress(payload_bytes)
|
||||
full_client_request = self.generate_header()
|
||||
full_client_request.extend((len(payload_bytes)).to_bytes(4, "big"))
|
||||
full_client_request.extend(payload_bytes)
|
||||
|
||||
logger.bind(tag=TAG).info(f"发送初始化请求: {request_params}")
|
||||
await self.asr_ws.send(full_client_request)
|
||||
|
||||
# 等待初始化响应
|
||||
init_res = await self.asr_ws.recv()
|
||||
result = self.parse_response(init_res)
|
||||
logger.bind(tag=TAG).info(f"收到初始化响应: {result}")
|
||||
|
||||
# 检查初始化响应
|
||||
if "code" in result and result["code"] != 1000:
|
||||
error_msg = f"ASR服务初始化失败: {result.get('payload_msg', {}).get('message', '未知错误')}"
|
||||
if "payload_msg" in result:
|
||||
error_msg += f"\n详细错误信息: {json.dumps(result['payload_msg'], ensure_ascii=False)}"
|
||||
logger.bind(tag=TAG).error(error_msg)
|
||||
raise Exception(error_msg)
|
||||
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"发送初始化请求失败: {str(e)}")
|
||||
if hasattr(e, "__cause__") and e.__cause__:
|
||||
logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
|
||||
raise e
|
||||
|
||||
# 启动接收ASR结果的异步任务
|
||||
self.forward_task = asyncio.create_task(self._forward_asr_results(conn))
|
||||
|
||||
# 发送缓存的音频数据
|
||||
if conn.asr_audio and len(conn.asr_audio) > 0:
|
||||
for cached_audio in conn.asr_audio[-10:]:
|
||||
try:
|
||||
pcm_frame = self.decoder.decode(cached_audio, 960)
|
||||
payload = gzip.compress(pcm_frame)
|
||||
audio_request = bytearray(
|
||||
self.generate_audio_default_header()
|
||||
)
|
||||
audio_request.extend(len(payload).to_bytes(4, "big"))
|
||||
audio_request.extend(payload)
|
||||
await self.asr_ws.send(audio_request)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).info(
|
||||
f"发送缓存音频数据时发生错误: {e}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"建立ASR连接失败: {str(e)}")
|
||||
if hasattr(e, "__cause__") and e.__cause__:
|
||||
logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
|
||||
if self.asr_ws:
|
||||
await self.asr_ws.close()
|
||||
self.asr_ws = None
|
||||
self.is_processing = False
|
||||
return
|
||||
|
||||
# 发送当前音频数据
|
||||
if self.asr_ws and self.is_processing:
|
||||
try:
|
||||
pcm_frame = self.decoder.decode(audio, 960)
|
||||
payload = gzip.compress(pcm_frame)
|
||||
audio_request = bytearray(self.generate_audio_default_header())
|
||||
audio_request.extend(len(payload).to_bytes(4, "big"))
|
||||
audio_request.extend(payload)
|
||||
await self.asr_ws.send(audio_request)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).info(f"发送音频数据时发生错误: {e}")
|
||||
|
||||
async def _forward_asr_results(self, conn):
|
||||
try:
|
||||
while self.asr_ws and not conn.stop_event.is_set():
|
||||
try:
|
||||
response = await self.asr_ws.recv()
|
||||
result = self.parse_response(response)
|
||||
logger.bind(tag=TAG).debug(f"收到ASR结果: {result}")
|
||||
|
||||
if "payload_msg" in result:
|
||||
payload = result["payload_msg"]
|
||||
if "result" in payload:
|
||||
utterances = payload["result"].get("utterances", [])
|
||||
# 检查duration和空文本的情况
|
||||
if (
|
||||
payload.get("audio_info", {}).get("duration", 0) > 2000
|
||||
and not utterances
|
||||
and not payload["result"].get("text")
|
||||
):
|
||||
logger.bind(tag=TAG).error(f"识别文本：空")
|
||||
self.text = ""
|
||||
conn.reset_vad_states()
|
||||
await self.handle_voice_stop(conn, None)
|
||||
break
|
||||
|
||||
for utterance in utterances:
|
||||
if utterance.get("definite", False):
|
||||
self.text = utterance["text"]
|
||||
logger.bind(tag=TAG).info(
|
||||
f"识别到文本: {self.text}"
|
||||
)
|
||||
conn.reset_vad_states()
|
||||
await self.handle_voice_stop(conn, None)
|
||||
break
|
||||
elif "error" in payload:
|
||||
error_msg = payload.get("error", "未知错误")
|
||||
logger.bind(tag=TAG).error(f"ASR服务返回错误: {error_msg}")
|
||||
break
|
||||
|
||||
except websockets.ConnectionClosed:
|
||||
logger.bind(tag=TAG).info("ASR服务连接已关闭")
|
||||
self.is_processing = False
|
||||
break
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"处理ASR结果时发生错误: {str(e)}")
|
||||
if hasattr(e, "__cause__") and e.__cause__:
|
||||
logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
|
||||
self.is_processing = False
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"ASR结果转发任务发生错误: {str(e)}")
|
||||
if hasattr(e, "__cause__") and e.__cause__:
|
||||
logger.bind(tag=TAG).error(f"错误原因: {str(e.__cause__)}")
|
||||
finally:
|
||||
if self.asr_ws:
|
||||
await self.asr_ws.close()
|
||||
self.asr_ws = None
|
||||
self.is_processing = False
|
||||
|
||||
def stop_ws_connection(self):
|
||||
if self.asr_ws:
|
||||
asyncio.create_task(self.asr_ws.close())
|
||||
self.asr_ws = None
|
||||
self.is_processing = False
|
||||
|
||||
def construct_request(self, reqid):
|
||||
req = {
|
||||
"app": {
|
||||
"appid": self.appid,
|
||||
"cluster": self.cluster,
|
||||
"token": self.access_token,
|
||||
},
|
||||
"user": {"uid": self.uid},
|
||||
"request": {
|
||||
"reqid": reqid,
|
||||
"workflow": self.workflow,
|
||||
"show_utterances": True,
|
||||
"result_type": self.result_type,
|
||||
"sequence": 1,
|
||||
"boosting_table_name": self.boosting_table_name,
|
||||
"correct_table_name": self.correct_table_name,
|
||||
"end_window_size": 200,
|
||||
},
|
||||
"audio": {
|
||||
"format": self.format,
|
||||
"codec": self.codec,
|
||||
"rate": self.rate,
|
||||
"language": self.language,
|
||||
"bits": self.bits,
|
||||
"channel": self.channel,
|
||||
"sample_rate": self.rate,
|
||||
},
|
||||
}
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"构造请求参数: {json.dumps(req, ensure_ascii=False)}"
|
||||
)
|
||||
return req
|
||||
|
||||
def token_auth(self):
|
||||
return {
|
||||
"X-Api-App-Key": self.appid,
|
||||
"X-Api-Access-Key": self.access_token,
|
||||
"X-Api-Resource-Id": "volc.bigasr.sauc.duration",
|
||||
"X-Api-Connect-Id": str(uuid.uuid4()),
|
||||
"Host": "openspeech.bytedance.com",
|
||||
}
|
||||
|
||||
def generate_header(
|
||||
self,
|
||||
version=0x01,
|
||||
message_type=0x01,
|
||||
message_type_specific_flags=0x00,
|
||||
serial_method=0x01,
|
||||
compression_type=0x01,
|
||||
reserved_data=0x00,
|
||||
extension_header: bytes = b"",
|
||||
):
|
||||
header = bytearray()
|
||||
header_size = int(len(extension_header) / 4) + 1
|
||||
header.append((version << 4) | header_size)
|
||||
header.append((message_type << 4) | message_type_specific_flags)
|
||||
header.append((serial_method << 4) | compression_type)
|
||||
header.append(reserved_data)
|
||||
header.extend(extension_header)
|
||||
return header
|
||||
|
||||
def generate_audio_default_header(self):
|
||||
return self.generate_header(
|
||||
version=0x01,
|
||||
message_type=0x02,
|
||||
message_type_specific_flags=0x00,
|
||||
serial_method=0x01,
|
||||
compression_type=0x01,
|
||||
)
|
||||
|
||||
def generate_last_audio_default_header(self):
|
||||
return self.generate_header(
|
||||
version=0x01,
|
||||
message_type=0x02,
|
||||
message_type_specific_flags=0x02,
|
||||
serial_method=0x01,
|
||||
compression_type=0x01,
|
||||
)
|
||||
|
||||
def parse_response(self, res: bytes) -> dict:
|
||||
try:
|
||||
# 检查响应长度
|
||||
if len(res) < 4:
|
||||
logger.bind(tag=TAG).error(f"响应数据长度不足: {len(res)}")
|
||||
return {"error": "响应数据长度不足"}
|
||||
|
||||
# 获取消息头
|
||||
header = res[:4]
|
||||
message_type = header[1] >> 4
|
||||
|
||||
# 如果是错误响应
|
||||
if message_type == 0x0F: # SERVER_ERROR_RESPONSE
|
||||
code = int.from_bytes(header[4:8], "big", signed=False)
|
||||
error_msg = res[8:].decode("utf-8")
|
||||
return {"code": code, "error": error_msg}
|
||||
|
||||
# 获取JSON数据（跳过12字节头部）
|
||||
try:
|
||||
json_data = res[12:].decode("utf-8")
|
||||
result = json.loads(json_data)
|
||||
logger.bind(tag=TAG).debug(f"成功解析JSON响应: {result}")
|
||||
return {"payload_msg": result}
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as e:
|
||||
logger.bind(tag=TAG).error(f"JSON解析失败: {str(e)}")
|
||||
logger.bind(tag=TAG).error(f"原始数据: {res}")
|
||||
raise
|
||||
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"解析响应失败: {str(e)}")
|
||||
logger.bind(tag=TAG).error(f"原始响应数据: {res.hex()}")
|
||||
raise
|
||||
|
||||
async def speech_to_text(self, opus_data, session_id, audio_format):
|
||||
result = self.text
|
||||
self.text = "" # 清空text
|
||||
return result, None
|
||||
|
||||
async def close(self):
|
||||
"""资源清理方法"""
|
||||
if self.asr_ws:
|
||||
await self.asr_ws.close()
|
||||
self.asr_ws = None
|
||||
if self.forward_task:
|
||||
self.forward_task.cancel()
|
||||
try:
|
||||
await self.forward_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self.forward_task = None
|
||||
self.is_processing = False
|
||||
@@ -0,0 +1,9 @@
|
||||
from enum import Enum
|
||||
from typing import Union, Optional
|
||||
|
||||
|
||||
class InterfaceType(Enum):
|
||||
# 接口类型
|
||||
STREAM = "STREAM" # 流式接口
|
||||
NON_STREAM = "NON_STREAM" # 非流式接口
|
||||
LOCAL = "LOCAL" # 本地服务
|
||||
@@ -1,15 +1,14 @@
|
||||
import time
|
||||
import wave
|
||||
import os
|
||||
import sys
|
||||
import io
|
||||
from config.logger import setup_logging
|
||||
from typing import Optional, Tuple, List
|
||||
import uuid
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from funasr import AutoModel
|
||||
from funasr.utils.postprocess_utils import rich_transcription_postprocess
|
||||
import shutil
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
@@ -38,6 +37,7 @@ class CaptureOutput:
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config: dict, delete_audio_file: bool):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.LOCAL
|
||||
self.model_dir = config.get("model_dir")
|
||||
self.output_dir = config.get("output_dir") # 修正配置键名
|
||||
self.delete_audio_file = delete_audio_file
|
||||
@@ -53,22 +53,8 @@ class ASRProvider(ASRProviderBase):
|
||||
# device="cuda:0", # 启用GPU加速
|
||||
)
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""语音转文本主处理逻辑"""
|
||||
file_path = None
|
||||
@@ -77,7 +63,7 @@ class ASRProvider(ASRProviderBase):
|
||||
while retry_count < MAX_RETRIES:
|
||||
try:
|
||||
# 合并所有opus数据包
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -1,11 +1,8 @@
|
||||
from typing import Optional, Tuple, List
|
||||
import opuslib_next
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
import os
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
import ssl
|
||||
import json
|
||||
import uuid
|
||||
import wave
|
||||
import websockets
|
||||
from config.logger import setup_logging
|
||||
import asyncio
|
||||
@@ -23,6 +20,7 @@ class ASRProvider(ASRProviderBase):
|
||||
:param delete_audio_file: Boolean to indicate whether to delete audio files after processing.
|
||||
"""
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.NON_STREAM
|
||||
self.host = config.get("host", "localhost")
|
||||
self.port = config.get("port", 10095)
|
||||
self.api_key = config.get("api_key", "none")
|
||||
@@ -43,20 +41,6 @@ class ASRProvider(ASRProviderBase):
|
||||
self.ssl_context.check_hostname = False
|
||||
self.ssl_context.verify_mode = ssl.CERT_NONE
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
async def _receive_responses(self, ws) -> None:
|
||||
"""
|
||||
Asynchronous generator to receive messages from the WebSocket.
|
||||
@@ -116,7 +100,7 @@ class ASRProvider(ASRProviderBase):
|
||||
logger.bind(tag=TAG).debug(f"Sent end message: {end_message}")
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""
|
||||
Convert speech data to text using FunASR.
|
||||
@@ -125,7 +109,7 @@ class ASRProvider(ASRProviderBase):
|
||||
:return: Tuple containing recognized text and optional timestamp.
|
||||
"""
|
||||
file_path = None
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -5,8 +5,7 @@ import sys
|
||||
import io
|
||||
from config.logger import setup_logging
|
||||
from typing import Optional, Tuple, List
|
||||
import uuid
|
||||
import opuslib_next
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
|
||||
import numpy as np
|
||||
@@ -38,6 +37,7 @@ class CaptureOutput:
|
||||
class ASRProvider(ASRProviderBase):
|
||||
def __init__(self, config: dict, delete_audio_file: bool):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.LOCAL
|
||||
self.model_dir = config.get("model_dir")
|
||||
self.output_dir = config.get("output_dir")
|
||||
self.delete_audio_file = delete_audio_file
|
||||
@@ -84,20 +84,6 @@ class ASRProvider(ASRProviderBase):
|
||||
use_itn=True,
|
||||
)
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
def read_wave(self, wave_filename: str) -> Tuple[np.ndarray, int]:
|
||||
"""
|
||||
Args:
|
||||
@@ -123,14 +109,14 @@ class ASRProvider(ASRProviderBase):
|
||||
return samples_float32, f.getframerate()
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""语音转文本主处理逻辑"""
|
||||
file_path = None
|
||||
try:
|
||||
# 保存音频文件
|
||||
start_time = time.time()
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -5,11 +5,8 @@ import json
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
import os
|
||||
import uuid
|
||||
from typing import Optional, Tuple, List
|
||||
import wave
|
||||
import opuslib_next
|
||||
|
||||
from core.providers.asr.dto.dto import InterfaceType
|
||||
import requests
|
||||
from core.providers.asr.base import ASRProviderBase
|
||||
from config.logger import setup_logging
|
||||
@@ -25,6 +22,7 @@ class ASRProvider(ASRProviderBase):
|
||||
|
||||
def __init__(self, config: dict, delete_audio_file: bool = True):
|
||||
super().__init__()
|
||||
self.interface_type = InterfaceType.NON_STREAM
|
||||
self.secret_id = config.get("secret_id")
|
||||
self.secret_key = config.get("secret_key")
|
||||
self.output_dir = config.get("output_dir")
|
||||
@@ -33,22 +31,8 @@ class ASRProvider(ASRProviderBase):
|
||||
# 确保输出目录存在
|
||||
os.makedirs(self.output_dir, exist_ok=True)
|
||||
|
||||
def save_audio_to_file(self, pcm_data: List[bytes], session_id: str) -> str:
|
||||
"""PCM数据保存为WAV文件"""
|
||||
module_name = __name__.split(".")[-1]
|
||||
file_name = f"asr_{module_name}_{session_id}_{uuid.uuid4()}.wav"
|
||||
file_path = os.path.join(self.output_dir, file_name)
|
||||
|
||||
with wave.open(file_path, "wb") as wf:
|
||||
wf.setnchannels(1)
|
||||
wf.setsampwidth(2) # 2 bytes = 16-bit
|
||||
wf.setframerate(16000)
|
||||
wf.writeframes(b"".join(pcm_data))
|
||||
|
||||
return file_path
|
||||
|
||||
async def speech_to_text(
|
||||
self, opus_data: List[bytes], session_id: str
|
||||
self, opus_data: List[bytes], session_id: str, audio_format="opus"
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""将语音数据转换为文本"""
|
||||
if not opus_data:
|
||||
@@ -63,7 +47,7 @@ class ASRProvider(ASRProviderBase):
|
||||
return None, file_path
|
||||
|
||||
# 将Opus音频数据解码为PCM
|
||||
if self.audio_format == "pcm":
|
||||
if audio_format == "pcm":
|
||||
pcm_data = opus_data
|
||||
else:
|
||||
pcm_data = self.decode_opus(opus_data)
|
||||
|
||||
@@ -122,6 +122,8 @@ class IntentProvider(IntentProviderBase):
|
||||
async def detect_intent(self, conn, dialogue_history: List[Dict], text: str) -> str:
|
||||
if not self.llm:
|
||||
raise ValueError("LLM provider not set")
|
||||
if conn.func_handler is None:
|
||||
return '{"function_call": {"name": "continue_chat"}}'
|
||||
|
||||
# 记录整体开始时间
|
||||
total_start_time = time.time()
|
||||
@@ -148,9 +150,8 @@ class IntentProvider(IntentProviderBase):
|
||||
self.clean_cache()
|
||||
|
||||
if self.promot == "":
|
||||
if hasattr(conn, "func_handler"):
|
||||
functions = conn.func_handler.get_functions()
|
||||
self.promot = self.get_intent_system_prompt(functions)
|
||||
functions = conn.func_handler.get_functions()
|
||||
self.promot = self.get_intent_system_prompt(functions)
|
||||
|
||||
music_config = initialize_music_handler(conn)
|
||||
music_file_names = music_config["music_file_names"]
|
||||
|
||||
@@ -52,6 +52,7 @@ class TTSProviderBase(ABC):
|
||||
)
|
||||
self.first_sentence_punctuations = (
|
||||
"，",
|
||||
"～",
|
||||
"~",
|
||||
"、",
|
||||
",",
|
||||
@@ -178,6 +179,9 @@ class TTSProviderBase(ABC):
|
||||
while not self.conn.stop_event.is_set():
|
||||
try:
|
||||
message = self.tts_text_queue.get(timeout=1)
|
||||
if self.conn.client_abort:
|
||||
logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
|
||||
continue
|
||||
if message.sentence_type == SentenceType.FIRST:
|
||||
# 初始化参数
|
||||
self.tts_stop_request = False
|
||||
@@ -216,6 +220,7 @@ class TTSProviderBase(ABC):
|
||||
logger.bind(tag=TAG).error(
|
||||
f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
|
||||
)
|
||||
continue
|
||||
|
||||
def _audio_play_priority_thread(self):
|
||||
while not self.conn.stop_event.is_set():
|
||||
|
||||
@@ -3,13 +3,13 @@ import uuid
|
||||
import json
|
||||
import queue
|
||||
import asyncio
|
||||
import threading
|
||||
import traceback
|
||||
import websockets
|
||||
from config.logger import setup_logging
|
||||
from core.utils import opus_encoder_utils
|
||||
from core.utils.util import check_model_key
|
||||
from core.providers.tts.base import TTSProviderBase
|
||||
from core.handle.abortHandle import handleAbortMessage
|
||||
from core.providers.tts.dto.dto import SentenceType, ContentType, InterfaceType
|
||||
|
||||
TAG = __name__
|
||||
@@ -138,6 +138,7 @@ class Response:
|
||||
class TTSProvider(TTSProviderBase):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
super().__init__(config, delete_audio_file)
|
||||
self.ws = None
|
||||
self.interface_type = InterfaceType.DUAL_STREAM
|
||||
self.appId = config.get("appid")
|
||||
self.access_token = config.get("access_token")
|
||||
@@ -152,73 +153,106 @@ class TTSProvider(TTSProviderBase):
|
||||
self.authorization = config.get("authorization")
|
||||
self.header = {"Authorization": f"{self.authorization}{self.access_token}"}
|
||||
self.enable_two_way = True
|
||||
self.start_connection_flag = False
|
||||
self.tts_text = ""
|
||||
# 合成文字语音后，播放的音频文件列表
|
||||
self.before_stop_play_files = []
|
||||
self.opus_encoder = opus_encoder_utils.OpusEncoderUtils(
|
||||
sample_rate=16000, channels=1, frame_size_ms=60
|
||||
)
|
||||
check_model_key("TTS", self.access_token)
|
||||
|
||||
# 添加会话状态控制
|
||||
self._session_lock = asyncio.Lock() # 会话操作的并发锁
|
||||
self._current_session_id = None # 当前会话ID
|
||||
self._session_started = False # 会话是否已开始
|
||||
self._session_finished = False # 会话是否已结束
|
||||
|
||||
###################################################################################
|
||||
# 火山双流式TTS重写父类的方法--开始
|
||||
###################################################################################
|
||||
|
||||
async def open_audio_channels(self, conn):
|
||||
await super().open_audio_channels(conn)
|
||||
ws_header = {
|
||||
"X-Api-App-Key": self.appId,
|
||||
"X-Api-Access-Key": self.access_token,
|
||||
"X-Api-Resource-Id": self.resource_id,
|
||||
"X-Api-Connect-Id": uuid.uuid4(),
|
||||
}
|
||||
self.ws = await websockets.connect(
|
||||
self.ws_url, additional_headers=ws_header, max_size=1000000000
|
||||
)
|
||||
tts_priority = threading.Thread(
|
||||
target=self._start_monitor_tts_response_thread, daemon=True
|
||||
)
|
||||
tts_priority.start()
|
||||
try:
|
||||
await super().open_audio_channels(conn)
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"Failed to open audio channels: {str(e)}")
|
||||
self.ws = None
|
||||
raise
|
||||
|
||||
async def _ensure_connection(self):
|
||||
"""建立新的WebSocket连接"""
|
||||
try:
|
||||
logger.bind(tag=TAG).info("开始建立新连接...")
|
||||
ws_header = {
|
||||
"X-Api-App-Key": self.appId,
|
||||
"X-Api-Access-Key": self.access_token,
|
||||
"X-Api-Resource-Id": self.resource_id,
|
||||
"X-Api-Connect-Id": uuid.uuid4(),
|
||||
}
|
||||
self.ws = await websockets.connect(
|
||||
self.ws_url, additional_headers=ws_header, max_size=1000000000
|
||||
)
|
||||
logger.bind(tag=TAG).info("WebSocket连接建立成功")
|
||||
return self.ws
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"建立连接失败: {str(e)}")
|
||||
self.ws = None
|
||||
raise
|
||||
|
||||
def tts_text_priority_thread(self):
|
||||
"""火山引擎双流式TTS的文本处理线程"""
|
||||
logger.bind(tag=TAG).info("TTS文本处理线程启动")
|
||||
while not self.conn.stop_event.is_set():
|
||||
try:
|
||||
logger.bind(tag=TAG).debug("等待TTS文本队列消息...")
|
||||
message = self.tts_text_queue.get(timeout=1)
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name}"
|
||||
f"收到TTS任务｜{message.sentence_type.name} ｜ {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
|
||||
)
|
||||
if self.conn.client_abort:
|
||||
logger.bind(tag=TAG).info("收到打断信息，终止TTS文本处理线程")
|
||||
continue
|
||||
|
||||
if message.sentence_type == SentenceType.FIRST:
|
||||
# 初始化参数
|
||||
future = asyncio.run_coroutine_threadsafe(
|
||||
self.start_session(self.conn.sentence_id), loop=self.conn.loop
|
||||
)
|
||||
future.result()
|
||||
self.tts_audio_first_sentence = True
|
||||
self.before_stop_play_files.clear()
|
||||
elif ContentType.TEXT == message.content_type:
|
||||
if message.content_detail:
|
||||
try:
|
||||
logger.bind(tag=TAG).info("开始启动TTS会话...")
|
||||
future = asyncio.run_coroutine_threadsafe(
|
||||
self.text_to_speak(message.content_detail, None),
|
||||
self.start_session(self.conn.sentence_id),
|
||||
loop=self.conn.loop,
|
||||
)
|
||||
future.result()
|
||||
self.tts_audio_first_sentence = True
|
||||
self.before_stop_play_files.clear()
|
||||
logger.bind(tag=TAG).info("TTS会话启动成功")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"启动TTS会话失败: {str(e)}")
|
||||
continue
|
||||
|
||||
elif ContentType.TEXT == message.content_type:
|
||||
if message.content_detail:
|
||||
try:
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"开始发送TTS文本: {message.content_detail}"
|
||||
)
|
||||
future = asyncio.run_coroutine_threadsafe(
|
||||
self.text_to_speak(message.content_detail, None),
|
||||
loop=self.conn.loop,
|
||||
)
|
||||
future.result()
|
||||
logger.bind(tag=TAG).debug("TTS文本发送成功")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
|
||||
continue
|
||||
|
||||
elif ContentType.FILE == message.content_type:
|
||||
logger.bind(tag=TAG).info(
|
||||
f"添加音频文件到待播放列表: {message.content_file}"
|
||||
)
|
||||
self.before_stop_play_files.append(
|
||||
(message.content_file, message.content_detail)
|
||||
)
|
||||
|
||||
if message.sentence_type == SentenceType.LAST:
|
||||
future = asyncio.run_coroutine_threadsafe(
|
||||
self.finish_session(self.conn.sentence_id), loop=self.conn.loop
|
||||
)
|
||||
future.result()
|
||||
try:
|
||||
logger.bind(tag=TAG).info("开始结束TTS会话...")
|
||||
future = asyncio.run_coroutine_threadsafe(
|
||||
self.finish_session(self.conn.sentence_id),
|
||||
loop=self.conn.loop,
|
||||
)
|
||||
future.result()
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"结束TTS会话失败: {str(e)}")
|
||||
continue
|
||||
|
||||
except queue.Empty:
|
||||
continue
|
||||
@@ -226,96 +260,220 @@ class TTSProvider(TTSProviderBase):
|
||||
logger.bind(tag=TAG).error(
|
||||
f"处理TTS文本失败: {str(e)}, 类型: {type(e).__name__}, 堆栈: {traceback.format_exc()}"
|
||||
)
|
||||
continue
|
||||
|
||||
async def text_to_speak(self, text, _):
|
||||
# 发送文本
|
||||
await self.send_text(self.speaker, text, self.conn.sentence_id)
|
||||
return
|
||||
"""发送文本到TTS服务"""
|
||||
try:
|
||||
# 建立新连接
|
||||
if self.ws is None:
|
||||
await handleAbortMessage(self.conn)
|
||||
logger.bind(tag=TAG).error(f"WebSocket连接不存在，终止发送文本")
|
||||
return
|
||||
# 发送文本
|
||||
await self.send_text(self.speaker, text, self.conn.sentence_id)
|
||||
return
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"发送TTS文本失败: {str(e)}")
|
||||
if self.ws:
|
||||
try:
|
||||
await self.ws.close()
|
||||
except:
|
||||
pass
|
||||
self.ws = None
|
||||
raise
|
||||
|
||||
###################################################################################
|
||||
# 火山双流式TTS重写父类的方法--结束
|
||||
###################################################################################
|
||||
def _start_monitor_tts_response_thread(self):
|
||||
# 初始化链接
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self._start_monitor_tts_response(), loop=self.conn.loop
|
||||
)
|
||||
async def start_session(self, session_id):
|
||||
logger.bind(tag=TAG).info(f"开始会话～～{session_id}")
|
||||
try:
|
||||
# 建立新连接
|
||||
await self._ensure_connection()
|
||||
|
||||
# 启动监听任务
|
||||
self._monitor_task = asyncio.create_task(self._start_monitor_tts_response())
|
||||
|
||||
header = Header(
|
||||
message_type=FULL_CLIENT_REQUEST,
|
||||
message_type_specific_flags=MsgTypeFlagWithEvent,
|
||||
serial_method=JSON,
|
||||
).as_bytes()
|
||||
optional = Optional(
|
||||
event=EVENT_StartSession, sessionId=session_id
|
||||
).as_bytes()
|
||||
payload = self.get_payload_bytes(
|
||||
event=EVENT_StartSession, speaker=self.speaker
|
||||
)
|
||||
await self.send_event(header, optional, payload)
|
||||
logger.bind(tag=TAG).info("会话启动请求已发送")
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
|
||||
# 确保清理资源
|
||||
if hasattr(self, "_monitor_task"):
|
||||
try:
|
||||
self._monitor_task.cancel()
|
||||
await self._monitor_task
|
||||
except:
|
||||
pass
|
||||
self._monitor_task = None
|
||||
if self.ws:
|
||||
try:
|
||||
await self.ws.close()
|
||||
except:
|
||||
pass
|
||||
self.ws = None
|
||||
raise
|
||||
|
||||
async def finish_session(self, session_id):
|
||||
logger.bind(tag=TAG).info(f"关闭会话～～{session_id}")
|
||||
try:
|
||||
if self.ws:
|
||||
header = Header(
|
||||
message_type=FULL_CLIENT_REQUEST,
|
||||
message_type_specific_flags=MsgTypeFlagWithEvent,
|
||||
serial_method=JSON,
|
||||
).as_bytes()
|
||||
optional = Optional(
|
||||
event=EVENT_FinishSession, sessionId=session_id
|
||||
).as_bytes()
|
||||
payload = str.encode("{}")
|
||||
await self.send_event(header, optional, payload)
|
||||
logger.bind(tag=TAG).info("会话结束请求已发送")
|
||||
|
||||
# 等待监听任务完成
|
||||
if hasattr(self, "_monitor_task"):
|
||||
try:
|
||||
await self._monitor_task
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(
|
||||
f"等待监听任务完成时发生错误: {str(e)}"
|
||||
)
|
||||
finally:
|
||||
self._monitor_task = None
|
||||
|
||||
# 关闭连接
|
||||
await self.close()
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
|
||||
# 确保清理资源
|
||||
if hasattr(self, "_monitor_task"):
|
||||
try:
|
||||
self._monitor_task.cancel()
|
||||
await self._monitor_task
|
||||
except:
|
||||
pass
|
||||
self._monitor_task = None
|
||||
if self.ws:
|
||||
try:
|
||||
await self.ws.close()
|
||||
except:
|
||||
pass
|
||||
self.ws = None
|
||||
raise
|
||||
|
||||
async def close(self):
|
||||
"""资源清理方法"""
|
||||
if self.ws:
|
||||
try:
|
||||
await self.ws.close()
|
||||
except:
|
||||
pass
|
||||
self.ws = None
|
||||
|
||||
async def _start_monitor_tts_response(self):
|
||||
"""监听TTS响应"""
|
||||
opus_datas_cache = []
|
||||
# 添加标志来区分是否是第一句话
|
||||
is_first_sentence = True
|
||||
while not self.conn.stop_event.is_set():
|
||||
try:
|
||||
# 确保 `recv()` 运行在同一个 event loop
|
||||
msg = await self.ws.recv()
|
||||
res = self.parser_response(msg)
|
||||
self.print_response(res, "send_text res:")
|
||||
try:
|
||||
while not self.conn.stop_event.is_set():
|
||||
try:
|
||||
# 确保 `recv()` 运行在同一个 event loop
|
||||
msg = await self.ws.recv()
|
||||
res = self.parser_response(msg)
|
||||
self.print_response(res, "send_text res:")
|
||||
|
||||
if res.optional.event == EVENT_TTSSentenceStart:
|
||||
json_data = json.loads(res.payload.decode("utf-8"))
|
||||
self.tts_text = json_data.get("text", "")
|
||||
logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
|
||||
self.tts_audio_queue.put((SentenceType.FIRST, [], self.tts_text))
|
||||
opus_datas_cache = []
|
||||
elif (
|
||||
res.optional.event == EVENT_TTSResponse
|
||||
and res.header.message_type == AUDIO_ONLY_RESPONSE
|
||||
):
|
||||
logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
|
||||
opus_datas = self.wav_to_opus_data_audio_raw(res.payload)
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"推送数据到队列里面帧数～～{len(opus_datas)}"
|
||||
)
|
||||
if is_first_sentence:
|
||||
# 第一句话直接发送
|
||||
# 检查客户端是否中止
|
||||
if self.conn.client_abort:
|
||||
logger.bind(tag=TAG).info("收到打断信息，终止监听TTS响应")
|
||||
break
|
||||
|
||||
if res.optional.event == EVENT_TTSSentenceStart:
|
||||
json_data = json.loads(res.payload.decode("utf-8"))
|
||||
self.tts_text = json_data.get("text", "")
|
||||
logger.bind(tag=TAG).debug(f"句子语音生成开始: {self.tts_text}")
|
||||
self.tts_audio_queue.put(
|
||||
(SentenceType.MIDDLE, opus_datas, self.tts_text)
|
||||
(SentenceType.FIRST, [], self.tts_text)
|
||||
)
|
||||
else:
|
||||
# 后续句子缓存
|
||||
opus_datas_cache = opus_datas_cache + opus_datas
|
||||
elif res.optional.event == EVENT_TTSSentenceEnd:
|
||||
logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
|
||||
if not is_first_sentence:
|
||||
# 只有非第一句话才发送缓存的数据
|
||||
self.tts_audio_queue.put(
|
||||
(SentenceType.MIDDLE, opus_datas_cache, self.tts_text)
|
||||
opus_datas_cache = []
|
||||
elif (
|
||||
res.optional.event == EVENT_TTSResponse
|
||||
and res.header.message_type == AUDIO_ONLY_RESPONSE
|
||||
):
|
||||
logger.bind(tag=TAG).debug(f"推送数据到队列里面～～")
|
||||
opus_datas = self.wav_to_opus_data_audio_raw(res.payload)
|
||||
logger.bind(tag=TAG).debug(
|
||||
f"推送数据到队列里面帧数～～{len(opus_datas)}"
|
||||
)
|
||||
# 第一句话结束后，将标志设置为False
|
||||
is_first_sentence = False
|
||||
elif res.optional.event == EVENT_SessionFinished:
|
||||
logger.bind(tag=TAG).debug(f"会话结束～～")
|
||||
for tts_file, text in self.before_stop_play_files:
|
||||
if tts_file and os.path.exists(tts_file):
|
||||
audio_datas = self._process_audio_file(tts_file)
|
||||
if is_first_sentence:
|
||||
# 第一句话直接发送
|
||||
self.tts_audio_queue.put(
|
||||
(SentenceType.MIDDLE, audio_datas, text)
|
||||
(SentenceType.MIDDLE, opus_datas, self.tts_text)
|
||||
)
|
||||
self.before_stop_play_files.clear()
|
||||
self.tts_audio_queue.put((SentenceType.LAST, [], None))
|
||||
|
||||
opus_datas_cache = []
|
||||
is_first_sentence = True
|
||||
continue
|
||||
except websockets.ConnectionClosed:
|
||||
break # 连接关闭时退出监听
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"Error in _start_monitor_tts_response: {e}")
|
||||
traceback.print_exc()
|
||||
continue
|
||||
else:
|
||||
# 后续句子缓存
|
||||
opus_datas_cache = opus_datas_cache + opus_datas
|
||||
elif res.optional.event == EVENT_TTSSentenceEnd:
|
||||
logger.bind(tag=TAG).info(f"句子语音生成成功：{self.tts_text}")
|
||||
if not is_first_sentence:
|
||||
# 只有非第一句话才发送缓存的数据
|
||||
self.tts_audio_queue.put(
|
||||
(SentenceType.MIDDLE, opus_datas_cache, self.tts_text)
|
||||
)
|
||||
# 第一句话结束后，将标志设置为False
|
||||
is_first_sentence = False
|
||||
elif res.optional.event == EVENT_SessionFinished:
|
||||
logger.bind(tag=TAG).debug(f"会话结束～～")
|
||||
for tts_file, text in self.before_stop_play_files:
|
||||
if tts_file and os.path.exists(tts_file):
|
||||
audio_datas = self._process_audio_file(tts_file)
|
||||
self.tts_audio_queue.put(
|
||||
(SentenceType.MIDDLE, audio_datas, text)
|
||||
)
|
||||
self.before_stop_play_files.clear()
|
||||
self.tts_audio_queue.put((SentenceType.LAST, [], None))
|
||||
break
|
||||
except websockets.ConnectionClosed:
|
||||
logger.bind(tag=TAG).warning("WebSocket连接已关闭")
|
||||
break
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(
|
||||
f"Error in _start_monitor_tts_response: {e}"
|
||||
)
|
||||
traceback.print_exc()
|
||||
break
|
||||
finally:
|
||||
# 确保清理资源
|
||||
if self.ws:
|
||||
try:
|
||||
await self.ws.close()
|
||||
except:
|
||||
pass
|
||||
self.ws = None
|
||||
|
||||
async def send_event(
|
||||
self, header: bytes, optional: bytes | None = None, payload: bytes = None
|
||||
):
|
||||
full_client_request = bytearray(header)
|
||||
if optional is not None:
|
||||
full_client_request.extend(optional)
|
||||
if payload is not None:
|
||||
payload_size = len(payload).to_bytes(4, "big", signed=True)
|
||||
full_client_request.extend(payload_size)
|
||||
full_client_request.extend(payload)
|
||||
await self.ws.send(full_client_request)
|
||||
try:
|
||||
full_client_request = bytearray(header)
|
||||
if optional is not None:
|
||||
full_client_request.extend(optional)
|
||||
if payload is not None:
|
||||
payload_size = len(payload).to_bytes(4, "big", signed=True)
|
||||
full_client_request.extend(payload_size)
|
||||
full_client_request.extend(payload)
|
||||
await self.ws.send(full_client_request)
|
||||
except websockets.ConnectionClosed:
|
||||
logger.bind(tag=TAG).error(f"ConnectionClosed")
|
||||
raise
|
||||
|
||||
async def send_text(self, speaker: str, text: str, session_id):
|
||||
header = Header(
|
||||
@@ -438,92 +596,6 @@ class TTSProvider(TTSProviderBase):
|
||||
)
|
||||
)
|
||||
|
||||
async def finish_connection(self):
|
||||
header = Header(
|
||||
message_type=FULL_CLIENT_REQUEST,
|
||||
message_type_specific_flags=MsgTypeFlagWithEvent,
|
||||
serial_method=JSON,
|
||||
).as_bytes()
|
||||
optional = Optional(event=EVENT_FinishConnection).as_bytes()
|
||||
payload = str.encode("{}")
|
||||
await self.send_event(header, optional, payload)
|
||||
return
|
||||
|
||||
async def start_session(self, session_id):
|
||||
logger.bind(tag=TAG).debug(f"开始会话～～{session_id}")
|
||||
async with self._session_lock:
|
||||
# 如果已有会话未结束，先关闭它
|
||||
if self._session_started and not self._session_finished:
|
||||
logger.bind(tag=TAG).warning(
|
||||
f"发现未关闭的会话 {self._current_session_id}，正在关闭..."
|
||||
)
|
||||
await self.finish_session(self._current_session_id)
|
||||
|
||||
# 重置会话状态
|
||||
self._current_session_id = session_id
|
||||
self._session_started = True
|
||||
self._session_finished = False
|
||||
|
||||
header = Header(
|
||||
message_type=FULL_CLIENT_REQUEST,
|
||||
message_type_specific_flags=MsgTypeFlagWithEvent,
|
||||
serial_method=JSON,
|
||||
).as_bytes()
|
||||
optional = Optional(
|
||||
event=EVENT_StartSession, sessionId=session_id
|
||||
).as_bytes()
|
||||
payload = self.get_payload_bytes(
|
||||
event=EVENT_StartSession, speaker=self.speaker
|
||||
)
|
||||
await self.send_event(header, optional, payload)
|
||||
|
||||
async def finish_session(self, session_id):
|
||||
logger.bind(tag=TAG).debug(f"关闭会话～～{session_id}")
|
||||
async with self._session_lock:
|
||||
# 检查会话状态
|
||||
if not self._session_started:
|
||||
logger.bind(tag=TAG).warning(f"尝试关闭未开始的会话 {session_id}")
|
||||
return
|
||||
|
||||
if self._session_finished:
|
||||
logger.bind(tag=TAG).warning(f"会话 {session_id} 已经关闭")
|
||||
return
|
||||
|
||||
if self._current_session_id != session_id:
|
||||
logger.bind(tag=TAG).warning(
|
||||
f"尝试关闭错误的会话 {session_id}，当前会话为 {self._current_session_id}"
|
||||
)
|
||||
return
|
||||
|
||||
header = Header(
|
||||
message_type=FULL_CLIENT_REQUEST,
|
||||
message_type_specific_flags=MsgTypeFlagWithEvent,
|
||||
serial_method=JSON,
|
||||
).as_bytes()
|
||||
optional = Optional(
|
||||
event=EVENT_FinishSession, sessionId=session_id
|
||||
).as_bytes()
|
||||
payload = str.encode("{}")
|
||||
await self.send_event(header, optional, payload)
|
||||
|
||||
# 更新会话状态
|
||||
self._session_finished = True
|
||||
self._session_started = False
|
||||
self._current_session_id = None
|
||||
|
||||
async def reset(self):
|
||||
# 关闭之前的对话
|
||||
if self.start_connection_flag:
|
||||
await self.finish_connection()
|
||||
self.start_connection_flag = False
|
||||
await self.start_connection()
|
||||
self.start_connection_flag = True
|
||||
|
||||
async def close(self):
|
||||
"""资源清理方法"""
|
||||
await self.finish_connection()
|
||||
await self.ws.close()
|
||||
|
||||
def wav_to_opus_data_audio_raw(self, raw_data_var, is_end=False):
|
||||
opus_datas = self.opus_encoder.encode_pcm_to_opus(raw_data_var, is_end)
|
||||
return opus_datas
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
from abc import ABC, abstractmethod
|
||||
from config.logger import setup_logging
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
|
||||
|
||||
class VLLMProviderBase(ABC):
|
||||
@abstractmethod
|
||||
def response(self, question, base64_image):
|
||||
"""VLLM response generator"""
|
||||
pass
|
||||
@@ -0,0 +1,65 @@
|
||||
import openai
|
||||
import json
|
||||
from config.logger import setup_logging
|
||||
from core.utils.util import check_model_key
|
||||
from core.providers.vllm.base import VLLMProviderBase
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
|
||||
|
||||
class VLLMProvider(VLLMProviderBase):
|
||||
def __init__(self, config):
|
||||
self.model_name = config.get("model_name")
|
||||
self.api_key = config.get("api_key")
|
||||
if "base_url" in config:
|
||||
self.base_url = config.get("base_url")
|
||||
else:
|
||||
self.base_url = config.get("url")
|
||||
|
||||
param_defaults = {
|
||||
"max_tokens": (500, int),
|
||||
"temperature": (0.7, lambda x: round(float(x), 1)),
|
||||
"top_p": (1.0, lambda x: round(float(x), 1)),
|
||||
}
|
||||
|
||||
for param, (default, converter) in param_defaults.items():
|
||||
value = config.get(param)
|
||||
try:
|
||||
setattr(
|
||||
self,
|
||||
param,
|
||||
converter(value) if value not in (None, "") else default,
|
||||
)
|
||||
except (ValueError, TypeError):
|
||||
setattr(self, param, default)
|
||||
|
||||
check_model_key("VLLM", self.api_key)
|
||||
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
|
||||
|
||||
def response(self, question, base64_image):
|
||||
try:
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": question},
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": f"data:image/jpeg;base64,{base64_image}"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model_name, messages=messages, stream=False
|
||||
)
|
||||
|
||||
return response.choices[0].message.content
|
||||
|
||||
except Exception as e:
|
||||
logger.bind(tag=TAG).error(f"Error in response generation: {e}")
|
||||
raise
|
||||
@@ -0,0 +1,43 @@
|
||||
import jwt
|
||||
import time
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, Tuple
|
||||
|
||||
|
||||
class AuthToken:
|
||||
def __init__(self, secret_key: str):
|
||||
self.secret_key = secret_key
|
||||
|
||||
def generate_token(self, device_id: str) -> str:
|
||||
"""
|
||||
生成JWT token
|
||||
:param device_id: 设备ID
|
||||
:return: JWT token字符串
|
||||
"""
|
||||
# 设置过期时间为1小时后
|
||||
expire_time = datetime.now(timezone.utc) + timedelta(hours=1)
|
||||
|
||||
# 创建payload
|
||||
payload = {"device_id": device_id, "exp": expire_time.timestamp()}
|
||||
|
||||
# 使用JWT进行编码
|
||||
token = jwt.encode(payload, self.secret_key, algorithm="HS256")
|
||||
return token
|
||||
|
||||
def verify_token(self, token: str) -> Tuple[bool, Optional[str]]:
|
||||
"""
|
||||
验证token
|
||||
:param token: JWT token字符串
|
||||
:return: (是否有效, 设备ID)
|
||||
"""
|
||||
try:
|
||||
# 解码token
|
||||
payload = jwt.decode(token, self.secret_key, algorithms=["HS256"])
|
||||
|
||||
# 检查是否过期
|
||||
if payload["exp"] < time.time():
|
||||
return False, None
|
||||
|
||||
return True, payload["device_id"]
|
||||
except jwt.InvalidTokenError:
|
||||
return False, None
|
||||
@@ -0,0 +1,128 @@
|
||||
from typing import Dict, Any
|
||||
from config.logger import setup_logging
|
||||
from core.utils import tts, llm, intent, memory, vad, asr
|
||||
|
||||
TAG = __name__
|
||||
logger = setup_logging()
|
||||
|
||||
|
||||
def initialize_modules(
|
||||
logger,
|
||||
config: Dict[str, Any],
|
||||
init_vad=False,
|
||||
init_asr=False,
|
||||
init_llm=False,
|
||||
init_tts=False,
|
||||
init_memory=False,
|
||||
init_intent=False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
初始化所有模块组件
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
|
||||
Returns:
|
||||
Dict[str, Any]: 包含所有初始化后的模块的字典
|
||||
"""
|
||||
modules = {}
|
||||
|
||||
# 初始化TTS模块
|
||||
if init_tts:
|
||||
select_tts_module = config["selected_module"]["TTS"]
|
||||
modules["tts"] = initialize_tts(config)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: tts成功 {select_tts_module}")
|
||||
|
||||
# 初始化LLM模块
|
||||
if init_llm:
|
||||
select_llm_module = config["selected_module"]["LLM"]
|
||||
llm_type = (
|
||||
select_llm_module
|
||||
if "type" not in config["LLM"][select_llm_module]
|
||||
else config["LLM"][select_llm_module]["type"]
|
||||
)
|
||||
modules["llm"] = llm.create_instance(
|
||||
llm_type,
|
||||
config["LLM"][select_llm_module],
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: llm成功 {select_llm_module}")
|
||||
|
||||
# 初始化Intent模块
|
||||
if init_intent:
|
||||
select_intent_module = config["selected_module"]["Intent"]
|
||||
intent_type = (
|
||||
select_intent_module
|
||||
if "type" not in config["Intent"][select_intent_module]
|
||||
else config["Intent"][select_intent_module]["type"]
|
||||
)
|
||||
modules["intent"] = intent.create_instance(
|
||||
intent_type,
|
||||
config["Intent"][select_intent_module],
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: intent成功 {select_intent_module}")
|
||||
|
||||
# 初始化Memory模块
|
||||
if init_memory:
|
||||
select_memory_module = config["selected_module"]["Memory"]
|
||||
memory_type = (
|
||||
select_memory_module
|
||||
if "type" not in config["Memory"][select_memory_module]
|
||||
else config["Memory"][select_memory_module]["type"]
|
||||
)
|
||||
modules["memory"] = memory.create_instance(
|
||||
memory_type,
|
||||
config["Memory"][select_memory_module],
|
||||
config.get("summaryMemory", None),
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: memory成功 {select_memory_module}")
|
||||
|
||||
# 初始化VAD模块
|
||||
if init_vad:
|
||||
select_vad_module = config["selected_module"]["VAD"]
|
||||
vad_type = (
|
||||
select_vad_module
|
||||
if "type" not in config["VAD"][select_vad_module]
|
||||
else config["VAD"][select_vad_module]["type"]
|
||||
)
|
||||
modules["vad"] = vad.create_instance(
|
||||
vad_type,
|
||||
config["VAD"][select_vad_module],
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: vad成功 {select_vad_module}")
|
||||
|
||||
# 初始化ASR模块
|
||||
if init_asr:
|
||||
select_asr_module = config["selected_module"]["ASR"]
|
||||
modules["asr"] = initialize_asr(config)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: asr成功 {select_asr_module}")
|
||||
return modules
|
||||
|
||||
|
||||
def initialize_tts(config):
|
||||
select_tts_module = config["selected_module"]["TTS"]
|
||||
tts_type = (
|
||||
select_tts_module
|
||||
if "type" not in config["TTS"][select_tts_module]
|
||||
else config["TTS"][select_tts_module]["type"]
|
||||
)
|
||||
new_tts = tts.create_instance(
|
||||
tts_type,
|
||||
config["TTS"][select_tts_module],
|
||||
str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
|
||||
)
|
||||
return new_tts
|
||||
|
||||
|
||||
def initialize_asr(config):
|
||||
select_asr_module = config["selected_module"]["ASR"]
|
||||
asr_type = (
|
||||
select_asr_module
|
||||
if "type" not in config["ASR"][select_asr_module]
|
||||
else config["ASR"][select_asr_module]["type"]
|
||||
)
|
||||
new_asr = asr.create_instance(
|
||||
asr_type,
|
||||
config["ASR"][select_asr_module],
|
||||
str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
|
||||
)
|
||||
return new_asr
|
||||
@@ -7,8 +7,6 @@ import numpy as np
|
||||
import requests
|
||||
import opuslib_next
|
||||
from pydub import AudioSegment
|
||||
from typing import Dict, Any
|
||||
from core.utils import tts, llm, intent, memory, vad, asr
|
||||
import copy
|
||||
|
||||
TAG = __name__
|
||||
@@ -245,122 +243,6 @@ def extract_json_from_string(input_string):
|
||||
return None
|
||||
|
||||
|
||||
def initialize_modules(
|
||||
logger,
|
||||
config: Dict[str, Any],
|
||||
init_vad=False,
|
||||
init_asr=False,
|
||||
init_llm=False,
|
||||
init_tts=False,
|
||||
init_memory=False,
|
||||
init_intent=False,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
初始化所有模块组件
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
|
||||
Returns:
|
||||
Dict[str, Any]: 包含所有初始化后的模块的字典
|
||||
"""
|
||||
modules = {}
|
||||
|
||||
# 初始化TTS模块
|
||||
if init_tts:
|
||||
select_tts_module = config["selected_module"]["TTS"]
|
||||
modules["tts"] = initialize_tts(config)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: tts成功 {select_tts_module}")
|
||||
|
||||
# 初始化LLM模块
|
||||
if init_llm:
|
||||
select_llm_module = config["selected_module"]["LLM"]
|
||||
llm_type = (
|
||||
select_llm_module
|
||||
if "type" not in config["LLM"][select_llm_module]
|
||||
else config["LLM"][select_llm_module]["type"]
|
||||
)
|
||||
modules["llm"] = llm.create_instance(
|
||||
llm_type,
|
||||
config["LLM"][select_llm_module],
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: llm成功 {select_llm_module}")
|
||||
|
||||
# 初始化Intent模块
|
||||
if init_intent:
|
||||
select_intent_module = config["selected_module"]["Intent"]
|
||||
intent_type = (
|
||||
select_intent_module
|
||||
if "type" not in config["Intent"][select_intent_module]
|
||||
else config["Intent"][select_intent_module]["type"]
|
||||
)
|
||||
modules["intent"] = intent.create_instance(
|
||||
intent_type,
|
||||
config["Intent"][select_intent_module],
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: intent成功 {select_intent_module}")
|
||||
|
||||
# 初始化Memory模块
|
||||
if init_memory:
|
||||
select_memory_module = config["selected_module"]["Memory"]
|
||||
memory_type = (
|
||||
select_memory_module
|
||||
if "type" not in config["Memory"][select_memory_module]
|
||||
else config["Memory"][select_memory_module]["type"]
|
||||
)
|
||||
modules["memory"] = memory.create_instance(
|
||||
memory_type,
|
||||
config["Memory"][select_memory_module],
|
||||
config.get("summaryMemory", None),
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: memory成功 {select_memory_module}")
|
||||
|
||||
# 初始化VAD模块
|
||||
if init_vad:
|
||||
select_vad_module = config["selected_module"]["VAD"]
|
||||
vad_type = (
|
||||
select_vad_module
|
||||
if "type" not in config["VAD"][select_vad_module]
|
||||
else config["VAD"][select_vad_module]["type"]
|
||||
)
|
||||
modules["vad"] = vad.create_instance(
|
||||
vad_type,
|
||||
config["VAD"][select_vad_module],
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: vad成功 {select_vad_module}")
|
||||
|
||||
# 初始化ASR模块
|
||||
if init_asr:
|
||||
select_asr_module = config["selected_module"]["ASR"]
|
||||
asr_type = (
|
||||
select_asr_module
|
||||
if "type" not in config["ASR"][select_asr_module]
|
||||
else config["ASR"][select_asr_module]["type"]
|
||||
)
|
||||
modules["asr"] = asr.create_instance(
|
||||
asr_type,
|
||||
config["ASR"][select_asr_module],
|
||||
str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
|
||||
)
|
||||
logger.bind(tag=TAG).info(f"初始化组件: asr成功 {select_asr_module}")
|
||||
return modules
|
||||
|
||||
|
||||
def initialize_tts(config):
|
||||
select_tts_module = config["selected_module"]["TTS"]
|
||||
tts_type = (
|
||||
select_tts_module
|
||||
if "type" not in config["TTS"][select_tts_module]
|
||||
else config["TTS"][select_tts_module]["type"]
|
||||
)
|
||||
new_tts = tts.create_instance(
|
||||
tts_type,
|
||||
config["TTS"][select_tts_module],
|
||||
str(config.get("delete_audio", True)).lower() in ("true", "1", "yes"),
|
||||
)
|
||||
return new_tts
|
||||
|
||||
|
||||
def analyze_emotion(text):
|
||||
"""
|
||||
分析文本情感并返回对应的emoji名称（支持中英文）
|
||||
@@ -1000,3 +882,51 @@ def filter_sensitive_info(config: dict) -> dict:
|
||||
return filtered
|
||||
|
||||
return _filter_dict(copy.deepcopy(config))
|
||||
|
||||
|
||||
def get_vision_url(config: dict) -> str:
|
||||
"""获取 vision URL
|
||||
|
||||
Args:
|
||||
config: 配置字典
|
||||
|
||||
Returns:
|
||||
str: vision URL
|
||||
"""
|
||||
server_config = config["server"]
|
||||
vision_explain = server_config.get("vision_explain", "")
|
||||
if "你的" in vision_explain:
|
||||
local_ip = get_local_ip()
|
||||
port = int(server_config.get("http_port", 8003))
|
||||
vision_explain = f"http://{local_ip}:{port}/mcp/vision/explain"
|
||||
return vision_explain
|
||||
|
||||
|
||||
def is_valid_image_file(file_data: bytes) -> bool:
|
||||
"""
|
||||
检查文件数据是否为有效的图片格式
|
||||
|
||||
Args:
|
||||
file_data: 文件的二进制数据
|
||||
|
||||
Returns:
|
||||
bool: 如果是有效的图片格式返回True，否则返回False
|
||||
"""
|
||||
# 常见图片格式的魔数（文件头）
|
||||
image_signatures = {
|
||||
b"\xff\xd8\xff": "JPEG",
|
||||
b"\x89PNG\r\n\x1a\n": "PNG",
|
||||
b"GIF87a": "GIF",
|
||||
b"GIF89a": "GIF",
|
||||
b"BM": "BMP",
|
||||
b"II*\x00": "TIFF",
|
||||
b"MM\x00*": "TIFF",
|
||||
b"RIFF": "WEBP",
|
||||
}
|
||||
|
||||
# 检查文件头是否匹配任何已知的图片格式
|
||||
for signature in image_signatures:
|
||||
if file_data.startswith(signature):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
import os
|
||||
import sys
|
||||
|
||||
# 添加项目根目录到Python路径
|
||||
current_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
project_root = os.path.abspath(os.path.join(current_dir, "..", ".."))
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from config.logger import setup_logging
|
||||
import importlib
|
||||
|
||||
logger = setup_logging()
|
||||
|
||||
|
||||
def create_instance(class_name, *args, **kwargs):
|
||||
# 创建LLM实例
|
||||
if os.path.exists(os.path.join("core", "providers", "vllm", f"{class_name}.py")):
|
||||
lib_name = f"core.providers.vllm.{class_name}"
|
||||
if lib_name not in sys.modules:
|
||||
sys.modules[lib_name] = importlib.import_module(f"{lib_name}")
|
||||
return sys.modules[lib_name].VLLMProvider(*args, **kwargs)
|
||||
|
||||
raise ValueError(f"不支持的VLLM类型: {class_name}，请检查该配置的type是否设置正确")
|
||||
@@ -2,8 +2,9 @@ import asyncio
|
||||
import websockets
|
||||
from config.logger import setup_logging
|
||||
from core.connection import ConnectionHandler
|
||||
from core.utils.util import initialize_modules, check_vad_update, check_asr_update
|
||||
from config.config_loader import get_config_from_api
|
||||
from core.utils.modules_initialize import initialize_modules
|
||||
from core.utils.util import check_vad_update, check_asr_update
|
||||
|
||||
TAG = __name__
|
||||
|
||||
|
||||
@@ -13,8 +13,8 @@ services:
|
||||
ports:
|
||||
# ws服务端
|
||||
- "8000:8000"
|
||||
# ota服务端
|
||||
- "8002:8002"
|
||||
# http服务的端口，用于简单OTA接口(单服务部署)，以及视觉分析接口
|
||||
- "8003:8003"
|
||||
volumes:
|
||||
# 配置文件目录
|
||||
- ./data:/opt/xiaozhi-esp32-server/data
|
||||
|
||||
@@ -15,6 +15,8 @@ services:
|
||||
ports:
|
||||
# ws服务端
|
||||
- "8000:8000"
|
||||
# http服务的端口，用于视觉分析接口
|
||||
- "8003:8003"
|
||||
security_opt:
|
||||
- seccomp:unconfined
|
||||
environment:
|
||||
|
||||
@@ -140,7 +140,9 @@ class AsyncPerformanceTester:
|
||||
|
||||
print(f"🎵 测试 STT: {stt_name}")
|
||||
|
||||
text, _ = await stt.speech_to_text([self.test_wav_list[0]], "1")
|
||||
text, _ = await stt.speech_to_text(
|
||||
[self.test_wav_list[0]], "1", stt.audio_format
|
||||
)
|
||||
|
||||
if text is None:
|
||||
print(f"❌ {stt_name} 连接失败")
|
||||
@@ -151,7 +153,7 @@ class AsyncPerformanceTester:
|
||||
|
||||
for i, sentence in enumerate(self.test_wav_list, 1):
|
||||
start = time.time()
|
||||
text, _ = await stt.speech_to_text([sentence], "1")
|
||||
text, _ = await stt.speech_to_text([sentence], "1", stt.audio_format)
|
||||
duration = time.time() - start
|
||||
total_time += duration
|
||||
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
import time
|
||||
import asyncio
|
||||
import logging
|
||||
import statistics
|
||||
import base64
|
||||
from typing import Dict
|
||||
from tabulate import tabulate
|
||||
from config.settings import load_config
|
||||
from core.utils.vllm import create_instance
|
||||
|
||||
# 设置全局日志级别为WARNING，抑制INFO级别日志
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
|
||||
class AsyncVisionPerformanceTester:
|
||||
def __init__(self):
|
||||
self.config = load_config()
|
||||
self.test_images = [
|
||||
"../../docs/images/demo1.png",
|
||||
"../../docs/images/demo2.png",
|
||||
]
|
||||
self.test_questions = [
|
||||
"这张图片里有什么？",
|
||||
"请详细描述这张图片的内容",
|
||||
]
|
||||
|
||||
# 加载测试图片
|
||||
self.results = {"vllm": {}}
|
||||
|
||||
async def _test_vllm(self, vllm_name: str, config: Dict) -> Dict:
|
||||
"""异步测试单个视觉大模型性能"""
|
||||
try:
|
||||
# 检查API密钥配置
|
||||
if "api_key" in config and any(
|
||||
x in config["api_key"] for x in ["你的", "placeholder", "sk-xxx"]
|
||||
):
|
||||
print(f"⏭️ VLLM {vllm_name} 未配置api_key，已跳过")
|
||||
return {"name": vllm_name, "type": "vllm", "errors": 1}
|
||||
|
||||
# 获取实际类型（兼容旧配置）
|
||||
module_type = config.get("type", vllm_name)
|
||||
vllm = create_instance(module_type, config)
|
||||
|
||||
print(f"🖼️ 测试 VLLM: {vllm_name}")
|
||||
|
||||
# 创建所有测试任务
|
||||
test_tasks = []
|
||||
for question in self.test_questions:
|
||||
for image in self.test_images:
|
||||
test_tasks.append(
|
||||
self._test_single_vision(vllm_name, vllm, question, image)
|
||||
)
|
||||
|
||||
# 并发执行所有测试
|
||||
test_results = await asyncio.gather(*test_tasks)
|
||||
|
||||
# 处理结果
|
||||
valid_results = [r for r in test_results if r is not None]
|
||||
if not valid_results:
|
||||
print(f"⚠️ {vllm_name} 无有效数据，可能配置错误")
|
||||
return {"name": vllm_name, "type": "vllm", "errors": 1}
|
||||
|
||||
response_times = [r["response_time"] for r in valid_results]
|
||||
|
||||
# 过滤异常数据
|
||||
mean = statistics.mean(response_times)
|
||||
stdev = statistics.stdev(response_times) if len(response_times) > 1 else 0
|
||||
filtered_times = [t for t in response_times if t <= mean + 3 * stdev]
|
||||
|
||||
if len(filtered_times) < len(test_tasks) * 0.5:
|
||||
print(f"⚠️ {vllm_name} 有效数据不足，可能网络不稳定")
|
||||
return {"name": vllm_name, "type": "vllm", "errors": 1}
|
||||
|
||||
return {
|
||||
"name": vllm_name,
|
||||
"type": "vllm",
|
||||
"avg_response": sum(response_times) / len(response_times),
|
||||
"std_response": (
|
||||
statistics.stdev(response_times) if len(response_times) > 1 else 0
|
||||
),
|
||||
"errors": 0,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(f"⚠️ VLLM {vllm_name} 测试失败: {str(e)}")
|
||||
return {"name": vllm_name, "type": "vllm", "errors": 1}
|
||||
|
||||
async def _test_single_vision(
|
||||
self, vllm_name: str, vllm, question: str, image: str
|
||||
) -> Dict:
|
||||
"""测试单个视觉问题的性能"""
|
||||
try:
|
||||
print(f"📝 {vllm_name} 开始测试: {question[:20]}...")
|
||||
start_time = time.time()
|
||||
|
||||
# 读取图片并转换为base64
|
||||
with open(image, "rb") as image_file:
|
||||
image_data = image_file.read()
|
||||
image_base64 = base64.b64encode(image_data).decode("utf-8")
|
||||
|
||||
# 直接获取响应
|
||||
response = vllm.response(question, image_base64)
|
||||
response_time = time.time() - start_time
|
||||
print(f"✓ {vllm_name} 完成响应: {response_time:.3f}s")
|
||||
|
||||
return {
|
||||
"name": vllm_name,
|
||||
"type": "vllm",
|
||||
"response_time": response_time,
|
||||
}
|
||||
except Exception as e:
|
||||
print(f"⚠️ {vllm_name} 测试失败: {str(e)}")
|
||||
return None
|
||||
|
||||
def _print_results(self):
|
||||
"""打印测试结果"""
|
||||
vllm_table = []
|
||||
for name, data in self.results["vllm"].items():
|
||||
if data["errors"] == 0:
|
||||
stability = data["std_response"] / data["avg_response"]
|
||||
vllm_table.append(
|
||||
[
|
||||
name,
|
||||
f"{data['avg_response']:.3f}秒",
|
||||
f"{stability:.3f}",
|
||||
]
|
||||
)
|
||||
|
||||
if vllm_table:
|
||||
print("\n视觉大模型性能排行:\n")
|
||||
print(
|
||||
tabulate(
|
||||
vllm_table,
|
||||
headers=["模型名称", "响应耗时", "稳定性"],
|
||||
tablefmt="github",
|
||||
colalign=("left", "right", "right"),
|
||||
disable_numparse=True,
|
||||
)
|
||||
)
|
||||
else:
|
||||
print("\n⚠️ 没有可用的视觉大模型进行测试。")
|
||||
|
||||
async def run(self):
|
||||
"""执行全量异步测试"""
|
||||
print("🔍 开始筛选可用视觉大模型...")
|
||||
|
||||
if not self.test_images:
|
||||
print(f"\n⚠️ {self.image_root} 路径下没有图片文件，无法进行测试")
|
||||
return
|
||||
|
||||
# 创建所有测试任务
|
||||
all_tasks = []
|
||||
|
||||
# VLLM测试任务
|
||||
if self.config.get("VLLM") is not None:
|
||||
for vllm_name, config in self.config.get("VLLM", {}).items():
|
||||
if "api_key" in config and any(
|
||||
x in config["api_key"] for x in ["你的", "placeholder", "sk-xxx"]
|
||||
):
|
||||
print(f"⏭️ VLLM {vllm_name} 未配置api_key，已跳过")
|
||||
continue
|
||||
print(f"🖼️ 添加VLLM测试任务: {vllm_name}")
|
||||
all_tasks.append(self._test_vllm(vllm_name, config))
|
||||
|
||||
print(f"\n✅ 找到 {len(all_tasks)} 个可用视觉大模型")
|
||||
print(f"✅ 使用 {len(self.test_images)} 张测试图片")
|
||||
print(f"✅ 使用 {len(self.test_questions)} 个测试问题")
|
||||
print("\n⏳ 开始并发测试所有模型...\n")
|
||||
|
||||
# 并发执行所有测试任务
|
||||
all_results = await asyncio.gather(*all_tasks, return_exceptions=True)
|
||||
|
||||
# 处理结果
|
||||
for result in all_results:
|
||||
if isinstance(result, dict) and result["errors"] == 0:
|
||||
self.results["vllm"][result["name"]] = result
|
||||
|
||||
# 打印结果
|
||||
print("\n📊 生成测试报告...")
|
||||
self._print_results()
|
||||
|
||||
|
||||
async def main():
|
||||
tester = AsyncVisionPerformanceTester()
|
||||
await tester.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -21,7 +21,7 @@ cozepy==0.12.0
|
||||
mem0ai==0.1.62
|
||||
bs4==0.0.2
|
||||
modelscope==1.23.2
|
||||
sherpa_onnx==1.11.0
|
||||
sherpa_onnx==1.12.0
|
||||
mcp==1.8.1
|
||||
cnlunar==0.2.0
|
||||
PySocks==1.7.1
|
||||
@@ -31,3 +31,4 @@ chardet==5.2.0
|
||||
aioconsole==0.8.1
|
||||
markitdown==0.1.1
|
||||
mcp-proxy==0.6.0
|
||||
PyJWT==2.8.0
|
||||
@@ -1484,6 +1484,23 @@
|
||||
if (message.text && message.text !== '😊') {
|
||||
addMessage(message.text);
|
||||
}
|
||||
}else if (message.type === 'mcp') {
|
||||
const payload = message.payload || {};
|
||||
log(`服务器下发: ${JSON.stringify(message)}`, 'info');
|
||||
if (payload) {
|
||||
// 模拟小智客户端行为
|
||||
if(payload.method === 'tools/list'){
|
||||
const replay_message = JSON.stringify({"session_id":"","type":"mcp","payload":{"jsonrpc":"2.0","id":2,"result":{"tools":[{"name":"self.get_device_status","description":"Provides the real-time information of the device, including the current status of the audio speaker, screen, battery, network, etc.\nUse this tool for: \n1. Answering questions about current condition (e.g. what is the current volume of the audio speaker?)\n2. As the first step to control the device (e.g. turn up / down the volume of the audio speaker, etc.)","inputSchema":{"type":"object","properties":{}}},{"name":"self.audio_speaker.set_volume","description":"Set the volume of the audio speaker. If the current volume is unknown, you must call `self.get_device_status` tool first and then call this tool.","inputSchema":{"type":"object","properties":{"volume":{"type":"integer","minimum":0,"maximum":100}},"required":["volume"]}},{"name":"self.screen.set_brightness","description":"Set the brightness of the screen.","inputSchema":{"type":"object","properties":{"brightness":{"type":"integer","minimum":0,"maximum":100}},"required":["brightness"]}},{"name":"self.screen.set_theme","description":"Set the theme of the screen. The theme can be 'light' or 'dark'.","inputSchema":{"type":"object","properties":{"theme":{"type":"string"}},"required":["theme"]}}]}}})
|
||||
websocket.send(replay_message);
|
||||
log(`回复MCP消息: ${replay_message}`, 'info');
|
||||
} else if(payload.method === 'tools/call'){
|
||||
// 模拟回复
|
||||
const replay_message = JSON.stringify({"session_id":"9f261599","type":"mcp","payload":{"jsonrpc":"2.0","id": payload.id,"result":{"content":[{"type":"text","text":"true"}],"isError":false}}})
|
||||
websocket.send(replay_message);
|
||||
log(`回复MCP消息: ${replay_message}`, 'info');
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
// 未知消息类型
|
||||
log(`未知消息类型: ${message.type}`, 'info');
|
||||
@@ -1523,7 +1540,10 @@
|
||||
device_id: config.deviceId,
|
||||
device_name: config.deviceName,
|
||||
device_mac: config.deviceMac,
|
||||
token: config.token
|
||||
token: config.token,
|
||||
features: {
|
||||
mcp: true
|
||||
}
|
||||
};
|
||||
|
||||
log('发送hello握手消息', 'info');
|
||||
|
||||
Reference in New Issue
Block a user