Merge branch 'main' into py_test_tts

This commit is contained in:
hrz
2025-07-10 13:47:53 +08:00
61 changed files with 1359 additions and 467 deletions
+1
View File
@@ -174,3 +174,4 @@ main/xiaozhi-server/mysql
uploadfile
*.json
.vscode
.cursor
+8 -3
View File
@@ -144,6 +144,11 @@
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1ZQKUzYExM" target="_blank">
<picture>
<img alt="MCP接入点" src="docs/images/demo13.png" />
</picture>
</a>
</td>
<td>
</td>
@@ -170,8 +175,8 @@
#### 🚀 部署方式选择
| 部署方式 | 特点 | 适用场景 | 部署文档 | 配置要求 | 视频教程 |
|---------|------|---------|---------|---------|---------|
| **最简化安装** | 智能对话、IOT、MCP、视觉感知,数据存储在配置文件 | 低配置环境,无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G,如果全API,要2核2G | - |
| **全模块安装** | 智能对话、IOT、MCP、视觉感知、OTA、智控台,数据存储在数据库 | 完整功能体验 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G,如果全API,要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
| **最简化安装** | 智能对话、IOT、MCP、视觉感知 | 低配置环境,数据存储在配置文件,无需数据库 | [①Docker版](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) / [②源码部署](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)| 如果使用`FunASR`要2核4G,如果全API,要2核2G | - |
| **全模块安装** | 智能对话、IOT、MCP接入点、视觉感知、OTA、智控台 | 完整功能体验,数据存储在数据库 |[①Docker版](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [②源码部署](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) / [③源码部署自动更新教程](./docs/dev-ops-integration.md) | 如果使用`FunASR`要4核8G,如果全API,要2核4G| [本地源码启动视频教程](https://www.bilibili.com/video/BV1wBJhz4Ewe) |
> 💡 提示:以下是按最新代码部署后的测试平台,有需要可烧录测试,并发为6个,每天会清空数据
@@ -226,7 +231,7 @@ Websocket接口地址: wss://2662r3426b.vicp.fun/xiaozhi/v1/
| 视觉感知系统 | 支持多种VLLM(视觉大模型),实现多模态交互 |
| 意图识别系统 | 支持LLM意图识别、Function Call函数调用,提供插件化意图处理机制 |
| 记忆系统 | 支持本地短期记忆、mem0ai接口记忆,具备记忆总结功能 |
| IOT/MCP控制协议 | 支持设备注册管理、智能控制接口,同时支持IOT、MCP控制协议 |
| 工具调用 | 支持客户端IOT协议、客户MCP协议、服务端MCP协议、MCP接入点协议、自定义工具函数 |
| 管理后台 | 提供Web管理界面,支持用户管理、系统配置和设备管理 |
| 测试工具 | 提供性能测试工具、视觉模型测试工具和音频交互测试工具 |
| 部署支持 | 支持Docker部署和本地部署,提供完整的配置文件管理 |
+13
View File
@@ -4,6 +4,8 @@
本项目的测试平台`https://2662r3426b.vicp.fun`,从开放以来就使用了该方法,效果良好。
教程可参考B站博主`毕乐labs`发布的视频教程:[《开源小智服务器xiaozhi-server自动更新以及最新版本MCP接入点配置保姆教程》](https://www.bilibili.com/video/BV15H37zHE7Q)
# 开始条件
- 你的电脑/服务器是linux操作系统
- 你已经跑通了整个流程
@@ -40,6 +42,9 @@ git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-serve
此刻你需要把`model.pt`文件复制到新的目录去,你可以这样
```
# 创建需要的目录
mkdir -p /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/
cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml
cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt
```
@@ -161,3 +166,11 @@ tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server
# 注意事项
测试平台`https://2662r3426b.vicp.fun`,是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791)
## 常见问题
### 1、为什么没有见到8001端口?
回答:8001是开发环境使用的,用于运行前端的端口。如果你是服务器部署,不建议使用`npm run serve`启动8001端口运行前端,而是像本教程一样编译成html文件,然后使用nginx来管理访问。
### 2、每次更新需要更新手动SQL语句吗?
回答:不需要,因为项目使用**Liquibase**管理数据库版本,会自动执行新的sql脚本。
+3 -3
View File
@@ -42,9 +42,9 @@ http {
proxy_set_header Referer $http_referer;
proxy_set_header Cookie $http_cookie;
proxy_connect_timeout 10;
proxy_send_timeout 10;
proxy_read_timeout 10;
proxy_connect_timeout 15;
proxy_send_timeout 15;
proxy_read_timeout 15;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+1
View File
@@ -6,6 +6,7 @@ java -jar /app/xiaozhi-esp32-api.jar \
--spring.datasource.druid.username=${SPRING_DATASOURCE_DRUID_USERNAME} \
--spring.datasource.druid.password=${SPRING_DATASOURCE_DRUID_PASSWORD} \
--spring.data.redis.host=${SPRING_DATA_REDIS_HOST} \
--spring.data.redis.password=${SPRING_DATA_REDIS_PASSWORD} \
--spring.data.redis.port=${SPRING_DATA_REDIS_PORT} &
# 启动Nginx(前台运行保持容器存活)
Binary file not shown.

After

Width:  |  Height:  |  Size: 359 KiB

+125
View File
@@ -0,0 +1,125 @@
# MCP 接入点部署使用指南
本教程包含2个部分
- 1、如何部署MCP接入点这个服务
- 1、全模块部署时,怎么配置MCP接入点
- 2、单模块部署时,怎么配置MCP接入点
# 1、如何部署MCP接入点这个服务
## 第一步,下载mcp接入点项目源码
浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server)
打开完,找到页面中一个绿色的按钮,写着`Code`的按钮,点开它,然后你就看到`Download ZIP`的按钮。
点击它,下载本项目源码压缩包。下载到你电脑后,解压它,此时它的名字可能叫`mcp-endpoint-server-main`
你需要把它重命名成`mcp-endpoint-server`
## 第二步,启动程序
这个项目是一个很简单的项目,建议使用docker运行。不过如果你不想使用docker运行,你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法
```
# 进入本项目源码根目录
cd mcp-endpoint-server
# 清除缓存
docker compose -f docker-compose.yml down
docker stop mcp-endpoint-server
docker rm mcp-endpoint-server
docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest
# 启动docker容器
docker compose -f docker-compose.yml up -d
# 查看日志
docker logs -f mcp-endpoint-server
```
此时,日志里会输出类似以下的日志
```
250705 INFO-=====下面的地址分别是智控台/单模块MCP接入点地址====
250705 INFO-智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc
250705 INFO-单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def
250705 INFO-=====请根据具体部署选择使用,请勿泄露给任何人======
```
请你把两个接口地址复制出来:
由于你是docker部署,切不可直接使用上面的地址!
由于你是docker部署,切不可直接使用上面的地址!
由于你是docker部署,切不可直接使用上面的地址!
你先把地址复制出来,放在一个草稿里,你要知道你的电脑的局域网ip是什么,例如我的电脑局域网ip是`192.168.1.25`,那么
原来我的接口地址
```
智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc
单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def
```
就要改成
```
智控台MCP参数配置: http://192.168.1.25:8004/mcp_endpoint/health?key=abc
单模块部署MCP接入点: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def
```
改好后,请使用浏览器直接访问`智控台MCP参数配置`。当浏览器出现类似这样的代码,说明是成功了。
```
{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"}
```
请你保留好上面两个`接口地址`,下一步要用到。
# 2、全模块部署时,怎么配置MCP接入点
如果你是全模块部署,使用管理员账号,登录智控台,点击顶部`参数字典`,选择`参数管理`功能。
然后搜索参数`server.mcp_endpoint`,此时,它的值应该是`null`值。
点击修改按钮,把上一步得来的`智控台MCP参数配置`粘贴到`参数值`里。然后保存。
如果能保存成功,说明一切顺利,你可以去智能体查看效果了。如果不成功,说明智控台无法访问mcp接入点,很大概率是网络防火墙,或者没有填写正确的局域网ip。
# 3、单模块部署时,怎么配置MCP接入点
如果你是单模块部署,找到你的配置文件`data/.config.yaml`
在配置文件搜索`mcp_endpoint`,如果没有找到,你就增加`mcp_endpoint`配置。类似我是就是这样
```
server:
websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
http_port: 8002
log:
log_level: INFO
# 此处可能还更多配置..
mcp_endpoint: 你的接入点websocket地址
```
这时,请你把`如何部署MCP接入点这个服务`中得到的`单模块部署MCP接入点` 粘贴到 `mcp_endpoint`中。类似这样
```
server:
websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/
http_port: 8002
log:
log_level: INFO
# 此处可能还更多配置
mcp_endpoint: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def
```
配置好后,启动单模块会输出如下的日志。
```
250705[__main__]-INFO-初始化组件: vad成功 SileroVAD
250705[__main__]-INFO-初始化组件: asr成功 FunASRServer
250705[__main__]-INFO-OTA接口是 http://192.168.1.25:8002/xiaozhi/ota/
250705[__main__]-INFO-视觉分析接口是 http://192.168.1.25:8002/mcp/vision/explain
250705[__main__]-INFO-mcp接入点是 ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
250705[__main__]-INFO-Websocket地址是 ws://192.168.1.25:8000/xiaozhi/v1/
250705[__main__]-INFO-=======上面的地址是websocket协议地址,请勿用浏览器访问=======
250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
250705[__main__]-INFO-=============================================================
```
如上,如果能输出类似的`mcp接入点是``ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`说明配置成功了。
+42 -82
View File
@@ -1,92 +1,36 @@
# MCP 接入点部署使用指南
# MCP 接入点使用指南
本教程包含2个部分
- 1、如何开启mcp接入点
- 2、如何为智能体接入一个简单的mcp功能,如计算器功能
本教程以虾哥开源的mcp计算器功能为示例,介绍如何将自己自定义的mcp服务接入到自己的接入点里。
部署的前提条件:
- 1、你已经部署了全模块,因为mcp接入点需要全模块中的智控台功能
- 2、你想在不修改xiaozhi-server项目的前提下,扩展小智的功能
# 如何开启mcp接入点
## 第一步,下载mcp接入点项目源码
浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server)
打开完,找到页面中一个绿色的按钮,写着`Code`的按钮,点开它,然后你就看到`Download ZIP`的按钮。
点击它,下载本项目源码压缩包。下载到你电脑后,解压它,此时它的名字可能叫`mcp-endpoint-server-main`
你需要把它重命名成`mcp-endpoint-server`
## 第二步,启动程序
这个项目是一个很简单的项目,建议使用docker运行。不过如果你不想使用docker运行,你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法
```
# 进入本项目源码根目录
cd mcp-endpoint-server
# 清除缓存
docker compose -f docker-compose.yml down
docker stop mcp-endpoint-server
docker rm mcp-endpoint-server
docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest
# 启动docker容器
docker compose -f docker-compose.yml up -d
# 查看日志
docker logs -f mcp-endpoint-server
```
此时,日志里会输出类似以下的日志
```
======================================================
接口地址: http://172.1.1.1:8004/mcp_endpoint/health?key=xxxx
=======上面的地址是MCP接入点地址,请勿泄露给任何人============
```
请你把接口地址复制出来:
由于你是docker部署,切不可直接使用上面的地址!
由于你是docker部署,切不可直接使用上面的地址!
由于你是docker部署,切不可直接使用上面的地址!
你先把地址复制出来,放在一个草稿里,你要知道你的电脑的局域网ip是什么,例如我的电脑局域网ip是`192.168.1.25`,那么
原来我的接口地址
```
http://172.1.1.1:8004/mcp_endpoint/health?key=xxxx
```
就要改成
```
http://192.168.1.25:8004/mcp_endpoint/health?key=xxxx
```
改好后,请使用浏览器直接访问这个接口。当浏览器出现类似这样的代码,说明是成功了。
```
{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"}
```
请你保留好这个`接口地址`,下一步要用到。
## 第三步,配置智控台
使用管理员账号,登录智控台,点击顶部`参数字典`,选择`参数管理`功能。
然后搜索参数`server.mcp_endpoint`,此时,它的值应该是`null`值。
点击修改按钮,把上一步得来的`接口地址`粘贴到`参数值`里。然后保存。
如果能保存成功,说明一切顺利,你可以去智能体查看效果了。如果不成功,说明智控台无法访问mcp接入点,很大概率是网络防火墙,或者没有填写正确的局域网ip。
本教程的前提是,你的`xiaozhi-server`已经启用了mcp接入点功能,如果你还没启用,可以先根据[这个教程](./mcp-endpoint-enable.md)启用。
# 如何为智能体接入一个简单的mcp功能,如计算器功能
如果以上步骤顺利,你可以进入智能体管理,点击`配置角色`,在`意图识别`的右边,有一个`编辑功能`的按钮。
### 如果你是全模块部署
如果你是全模块部署,你可以进入智控台,智能体管理,点击`配置角色`,在`意图识别`的右边,有一个`编辑功能`的按钮。
点击这个按钮。在弹出的页面里,位于底部,会有`MCP接入点`,正常来说,会显示这个智能体的`MCP接入点地址`,接下来,我们来给这个智能体扩展一个基于MCP技术的计算器的功能。
这个`MCP接入点地址`很重要,你等一下会用到。
### 如果你是单模块部署
如果你是单模块部署,且你已经在配置文件里配置了MCP接入点地址,那么正常来说,单模块部署启动的时候,会输出如下的日志。
```
250705[__main__]-INFO-初始化组件: vad成功 SileroVAD
250705[__main__]-INFO-初始化组件: asr成功 FunASRServer
250705[__main__]-INFO-OTA接口是 http://192.168.1.25:8002/xiaozhi/ota/
250705[__main__]-INFO-视觉分析接口是 http://192.168.1.25:8002/mcp/vision/explain
250705[__main__]-INFO-mcp接入点是 ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
250705[__main__]-INFO-Websocket地址是 ws://192.168.1.25:8000/xiaozhi/v1/
250705[__main__]-INFO-=======上面的地址是websocket协议地址,请勿用浏览器访问=======
250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html
250705[__main__]-INFO-=============================================================
```
如上,输出`mcp接入点是``ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`就是你的`MCP接入点地址`
这个`MCP接入点地址`很重要,你等一下会用到。
## 第一步 下载虾哥MCP计算器项目代码
浏览器打开虾哥写的[计算器项目](https://github.com/78/mcp-calculator)
@@ -114,13 +58,13 @@ pip install -r requirements.txt
例如我的智能体的mcp地址是
```
ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc
ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
```
开始输入命令
```bash
export MCP_ENDPOINT=ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc
export MCP_ENDPOINT=ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc
```
输入完后,启动程序
@@ -129,6 +73,22 @@ export MCP_ENDPOINT=ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc
python mcp_pipe.py calculator.py
```
### 如果你是智控台部署
如果你是智控台部署,启动完后,你再进入智控台,点击刷新MCP的接入状态,就会看到你扩展的功能列表了。
启动完后,你再进入智控台,点击刷新MCP的接入状态,就会看到你扩展的功能列表了。
### 如果你是单模块部署
如果你是单模块部署,当设备连接后,会输出类似的日志,说明成功了
```
250705 -INFO-正在初始化MCP接入点: wss://2662r3426b.vicp.fun/mcp_e
250705 -INFO-发送MCP接入点初始化消息
250705 -INFO-MCP接入点连接成功
250705 -INFO-MCP接入点初始化成功
250705 -INFO-统一工具处理器初始化完成
250705 -INFO-MCP接入点服务器信息: name=Calculator, version=1.9.4
250705 -INFO-MCP接入点支持的工具数量: 1
250705 -INFO-所有MCP接入点工具已获取,客户端准备就绪
250705 -INFO-工具缓存已刷新
250705 -INFO-当前支持的函数列表: [ 'get_time', 'get_lunar', 'play_music', 'get_weather', 'handle_exit_intent', 'calculator']
```
如果包含了 `'calculator'`,说明设备将可以根据意图识别,调用计算器这个工具。
+105
View File
@@ -0,0 +1,105 @@
# get_news_from_newsnow 插件新闻源配置指南
## 概述
`get_news_from_newsnow` 插件现在支持通过Web管理界面动态配置新闻源,不再需要修改代码。用户可以在智控台中为每个智能体配置不同的新闻源。
## 配置方式
### 1. 通过Web管理界面配置(推荐)
1. 登录智控台
2. 进入"角色配置"页面
3. 选择要配置的智能体
4. 点击"编辑功能"按钮
5. 在右侧参数配置区域找到"newsnow新闻聚合"插件
6. 在"新闻源配置"字段中输入分号分隔的中文名称
### 2. 配置文件方式
`config.yaml` 中配置:
```yaml
plugins:
get_news_from_newsnow:
url: "https://newsnow.busiyi.world/api/s?id="
news_sources: "澎湃新闻;百度热搜;财联社;微博;抖音"
```
## 新闻源配置格式
新闻源配置使用分号分隔的中文名称,格式为:
```
中文名称1;中文名称2;中文名称3
```
### 配置示例
```
澎湃新闻;百度热搜;财联社;微博;抖音;知乎;36氪
```
## 支持的新闻源
插件支持以下新闻源的中文名称:
- 澎湃新闻
- 百度热搜
- 财联社
- 微博
- 抖音
- 知乎
- 36氪
- 华尔街见闻
- IT之家
- 今日头条
- 虎扑
- 哔哩哔哩
- 快手
- 雪球
- 格隆汇
- 法布财经
- 金十数据
- 牛客
- 少数派
- 稀土掘金
- 凤凰网
- 虫部落
- 联合早报
- 酷安
- 远景论坛
- 参考消息
- 卫星通讯社
- 百度贴吧
- 靠谱新闻
- 以及更多...
## 默认配置
如果未配置新闻源,插件将使用以下默认配置:
```
澎湃新闻;百度热搜;财联社
```
## 使用说明
1. **配置新闻源**:在Web界面或配置文件中设置新闻源的中文名称,用分号分隔
2. **调用插件**:用户可以说"播报新闻"或"获取新闻"
3. **指定新闻源**:用户可以说"播报澎湃新闻"或"获取百度热搜"
4. **获取详情**:用户可以说"详细介绍这条新闻"
## 工作原理
1. 插件接受中文名称作为参数(如"澎湃新闻")
2. 根据配置的新闻源列表,将中文名称转换为对应的英文ID(如"thepaper"
3. 使用英文ID调用API获取新闻数据
4. 返回新闻内容给用户
## 注意事项
1. 配置的中文名称必须与 CHANNEL_MAP 中定义的名称完全一致
2. 配置更改后需要重启服务或重新加载配置
3. 如果配置的新闻源无效,插件会自动使用默认新闻源
4. 多个新闻源之间使用英文分号(;)分隔,不要使用中文分号(;)
@@ -232,7 +232,7 @@ public interface Constant {
/**
* 版本号
*/
public static final String VERSION = "0.6.1";
public static final String VERSION = "0.6.3";
/**
* 无效固件URL
@@ -19,6 +19,8 @@ import xiaozhi.modules.agent.service.AgentChatAudioService;
import xiaozhi.modules.agent.service.AgentChatHistoryService;
import xiaozhi.modules.agent.service.AgentService;
import xiaozhi.modules.agent.service.biz.AgentChatHistoryBizService;
import xiaozhi.modules.device.entity.DeviceEntity;
import xiaozhi.modules.device.service.DeviceService;
/**
* {@link AgentChatHistoryBizService} impl
@@ -35,6 +37,7 @@ public class AgentChatHistoryBizServiceImpl implements AgentChatHistoryBizServic
private final AgentChatHistoryService agentChatHistoryService;
private final AgentChatAudioService agentChatAudioService;
private final RedisUtils redisUtils;
private final DeviceService deviceService;
/**
* 处理聊天记录上报,包括文件上传和相关信息记录
@@ -68,6 +71,15 @@ public class AgentChatHistoryBizServiceImpl implements AgentChatHistoryBizServic
// 更新设备最后对话时间
redisUtils.set(RedisKeys.getAgentDeviceLastConnectedAtById(agentId), new Date());
// 更新设备最后连接时间
DeviceEntity device = deviceService.getDeviceByMacAddress(macAddress);
if (device != null) {
deviceService.updateDeviceConnectionInfo(agentId, device.getId(), null);
} else {
log.warn("聊天记录上报时,未找到mac地址为 {} 的设备", macAddress);
}
return Boolean.TRUE;
}
@@ -5,6 +5,7 @@ import java.net.URISyntaxException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
@@ -18,7 +19,6 @@ import xiaozhi.common.utils.AESUtils;
import xiaozhi.common.utils.HashEncryptionUtil;
import xiaozhi.common.utils.JsonUtils;
import xiaozhi.modules.agent.dto.McpJsonRpcRequest;
import xiaozhi.modules.agent.dto.McpJsonRpcResponse;
import xiaozhi.modules.agent.service.AgentMcpAccessPointService;
import xiaozhi.modules.sys.service.SysParamsService;
import xiaozhi.modules.sys.utils.WebSocketClientManager;
@@ -61,53 +61,119 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic
wsUrl = wsUrl.replace("/mcp/", "/call/");
try {
// 创建 WebSocket 连接
// 创建 WebSocket 连接,增加超时时间到15秒
try (WebSocketClientManager client = WebSocketClientManager.build(
new WebSocketClientManager.Builder()
.uri(wsUrl)
.connectTimeout(5, TimeUnit.SECONDS)
.maxSessionDuration(20, TimeUnit.SECONDS))) {
.connectTimeout(8, TimeUnit.SECONDS)
.maxSessionDuration(10, TimeUnit.SECONDS))) {
// 发送初始化通知
McpJsonRpcRequest initRequest = new McpJsonRpcRequest("notifications/initialized");
client.sendJson(initRequest);
// 步骤1: 发送初始化消息并等待响应
log.info("发送MCP初始化消息,智能体ID: {}", id);
McpJsonRpcRequest initializeRequest = new McpJsonRpcRequest("initialize",
Map.of(
"protocolVersion", "2024-11-05",
"capabilities", Map.of(
"roots", Map.of("listChanged", false),
"sampling", Map.of()),
"clientInfo", Map.of(
"name", "xz-mcp-broker",
"version", "0.0.1")),
1);
client.sendJson(initializeRequest);
// 等待 0.2 秒
Thread.sleep(200);
// 发送工具列表请求
McpJsonRpcRequest toolsRequest = new McpJsonRpcRequest("tools/list", null, 1);
client.sendJson(toolsRequest);
// 监听响应,直到收到包含 id=1 的响应
List<String> responses = client.listener(response -> {
// 等待初始化响应 (id=1) - 移除固定延迟,改为响应驱动
List<String> initResponses = client.listenerWithoutClose(response -> {
try {
McpJsonRpcResponse jsonResponse = JsonUtils.parseObject(response, McpJsonRpcResponse.class);
return jsonResponse != null && Integer.valueOf(1).equals(jsonResponse.getId());
Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) {
// 检查是否有result字段,表示初始化成功
return jsonMap.containsKey("result") && !jsonMap.containsKey("error");
}
return false;
} catch (Exception e) {
log.warn("解析响应失败: {}", response, e);
log.warn("解析初始化响应失败: {}", response, e);
return false;
}
});
// 处理响应
for (String response : responses) {
// 验证初始化响应
boolean initSucceeded = false;
for (String response : initResponses) {
try {
McpJsonRpcResponse jsonResponse = JsonUtils.parseObject(response, McpJsonRpcResponse.class);
if (jsonResponse != null && Integer.valueOf(1).equals(jsonResponse.getId())
&& jsonResponse.getResult() != null && jsonResponse.getResult().getTools() != null) {
Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) {
if (jsonMap.containsKey("result")) {
log.info("MCP初始化成功,智能体ID: {}", id);
initSucceeded = true;
break;
} else if (jsonMap.containsKey("error")) {
log.error("MCP初始化失败,智能体ID: {}, 错误: {}", id, jsonMap.get("error"));
return List.of();
}
}
} catch (Exception e) {
log.warn("处理初始化响应失败: {}", response, e);
}
}
// 提取工具名称列表
return java.util.Arrays.stream(jsonResponse.getResult().getTools())
.map(McpJsonRpcResponse.McpTool::getName)
.collect(Collectors.toList());
if (!initSucceeded) {
log.error("未收到有效的MCP初始化响应,智能体ID: {}", id);
return List.of();
}
// 步骤2: 发送初始化完成通知 - 只有在收到initialize响应后才发送
log.info("发送MCP初始化完成通知,智能体ID: {}", id);
String notificationJson = "{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\"}";
client.sendText(notificationJson);
// 步骤3: 发送工具列表请求 - 立即发送,无需额外延迟
log.info("发送MCP工具列表请求,智能体ID: {}", id);
McpJsonRpcRequest toolsRequest = new McpJsonRpcRequest("tools/list", null, 2);
client.sendJson(toolsRequest);
// 等待工具列表响应 (id=2)
List<String> toolsResponses = client.listener(response -> {
try {
Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
return jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"));
} catch (Exception e) {
log.warn("解析工具列表响应失败: {}", response, e);
return false;
}
});
// 处理工具列表响应
for (String response : toolsResponses) {
try {
Map<String, Object> jsonMap = JsonUtils.parseObject(response, Map.class);
if (jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"))) {
// 检查是否有result字段
Object resultObj = jsonMap.get("result");
if (resultObj instanceof Map) {
Map<String, Object> resultMap = (Map<String, Object>) resultObj;
Object toolsObj = resultMap.get("tools");
if (toolsObj instanceof List) {
List<Map<String, Object>> toolsList = (List<Map<String, Object>>) toolsObj;
// 提取工具名称列表
List<String> result = toolsList.stream()
.map(tool -> (String) tool.get("name"))
.filter(name -> name != null)
.collect(Collectors.toList());
log.info("成功获取MCP工具列表,智能体ID: {}, 工具数量: {}", id, result.size());
return result;
}
} else if (jsonMap.containsKey("error")) {
log.error("获取工具列表失败,智能体ID: {}, 错误: {}", id, jsonMap.get("error"));
return List.of();
}
}
} catch (Exception e) {
log.warn("处理工具列表响应失败: {}", response, e);
}
}
log.warn("未找到有效的工具列表响应");
log.warn("未找到有效的工具列表响应,智能体ID: {}", id);
return List.of();
}
@@ -177,4 +243,4 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic
// 加密后成token值
return AESUtils.encrypt(key, json);
}
}
}
@@ -72,6 +72,8 @@ public class ConfigServiceImpl implements ConfigService {
null,
null,
null,
null,
null,
agent.getVadModelId(),
agent.getAsrModelId(),
null,
@@ -108,9 +110,13 @@ public class ConfigServiceImpl implements ConfigService {
}
// 获取音色信息
String voice = null;
String referenceAudio = null;
String referenceText = null;
TimbreDetailsVO timbre = timbreService.get(agent.getTtsVoiceId());
if (timbre != null) {
voice = timbre.getTtsVoice();
referenceAudio = timbre.getReferenceAudio();
referenceText = timbre.getReferenceText();
}
// 构建返回数据
Map<String, Object> result = new HashMap<>();
@@ -163,6 +169,8 @@ public class ConfigServiceImpl implements ConfigService {
agent.getSystemPrompt(),
agent.getSummaryMemory(),
voice,
referenceAudio,
referenceText,
agent.getVadModelId(),
agent.getAsrModelId(),
agent.getLlmModelId(),
@@ -179,7 +187,7 @@ public class ConfigServiceImpl implements ConfigService {
/**
* 构建配置信息
*
* @param paramsList 系统参数列表
* @param config 系统参数列表
* @return 配置信息
*/
private Object buildConfig(Map<String, Object> config) {
@@ -250,21 +258,25 @@ public class ConfigServiceImpl implements ConfigService {
/**
* 构建模块配置
*
* @param prompt 提示词
* @param voice 音色
* @param vadModelId VAD模型ID
* @param asrModelId ASR模型ID
* @param llmModelId LLM模型ID
* @param ttsModelId TTS模型ID
* @param memModelId 记忆模型ID
* @param intentModelId 意图模型ID
* @param result 结果Map
* @param prompt 提示词
* @param voice 音色
* @param referenceAudio 参考音频路径
* @param referenceText 参考文本
* @param vadModelId VAD模型ID
* @param asrModelId ASR模型ID
* @param llmModelId LLM模型ID
* @param ttsModelId TTS模型ID
* @param memModelId 记忆模型ID
* @param intentModelId 意图模型ID
* @param result 结果Map
*/
private void buildModuleConfig(
String assistantName,
String prompt,
String summaryMemory,
String voice,
String referenceAudio,
String referenceText,
String vadModelId,
String asrModelId,
String llmModelId,
@@ -286,12 +298,20 @@ public class ConfigServiceImpl implements ConfigService {
continue;
}
ModelConfigEntity model = modelConfigService.getModelById(modelIds[i], isCache);
if (model == null) {
continue;
}
Map<String, Object> typeConfig = new HashMap<>();
if (model.getConfigJson() != null) {
typeConfig.put(model.getId(), model.getConfigJson());
// 如果是TTS类型,添加private_voice属性
if ("TTS".equals(modelTypes[i]) && voice != null) {
((Map<String, Object>) model.getConfigJson()).put("private_voice", voice);
if ("TTS".equals(modelTypes[i])) {
if (voice != null)
((Map<String, Object>) model.getConfigJson()).put("private_voice", voice);
if (referenceAudio != null)
((Map<String, Object>) model.getConfigJson()).put("ref_audio", referenceAudio);
if (referenceText != null)
((Map<String, Object>) model.getConfigJson()).put("ref_text", referenceText);
}
// 如果是Intent类型,且type=intent_llm,则给他添加附加模型
if ("Intent".equals(modelTypes[i])) {
@@ -25,6 +25,7 @@ import xiaozhi.common.utils.Result;
import xiaozhi.modules.device.dto.DeviceRegisterDTO;
import xiaozhi.modules.device.dto.DeviceUnBindDTO;
import xiaozhi.modules.device.dto.DeviceUpdateDTO;
import xiaozhi.modules.device.dto.DeviceManualAddDTO;
import xiaozhi.modules.device.entity.DeviceEntity;
import xiaozhi.modules.device.service.DeviceService;
import xiaozhi.modules.security.user.SecurityUser;
@@ -99,4 +100,13 @@ public class DeviceController {
deviceService.updateById(entity);
return new Result<Void>();
}
@PostMapping("/manual-add")
@Operation(summary = "手动添加设备")
@RequiresPermissions("sys:role:normal")
public Result<Void> manualAddDevice(@RequestBody @Valid DeviceManualAddDTO dto) {
UserDetail user = SecurityUser.getUser();
deviceService.manualAddDevice(user.getId(), dto);
return new Result<>();
}
}
@@ -51,13 +51,12 @@ public class OTAController {
if (StringUtils.isBlank(clientId)) {
clientId = deviceId;
}
String macAddress = deviceReportReqDTO.getMacAddress();
boolean macAddressValid = isMacAddressValid(macAddress);
boolean macAddressValid = isMacAddressValid(deviceId);
// 设备Id和Mac地址应是一致的, 并且必须需要application字段
if (!deviceId.equals(macAddress) || !macAddressValid || deviceReportReqDTO.getApplication() == null) {
return createResponse(DeviceReportRespDTO.createError("Invalid OTA request"));
if (!macAddressValid) {
return createResponse(DeviceReportRespDTO.createError("Invalid device ID"));
}
return createResponse(deviceService.checkDeviceActive(macAddress, clientId, deviceReportReqDTO));
return createResponse(deviceService.checkDeviceActive(deviceId, clientId, deviceReportReqDTO));
}
@Operation(summary = "设备快速检查激活状态")
@@ -0,0 +1,11 @@
package xiaozhi.modules.device.dto;
import lombok.Data;
@Data
public class DeviceManualAddDTO {
private String agentId;
private String board; // 设备型号
private String appVersion; // 固件版本
private String macAddress; // Mac地址
}
@@ -8,6 +8,7 @@ import xiaozhi.common.service.BaseService;
import xiaozhi.modules.device.dto.DevicePageUserDTO;
import xiaozhi.modules.device.dto.DeviceReportReqDTO;
import xiaozhi.modules.device.dto.DeviceReportRespDTO;
import xiaozhi.modules.device.dto.DeviceManualAddDTO;
import xiaozhi.modules.device.entity.DeviceEntity;
import xiaozhi.modules.device.vo.UserShowDeviceListVO;
@@ -87,5 +88,14 @@ public interface DeviceService extends BaseService<DeviceEntity> {
*/
Date getLatestLastConnectionTime(String agentId);
/**
* 手动添加设备
*/
void manualAddDevice(Long userId, DeviceManualAddDTO dto);
/**
* 更新设备连接信息
*/
void updateDeviceConnectionInfo(String agentId, String deviceId, String appVersion);
}
@@ -44,6 +44,7 @@ import xiaozhi.modules.device.vo.UserShowDeviceListVO;
import xiaozhi.modules.security.user.SecurityUser;
import xiaozhi.modules.sys.service.SysParamsService;
import xiaozhi.modules.sys.service.SysUserUtilService;
import xiaozhi.modules.device.dto.DeviceManualAddDTO;
@Slf4j
@Service
@@ -410,4 +411,30 @@ public class DeviceServiceImpl extends BaseServiceImpl<DeviceDao, DeviceEntity>
}
return 0;
}
@Override
public void manualAddDevice(Long userId, DeviceManualAddDTO dto) {
// 检查mac是否已存在
QueryWrapper<DeviceEntity> wrapper = new QueryWrapper<>();
wrapper.eq("mac_address", dto.getMacAddress());
DeviceEntity exist = baseDao.selectOne(wrapper);
if (exist != null) {
throw new RenException("该Mac地址已存在");
}
Date now = new Date();
DeviceEntity entity = new DeviceEntity();
entity.setId(dto.getMacAddress());
entity.setUserId(userId);
entity.setAgentId(dto.getAgentId());
entity.setBoard(dto.getBoard());
entity.setAppVersion(dto.getAppVersion());
entity.setMacAddress(dto.getMacAddress());
entity.setCreateDate(now);
entity.setUpdateDate(now);
entity.setLastConnectedAt(now);
entity.setCreator(userId);
entity.setUpdater(userId);
entity.setAutoUpdate(1);
baseDao.insert(entity);
}
}
@@ -218,7 +218,7 @@ public class SysParamsController {
if (response.getStatusCode() != HttpStatus.OK) {
throw new RenException("MCP接口访问失败,状态码:" + response.getStatusCode());
}
// 检查响应内容是否包含OTA相关信息
// 检查响应内容是否包含mcp相关信息
String body = response.getBody();
if (body == null || !body.contains("success")) {
throw new RenException("MCP接口返回内容格式不正确,可能不是一个真实的MCP接口");
@@ -137,6 +137,37 @@ public class WebSocketClientManager implements Closeable {
return collected;
}
private <T> List<T> listenerCustomWithoutClose(
BlockingQueue<T> queue,
Predicate<T> predicate)
throws InterruptedException, TimeoutException, ExecutionException {
List<T> collected = new ArrayList<>();
long deadline = System.currentTimeMillis() + maxSessionDurationUnit.toMillis(maxSessionDuration);
while (true) {
if (errorFuture.isDone()) {
errorFuture.get();
}
long remaining = deadline - System.currentTimeMillis();
if (remaining <= 0) {
throw new TimeoutException("等待批量消息超时");
}
T msg = queue.poll(remaining, TimeUnit.MILLISECONDS);
if (msg == null) {
throw new TimeoutException("等待批量消息超时");
}
collected.add(msg);
if (predicate.test(msg)) {
break;
}
}
// 不调用 close(),保持连接开放
return collected;
}
/**
* 同步接收多条消息,直到 predicate 为 true 或超时抛异常;
*
@@ -147,6 +178,17 @@ public class WebSocketClientManager implements Closeable {
return listenerCustom(textMessageQueue, predicate);
}
/**
* 同步接收多条消息,直到 predicate 为 true 或超时抛异常;
* 不自动关闭连接,适用于需要在同一连接上发送多个消息的场景
*
* @return 返回监听期间的所有消息列表
*/
public List<String> listenerWithoutClose(Predicate<String> predicate)
throws InterruptedException, TimeoutException, ExecutionException {
return listenerCustomWithoutClose(textMessageQueue, predicate);
}
public List<byte[]> listenerBinary(Predicate<byte[]> predicate)
throws InterruptedException, TimeoutException, ExecutionException {
return listenerCustom(binaryMessageQueue, predicate);
@@ -26,6 +26,12 @@ public class TimbreDataDTO {
@Schema(description = "备注")
private String remark;
@Schema(description = "参考音频路径")
private String referenceAudio;
@Schema(description = "參考文本")
private String referenceText;
@Schema(description = "排序")
@Min(value = 0, message = "{sort.number}")
private long sort;
@@ -34,6 +34,12 @@ public class TimbreEntity {
@Schema(description = "备注")
private String remark;
@Schema(description = "参考音频路径")
private String referenceAudio;
@Schema(description = "參考文本")
private String referenceText;
@Schema(description = "排序")
private long sort;
@@ -25,6 +25,12 @@ public class TimbreDetailsVO implements Serializable {
@Schema(description = "备注")
private String remark;
@Schema(description = "参考音频路径")
private String referenceAudio;
@Schema(description = "參考文本")
private String referenceText;
@Schema(description = "排序")
private long sort;
@@ -0,0 +1,3 @@
ALTER TABLE `ai_tts_voice`
ADD COLUMN `reference_audio` VARCHAR(500) DEFAULT NULL COMMENT '参考音频路径' AFTER `remark`,
ADD COLUMN `reference_text` VARCHAR(500) DEFAULT NULL COMMENT '参考文本' AFTER `reference_audio`;
@@ -0,0 +1,18 @@
-- 更新现有的 get_news_from_newsnow 插件配置
UPDATE ai_model_provider
SET fields = JSON_ARRAY(
JSON_OBJECT(
'key', 'url',
'type', 'string',
'label', '接口地址',
'default', 'https://newsnow.busiyi.world/api/s?id='
),
JSON_OBJECT(
'key', 'news_sources',
'type', 'string',
'label', '新闻源配置',
'default', '澎湃新闻;百度热搜;财联社'
)
)
WHERE provider_code = 'get_news_from_newsnow'
AND model_type = 'Plugin';
@@ -205,6 +205,13 @@ databaseChangeLog:
- sqlFile:
encoding: utf8
path: classpath:db/changelog/202506080955.sql
- changeSet:
id: 202506091720
author: shane0411
changes:
- sqlFile:
encoding: utf8
path: classpath:db/changelog/202506091720.sql
- changeSet:
id: 202506161101
author: hrz
@@ -219,6 +226,13 @@ databaseChangeLog:
- sqlFile:
encoding: utf8
path: classpath:db/changelog/202506191643.sql
- changeSet:
id: 202506251620
author: Tink
changes:
- sqlFile:
encoding: utf8
path: classpath:db/changelog/202506251620.sql
- changeSet:
id: 202506261637
author: hrz
@@ -1 +1 @@
redis.call('FLUSHALL')
redis.call('FLUSHDB')
+1 -1
View File
@@ -7,6 +7,7 @@ import model from './module/model.js'
import ota from './module/ota.js'
import timbre from "./module/timbre.js"
import user from './module/user.js'
/**
* 接口地址
* 开发时自动读取使用.env.development文件
@@ -22,7 +23,6 @@ export function getServiceUrl() {
return DEV_API_SERVICE
}
/** request服务封装 */
export default {
getServiceUrl,
@@ -68,4 +68,21 @@ export default {
})
}).send()
},
// 手动添加设备
manualAddDevice(params, callback) {
RequestService.sendRequest()
.url(`${getServiceUrl()}/device/manual-add`)
.method('POST')
.data(params)
.success((res) => {
RequestService.clearRequestTime();
callback(res);
})
.networkFail((err) => {
console.error('手动添加设备失败:', err);
RequestService.reAjaxFun(() => {
this.manualAddDevice(params, callback);
});
}).send();
},
}
@@ -34,6 +34,8 @@ export default {
languages: params.languageType,
name: params.voiceName,
remark: params.remark,
referenceAudio: params.referenceAudio,
referenceText: params.referenceText,
sort: params.sort,
ttsModelId: params.ttsModelId,
ttsVoice: params.voiceCode,
@@ -75,6 +77,8 @@ export default {
languages: params.languageType,
name: params.voiceName,
remark: params.remark,
referenceAudio: params.referenceAudio,
referenceText: params.referenceText,
ttsModelId: params.ttsModelId,
ttsVoice: params.voiceCode,
voiceDemo: params.voiceDemo || ''
@@ -110,8 +110,10 @@
<div class="url-header">
<div class="address-desc">
<span>以下是智能体的MCP接入点地址</span>
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-enable.md"
target="_blank" class="doc-link">如何部署MCP接入点</a> &nbsp;&nbsp;|&nbsp;&nbsp;
<a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md"
target="_blank" class="doc-link">查看接入点使用文档</a>
target="_blank" class="doc-link">如何接入MCP功能</a> &nbsp;
</div>
</div>
<el-input v-model="mcpUrl" readonly class="url-input">
@@ -660,6 +662,7 @@ export default {
.mcp-left,
.mcp-right {
flex: 1;
padding-bottom: 50px;
}
.url-header {
@@ -0,0 +1,158 @@
<template>
<el-dialog title="手动添加设备" :visible="visible" @close="handleClose" width="30%" center>
<div class="dialog-content">
<el-form :model="deviceForm" :rules="rules" ref="deviceForm" label-width="100px">
<el-form-item label="设备型号" prop="board">
<el-select v-model="deviceForm.board" placeholder="请选择设备型号" style="width: 100%">
<el-option
v-for="item in firmwareTypes"
:key="item.key"
:label="item.name"
:value="item.key">
</el-option>
</el-select>
</el-form-item>
<el-form-item label="固件版本" prop="appVersion">
<el-input v-model="deviceForm.appVersion" placeholder="请输入固件版本"></el-input>
</el-form-item>
<el-form-item label="Mac地址" prop="macAddress">
<el-input v-model="deviceForm.macAddress" placeholder="请输入Mac地址"></el-input>
</el-form-item>
</el-form>
</div>
<div style="display: flex;margin: 15px 15px;gap: 7px;">
<div class="dialog-btn" @click="submitForm">确定</div>
<div class="dialog-btn cancel-btn" @click="cancel">取消</div>
</div>
</el-dialog>
</template>
<script>
import Api from '@/apis/api';
export default {
name: 'ManualAddDeviceDialog',
props: {
visible: { type: Boolean, required: true },
agentId: { type: String, required: true }
},
data() {
// MAC地址验证规则
const validateMac = (rule, value, callback) => {
const macRegex = /^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$/;
if (!value) {
callback(new Error('请输入Mac地址'));
} else if (!macRegex.test(value)) {
callback(new Error('请输入正确的Mac地址格式,例如:00:1A:2B:3C:4D:5E'));
} else {
callback();
}
};
return {
deviceForm: {
board: '',
appVersion: '',
macAddress: ''
},
firmwareTypes: [],
rules: {
board: [
{ required: true, message: '请选择设备型号', trigger: 'change' }
],
appVersion: [
{ required: true, message: '请输入固件版本', trigger: 'blur' }
],
macAddress: [
{ required: true, validator: validateMac, trigger: 'blur' }
]
}
}
},
created() {
this.getFirmwareTypes();
},
methods: {
async getFirmwareTypes() {
try {
const res = await Api.dict.getDictDataByType('FIRMWARE_TYPE');
this.firmwareTypes = res.data;
} catch (error) {
console.error('获取固件类型失败:', error);
this.$message.error(error.message || '获取固件类型失败');
}
},
submitForm() {
this.$refs.deviceForm.validate((valid) => {
if (valid) {
this.addDevice();
}
});
},
addDevice() {
const params = {
agentId: this.agentId,
...this.deviceForm
};
Api.device.manualAddDevice(params, ({ data }) => {
if (data.code === 0) {
this.$message.success('设备添加成功');
this.$emit('refresh');
this.closeDialog();
} else {
this.$message.error(data.msg || '添加失败');
}
});
},
closeDialog() {
this.$emit('update:visible', false);
this.$refs.deviceForm.resetFields();
},
cancel() {
this.closeDialog();
},
handleClose() {
this.closeDialog();
}
}
}
</script>
<style scoped>
.dialog-content {
padding: 0 20px;
}
.dialog-btn {
cursor: pointer;
flex: 1;
border-radius: 23px;
background: #5778ff;
height: 40px;
font-weight: 500;
font-size: 12px;
color: #fff;
line-height: 40px;
text-align: center;
}
.cancel-btn {
background: #e6ebff;
border: 1px solid #adbdff;
color: #5778ff;
}
::v-deep .el-dialog {
border-radius: 15px;
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
}
::v-deep .el-dialog__body {
padding: 20px 6px;
}
::v-deep .el-form-item {
margin-bottom: 20px;
}
</style>
+118 -106
View File
@@ -1,79 +1,66 @@
<template>
<el-dialog
:visible.sync="localVisible"
width="85%"
@close="handleClose"
:show-close="false"
:append-to-body="true"
:close-on-click-modal="true">
<el-dialog :visible.sync="localVisible" width="90%" @close="handleClose" :show-close="false" :append-to-body="true"
:close-on-click-modal="true">
<button class="custom-close-btn" @click="handleClose">
×
</button>
<div class="scroll-wrapper">
<div
class="table-container"
ref="tableContainer"
@scroll="handleScroll">
<el-table
v-loading="loading"
:data="filteredTtsModels"
style="width: 100%;"
class="data-table"
header-row-class-name="table-header"
:fit="true"
element-loading-text="拼命加载中"
element-loading-spinner="el-icon-loading"
element-loading-background="rgba(0, 0, 0, 0.8)">
<div class="table-container" ref="tableContainer" @scroll="handleScroll">
<el-table v-loading="loading" :data="filteredTtsModels" style="width: 100%;" class="data-table"
header-row-class-name="table-header" :fit="true" element-loading-text="拼命加载中"
element-loading-spinner="el-icon-loading" element-loading-background="rgba(0, 0, 0, 0.8)">
<el-table-column label="选择" width="50" align="center">
<template slot-scope="scope">
<el-checkbox v-model="scope.row.selected"></el-checkbox>
</template>
</el-table-column>
<el-table-column label="音色编码" align="center" min-width="50">
<el-table-column label="音色编码" align="center">
<template slot-scope="scope">
<el-input v-if="scope.row.editing" v-model="scope.row.voiceCode"></el-input>
<span v-else>{{ scope.row.voiceCode }}</span>
</template>
</el-table-column>
<el-table-column label="音色名称" align="center" min-width="50">
<el-table-column label="音色名称" align="center">
<template slot-scope="scope">
<el-input v-if="scope.row.editing" v-model="scope.row.voiceName"></el-input>
<span v-else>{{ scope.row.voiceName }}</span>
</template>
</el-table-column>
<el-table-column label="语言类型" align="center" min-width="50">
<el-table-column label="语言类型" align="center">
<template slot-scope="scope">
<el-input v-if="scope.row.editing" v-model="scope.row.languageType"></el-input>
<span v-else>{{ scope.row.languageType }}</span>
</template>
</el-table-column>
<el-table-column label="试听" align="center" min-width="100px" class-name="audio-column">
<el-table-column v-if="!showReferenceColumns" label="试听" align="center" class-name="audio-column">
<template slot-scope="scope">
<div class="custom-audio-container">
<el-input
v-if="scope.row.editing"
v-model="scope.row.voiceDemo"
placeholder="请输入MP3地址"
size="mini"
class="audio-input">
<el-input v-if="scope.row.editing" v-model="scope.row.voiceDemo" placeholder="请输入MP3地址"
class="audio-input">
</el-input>
<AudioPlayer v-else-if="isValidAudioUrl(scope.row.voiceDemo)" :audioUrl="scope.row.voiceDemo"/>
<AudioPlayer v-else-if="isValidAudioUrl(scope.row.voiceDemo)" :audioUrl="scope.row.voiceDemo" />
</div>
</template>
</el-table-column>
<el-table-column label="备注" align="center">
<el-table-column v-if="!showReferenceColumns" label="备注" align="center">
<template slot-scope="scope">
<el-input
v-if="scope.row.editing"
type="textarea"
:rows="1"
autosize
v-model="scope.row.remark"
placeholder="这里是备注"
class="remark-input"></el-input>
<el-input v-if="scope.row.editing" type="textarea" :rows="1" autosize v-model="scope.row.remark"
placeholder="这里是备注" class="remark-input"></el-input>
<span v-else>{{ scope.row.remark }}</span>
</template>
</el-table-column>
<el-table-column v-if="showReferenceColumns" label="克隆音频路径" align="center">
<template slot-scope="scope">
<el-input v-if="scope.row.editing" v-model="scope.row.referenceAudio" placeholder="这里是克隆音频路径"></el-input>
<span v-else>{{ scope.row.referenceAudio }}</span>
</template>
</el-table-column>
<el-table-column v-if="showReferenceColumns" label="克隆音频文本" align="center">
<template slot-scope="scope">
<el-input v-if="scope.row.editing" v-model="scope.row.referenceText" placeholder="这里是克隆音频对应文本"></el-input>
<span v-else>{{ scope.row.referenceText }}</span>
</template>
</el-table-column>
<el-table-column label="操作" align="center" width="150">
<template slot-scope="scope">
<template v-if="!scope.row.editing">
@@ -84,12 +71,7 @@
删除
</el-button>
</template>
<el-button
v-else
type="success"
size="mini"
@click="saveEdit(scope.row)"
class="save-Tts">保存
<el-button v-else type="success" size="mini" @click="saveEdit(scope.row)" class="save-Tts">保存
</el-button>
</template>
</el-table-column>
@@ -110,21 +92,19 @@
<el-button type="primary" size="mini" @click="addNew" style="background: #5bc98c;border: None;">
新增
</el-button>
<el-button type="primary"
size="mini"
@click="deleteRow(filteredTtsModels.filter(row => row.selected))"
style="background: red;border:None">删除
<el-button type="primary" size="mini" @click="deleteRow(filteredTtsModels.filter(row => row.selected))"
style="background: red;border:None">删除
</el-button>
</div>
</el-dialog>
</template>
<script>
import AudioPlayer from './AudioPlayer.vue'
import Api from "@/apis/api";
import AudioPlayer from './AudioPlayer.vue';
export default {
components: {AudioPlayer},
components: { AudioPlayer },
props: {
visible: {
type: Boolean,
@@ -133,6 +113,10 @@ export default {
ttsModelId: {
type: String,
required: true
},
modelConfig: {
type: Object,
default: null
}
},
data() {
@@ -151,6 +135,7 @@ export default {
selectAll: false,
selectedRows: [],
loading: false,
showReferenceColumns: false, // 控制是否显示参考列
};
},
watch: {
@@ -158,12 +143,19 @@ export default {
this.localVisible = newVal;
if (newVal) {
this.currentPage = 1;
this.updateShowReferenceColumns(); // 更新显示状态
this.loadData(); // 对话框显示时加载数据
this.$nextTick(() => {
this.updateScrollbar();
});
}
},
modelConfig: {
handler(newVal) {
this.updateShowReferenceColumns();
},
immediate: true
},
filteredTtsModels() {
this.$nextTick(() => {
this.updateScrollbar();
@@ -173,7 +165,7 @@ export default {
computed: {
filteredTtsModels() {
return this.ttsModels.filter(model =>
model.voiceName.toLowerCase().includes(this.searchQuery.toLowerCase())
model.voiceName.toLowerCase().includes(this.searchQuery.toLowerCase())
);
}
},
@@ -189,6 +181,16 @@ export default {
window.removeEventListener('mousemove', this.handleDrag);
},
methods: {
// 更新是否显示参考列
updateShowReferenceColumns() {
if (this.modelConfig && this.modelConfig.configJson) {
const providerType = this.modelConfig.configJson.type;
this.showReferenceColumns = ['fishspeech', 'gpt_sovits_v2', 'gpt_sovits_v3'].includes(providerType);
} else {
this.showReferenceColumns = false;
}
},
loadData() {
this.loading = true;
const params = {
@@ -206,6 +208,8 @@ export default {
voiceName: item.name || '未命名音色',
languageType: item.languages || '',
remark: item.remark || '',
referenceAudio: item.referenceAudio || '',
referenceText: item.referenceText || '',
voiceDemo: item.voiceDemo || '',
selected: false,
editing: false,
@@ -237,6 +241,7 @@ export default {
this.total = 0;
this.selectAll = false;
this.searchQuery = '';
this.showReferenceColumns = false;
this.localVisible = false;
this.$emit('update:visible', false);
@@ -249,7 +254,7 @@ export default {
if (!container || !scrollbarThumb || !scrollbarTrack) return;
const {scrollHeight, clientHeight} = container;
const { scrollHeight, clientHeight } = container;
const trackHeight = scrollbarTrack.clientHeight;
const thumbHeight = Math.max((clientHeight / scrollHeight) * trackHeight, 20);
@@ -264,7 +269,7 @@ export default {
if (!container || !scrollbarThumb || !scrollbarTrack) return;
const {scrollHeight, clientHeight, scrollTop} = container;
const { scrollHeight, clientHeight, scrollTop } = container;
const trackHeight = scrollbarTrack.clientHeight;
const thumbHeight = scrollbarThumb.clientHeight;
const maxTop = trackHeight - thumbHeight;
@@ -332,7 +337,7 @@ export default {
startEdit(row) {
row.editing = true;
this.$set(row, 'originalData', {...row});
this.$set(row, 'originalData', { ...row });
},
saveEdit(row) {
@@ -356,6 +361,12 @@ export default {
sort: row.sort
};
// 只有在显示参考列的情况下才添加参考字段
if (this.showReferenceColumns) {
params.referenceAudio = row.referenceAudio;
params.referenceText = row.referenceText;
}
let res;
if (row.id) {
// 已有ID,执行更新操作
@@ -423,8 +434,8 @@ export default {
}
const maxSort = this.ttsModels.length > 0
? Math.max(...this.ttsModels.map(item => Number(item.sort) || 0))
: 0;
? Math.max(...this.ttsModels.map(item => Number(item.sort) || 0))
: 0;
const newRow = {
voiceCode: '',
@@ -432,6 +443,8 @@ export default {
languageType: '中文',
voiceDemo: '',
remark: '',
referenceAudio: '',
referenceText: '',
selected: false,
editing: true,
sort: maxSort + 1
@@ -441,56 +454,56 @@ export default {
},
deleteRow(row) {
// 处理单个音色或音色数组
const voices = Array.isArray(row) ? row : [row];
// 处理单个音色或音色数组
const voices = Array.isArray(row) ? row : [row];
if (Array.isArray(row) && row.length === 0) {
this.$message.warning("请先选择需要删除的音色");
return;
if (Array.isArray(row) && row.length === 0) {
this.$message.warning("请先选择需要删除的音色");
return;
}
const voiceCount = voices.length;
this.$confirm(`确定要删除选中的${voiceCount}个音色吗?`, "警告", {
confirmButtonText: "确定",
cancelButtonText: "取消",
type: "warning",
distinguishCancelAndClose: true
}).then(() => {
const ids = voices.map(voice => voice.id);
if (ids.some(id => !id)) {
this.$message.error("存在无效的音色ID");
return;
}
const voiceCount = voices.length;
this.$confirm(`确定要删除选中的${voiceCount}个音色吗?`, "警告", {
confirmButtonText: "确定",
cancelButtonText: "取消",
type: "warning",
distinguishCancelAndClose: true
}).then(() => {
const ids = voices.map(voice => voice.id);
if (ids.some(id => !id)) {
this.$message.error("存在无效的音色ID");
return;
}
Api.timbre.deleteVoice(ids, ({data}) => {
if (data.code === 0) {
this.$message.success({
message: `成功删除${voiceCount}个参数`,
showClose: true
});
this.loadData(); // 刷新参数列表
} else {
this.$message.error({
message: data.msg || '删除失败,请重试',
showClose: true
});
}
Api.timbre.deleteVoice(ids, ({ data }) => {
if (data.code === 0) {
this.$message.success({
message: `成功删除${voiceCount}个参数`,
showClose: true
});
this.loadData(); // 刷新参数列表
} else {
this.$message.error({
message: data.msg || '删除失败,请重试',
showClose: true
});
}
});
}).catch(action => {
if (action === 'cancel') {
this.$message({
type: 'info',
message: '已取消删除操作',
duration: 1000
});
} else {
this.$message({
type: 'info',
message: '操作已关闭',
duration: 1000
});
}
if (action === 'cancel') {
this.$message({
type: 'info',
message: '已取消删除操作',
duration: 1000
});
} else {
this.$message({
type: 'info',
message: '操作已关闭',
duration: 1000
});
}
});
},
@@ -502,7 +515,6 @@ export default {
</script>
<style lang="scss" scoped>
::v-deep .el-dialog {
border-radius: 8px !important;
overflow: hidden;
@@ -695,6 +707,7 @@ export default {
white-space: pre-wrap !important;
word-break: break-all !important;
}
/* 按钮组定位调整 */
.action-buttons {
position: static;
@@ -719,5 +732,4 @@ export default {
flex-shrink: 0;
margin: 2px !important;
}
</style>
@@ -77,7 +77,10 @@
{{ isAllSelected ? '取消全选' : '全选' }}
</el-button>
<el-button type="success" size="mini" class="add-device-btn" @click="handleAddDevice">
新增
验证码绑定
</el-button>
<el-button type="success" size="mini" class="add-device-btn" @click="handleManualAddDevice">
手动添加
</el-button>
<el-button size="mini" type="danger" icon="el-icon-delete" @click="deleteSelected">解绑</el-button>
</div>
@@ -103,6 +106,8 @@
<AddDeviceDialog :visible.sync="addDeviceDialogVisible" :agent-id="currentAgentId"
@refresh="fetchBindDevices(currentAgentId)"/>
<ManualAddDeviceDialog :visible.sync="manualAddDeviceDialogVisible" :agent-id="currentAgentId"
@refresh="fetchBindDevices(currentAgentId)"/>
</div>
</template>
@@ -110,13 +115,19 @@
<script>
import Api from '@/apis/api';
import AddDeviceDialog from "@/components/AddDeviceDialog.vue";
import ManualAddDeviceDialog from "@/components/ManualAddDeviceDialog.vue";
import HeaderBar from "@/components/HeaderBar.vue";
export default {
components: {HeaderBar, AddDeviceDialog},
components: {
HeaderBar,
AddDeviceDialog,
ManualAddDeviceDialog
},
data() {
return {
addDeviceDialogVisible: false,
manualAddDeviceDialogVisible: false,
selectedDevices: [],
isAllSelected: false,
searchKeyword: "",
@@ -252,6 +263,9 @@ export default {
handleAddDevice() {
this.addDeviceDialogVisible = true;
},
handleManualAddDevice() {
this.manualAddDeviceDialogVisible = true;
},
submitRemark(row) {
if (row._submitting) return;
+4 -2
View File
@@ -128,7 +128,7 @@
<ModelEditDialog :modelType="activeTab" :visible.sync="editDialogVisible" :modelData="editModelData"
@save="handleModelSave" />
<TtsModel :visible.sync="ttsDialogVisible" :ttsModelId="selectedTtsModelId" />
<TtsModel :visible.sync="ttsDialogVisible" :ttsModelId="selectedTtsModelId" :modelConfig="selectedModelConfig" />
<AddModelDialog :modelType="activeTab" :visible.sync="addDialogVisible" @confirm="handleAddConfirm" />
</div>
<el-footer>
@@ -162,7 +162,8 @@ export default {
total: 0,
selectedModels: [],
isAllSelected: false,
loading: false
loading: false,
selectedModelConfig: {}
};
},
@@ -211,6 +212,7 @@ export default {
},
openTtsDialog(row) {
this.selectedTtsModelId = row.id;
this.selectedModelConfig = row;
this.ttsDialogVisible = true;
},
headerCellClassName({ column, columnIndex }) {
+1 -1
View File
@@ -43,7 +43,7 @@
<div class="input-box">
<img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
<el-input v-model="form.password" placeholder="请输入密码" type="password" />
<el-input v-model="form.password" placeholder="请输入密码" type="password" show-password />
</div>
<div style="display: flex; align-items: center; margin-top: 20px; width: 100%; gap: 10px;">
<div class="input-box" style="width: calc(100% - 130px); margin-top: 0;">
+2 -2
View File
@@ -70,13 +70,13 @@
<!-- 密码输入框 -->
<div class="input-box">
<img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
<el-input v-model="form.password" placeholder="请输入密码" type="password" />
<el-input v-model="form.password" placeholder="请输入密码" type="password" show-password />
</div>
<!-- 新增确认密码 -->
<div class="input-box">
<img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
<el-input v-model="form.confirmPassword" placeholder="请确认密码" type="password" />
<el-input v-model="form.confirmPassword" placeholder="请确认密码" type="password" show-password />
</div>
<!-- 验证码部分保持相同 -->
@@ -61,13 +61,13 @@
<!-- 新密码 -->
<div class="input-box">
<img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
<el-input v-model="form.newPassword" placeholder="请输入新密码" type="password" />
<el-input v-model="form.newPassword" placeholder="请输入新密码" type="password" show-password />
</div>
<!-- 确认新密码 -->
<div class="input-box">
<img loading="lazy" alt="" class="input-icon" src="@/assets/login/password.png" />
<el-input v-model="form.confirmPassword" placeholder="请确认新密码" type="password" />
<el-input v-model="form.confirmPassword" placeholder="请确认新密码" type="password" show-password />
</div>
<!-- 修改底部链接 -->
+12 -1
View File
@@ -5,7 +5,7 @@ import asyncio
from aioconsole import ainput
from config.settings import load_config
from config.logger import setup_logging
from core.utils.util import get_local_ip
from core.utils.util import get_local_ip, validate_mcp_endpoint
from core.http_server import SimpleHttpServer
from core.websocket_server import WebSocketServer
from core.utils.util import check_ffmpeg_installed
@@ -77,6 +77,17 @@ async def main():
get_local_ip(),
port,
)
mcp_endpoint = config.get("mcp_endpoint", None)
if mcp_endpoint is not None and "" not in mcp_endpoint:
# 校验MCP接入点格式
if validate_mcp_endpoint(mcp_endpoint):
logger.bind(tag=TAG).info("mcp接入点是\t{}", mcp_endpoint)
# 将mcp计入点地址转成调用点
mcp_endpoint = mcp_endpoint.replace("/mcp/", "/call/")
config["mcp_endpoint"] = mcp_endpoint
else:
logger.bind(tag=TAG).error("mcp接入点不符合规范")
config["mcp_endpoint"] = "你的接入点 websocket地址"
# 获取WebSocket配置,使用安全的默认值
websocket_port = 8000
+10 -6
View File
@@ -104,7 +104,8 @@ wakeup_words:
- "喵喵同学"
- "小滨小滨"
- "小冰小冰"
# MCP接入点地址
# MCP接入点地址,地址格式为:ws://你的mcp接入点ip或者域名:端口号/mcp/?token=你的token
# 详细教程 https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md
mcp_endpoint: 你的接入点 websocket地址
# 插件的基础配置
plugins:
@@ -121,7 +122,9 @@ plugins:
society_rss_url: "https://www.chinanews.com.cn/rss/society.xml"
world_rss_url: "https://www.chinanews.com.cn/rss/world.xml"
finance_rss_url: "https://www.chinanews.com.cn/rss/finance.xml"
get_news_from_newsnow: {"url": "https://newsnow.busiyi.world/api/s?id="}
get_news_from_newsnow:
url: "https://newsnow.busiyi.world/api/s?id="
news_sources: "澎湃新闻;百度热搜;财联社"
home_assistant:
devices:
- 客厅,玩具灯,switch.cuco_cn_460494544_cp1_on_p_2_1
@@ -159,7 +162,7 @@ end_prompt:
enable: true # 是否开启结束语
# 结束语
prompt: |
请你以时间过得真快未来头,用富有感情、依依不舍的话来结束这场对话吧!
请你以"时间过得真快"未来头,用富有感情、依依不舍的话来结束这场对话吧!
# 具体处理时选择的模块(The module selected for specific processing)
selected_module:
@@ -196,7 +199,7 @@ Intent:
# 如果你的不想使用selected_module.LLM意图识别,这里最好使用独立的LLM作为意图识别,例如使用免费的ChatGLMLLM
llm: ChatGLMLLM
# plugins_func/functions下的模块,可以通过配置,选择加载哪个模块,加载后对话支持相应的function调用
# 系统默认已经记载handle_exit_intent(退出识别)”、“play_music(音乐播放)插件,请勿重复加载
# 系统默认已经记载"handle_exit_intent(退出识别)"、"play_music(音乐播放)"插件,请勿重复加载
# 下面是加载查天气、角色切换、加载查新闻的插件示例
functions:
- get_weather
@@ -206,7 +209,7 @@ Intent:
# 不需要动type
type: function_call
# plugins_func/functions下的模块,可以通过配置,选择加载哪个模块,加载后对话支持相应的function调用
# 系统默认已经记载handle_exit_intent(退出识别)”、“play_music(音乐播放)插件,请勿重复加载
# 系统默认已经记载"handle_exit_intent(退出识别)"、"play_music(音乐播放)"插件,请勿重复加载
# 下面是加载查天气、角色切换、加载查新闻的插件示例
functions:
- change_role
@@ -347,6 +350,7 @@ VAD:
SileroVAD:
type: silero
threshold: 0.5
threshold_low: 0.3
model_dir: models/snakers4_silero-vad
min_silence_duration_ms: 200 # 如果说话停顿比较长,可以把这个值设置大一些
@@ -812,4 +816,4 @@ TTS:
# 支持声音克隆,可自行上传音频,填入voice参数,voice参数为空时,使用默认声音
access_token: "U4YdYXVfpwWnk2t5Gp822zWPCuORyeJL"
voice: "OUeAo1mhq6IBExi"
output_dir: tmp/
output_dir: tmp/
+1 -1
View File
@@ -5,7 +5,7 @@ from config.config_loader import load_config
from config.settings import check_config_file
from datetime import datetime
SERVER_VERSION = "0.6.1"
SERVER_VERSION = "0.6.3"
_logger_initialized = False
+1
View File
@@ -19,6 +19,7 @@ server:
vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain
manager-api:
# 你的manager-api的地址,最好使用局域网ip
# 如果使用docker部署,请使用填写成 http://xiaozhi-esp32-server-web:8002/xiaozhi
url: http://127.0.0.1:8002/xiaozhi
# 你的manager-api的token,就是刚才复制出来的server.secret
secret: 你的server.secret值
+132 -43
View File
@@ -17,6 +17,7 @@ from core.utils.util import (
filter_sensitive_info,
)
from typing import Dict, Any
from collections import deque
from core.utils.modules_initialize import (
initialize_modules,
initialize_tts,
@@ -110,10 +111,10 @@ class ConnectionHandler:
# vad相关变量
self.client_audio_buffer = bytearray()
self.client_have_voice = False
self.client_have_voice_last_time = 0.0
self.client_no_voice_last_time = 0.0
self.last_activity_time = 0.0 # 统一的活动时间戳(毫秒)
self.client_voice_stop = False
self.client_voice_frame_count = 0
self.client_voice_window = deque(maxlen=5)
self.last_is_voice = False
# asr相关变量
# 因为实际部署时可能会用到公共的本地ASR,不能把变量暴露给公共ASR
@@ -143,10 +144,10 @@ class ConnectionHandler:
self.load_function_plugin = False
self.intent_type = "nointent"
self.timeout_task = None
self.timeout_seconds = (
int(self.config.get("close_connection_no_voice_time", 120)) + 60
) # 在原来第一道关闭的基础上加60秒,进行二道关闭
self.timeout_task = None
# {"mcp":true} 表示启用MCP功能
self.features = None
@@ -187,12 +188,14 @@ class ConnectionHandler:
self.websocket = ws
self.device_id = self.headers.get("device-id", None)
# 初始化活动时间戳
self.last_activity_time = time.time() * 1000
# 启动超时检查任务
self.timeout_task = asyncio.create_task(self._check_timeout())
self.welcome_msg = self.config["xiaozhi"]
self.welcome_msg["session_id"] = self.session_id
await self.websocket.send(json.dumps(self.welcome_msg))
# 获取差异化配置
self._initialize_private_config()
@@ -213,7 +216,17 @@ class ConnectionHandler:
self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}")
return
finally:
await self._save_and_close(ws)
try:
await self._save_and_close(ws)
except Exception as final_error:
self.logger.bind(tag=TAG).error(f"最终清理时出错: {final_error}")
# 确保即使保存记忆失败,也要关闭连接
try:
await self.close(ws)
except Exception as close_error:
self.logger.bind(tag=TAG).error(
f"强制关闭连接时出错: {close_error}"
)
async def _save_and_close(self, ws):
"""保存记忆并关闭连接"""
@@ -231,7 +244,10 @@ class ConnectionHandler:
except Exception as e:
self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}")
finally:
loop.close()
try:
loop.close()
except Exception:
pass
# 启动线程保存记忆,不等待完成
threading.Thread(target=save_memory_task, daemon=True).start()
@@ -239,16 +255,17 @@ class ConnectionHandler:
self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}")
finally:
# 立即关闭连接,不等待记忆保存完成
await self.close(ws)
try:
await self.close(ws)
except Exception as close_error:
self.logger.bind(tag=TAG).error(
f"保存记忆后关闭连接失败: {close_error}"
)
async def _route_message(self, message):
"""消息路由"""
# 重置超时计时器
if self.timeout_task:
self.timeout_task.cancel()
self.timeout_task = asyncio.create_task(self._check_timeout())
if isinstance(message, str):
self.last_activity_time = time.time() * 1000
await handleTextMessage(self, message)
elif isinstance(message, bytes):
if self.vad is None:
@@ -594,13 +611,24 @@ class ConnectionHandler:
# 更新系统prompt至上下文
self.dialogue.update_system_message(self.prompt)
def chat(self, query, tool_call=False):
def chat(self, query, tool_call=False, depth=0):
self.logger.bind(tag=TAG).info(f"大模型收到用户消息: {query}")
self.llm_finish_task = False
if not tool_call:
self.dialogue.put(Message(role="user", content=query))
# 为最顶层时新建会话ID和发送FIRST请求
if depth == 0:
self.sentence_id = str(uuid.uuid4().hex)
self.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=self.sentence_id,
sentence_type=SentenceType.FIRST,
content_type=ContentType.ACTION,
)
)
# Define intent functions
functions = None
if self.intent_type == "function_call" and hasattr(self, "func_handler"):
@@ -616,8 +644,6 @@ class ConnectionHandler:
)
memory_str = future.result()
self.sentence_id = str(uuid.uuid4().hex)
if self.intent_type == "function_call" and functions is not None:
# 使用支持functions的streaming接口
llm_responses = self.llm.response_with_functions(
@@ -640,7 +666,6 @@ class ConnectionHandler:
function_id = None
function_arguments = ""
content_arguments = ""
text_index = 0
self.client_abort = False
for response in llm_responses:
if self.client_abort:
@@ -670,14 +695,6 @@ class ConnectionHandler:
if content is not None and len(content) > 0:
if not tool_call_flag:
response_message.append(content)
if text_index == 0:
self.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=self.sentence_id,
sentence_type=SentenceType.FIRST,
content_type=ContentType.ACTION,
)
)
self.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=self.sentence_id,
@@ -686,7 +703,6 @@ class ConnectionHandler:
content_detail=content,
)
)
text_index += 1
# 处理function call
if tool_call_flag:
bHasError = False
@@ -711,6 +727,11 @@ class ConnectionHandler:
f"function call error: {content_arguments}"
)
if not bHasError:
# 如需要大模型先处理一轮,添加相关处理后的日志情况
if len(response_message) > 0:
self.dialogue.put(
Message(role="assistant", content="".join(response_message))
)
response_message.clear()
self.logger.bind(tag=TAG).debug(
f"function_name={function_name}, function_id={function_id}, function_arguments={function_arguments}"
@@ -728,14 +749,14 @@ class ConnectionHandler:
),
self.loop,
).result()
self._handle_function_result(result, function_call_data)
self._handle_function_result(result, function_call_data, depth=depth)
# 存储对话内容
if len(response_message) > 0:
self.dialogue.put(
Message(role="assistant", content="".join(response_message))
)
if text_index > 0:
if depth == 0:
self.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=self.sentence_id,
@@ -750,7 +771,7 @@ class ConnectionHandler:
return True
def _handle_function_result(self, result, function_call_data):
def _handle_function_result(self, result, function_call_data, depth):
if result.action == Action.RESPONSE: # 直接回复前端
text = result.response
self.tts.tts_one_sentence(self, ContentType.TEXT, content_detail=text)
@@ -787,7 +808,7 @@ class ConnectionHandler:
content=text,
)
)
self.chat(text, tool_call=True)
self.chat(text, tool_call=True, depth=depth + 1)
elif result.action == Action.NOTFOUND or result.action == Action.ERROR:
text = result.response if result.response else result.result
self.tts.tts_one_sentence(self, ContentType.TEXT, content_detail=text)
@@ -840,13 +861,22 @@ class ConnectionHandler:
"""资源清理方法"""
try:
# 取消超时任务
if self.timeout_task:
if self.timeout_task and not self.timeout_task.done():
self.timeout_task.cancel()
try:
await self.timeout_task
except asyncio.CancelledError:
pass
self.timeout_task = None
# 清理工具处理器资源
if hasattr(self, "func_handler") and self.func_handler:
await self.func_handler.cleanup()
try:
await self.func_handler.cleanup()
except Exception as cleanup_error:
self.logger.bind(tag=TAG).error(
f"清理工具处理器时出错: {cleanup_error}"
)
# 触发停止事件
if self.stop_event:
@@ -856,19 +886,61 @@ class ConnectionHandler:
self.clear_queues()
# 关闭WebSocket连接
if ws:
await ws.close()
elif self.websocket:
await self.websocket.close()
try:
if ws:
# 安全地检查WebSocket状态并关闭
try:
if hasattr(ws, "closed") and not ws.closed:
await ws.close()
elif hasattr(ws, "state") and ws.state.name != "CLOSED":
await ws.close()
else:
# 如果没有closed属性,直接尝试关闭
await ws.close()
except Exception:
# 如果关闭失败,忽略错误
pass
elif self.websocket:
try:
if (
hasattr(self.websocket, "closed")
and not self.websocket.closed
):
await self.websocket.close()
elif (
hasattr(self.websocket, "state")
and self.websocket.state.name != "CLOSED"
):
await self.websocket.close()
else:
# 如果没有closed属性,直接尝试关闭
await self.websocket.close()
except Exception:
# 如果关闭失败,忽略错误
pass
except Exception as ws_error:
self.logger.bind(tag=TAG).error(f"关闭WebSocket连接时出错: {ws_error}")
if self.tts:
await self.tts.close()
# 最后关闭线程池(避免阻塞)
if self.executor:
self.executor.shutdown(wait=False)
try:
self.executor.shutdown(wait=False)
except Exception as executor_error:
self.logger.bind(tag=TAG).error(
f"关闭线程池时出错: {executor_error}"
)
self.executor = None
self.logger.bind(tag=TAG).info("连接资源已释放")
except Exception as e:
self.logger.bind(tag=TAG).error(f"关闭连接时出错: {e}")
finally:
# 确保停止事件被设置
if self.stop_event:
self.stop_event.set()
def clear_queues(self):
"""清空所有任务队列"""
@@ -898,7 +970,6 @@ class ConnectionHandler:
def reset_vad_states(self):
self.client_audio_buffer = bytearray()
self.client_have_voice = False
self.client_have_voice_last_time = 0
self.client_voice_stop = False
self.logger.bind(tag=TAG).debug("VAD states reset.")
@@ -917,10 +988,28 @@ class ConnectionHandler:
"""检查连接超时"""
try:
while not self.stop_event.is_set():
await asyncio.sleep(self.timeout_seconds)
if not self.stop_event.is_set():
self.logger.bind(tag=TAG).info("连接超时,准备关闭")
await self.close(self.websocket)
break
# 检查是否超时(只有在时间戳已初始化的情况下)
if self.last_activity_time > 0.0:
current_time = time.time() * 1000
if (
current_time - self.last_activity_time
> self.timeout_seconds * 1000
):
if not self.stop_event.is_set():
self.logger.bind(tag=TAG).info("连接超时,准备关闭")
# 设置停止事件,防止重复处理
self.stop_event.set()
# 使用 try-except 包装关闭操作,确保不会因为异常而阻塞
try:
await self.close(self.websocket)
except Exception as close_error:
self.logger.bind(tag=TAG).error(
f"超时关闭连接时出错: {close_error}"
)
break
# 每10秒检查一次,避免过于频繁
await asyncio.sleep(10)
except Exception as e:
self.logger.bind(tag=TAG).error(f"超时检查任务出错: {e}")
finally:
self.logger.bind(tag=TAG).info("超时检查任务已退出")
@@ -6,9 +6,8 @@ from core.handle.helloHandle import checkWakeupWords
from core.utils.util import remove_punctuation_and_length
from core.providers.tts.dto.dto import ContentType
from core.utils.dialogue import Message
from core.providers.tools.device_mcp import call_mcp_tool
from plugins_func.register import Action, ActionResponse
from loguru import logger
from core.providers.tts.dto.dto import TTSMessageDTO, SentenceType
TAG = __name__
@@ -29,6 +28,8 @@ async def handle_user_intent(conn, text):
intent_result = await analyze_intent_with_llm(conn, text)
if not intent_result:
return False
# 会话开始时生成sentence_id
conn.sentence_id = str(uuid.uuid4().hex)
# 处理各种意图
return await process_intent_result(conn, intent_result, text)
@@ -79,11 +80,6 @@ async def process_intent_result(conn, intent_result, original_text):
if function_name == "continue_chat":
return False
if function_name == "play_music":
funcItem = conn.func_handler.get_function(function_name)
if not funcItem:
conn.func_handler.function_registry.register_function("play_music")
function_args = {}
if "arguments" in intent_data["function_call"]:
function_args = intent_data["function_call"]["arguments"]
@@ -158,5 +154,19 @@ async def process_intent_result(conn, intent_result, original_text):
def speak_txt(conn, text):
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
sentence_type=SentenceType.FIRST,
content_type=ContentType.ACTION,
)
)
conn.tts.tts_one_sentence(conn, ContentType.TEXT, content_detail=text)
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
sentence_type=SentenceType.LAST,
content_type=ContentType.ACTION,
)
)
conn.dialogue.put(Message(role="assistant", content=text))
@@ -66,12 +66,11 @@ async def startToChat(conn, text):
async def no_voice_close_connect(conn, have_voice):
if have_voice:
conn.client_no_voice_last_time = 0.0
conn.last_activity_time = time.time() * 1000
return
if conn.client_no_voice_last_time == 0.0:
conn.client_no_voice_last_time = time.time() * 1000
else:
no_voice_time = time.time() * 1000 - conn.client_no_voice_last_time
# 只有在已经初始化过时间戳的情况下才进行超时检查
if conn.last_activity_time > 0.0:
no_voice_time = time.time() * 1000 - conn.last_activity_time
close_connection_no_voice_time = int(
conn.config.get("close_connection_no_voice_time", 120)
)
@@ -93,7 +93,7 @@ async def sendAudio(conn, audios, pre_buffer=True):
break
# 重置没有声音的状态
conn.client_no_voice_last_time = 0.0
conn.last_activity_time = time.time() * 1000
# 计算预期发送时间
expected_time = start_time + (play_position / 1000)
@@ -1,3 +1,4 @@
import httpx
import openai
from openai.types import CompletionUsage
from config.logger import setup_logging
@@ -16,6 +17,9 @@ class LLMProvider(LLMProviderBase):
self.base_url = config.get("base_url")
else:
self.base_url = config.get("url")
# 增加timeout的配置项,单位为秒
timeout = config.get("timeout", 300)
self.timeout = int(timeout) if timeout else 300
param_defaults = {
"max_tokens": (500, int),
@@ -42,7 +46,7 @@ class LLMProvider(LLMProviderBase):
model_key_msg = check_model_key("LLM", self.api_key)
if model_key_msg:
logger.bind(tag=TAG).error(model_key_msg)
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url, timeout=httpx.Timeout(self.timeout))
def response(self, session_id, dialogue, **kwargs):
try:
+4 -18
View File
@@ -161,13 +161,6 @@ class TTSProviderBase(ABC):
else:
sentence_id = str(uuid.uuid4()).replace("-", "")
conn.sentence_id = sentence_id
self.tts_text_queue.put(
TTSMessageDTO(
sentence_id=sentence_id,
sentence_type=SentenceType.FIRST,
content_type=ContentType.ACTION,
)
)
# 对于单句的文本,进行分段处理
segments = re.split(r"([。!?!?;\n])", content_detail)
for seg in segments:
@@ -180,13 +173,6 @@ class TTSProviderBase(ABC):
content_file=content_file,
)
)
self.tts_text_queue.put(
TTSMessageDTO(
sentence_id=sentence_id,
sentence_type=SentenceType.LAST,
content_type=ContentType.ACTION,
)
)
async def open_audio_channels(self, conn):
self.conn = conn
@@ -209,6 +195,8 @@ class TTSProviderBase(ABC):
while not self.conn.stop_event.is_set():
try:
message = self.tts_text_queue.get(timeout=1)
if message.sentence_type == SentenceType.FIRST:
self.conn.client_abort = False
if self.conn.client_abort:
logger.bind(tag=TAG).info("收到打断信息,终止TTS文本处理线程")
continue
@@ -362,10 +350,8 @@ class TTSProviderBase(ABC):
return audio_datas
def _process_before_stop_play_files(self):
for tts_file, text in self.before_stop_play_files:
if tts_file and os.path.exists(tts_file):
audio_datas = self._process_audio_file(tts_file)
self.tts_audio_queue.put((SentenceType.MIDDLE, audio_datas, text))
for audio_datas, text in self.before_stop_play_files:
self.tts_audio_queue.put((SentenceType.MIDDLE, audio_datas, text))
self.before_stop_play_files.clear()
self.tts_audio_queue.put((SentenceType.LAST, [], None))
@@ -85,8 +85,12 @@ class TTSProvider(TTSProviderBase):
self.reference_id = (
None if not config.get("reference_id") else config.get("reference_id")
)
self.reference_audio = parse_string_to_list(config.get("reference_audio"))
self.reference_text = parse_string_to_list(config.get("reference_text"))
self.reference_audio = parse_string_to_list(
config.get('ref_audio')if config.get('ref_audio') else config.get("reference_audio")
)
self.reference_text = parse_string_to_list(
config.get('ref_text')if config.get('ref_text') else config.get("reference_text")
)
self.format = config.get("response_format", "wav")
self.audio_file_type = config.get("response_format", "wav")
self.api_key = config.get("api_key", "YOUR_API_KEY")
@@ -12,8 +12,8 @@ class TTSProvider(TTSProviderBase):
super().__init__(config, delete_audio_file)
self.url = config.get("url")
self.text_lang = config.get("text_lang", "zh")
self.ref_audio_path = config.get("ref_audio_path")
self.prompt_text = config.get("prompt_text")
self.ref_audio_path = config.get('ref_audio') if config.get('ref_audio') else config.get("ref_audio_path")
self.prompt_text = config.get('ref_text') if config.get('ref_text') else config.get("prompt_text")
self.prompt_lang = config.get("prompt_lang", "zh")
# 处理空字符串的情况
@@ -11,8 +11,8 @@ class TTSProvider(TTSProviderBase):
def __init__(self, config, delete_audio_file):
super().__init__(config, delete_audio_file)
self.url = config.get("url")
self.refer_wav_path = config.get("refer_wav_path")
self.prompt_text = config.get("prompt_text")
self.refer_wav_path = config.get('ref_audio')if config.get('ref_audio') else config.get("refer_wav_path")
self.prompt_text = config.get('ref_text')if config.get('ref_text') else config.get("prompt_text")
self.prompt_language = config.get("prompt_language")
self.text_language = config.get("text_language", "audo")
@@ -10,7 +10,6 @@ from config.logger import setup_logging
from core.utils import opus_encoder_utils
from core.utils.util import check_model_key
from core.providers.tts.base import TTSProviderBase
from core.handle.abortHandle import handleAbortMessage
from core.providers.tts.dto.dto import SentenceType, ContentType, InterfaceType
from asyncio import Task
@@ -175,6 +174,9 @@ class TTSProvider(TTSProviderBase):
async def _ensure_connection(self):
"""建立新的WebSocket连接"""
try:
if self.ws:
logger.bind(tag=TAG).info(f"使用已有链接...")
return self.ws
logger.bind(tag=TAG).info("开始建立新连接...")
ws_header = {
"X-Api-App-Key": self.appId,
@@ -200,6 +202,10 @@ class TTSProvider(TTSProviderBase):
logger.bind(tag=TAG).debug(
f"收到TTS任务|{message.sentence_type.name} {message.content_type.name} | 会话ID: {self.conn.sentence_id}"
)
if message.sentence_type == SentenceType.FIRST:
self.conn.client_abort = False
if self.conn.client_abort:
logger.bind(tag=TAG).info("收到打断信息,终止TTS文本处理线程")
continue
@@ -244,9 +250,12 @@ class TTSProvider(TTSProviderBase):
logger.bind(tag=TAG).info(
f"添加音频文件到待播放列表: {message.content_file}"
)
self.before_stop_play_files.append(
(message.content_file, message.content_detail)
)
if message.content_file and os.path.exists(message.content_file):
# 先处理文件音频数据
file_audio = self._process_audio_file(message.content_file)
self.before_stop_play_files.append(
(file_audio, message.content_detail)
)
if message.sentence_type == SentenceType.LAST:
try:
@@ -295,25 +304,15 @@ class TTSProvider(TTSProviderBase):
async def start_session(self, session_id):
logger.bind(tag=TAG).info(f"开始会话~~{session_id}")
try:
task = self._monitor_task
# 会话开始时检测上个会话的监听状态
if (
task is not None
and isinstance(task, Task)
and not task.done()
self._monitor_task is not None
and isinstance(self._monitor_task, Task)
and not self._monitor_task.done()
):
logger.bind(tag=TAG).info("等待上一个监听任务结束...")
if self.ws is not None:
logger.bind(tag=TAG).info("强制关闭上一个WebSocket连接以唤醒监听任务...")
try:
await self.ws.close()
except Exception as e:
logger.bind(tag=TAG).warning(f"关闭上一个ws异常: {e}")
self.ws = None
try:
await asyncio.wait_for(task, timeout=8)
except Exception as e:
logger.bind(tag=TAG).warning(f"等待监听任务异常: {e}")
self._monitor_task = None
logger.bind(tag=TAG).info("检测到未完成的上个会话,关闭监听任务和连接...")
await self.close()
# 建立新连接
await self._ensure_connection()
@@ -336,19 +335,7 @@ class TTSProvider(TTSProviderBase):
except Exception as e:
logger.bind(tag=TAG).error(f"启动会话失败: {str(e)}")
# 确保清理资源
if hasattr(self, "_monitor_task"):
try:
self._monitor_task.cancel()
await self._monitor_task
except:
pass
self._monitor_task = None
if self.ws:
try:
await self.ws.close()
except:
pass
self.ws = None
await self.close()
raise
async def finish_session(self, session_id):
@@ -368,7 +355,7 @@ class TTSProvider(TTSProviderBase):
logger.bind(tag=TAG).info("会话结束请求已发送")
# 等待监听任务完成
if hasattr(self, "_monitor_task"):
if self._monitor_task:
try:
await self._monitor_task
except Exception as e:
@@ -378,28 +365,25 @@ class TTSProvider(TTSProviderBase):
finally:
self._monitor_task = None
# 关闭连接
await self.close()
except Exception as e:
logger.bind(tag=TAG).error(f"关闭会话失败: {str(e)}")
# 确保清理资源
if hasattr(self, "_monitor_task"):
try:
self._monitor_task.cancel()
await self._monitor_task
except:
pass
self._monitor_task = None
if self.ws:
try:
await self.ws.close()
except:
pass
self.ws = None
await self.close()
raise
async def close(self):
"""资源清理方法"""
# 取消监听任务
if self._monitor_task:
try:
self._monitor_task.cancel()
await self._monitor_task
except asyncio.CancelledError:
pass
except Exception as e:
logger.bind(tag=TAG).warning(f"关闭时取消监听任务错误: {e}")
self._monitor_task = None
if self.ws:
try:
await self.ws.close()
@@ -413,6 +397,7 @@ class TTSProvider(TTSProviderBase):
is_first_sentence = True
first_sentence_segment_count = 0 # 添加计数器
try:
session_finished = False # 标记会话是否正常结束
while not self.conn.stop_event.is_set():
try:
# 确保 `recv()` 运行在同一个 event loop
@@ -466,6 +451,7 @@ class TTSProvider(TTSProviderBase):
elif res.optional.event == EVENT_SessionFinished:
logger.bind(tag=TAG).debug(f"会话结束~~")
self._process_before_stop_play_files()
session_finished = True
break
except websockets.ConnectionClosed:
logger.bind(tag=TAG).warning("WebSocket连接已关闭")
@@ -476,15 +462,15 @@ class TTSProvider(TTSProviderBase):
)
traceback.print_exc()
break
finally:
# 确保清理资源
if self.ws:
# 仅在连接异常时才关闭
if not session_finished and self.ws:
try:
await self.ws.close()
except:
pass
self.ws = None
# 监听任务退出时清理引用
finally:
self._monitor_task = None
async def send_event(
@@ -1,3 +1,4 @@
import os
import queue
import asyncio
import traceback
@@ -63,9 +64,12 @@ class TTSProvider(TTSProviderBase):
logger.bind(tag=TAG).info(
f"添加音频文件到待播放列表: {message.content_file}"
)
self.before_stop_play_files.append(
(message.content_file, message.content_detail)
)
if message.content_file and os.path.exists(message.content_file):
# 先处理文件音频数据
file_audio = self._process_audio_file(message.content_file)
self.before_stop_play_files.append(
(file_audio, message.content_detail)
)
if message.sentence_type == SentenceType.LAST:
# 处理剩余的文本
@@ -23,22 +23,24 @@ class VADProvider(VADProviderBase):
# 处理空字符串的情况
threshold = config.get("threshold", "0.5")
threshold_low = config.get("threshold_low", "0.2")
min_silence_duration_ms = config.get("min_silence_duration_ms", "1000")
self.vad_threshold = float(threshold) if threshold else 0.5
self.vad_threshold_low = float(threshold_low) if threshold_low else 0.2
self.silence_threshold_ms = (
int(min_silence_duration_ms) if min_silence_duration_ms else 1000
)
# 至少要多少帧才算有语音
self.frame_window_threshold = 3
def is_vad(self, conn, opus_packet):
try:
pcm_frame = self.decoder.decode(opus_packet, 960)
conn.client_audio_buffer.extend(pcm_frame) # 将新数据加入缓冲区
# 初始化帧计数器
if not hasattr(conn, "client_voice_frame_count"):
conn.client_voice_frame_count = 0
# 处理缓冲区中的完整帧(每次处理512采样点)
client_have_voice = False
while len(conn.client_audio_buffer) >= 512 * 2:
@@ -54,26 +56,30 @@ class VADProvider(VADProviderBase):
# 检测语音活动
with torch.no_grad():
speech_prob = self.model(audio_tensor, 16000).item()
is_voice = speech_prob >= self.vad_threshold
if is_voice:
conn.client_voice_frame_count += 1
# 双阈值判断
if speech_prob >= self.vad_threshold:
is_voice = True
elif speech_prob <= self.vad_threshold_low:
is_voice = False
else:
conn.client_voice_frame_count = 0
is_voice = conn.last_is_voice
# 只有连续4帧检测到语音才认为有
client_have_voice = conn.client_voice_frame_count >= 4
# 声音没低于最低值则延续前一个状态,判断为有
conn.last_is_voice = is_voice
# 更新滑动窗口
conn.client_voice_window.append(is_voice)
client_have_voice = (conn.client_voice_window.count(True) >= self.frame_window_threshold)
# 如果之前有声音,但本次没有声音,且与上次有声音的时间差已经超过了静默阈值,则认为已经说完一句话
if conn.client_have_voice and not client_have_voice:
stop_duration = (
time.time() * 1000 - conn.client_have_voice_last_time
)
stop_duration = time.time() * 1000 - conn.last_activity_time
if stop_duration >= self.silence_threshold_ms:
conn.client_voice_stop = True
if client_have_voice:
conn.client_have_voice = True
conn.client_have_voice_last_time = time.time() * 1000
conn.last_activity_time = time.time() * 1000
return client_have_voice
except opuslib_next.OpusError as e:
@@ -40,6 +40,7 @@ class VLLMProvider(VLLMProviderBase):
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
def response(self, question, base64_image):
question = question + "(请使用中文回复)"
try:
messages = [
{
+27 -1
View File
@@ -980,4 +980,30 @@ def is_valid_image_file(file_data: bytes) -> bool:
def sanitize_tool_name(name: str) -> str:
"""Sanitize tool names for OpenAI compatibility."""
return re.sub(r"[^a-zA-Z0-9_-]", "_", name)
# 支持中文、英文字母、数字、下划线和连字符
return re.sub(r"[^a-zA-Z0-9_\-\u4e00-\u9fff]", "_", name)
def validate_mcp_endpoint(mcp_endpoint: str) -> bool:
"""
校验MCP接入点格式
Args:
mcp_endpoint: MCP接入点字符串
Returns:
bool: 是否有效
"""
# 1. 检查是否以ws开头
if not mcp_endpoint.startswith("ws"):
return False
# 2. 检查是否包含key、call字样
if "key" in mcp_endpoint.lower() or "call" in mcp_endpoint.lower():
return False
# 3. 检查是否包含/mcp/字样
if "/mcp/" not in mcp_endpoint:
return False
return True
@@ -57,8 +57,25 @@ class WebSocketServer:
self.active_connections.add(handler)
try:
await handler.handle_connection(websocket)
except Exception as e:
self.logger.bind(tag=TAG).error(f"处理连接时出错: {e}")
finally:
# 确保从活动连接集合中移除
self.active_connections.discard(handler)
# 强制关闭连接(如果还没有关闭的话)
try:
# 安全地检查WebSocket状态并关闭
if hasattr(websocket, "closed") and not websocket.closed:
await websocket.close()
elif hasattr(websocket, "state") and websocket.state.name != "CLOSED":
await websocket.close()
else:
# 如果没有closed属性,直接尝试关闭
await websocket.close()
except Exception as close_error:
self.logger.bind(tag=TAG).error(
f"服务器端强制关闭连接时出错: {close_error}"
)
async def _http_response(self, websocket, request_headers):
# 检查是否为 WebSocket 升级请求
@@ -48,6 +48,7 @@ services:
- SPRING_DATASOURCE_DRUID_USERNAME=root
- SPRING_DATASOURCE_DRUID_PASSWORD=123456
- SPRING_DATA_REDIS_HOST=xiaozhi-esp32-server-redis
- SPRING_DATA_REDIS_PASSWORD=
- SPRING_DATA_REDIS_PORT=6379
volumes:
# 配置文件目录
@@ -8,30 +8,90 @@ from markitdown import MarkItDown
TAG = __name__
logger = setup_logging()
# 新闻来源字典,包含名称和对应的API ID
NEWS_SOURCES = {
"thepaper": "澎湃新闻",
"baidu": "百度热搜",
"cls-depth": "财联社",
CHANNEL_MAP = {
"V2EX": "v2ex-share",
"知乎": "zhihu",
"微博": "weibo",
"联合早报": "zaobao",
"酷安": "coolapk",
"MKTNews": "mktnews-flash",
"华尔街见闻": "wallstreetcn-quick",
"36氪": "36kr-quick",
"抖音": "douyin",
"虎扑": "hupu",
"百度贴吧": "tieba",
"今日头条": "toutiao",
"IT之家": "ithome",
"澎湃新闻": "thepaper",
"卫星通讯社": "sputniknewscn",
"参考消息": "cankaoxiaoxi",
"远景论坛": "pcbeta-windows11",
"财联社": "cls-depth",
"雪球": "xueqiu-hotstock",
"格隆汇": "gelonghui",
"法布财经": "fastbull-express",
"Solidot": "solidot",
"Hacker News": "hackernews",
"Product Hunt": "producthunt",
"Github": "github-trending-today",
"哔哩哔哩": "bilibili-hot-search",
"快手": "kuaishou",
"靠谱新闻": "kaopu",
"金十数据": "jin10",
"百度热搜": "baidu",
"牛客": "nowcoder",
"少数派": "sspai",
"稀土掘金": "juejin",
"凤凰网": "ifeng",
"虫部落": "chongbuluo-latest",
}
# 动态生成新闻源描述
def generate_news_sources_description():
sources_desc = []
for source_id, source_name in NEWS_SOURCES.items():
sources_desc.append(f"{source_name}({source_id})")
return "".join(sources_desc)
# 默认新闻来源字典,当配置中没有指定时使用
DEFAULT_NEWS_SOURCES = "澎湃新闻;百度热搜;财联社"
def get_news_sources_from_config(conn):
"""从配置中获取新闻源字符串"""
try:
# 尝试从插件配置中获取新闻源
if (
conn.config.get("plugins")
and conn.config["plugins"].get("get_news_from_newsnow")
and conn.config["plugins"]["get_news_from_newsnow"].get("news_sources")
):
# 获取配置的新闻源字符串
news_sources_config = conn.config["plugins"]["get_news_from_newsnow"][
"news_sources"
]
if isinstance(news_sources_config, str) and news_sources_config.strip():
logger.bind(tag=TAG).debug(f"使用配置的新闻源: {news_sources_config}")
return news_sources_config
else:
logger.bind(tag=TAG).warning("新闻源配置为空或格式错误,使用默认配置")
else:
logger.bind(tag=TAG).debug("未找到新闻源配置,使用默认配置")
return DEFAULT_NEWS_SOURCES
except Exception as e:
logger.bind(tag=TAG).error(f"获取新闻源配置失败: {e},使用默认配置")
return DEFAULT_NEWS_SOURCES
# 从CHANNEL_MAP获取所有可用的新闻源名称
available_sources = list(CHANNEL_MAP.keys())
example_sources_str = "".join(available_sources)
GET_NEWS_FROM_NEWSNOW_FUNCTION_DESC = {
"type": "function",
"function": {
"name": "get_news_from_newsnow",
"description": (
"获取最新新闻,随机选择一条新闻进行播报。"
f"用户可以选择不同的新闻源,{generate_news_sources_description()}等。"
"如果没有指定,默认从澎湃新闻获取。"
f"用户可以选择不同的新闻源,标准的名称是:{example_sources_str}"
"例如用户要求百度新闻,其实就是百度热搜。如果没有指定,默认从澎湃新闻获取。"
"用户可以要求获取详细内容,此时会获取新闻的详细内容。"
),
"parameters": {
@@ -39,7 +99,7 @@ GET_NEWS_FROM_NEWSNOW_FUNCTION_DESC = {
"properties": {
"source": {
"type": "string",
"description": f"新闻源,例如{generate_news_sources_description()}等。可选参数,如果不提供则使用默认新闻源",
"description": f"新闻源的标准中文名称,例如{example_sources_str}等。可选参数,如果不提供则使用默认新闻源",
},
"detail": {
"type": "boolean",
@@ -111,10 +171,13 @@ def fetch_news_detail(url):
ToolType.SYSTEM_CTL,
)
def get_news_from_newsnow(
conn, source: str = "thepaper", detail: bool = False, lang: str = "zh_CN"
conn, source: str = "澎湃新闻", detail: bool = False, lang: str = "zh_CN"
):
"""获取新闻并随机选择一条进行播报,或获取上一条新闻的详细内容"""
try:
# 获取当前配置的新闻源
news_sources = get_news_sources_from_config(conn)
# 如果detail为True,获取上一条新闻的详细内容
detail = str(detail).lower() == "true"
if detail:
@@ -132,7 +195,7 @@ def get_news_from_newsnow(
url = conn.last_newsnow_link.get("url")
title = conn.last_newsnow_link.get("title", "未知标题")
source_id = conn.last_newsnow_link.get("source_id", "thepaper")
source_name = NEWS_SOURCES.get(source_id, "未知来源")
source_name = CHANNEL_MAP.get(source_id, "未知来源")
if not url or url == "#":
return ActionResponse(
@@ -166,21 +229,32 @@ def get_news_from_newsnow(
return ActionResponse(Action.REQLLM, detail_report, None)
# 否则,获取新闻列表并随机选择一条
# 验证新闻源是否有效,如果无效则使用默认源
if source not in NEWS_SOURCES:
logger.bind(tag=TAG).warning(f"无效的新闻源: {source},使用默认源thepaper")
source = "thepaper"
# 将中文名称转换为英文ID
english_source_id = None
source_name = NEWS_SOURCES.get(source, "澎湃新闻")
logger.bind(tag=TAG).info(f"获取新闻: 新闻源={source}({source_name})")
# 检查输入的中文名称是否在配置的新闻源中
news_sources_list = [
name.strip() for name in news_sources.split(";") if name.strip()
]
if source in news_sources_list:
# 如果输入的中文名称在配置的新闻源中,在 CHANNEL_MAP 中查找对应的英文ID
english_source_id = CHANNEL_MAP.get(source)
# 如果找不到对应的英文ID,使用默认源
if not english_source_id:
logger.bind(tag=TAG).warning(f"无效的新闻源: {source},使用默认源澎湃新闻")
english_source_id = "thepaper"
source = "澎湃新闻"
logger.bind(tag=TAG).info(f"获取新闻: 新闻源={source}({english_source_id})")
# 获取新闻列表
news_items = fetch_news_from_api(conn, source)
news_items = fetch_news_from_api(conn, english_source_id)
if not news_items:
return ActionResponse(
Action.REQLLM,
f"抱歉,未能从{source_name}获取到新闻信息,请稍后再试或尝试其他新闻源。",
f"抱歉,未能从{source}获取到新闻信息,请稍后再试或尝试其他新闻源。",
None,
)
@@ -193,14 +267,14 @@ def get_news_from_newsnow(
conn.last_newsnow_link = {
"url": selected_news.get("url", "#"),
"title": selected_news.get("title", "未知标题"),
"source_id": source,
"source_id": english_source_id,
}
# 构建新闻报告
news_report = (
f"根据下列数据,用{lang}回应用户的新闻查询请求:\n\n"
f"新闻标题: {selected_news['title']}\n"
# f"新闻来源: {source_name}\n"
# f"新闻来源: {source}\n"
f"(请以自然、流畅的方式向用户播报这条新闻标题,"
f"提示用户可以要求获取详细内容,此时会获取新闻的详细内容。)"
)
@@ -1,13 +1,10 @@
from config.logger import setup_logging
import os
import re
import time
import random
import asyncio
import difflib
import traceback
from pathlib import Path
from core.utils import p3
from core.handle.sendAudioHandle import send_stt_message
from plugins_func.register import register_function, ToolType, ActionResponse, Action
from core.utils.dialogue import Message
@@ -51,8 +48,8 @@ def play_music(conn, song_name: str):
)
# 提交异步任务
future = asyncio.run_coroutine_threadsafe(
handle_music_command(conn, music_intent), conn.loop
task = conn.loop.create_task(
handle_music_command(conn, music_intent) # 封装异步逻辑
)
# 非阻塞回调处理
@@ -63,7 +60,7 @@ def play_music(conn, song_name: str):
except Exception as e:
conn.logger.bind(tag=TAG).error(f"播放失败: {e}")
future.add_done_callback(handle_done)
task.add_done_callback(handle_done)
return ActionResponse(
action=Action.NONE, result="指令已接收", response="正在为您播放音乐"
@@ -178,13 +175,13 @@ def _get_random_play_prompt(song_name):
# 移除文件扩展名
clean_name = os.path.splitext(song_name)[0]
prompts = [
f"正在为您播放,{clean_name}",
f"请欣赏歌曲,{clean_name}",
f"即将为您播放,{clean_name}",
f"为您带来,{clean_name}",
f"让我们聆听,{clean_name}",
f"接下来请欣赏,{clean_name}",
f"为您献上,{clean_name}",
f"正在为您播放,{clean_name}",
f"请欣赏歌曲,{clean_name}",
f"即将为您播放,{clean_name}",
f"现在为您带来,{clean_name}",
f"让我们一起聆听,{clean_name}",
f"接下来请欣赏,{clean_name}",
f"此刻为您献上,{clean_name}",
]
# 直接使用random.choice,不设置seed
return random.choice(prompts)
@@ -218,13 +215,14 @@ async def play_local_music(conn, specific_file=None):
await send_stt_message(conn, text)
conn.dialogue.put(Message(role="assistant", content=text))
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
sentence_type=SentenceType.FIRST,
content_type=ContentType.ACTION,
if conn.intent_type == "intent_llm":
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
sentence_type=SentenceType.FIRST,
content_type=ContentType.ACTION,
)
)
)
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
@@ -241,13 +239,14 @@ async def play_local_music(conn, specific_file=None):
content_file=music_path,
)
)
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
sentence_type=SentenceType.LAST,
content_type=ContentType.ACTION,
if conn.intent_type == "intent_llm":
conn.tts.tts_text_queue.put(
TTSMessageDTO(
sentence_id=conn.sentence_id,
sentence_type=SentenceType.LAST,
content_type=ContentType.ACTION,
)
)
)
except Exception as e:
conn.logger.bind(tag=TAG).error(f"播放音乐失败: {str(e)}")
+1 -1
View File
@@ -21,7 +21,7 @@ cozepy==0.12.0
mem0ai==0.1.62
bs4==0.0.2
modelscope==1.23.2
sherpa_onnx==1.12.0
sherpa_onnx==1.12.4
mcp==1.8.1
cnlunar==0.2.0
PySocks==1.7.1