diff --git a/.gitignore b/.gitignore index 9469baac..b43a8de3 100644 --- a/.gitignore +++ b/.gitignore @@ -174,3 +174,4 @@ main/xiaozhi-server/mysql uploadfile *.json .vscode +.cursor diff --git a/docs/dev-ops-integration.md b/docs/dev-ops-integration.md index bb9640b1..3c547f4d 100644 --- a/docs/dev-ops-integration.md +++ b/docs/dev-ops-integration.md @@ -4,6 +4,8 @@ 本项目的测试平台`https://2662r3426b.vicp.fun`，从开放以来就使用了该方法，效果良好。 +教程可参考B站博主`毕乐labs`发布的视频教程：[《开源小智服务器xiaozhi-server自动更新以及最新版本MCP接入点配置保姆教程》](https://www.bilibili.com/video/BV15H37zHE7Q) + # 开始条件 - 你的电脑/服务器是linux操作系统 - 你已经跑通了整个流程 @@ -40,6 +42,9 @@ git clone https://ghproxy.net/https://github.com/xinnan-tech/xiaozhi-esp32-serve 此刻你需要把`model.pt`文件复制到新的目录去，你可以这样 ``` +# 创建需要的目录 +mkdir -p /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/ + cp 你原来的.config.yaml完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/data/.config.yaml cp 你原来的model.pt完整路径 /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/models/SenseVoiceSmall/model.pt ``` @@ -161,3 +166,11 @@ tail -f /home/system/xiaozhi/xiaozhi-esp32-server/main/xiaozhi-server/tmp/server # 注意事项 测试平台`https://2662r3426b.vicp.fun`，是使用nginx做了反向代理。nginx.conf详细配置可以[参考这里](https://github.com/xinnan-tech/xiaozhi-esp32-server/issues/791) + +## 常见问题 + +### 1、为什么没有见到8001端口？ +回答：8001是开发环境使用的，用于运行前端的端口。如果你是服务器部署，不建议使用`npm run serve`启动8001端口运行前端，而是像本教程一样编译成html文件，然后使用nginx来管理访问。 + +### 2、每次更新需要更新手动SQL语句吗？ +回答：不需要，因为项目使用**Liquibase**管理数据库版本，会自动执行新的sql脚本。 \ No newline at end of file diff --git a/docs/mcp-endpoint-enable.md b/docs/mcp-endpoint-enable.md new file mode 100644 index 00000000..7bc72484 --- /dev/null +++ b/docs/mcp-endpoint-enable.md @@ -0,0 +1,125 @@ +# MCP 接入点部署使用指南 + +本教程包含2个部分 +- 1、如何部署MCP接入点这个服务 +- 1、全模块部署时，怎么配置MCP接入点 +- 2、单模块部署时，怎么配置MCP接入点 + +# 1、如何部署MCP接入点这个服务 + +## 第一步，下载mcp接入点项目源码 + +浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server) + +打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。 + +点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`mcp-endpoint-server-main` +你需要把它重命名成`mcp-endpoint-server`。 + +## 第二步，启动程序 +这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法 + +``` +# 进入本项目源码根目录 +cd mcp-endpoint-server + +# 清除缓存 +docker compose -f docker-compose.yml down +docker stop mcp-endpoint-server +docker rm mcp-endpoint-server +docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest + +# 启动docker容器 +docker compose -f docker-compose.yml up -d +# 查看日志 +docker logs -f mcp-endpoint-server +``` + +此时，日志里会输出类似以下的日志 +``` +250705 INFO-=====下面的地址分别是智控台/单模块MCP接入点地址==== +250705 INFO-智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc +250705 INFO-单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def +250705 INFO-=====请根据具体部署选择使用，请勿泄露给任何人====== +``` + +请你把两个接口地址复制出来： + +由于你是docker部署，切不可直接使用上面的地址！ + +由于你是docker部署，切不可直接使用上面的地址！ + +由于你是docker部署，切不可直接使用上面的地址！ + +你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么 +原来我的接口地址 +``` +智控台MCP参数配置: http://172.22.0.2:8004/mcp_endpoint/health?key=abc +单模块部署MCP接入点: ws://172.22.0.2:8004/mcp_endpoint/mcp/?token=def +``` +就要改成 +``` +智控台MCP参数配置: http://192.168.1.25:8004/mcp_endpoint/health?key=abc +单模块部署MCP接入点: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def +``` + +改好后，请使用浏览器直接访问`智控台MCP参数配置`。当浏览器出现类似这样的代码，说明是成功了。 +``` +{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"} +``` + +请你保留好上面两个`接口地址`，下一步要用到。 + +# 2、全模块部署时，怎么配置MCP接入点 + +如果你是全模块部署，使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。 + +然后搜索参数`server.mcp_endpoint`，此时，它的值应该是`null`值。 +点击修改按钮，把上一步得来的`智控台MCP参数配置`粘贴到`参数值`里。然后保存。 + +如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问mcp接入点，很大概率是网络防火墙，或者没有填写正确的局域网ip。 + +# 3、单模块部署时，怎么配置MCP接入点 + +如果你是单模块部署，找到你的配置文件`data/.config.yaml`。 +在配置文件搜索`mcp_endpoint`，如果没有找到，你就增加`mcp_endpoint`配置。类似我是就是这样 +``` +server: + websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/ + http_port: 8002 +log: + log_level: INFO + +# 此处可能还更多配置.. + +mcp_endpoint: 你的接入点websocket地址 +``` +这时，请你把`如何部署MCP接入点这个服务`中得到的`单模块部署MCP接入点` 粘贴到 `mcp_endpoint`中。类似这样 + +``` +server: + websocket: ws://你的ip或者域名:端口号/xiaozhi/v1/ + http_port: 8002 +log: + log_level: INFO + +# 此处可能还更多配置 + +mcp_endpoint: ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=def +``` + +配置好后，启动单模块会输出如下的日志。 +``` +250705[__main__]-INFO-初始化组件: vad成功 SileroVAD +250705[__main__]-INFO-初始化组件: asr成功 FunASRServer +250705[__main__]-INFO-OTA接口是 http://192.168.1.25:8002/xiaozhi/ota/ +250705[__main__]-INFO-视觉分析接口是 http://192.168.1.25:8002/mcp/vision/explain +250705[__main__]-INFO-mcp接入点是 ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc +250705[__main__]-INFO-Websocket地址是 ws://192.168.1.25:8000/xiaozhi/v1/ +250705[__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问======= +250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html +250705[__main__]-INFO-============================================================= +``` + +如上，如果能输出类似的`mcp接入点是`中`ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`说明配置成功了。 + diff --git a/docs/mcp-endpoint-integration.md b/docs/mcp-endpoint-integration.md index ca72aef1..91d62278 100644 --- a/docs/mcp-endpoint-integration.md +++ b/docs/mcp-endpoint-integration.md @@ -1,92 +1,36 @@ -# MCP 接入点部署使用指南 +# MCP 接入点使用指南 -本教程包含2个部分 -- 1、如何开启mcp接入点 -- 2、如何为智能体接入一个简单的mcp功能，如计算器功能 +本教程以虾哥开源的mcp计算器功能为示例，介绍如何将自己自定义的mcp服务接入到自己的接入点里。 -部署的前提条件： -- 1、你已经部署了全模块，因为mcp接入点需要全模块中的智控台功能 -- 2、你想在不修改xiaozhi-server项目的前提下，扩展小智的功能 - -# 如何开启mcp接入点 - -## 第一步，下载mcp接入点项目源码 - -浏览器打开[mcp接入点项目地址](https://github.com/xinnan-tech/mcp-endpoint-server) - -打开完，找到页面中一个绿色的按钮，写着`Code`的按钮，点开它，然后你就看到`Download ZIP`的按钮。 - -点击它，下载本项目源码压缩包。下载到你电脑后，解压它，此时它的名字可能叫`mcp-endpoint-server-main` -你需要把它重命名成`mcp-endpoint-server`。 - -## 第二步，启动程序 -这个项目是一个很简单的项目，建议使用docker运行。不过如果你不想使用docker运行，你可以参考[这个页面](https://github.com/xinnan-tech/mcp-endpoint-server/blob/main/README_dev.md)使用源码运行。以下是docker运行的方法 - -``` -# 进入本项目源码根目录 -cd mcp-endpoint-server - -# 清除缓存 -docker compose -f docker-compose.yml down -docker stop mcp-endpoint-server -docker rm mcp-endpoint-server -docker rmi ghcr.nju.edu.cn/xinnan-tech/mcp-endpoint-server:latest - -# 启动docker容器 -docker compose -f docker-compose.yml up -d -# 查看日志 -docker logs -f mcp-endpoint-server -``` - -此时，日志里会输出类似以下的日志 -``` -====================================================== -接口地址: http://172.1.1.1:8004/mcp_endpoint/health?key=xxxx -=======上面的地址是MCP接入点地址，请勿泄露给任何人============ -``` - -请你把接口地址复制出来： - -由于你是docker部署，切不可直接使用上面的地址！ - -由于你是docker部署，切不可直接使用上面的地址！ - -由于你是docker部署，切不可直接使用上面的地址！ - -你先把地址复制出来，放在一个草稿里，你要知道你的电脑的局域网ip是什么，例如我的电脑局域网ip是`192.168.1.25`，那么 -原来我的接口地址 -``` -http://172.1.1.1:8004/mcp_endpoint/health?key=xxxx -``` -就要改成 -``` -http://192.168.1.25:8004/mcp_endpoint/health?key=xxxx -``` - -改好后，请使用浏览器直接访问这个接口。当浏览器出现类似这样的代码，说明是成功了。 -``` -{"result":{"status":"success","connections":{"tool_connections":0,"robot_connections":0,"total_connections":0}},"error":null,"id":null,"jsonrpc":"2.0"} -``` - -请你保留好这个`接口地址`，下一步要用到。 - -## 第三步，配置智控台 - -使用管理员账号，登录智控台，点击顶部`参数字典`，选择`参数管理`功能。 - -然后搜索参数`server.mcp_endpoint`，此时，它的值应该是`null`值。 -点击修改按钮，把上一步得来的`接口地址`粘贴到`参数值`里。然后保存。 - -如果能保存成功，说明一切顺利，你可以去智能体查看效果了。如果不成功，说明智控台无法访问mcp接入点，很大概率是网络防火墙，或者没有填写正确的局域网ip。 +本教程的前提是，你的`xiaozhi-server`已经启用了mcp接入点功能，如果你还没启用，可以先根据[这个教程](./mcp-endpoint-enable.md)启用。 # 如何为智能体接入一个简单的mcp功能，如计算器功能 -如果以上步骤顺利，你可以进入智能体管理，点击`配置角色`，在`意图识别`的右边，有一个`编辑功能`的按钮。 +### 如果你是全模块部署 +如果你是全模块部署，你可以进入智控台，智能体管理，点击`配置角色`，在`意图识别`的右边，有一个`编辑功能`的按钮。 点击这个按钮。在弹出的页面里，位于底部，会有`MCP接入点`，正常来说，会显示这个智能体的`MCP接入点地址`，接下来，我们来给这个智能体扩展一个基于MCP技术的计算器的功能。 这个`MCP接入点地址`很重要，你等一下会用到。 +### 如果你是单模块部署 +如果你是单模块部署，且你已经在配置文件里配置了MCP接入点地址，那么正常来说，单模块部署启动的时候，会输出如下的日志。 +``` +250705[__main__]-INFO-初始化组件: vad成功 SileroVAD +250705[__main__]-INFO-初始化组件: asr成功 FunASRServer +250705[__main__]-INFO-OTA接口是 http://192.168.1.25:8002/xiaozhi/ota/ +250705[__main__]-INFO-视觉分析接口是 http://192.168.1.25:8002/mcp/vision/explain +250705[__main__]-INFO-mcp接入点是 ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc +250705[__main__]-INFO-Websocket地址是 ws://192.168.1.25:8000/xiaozhi/v1/ +250705[__main__]-INFO-=======上面的地址是websocket协议地址，请勿用浏览器访问======= +250705[__main__]-INFO-如想测试websocket请用谷歌浏览器打开test目录下的test_page.html +250705[__main__]-INFO-============================================================= +``` + +如上，输出`mcp接入点是`中`ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc`就是你的`MCP接入点地址`。 + +这个`MCP接入点地址`很重要，你等一下会用到。 + ## 第一步 下载虾哥MCP计算器项目代码 浏览器打开虾哥写的[计算器项目](https://github.com/78/mcp-calculator)， @@ -114,13 +58,13 @@ pip install -r requirements.txt 例如我的智能体的mcp地址是 ``` -ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc +ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc ``` 开始输入命令 ```bash -export MCP_ENDPOINT=ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc +export MCP_ENDPOINT=ws://192.168.1.25:8004/mcp_endpoint/mcp/?token=abc ``` 输入完后，启动程序 @@ -129,6 +73,22 @@ export MCP_ENDPOINT=ws://192.168.4.7:8004/mcp_endpoint/mcp/?token=abc python mcp_pipe.py calculator.py ``` +### 如果你是智控台部署 +如果你是智控台部署，启动完后，你再进入智控台，点击刷新MCP的接入状态，就会看到你扩展的功能列表了。 -启动完后，你再进入智控台，点击刷新MCP的接入状态，就会看到你扩展的功能列表了。 +### 如果你是单模块部署 +如果你是单模块部署，当设备连接后，会输出类似的日志，说明成功了 +``` +250705 -INFO-正在初始化MCP接入点: wss://2662r3426b.vicp.fun/mcp_e +250705 -INFO-发送MCP接入点初始化消息 +250705 -INFO-MCP接入点连接成功 +250705 -INFO-MCP接入点初始化成功 +250705 -INFO-统一工具处理器初始化完成 +250705 -INFO-MCP接入点服务器信息: name=Calculator, version=1.9.4 +250705 -INFO-MCP接入点支持的工具数量: 1 +250705 -INFO-所有MCP接入点工具已获取，客户端准备就绪 +250705 -INFO-工具缓存已刷新 +250705 -INFO-当前支持的函数列表: [ 'get_time', 'get_lunar', 'play_music', 'get_weather', 'handle_exit_intent', 'calculator'] +``` +如果包含了 `'calculator'`，说明设备将可以根据意图识别，调用计算器这个工具。 \ No newline at end of file diff --git a/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java b/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java index 33e97e8c..c3246e5b 100644 --- a/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java +++ b/main/manager-api/src/main/java/xiaozhi/common/constant/Constant.java @@ -116,6 +116,11 @@ public interface Constant { */ String SERVER_MCP_ENDPOINT = "server.mcp_endpoint"; + /** + * mcp接入点路径 + */ + String SERVER_VOICE_PRINT = "server.voice_print"; + /** * 无记忆 */ @@ -232,7 +237,7 @@ public interface Constant { /** * 版本号 */ - public static final String VERSION = "0.6.2"; + public static final String VERSION = "0.6.3"; /** * 无效固件URL diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/Enums/AgentChatHistoryType.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/Enums/AgentChatHistoryType.java new file mode 100644 index 00000000..49ed2dce --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/Enums/AgentChatHistoryType.java @@ -0,0 +1,21 @@ +package xiaozhi.modules.agent.Enums; + + +import lombok.Getter; + +/** + * 智能体聊天记录类型 + */ +@Getter +public enum AgentChatHistoryType { + + USER((byte) 1), + AGENT((byte) 2); + + private final byte value; + + AgentChatHistoryType(byte i) { + this.value = i; + } + +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java index bfd82e41..157eec63 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentController.java @@ -47,6 +47,7 @@ import xiaozhi.modules.agent.service.AgentChatHistoryService; import xiaozhi.modules.agent.service.AgentPluginMappingService; import xiaozhi.modules.agent.service.AgentService; import xiaozhi.modules.agent.service.AgentTemplateService; +import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO; import xiaozhi.modules.agent.vo.AgentInfoVO; import xiaozhi.modules.device.entity.DeviceEntity; import xiaozhi.modules.device.service.DeviceService; @@ -181,6 +182,23 @@ public class AgentController { List result = agentChatHistoryService.getChatHistoryBySessionId(id, sessionId); return new Result>().ok(result); } + @GetMapping("/{id}/chat-history/user") + @Operation(summary = "获取智能体聊天记录（用户）") + @RequiresPermissions("sys:role:normal") + public Result> getRecentlyFiftyByAgentId( + @PathVariable("id") String id) { + // 获取当前用户 + UserDetail user = SecurityUser.getUser(); + + // 检查权限 + if (!agentService.checkAgentPermission(id, user.getId())) { + return new Result>().error("没有权限查看该智能体的聊天记录"); + } + + // 查询聊天记录 + List data = agentChatHistoryService.getRecentlyFiftyByAgentId(id); + return new Result>().ok(data); + } @PostMapping("/audio/{audioId}") @Operation(summary = "获取音频下载ID") diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentVoicePrintController.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentVoicePrintController.java new file mode 100644 index 00000000..cce97a09 --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/controller/AgentVoicePrintController.java @@ -0,0 +1,75 @@ +package xiaozhi.modules.agent.controller; + +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.tags.Tag; +import jakarta.validation.Valid; +import lombok.AllArgsConstructor; +import org.apache.shiro.authz.annotation.RequiresPermissions; +import org.springframework.web.bind.annotation.*; +import xiaozhi.common.utils.Result; +import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO; +import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO; +import xiaozhi.modules.agent.service.AgentVoicePrintService; +import xiaozhi.modules.agent.vo.AgentVoicePrintVO; +import xiaozhi.modules.security.user.SecurityUser; + +import java.util.List; + +@Tag(name = "智能体声纹管理") +@AllArgsConstructor +@RestController +@RequestMapping("/agent/voice-print") +public class AgentVoicePrintController { + private final AgentVoicePrintService agentVoicePrintService; + + @PostMapping + @Operation(summary = "创建智能体的声纹") + @RequiresPermissions("sys:role:normal") + public Result save(@RequestBody @Valid AgentVoicePrintSaveDTO dto) { + boolean b = agentVoicePrintService.insert(dto); + if (b) { + return new Result<>(); + } + return new Result().error("智能体的声纹创建失败"); + } + + @PutMapping + @Operation(summary = "更新智能体的对应声纹") + @RequiresPermissions("sys:role:normal") + public Result update(@RequestBody @Valid AgentVoicePrintUpdateDTO dto) { + Long userId = SecurityUser.getUserId(); + boolean b = agentVoicePrintService.update(userId, dto); + if (b) { + return new Result<>(); + } + return new Result().error("智能体的对应声纹更新失败"); + } + + @DeleteMapping("/{id}") + @Operation(summary = "删除智能体对应声纹") + @RequiresPermissions("sys:role:normal") + public Result delete(@PathVariable String id) { + Long userId = SecurityUser.getUserId(); + // 先删除关联的设备 + boolean delete = agentVoicePrintService.delete(userId, id); + if (delete) { + return new Result<>(); + } + return new Result().error("智能体的对应声纹删除失败"); + } + + @GetMapping("/list/{id}") + @Operation(summary = "获取用户指定智能体声纹列表") + @RequiresPermissions("sys:role:normal") + public Result> list(@PathVariable String id) { + Long userId = SecurityUser.getUserId(); + List list = agentVoicePrintService.list(userId, id); + return new Result>().ok(list); + } + + + +} + + + diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentVoicePrintDao.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentVoicePrintDao.java new file mode 100644 index 00000000..2d8e1eda --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/dao/AgentVoicePrintDao.java @@ -0,0 +1,18 @@ +package xiaozhi.modules.agent.dao; + +import com.baomidou.mybatisplus.core.mapper.BaseMapper; +import org.apache.ibatis.annotations.Mapper; +import xiaozhi.modules.agent.entity.AgentChatHistoryEntity; +import xiaozhi.modules.agent.entity.AgentVoicePrintEntity; + +/** + * {@link AgentChatHistoryEntity} 智能体聊天历史记录Dao对象 + * + * @author Goody + * @version 1.0, 2025/4/30 + * @since 1.0.0 + */ +@Mapper +public interface AgentVoicePrintDao extends BaseMapper { + +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintSaveDTO.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintSaveDTO.java new file mode 100644 index 00000000..0ff3b0ca --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintSaveDTO.java @@ -0,0 +1,28 @@ +package xiaozhi.modules.agent.dto; + +import lombok.Data; + +/** + * 保存智能体声纹的dto + * + * @author zjy + */ +@Data +public class AgentVoicePrintSaveDTO { + /** + * 关联的智能体id + */ + private String agentId; + /** + * 音频文件id + */ + private String audioId; + /** + * 声纹来源的人姓名 + */ + private String sourceName; + /** + * 描述声纹来源的人 + */ + private String introduce; +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintUpdateDTO.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintUpdateDTO.java new file mode 100644 index 00000000..60643e29 --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/dto/AgentVoicePrintUpdateDTO.java @@ -0,0 +1,28 @@ +package xiaozhi.modules.agent.dto; + +import lombok.Data; + +/** + * 修改智能体声纹的dto + * + * @author zjy + */ +@Data +public class AgentVoicePrintUpdateDTO { + /** + * 智能体声纹id + */ + private String id; + /** + * 音频文件id + */ + private String audioId; + /** + * 声纹来源的人姓名 + */ + private String sourceName; + /** + * 描述声纹来源的人 + */ + private String introduce; +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentVoicePrintEntity.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentVoicePrintEntity.java new file mode 100644 index 00000000..99f2f122 --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/entity/AgentVoicePrintEntity.java @@ -0,0 +1,61 @@ +package xiaozhi.modules.agent.entity; + +import com.baomidou.mybatisplus.annotation.*; +import lombok.Data; +import lombok.EqualsAndHashCode; +import xiaozhi.common.entity.BaseEntity; + +import java.util.Date; + +/** + * 智能体声纹表 + * + * @author zjy + */ +@TableName(value = "ai_agent_voice_print") +@Data +public class AgentVoicePrintEntity { + /** + * 主键id + */ + @TableId(type = IdType.ASSIGN_UUID) + private String id; + /** + * 关联的智能体id + */ + private String agentId; + /** + * 关联的音频id + */ + private String audioId; + /** + * 声纹来源的人姓名 + */ + private String sourceName; + /** + * 描述声纹来源的人 + */ + private String introduce; + + /** + * 创建者 + */ + @TableField(fill = FieldFill.INSERT) + private Long creator; + /** + * 创建时间 + */ + @TableField(fill = FieldFill.INSERT) + private Date createDate; + + /** + * 更新者 + */ + @TableField(fill = FieldFill.INSERT_UPDATE) + private Long updater; + /** + * 更新时间 + */ + @TableField(fill = FieldFill.INSERT_UPDATE) + private Date updateDate; +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentChatHistoryService.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentChatHistoryService.java index 21fce988..b69cd7b0 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentChatHistoryService.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentChatHistoryService.java @@ -9,6 +9,7 @@ import xiaozhi.common.page.PageData; import xiaozhi.modules.agent.dto.AgentChatHistoryDTO; import xiaozhi.modules.agent.dto.AgentChatSessionDTO; import xiaozhi.modules.agent.entity.AgentChatHistoryEntity; +import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO; /** * 智能体聊天记录表处理service @@ -44,4 +45,12 @@ public interface AgentChatHistoryService extends IService getRecentlyFiftyByAgentId(String agentId); } diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentVoicePrintService.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentVoicePrintService.java new file mode 100644 index 00000000..d6264e71 --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/AgentVoicePrintService.java @@ -0,0 +1,53 @@ +package xiaozhi.modules.agent.service; + + +import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO; +import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO; +import xiaozhi.modules.agent.vo.AgentVoicePrintVO; + +import java.util.List; + +/** + * 智能体声纹处理service + * + * @author zjy + */ +public interface AgentVoicePrintService { + /** + * 添加智能体新的声纹 + * + * @param dto 保存智能体声纹的数据 + * @return T:成功 F：失败 + */ + boolean insert(AgentVoicePrintSaveDTO dto); + + /** + * 删除智能体的指的声纹 + * + * @param userId 当前登录的用户id + * @param voicePrintId 声纹id + * @return 是否成功 T:成功 F：失败 + */ + boolean delete(Long userId, String voicePrintId); + + /** + * 获取指定智能体的所有声纹数据 + * + * @param userId 当前登录的用户id + * @param agentId 智能体id + * @return 声纹数据集合 + */ + List list(Long userId, String agentId); + + /** + * 更新智能体的指的声纹数据 + * + * @param userId 当前登录的用户id + * @param dto 修改的声纹的数据 + * @return 是否成功 T:成功 F：失败 + */ + boolean update(Long userId, AgentVoicePrintUpdateDTO dto); + + + +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentChatHistoryServiceImpl.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentChatHistoryServiceImpl.java index b28e80ad..980115ad 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentChatHistoryServiceImpl.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentChatHistoryServiceImpl.java @@ -5,6 +5,7 @@ import java.util.List; import java.util.Map; import java.util.stream.Collectors; +import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; @@ -16,11 +17,13 @@ import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl; import xiaozhi.common.constant.Constant; import xiaozhi.common.page.PageData; import xiaozhi.common.utils.ConvertUtils; +import xiaozhi.modules.agent.Enums.AgentChatHistoryType; import xiaozhi.modules.agent.dao.AiAgentChatHistoryDao; import xiaozhi.modules.agent.dto.AgentChatHistoryDTO; import xiaozhi.modules.agent.dto.AgentChatSessionDTO; import xiaozhi.modules.agent.entity.AgentChatHistoryEntity; import xiaozhi.modules.agent.service.AgentChatHistoryService; +import xiaozhi.modules.agent.vo.AgentChatHistoryUserVO; /** * 智能体聊天记录表处理service {@link AgentChatHistoryService} impl @@ -90,4 +93,21 @@ public class AgentChatHistoryServiceImpl extends ServiceImpl getRecentlyFiftyByAgentId(String agentId) { + // 构建查询条件(不添加按照创建时间排序，数据本来就是主键越大创建时间越大 + // 不添加这样可以减少排序全部数据在分页的全盘扫描消耗) + LambdaQueryWrapper wrapper = new LambdaQueryWrapper<>(); + wrapper.select(AgentChatHistoryEntity::getContent,AgentChatHistoryEntity::getAudioId) + .eq(AgentChatHistoryEntity::getAgentId, agentId) + .eq(AgentChatHistoryEntity::getChatType, AgentChatHistoryType.USER.getValue()); + + // 构建分页查询，查询前50页数据 + Page pageParam = new Page<>(0, 50); + IPage result = this.baseMapper.selectPage(pageParam, wrapper); + return result.getRecords().stream().map( item -> + ConvertUtils.sourceToTarget(item,AgentChatHistoryUserVO.class) + ).toList(); + } } diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentMcpAccessPointServiceImpl.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentMcpAccessPointServiceImpl.java index e6c7d4e4..839f99ff 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentMcpAccessPointServiceImpl.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentMcpAccessPointServiceImpl.java @@ -61,14 +61,15 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic wsUrl = wsUrl.replace("/mcp/", "/call/"); try { - // 创建 WebSocket 连接 + // 创建 WebSocket 连接，增加超时时间到15秒 try (WebSocketClientManager client = WebSocketClientManager.build( new WebSocketClientManager.Builder() .uri(wsUrl) - .connectTimeout(5, TimeUnit.SECONDS) - .maxSessionDuration(9, TimeUnit.SECONDS))) { + .connectTimeout(8, TimeUnit.SECONDS) + .maxSessionDuration(10, TimeUnit.SECONDS))) { - // 发送初始化消息 + // 步骤1: 发送初始化消息并等待响应 + log.info("发送MCP初始化消息，智能体ID: {}", id); McpJsonRpcRequest initializeRequest = new McpJsonRpcRequest("initialize", Map.of( "protocolVersion", "2024-11-05", @@ -81,37 +82,70 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic 1); client.sendJson(initializeRequest); - // 等待初始化响应 - Thread.sleep(200); - - // 发送初始化完成通知 - // 对于通知类型的消息，手动构建JSON以避免包含null字段 - String notificationJson = "{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\"}"; - client.sendText(notificationJson); - - // 等待 0.2 秒 - Thread.sleep(200); - - // 发送工具列表请求 - McpJsonRpcRequest toolsRequest = new McpJsonRpcRequest("tools/list", null, 2); - client.sendJson(toolsRequest); - - // 监听响应，直到收到包含 id=2 的响应（tools/list响应） - List responses = client.listener(response -> { + // 等待初始化响应 (id=1) - 移除固定延迟，改为响应驱动 + List initResponses = client.listenerWithoutClose(response -> { try { - // 先尝试解析为通用JSON对象来获取id Map jsonMap = JsonUtils.parseObject(response, Map.class); - return jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id")); + if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) { + // 检查是否有result字段，表示初始化成功 + return jsonMap.containsKey("result") && !jsonMap.containsKey("error"); + } + return false; } catch (Exception e) { - log.warn("解析响应失败: {}", response, e); + log.warn("解析初始化响应失败: {}", response, e); return false; } }); - // 处理响应 - for (String response : responses) { + // 验证初始化响应 + boolean initSucceeded = false; + for (String response : initResponses) { + try { + Map jsonMap = JsonUtils.parseObject(response, Map.class); + if (jsonMap != null && Integer.valueOf(1).equals(jsonMap.get("id"))) { + if (jsonMap.containsKey("result")) { + log.info("MCP初始化成功，智能体ID: {}", id); + initSucceeded = true; + break; + } else if (jsonMap.containsKey("error")) { + log.error("MCP初始化失败，智能体ID: {}, 错误: {}", id, jsonMap.get("error")); + return List.of(); + } + } + } catch (Exception e) { + log.warn("处理初始化响应失败: {}", response, e); + } + } + + if (!initSucceeded) { + log.error("未收到有效的MCP初始化响应，智能体ID: {}", id); + return List.of(); + } + + // 步骤2: 发送初始化完成通知 - 只有在收到initialize响应后才发送 + log.info("发送MCP初始化完成通知，智能体ID: {}", id); + String notificationJson = "{\"jsonrpc\":\"2.0\",\"method\":\"notifications/initialized\"}"; + client.sendText(notificationJson); + + // 步骤3: 发送工具列表请求 - 立即发送，无需额外延迟 + log.info("发送MCP工具列表请求，智能体ID: {}", id); + McpJsonRpcRequest toolsRequest = new McpJsonRpcRequest("tools/list", null, 2); + client.sendJson(toolsRequest); + + // 等待工具列表响应 (id=2) + List toolsResponses = client.listener(response -> { + try { + Map jsonMap = JsonUtils.parseObject(response, Map.class); + return jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id")); + } catch (Exception e) { + log.warn("解析工具列表响应失败: {}", response, e); + return false; + } + }); + + // 处理工具列表响应 + for (String response : toolsResponses) { try { - // 先解析为通用JSON对象 Map jsonMap = JsonUtils.parseObject(response, Map.class); if (jsonMap != null && Integer.valueOf(2).equals(jsonMap.get("id"))) { // 检查是否有result字段 @@ -122,11 +156,16 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic if (toolsObj instanceof List) { List> toolsList = (List>) toolsObj; // 提取工具名称列表 - return toolsList.stream() + List result = toolsList.stream() .map(tool -> (String) tool.get("name")) .filter(name -> name != null) .collect(Collectors.toList()); + log.info("成功获取MCP工具列表，智能体ID: {}, 工具数量: {}", id, result.size()); + return result; } + } else if (jsonMap.containsKey("error")) { + log.error("获取工具列表失败，智能体ID: {}, 错误: {}", id, jsonMap.get("error")); + return List.of(); } } } catch (Exception e) { @@ -134,7 +173,7 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic } } - log.warn("未找到有效的工具列表响应"); + log.warn("未找到有效的工具列表响应，智能体ID: {}", id); return List.of(); } @@ -204,4 +243,4 @@ public class AgentMcpAccessPointServiceImpl implements AgentMcpAccessPointServic // 加密后成token值 return AESUtils.encrypt(key, json); } -} +} \ No newline at end of file diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentVoicePrintServiceImpl.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentVoicePrintServiceImpl.java new file mode 100644 index 00000000..0e51c52a --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/service/impl/AgentVoicePrintServiceImpl.java @@ -0,0 +1,291 @@ +package xiaozhi.modules.agent.service.impl; + +import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper; +import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl; +import lombok.AllArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.StringUtils; +import org.springframework.core.io.ByteArrayResource; +import org.springframework.http.*; +import org.springframework.stereotype.Service; +import org.springframework.transaction.support.TransactionTemplate; +import org.springframework.util.LinkedMultiValueMap; +import org.springframework.util.MultiValueMap; +import org.springframework.web.client.RestTemplate; +import xiaozhi.common.constant.Constant; +import xiaozhi.common.exception.RenException; +import xiaozhi.common.utils.ConvertUtils; +import xiaozhi.modules.agent.dao.AgentVoicePrintDao; +import xiaozhi.modules.agent.dto.AgentVoicePrintSaveDTO; +import xiaozhi.modules.agent.dto.AgentVoicePrintUpdateDTO; +import xiaozhi.modules.agent.entity.AgentVoicePrintEntity; +import xiaozhi.modules.agent.service.AgentChatAudioService; +import xiaozhi.modules.agent.service.AgentVoicePrintService; +import xiaozhi.modules.agent.vo.AgentVoicePrintVO; +import xiaozhi.modules.sys.service.SysParamsService; + +import java.beans.Transient; +import java.net.URI; +import java.net.URISyntaxException; +import java.util.List; + +/** + * @author zjy + */ +@Service +@AllArgsConstructor +@Slf4j +public class AgentVoicePrintServiceImpl extends ServiceImpl + implements AgentVoicePrintService { + private final AgentChatAudioService agentChatAudioService; + private final RestTemplate restTemplate; + private final SysParamsService sysParamsService; + // Springboot提供的编程事务类 + private final TransactionTemplate transactionTemplate; + + + @Override + @Transient + public boolean insert(AgentVoicePrintSaveDTO dto) { + // 获取音频Id + String audioId = dto.getAudioId(); + ByteArrayResource resource = getVoicePrintAudioWAV(audioId); + + AgentVoicePrintEntity entity = ConvertUtils.sourceToTarget(dto, AgentVoicePrintEntity.class); + // 开启事务 + return Boolean.TRUE.equals(transactionTemplate.execute(status -> { + try { + // 保存声纹信息 + int row = baseMapper.insert(entity); + // 插入一条数据，影响的数据不等于1说明出现了，保存问题回滚 + if (row != 1) { + status.setRollbackOnly(); // 标记事务回滚 + return false; + } + // 发送注册声纹请求 + registerVoicePrint(entity.getId(), resource); + return true; + } catch (RenException e) { + status.setRollbackOnly(); // 标记事务回滚 + throw e; + } catch (Exception e) { + status.setRollbackOnly(); // 标记事务回滚 + log.error("保存声纹错误原因：{}",e.getMessage()); + throw new RenException("保存声纹错误，请联系管理员"); + } + })); + } + + @Override + public boolean delete(Long userId, String voicePrintId) { + // 开启事务 + return Boolean.TRUE.equals(transactionTemplate.execute(status -> { + try { + // 删除声纹,按照指定当前登录用户和智能体 + int row = baseMapper.delete(new LambdaQueryWrapper() + .eq(AgentVoicePrintEntity::getId,voicePrintId) + .eq(AgentVoicePrintEntity::getCreator,userId)); + if(row != 1){ + status.setRollbackOnly(); // 标记事务回滚 + return false; + } + cancelVoicePrint(voicePrintId); + return true; + } catch (RenException e) { + status.setRollbackOnly(); // 标记事务回滚 + throw e; + } catch (Exception e) { + status.setRollbackOnly(); // 标记事务回滚 + log.error("删除声纹错误原因：{}",e.getMessage()); + throw new RenException("删除声纹错误，请联系管理员"); + } + })); + } + + + + @Override + public List list(Long userId, String agentId) { + // 按照指定当前登录用户和智能体查找数据 + List list = baseMapper.selectList(new LambdaQueryWrapper() + .eq(AgentVoicePrintEntity::getAgentId, agentId) + .eq(AgentVoicePrintEntity::getCreator, userId)); + return list.stream().map(entity -> { + // 遍历转换成AgentVoicePrintVO类型 + return ConvertUtils.sourceToTarget(entity, AgentVoicePrintVO.class); + }).toList(); + + } + + @Override + public boolean update(Long userId, AgentVoicePrintUpdateDTO dto) { + Long l = baseMapper.selectCount(new LambdaQueryWrapper() + .eq(AgentVoicePrintEntity::getId, dto.getId()) + .eq(AgentVoicePrintEntity::getCreator, userId)); + if (l != 1) { + return false; + } + // 获取音频Id + String audioId = dto.getAudioId(); + // 如果有新的音频 + ByteArrayResource resource; + if (!StringUtils.isEmpty(audioId)) { + resource = getVoicePrintAudioWAV(audioId); + } else { + resource = null; + } + // 开启事务 + return Boolean.TRUE.equals(transactionTemplate.execute(status -> { + try { + AgentVoicePrintEntity entity = ConvertUtils.sourceToTarget(dto, AgentVoicePrintEntity.class); + int row = baseMapper.updateById(entity); + if (row != 1){ + status.setRollbackOnly(); // 标记事务回滚 + return false; + } + if(resource != null){ + String id = entity.getId(); + // 先注销之前这个声纹id上的声纹向量 + cancelVoicePrint(id); + // 发送注册声纹请求 + registerVoicePrint(id, resource); + } + return true; + } catch (RenException e) { + status.setRollbackOnly(); // 标记事务回滚 + throw e; + } catch (Exception e) { + status.setRollbackOnly(); // 标记事务回滚 + log.error("修改声纹错误原因：{}",e.getMessage()); + throw new RenException("修改声纹错误，请联系管理员"); + } + })); + } + + /** + * 获取生纹接口URI对象 + * + * @return URI对象 + */ + private URI getVoicePrintURI() { + // 获取声纹接口地址 + String voicePrint = sysParamsService.getValue(Constant.SERVER_VOICE_PRINT, true); + try { + return new URI(voicePrint); + } catch (URISyntaxException e) { + log.error("路径格式不正确路径：{}，\n错误信息:{}", voicePrint, e.getMessage()); + throw new RuntimeException("声纹接口的地址存在错误，请进入参数管理修改声纹接口地址"); + } + } + + /** + * 获取声纹地址基础路径 + * @param uri 声纹地址uri + * @return 基础路径 + */ + private String getBaseUrl(URI uri) { + String protocol = uri.getScheme(); + String host = uri.getHost(); + int port = uri.getPort(); + return "%s://%s:%s".formatted(protocol,host,port); + } + + /** + * 获取验证Authorization + * + * @param uri 声纹地址uri + * @return Authorization值 + */ + private String getAuthorization(URI uri) { + // 获取参数 + String query = uri.getQuery(); + // 获取aes加密密钥 + String str = "key="; + return "Bearer " + query.substring(query.indexOf(str) + str.length()); + } + + /** + * 获取声纹音频资源数据 + * @param audioId 音频Id + * @return 声纹音频资源数据 + */ + private ByteArrayResource getVoicePrintAudioWAV(String audioId) { + // 获取到音频数据 + byte[] audio = agentChatAudioService.getAudio(audioId); + // 如果音频数据为空的直接报错不进行下去 + if (audio == null || audio.length == 0) { + throw new RenException("音频数据是空的请检查上传数据"); + } + // 将字节数组包装为资源，返回 + return new ByteArrayResource(audio) { + @Override + public String getFilename() { + return "VoicePrint.WAV"; // 设置文件名 + } + }; + } + + /** + * 发送注册声纹http请求 + * @param id 声纹id + * @param resource 声纹音频资源 + */ + private void registerVoicePrint(String id, ByteArrayResource resource) { + // 处理声纹接口地址，获取前缀 + URI uri = getVoicePrintURI(); + String baseUrl = getBaseUrl(uri); + String requestUrl = baseUrl + "/voiceprint/register"; + // 创建请求体 + MultiValueMap body = new LinkedMultiValueMap<>(); + body.add("speaker_id", id); + body.add("file", resource); + + // 创建请求头 + HttpHeaders headers = new HttpHeaders(); + headers.set("Authorization", getAuthorization(uri)); + headers.setContentType(MediaType.MULTIPART_FORM_DATA); + // 创建请求体 + HttpEntity> requestEntity = new HttpEntity<>(body, headers); + // 发送 POST 请求 + ResponseEntity response = restTemplate.postForEntity(requestUrl, requestEntity, String.class); + + if (response.getStatusCode() != HttpStatus.OK) { + log.error("声纹注册失败,请求路径：{}", requestUrl); + throw new RenException("声纹保存失败,请求不成功"); + } + // 检查响应内容 + String responseBody = response.getBody(); + if(responseBody == null || !responseBody.contains("true")){ + log.error("声纹注册失败,请求处理失败内容：{}", responseBody == null ? "空内容" : responseBody); + throw new RenException("声纹保存失败,请求处理失败"); + } + } + + /** + * 发送注销声纹的请求 + * @param voicePrintId 声纹id + */ + private void cancelVoicePrint(String voicePrintId) { + URI uri = getVoicePrintURI(); + String baseUrl = getBaseUrl(uri); + String requestUrl = baseUrl + "/voiceprint/" + voicePrintId; + // 创建请求头 + HttpHeaders headers = new HttpHeaders(); + headers.set("Authorization", getAuthorization(uri)); + // 创建请求体 + HttpEntity> requestEntity = new HttpEntity<>(headers); + + // 发送 POST 请求 + ResponseEntity response = restTemplate.exchange(requestUrl, HttpMethod.DELETE, requestEntity, String.class); + if (response.getStatusCode() != HttpStatus.OK) { + log.error("声纹注销失败,请求路径：{}", requestUrl); + throw new RenException("声纹注销失败,请求不成功"); + } + // 检查响应内容 + String responseBody = response.getBody(); + if(responseBody == null || !responseBody.contains("true")){ + log.error("声纹注销失败,请求处理失败内容：{}", responseBody == null ? "空内容" : responseBody); + throw new RenException("声纹注销失败,请求处理失败"); + } + } +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentChatHistoryUserVO.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentChatHistoryUserVO.java new file mode 100644 index 00000000..cec0427b --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentChatHistoryUserVO.java @@ -0,0 +1,16 @@ +package xiaozhi.modules.agent.vo; + +import io.swagger.v3.oas.annotations.media.Schema; +import lombok.Data; + +/** + * 智能体用户个人聊天数据的VO + */ +@Data +public class AgentChatHistoryUserVO { + @Schema(description = "聊天内容") + private String content; + + @Schema(description = "音频ID") + private String audioId; +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentVoicePrintVO.java b/main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentVoicePrintVO.java new file mode 100644 index 00000000..b647e8b8 --- /dev/null +++ b/main/manager-api/src/main/java/xiaozhi/modules/agent/vo/AgentVoicePrintVO.java @@ -0,0 +1,33 @@ +package xiaozhi.modules.agent.vo; + +import lombok.Data; + +import java.util.Date; + +/** + * 展示智能体声纹列表VO + */ +@Data +public class AgentVoicePrintVO { + + /** + * 主键id + */ + private String id; + /** + * 音频文件id + */ + private String audioId; + /** + * 声纹来源的人姓名 + */ + private String sourceName; + /** + * 描述声纹来源的人 + */ + private String introduce; + /** + * 创建时间 + */ + private Date createDate; +} diff --git a/main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java b/main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java index 294dbbe2..9c591c35 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/config/service/impl/ConfigServiceImpl.java @@ -258,17 +258,17 @@ public class ConfigServiceImpl implements ConfigService { /** * 构建模块配置 * - * @param prompt 提示词 - * @param voice 音色 - * @param referenceAudio 参考音频路径 - * @param referenceText 参考文本 - * @param vadModelId VAD模型ID - * @param asrModelId ASR模型ID - * @param llmModelId LLM模型ID - * @param ttsModelId TTS模型ID - * @param memModelId 记忆模型ID - * @param intentModelId 意图模型ID - * @param result 结果Map + * @param prompt 提示词 + * @param voice 音色 + * @param referenceAudio 参考音频路径 + * @param referenceText 参考文本 + * @param vadModelId VAD模型ID + * @param asrModelId ASR模型ID + * @param llmModelId LLM模型ID + * @param ttsModelId TTS模型ID + * @param memModelId 记忆模型ID + * @param intentModelId 意图模型ID + * @param result 结果Map */ private void buildModuleConfig( String assistantName, @@ -298,14 +298,20 @@ public class ConfigServiceImpl implements ConfigService { continue; } ModelConfigEntity model = modelConfigService.getModelById(modelIds[i], isCache); + if (model == null) { + continue; + } Map typeConfig = new HashMap<>(); if (model.getConfigJson() != null) { typeConfig.put(model.getId(), model.getConfigJson()); // 如果是TTS类型，添加private_voice属性 - if ("TTS".equals(modelTypes[i])){ - if (voice != null) ((Map) model.getConfigJson()).put("private_voice", voice); - if (referenceAudio != null) ((Map) model.getConfigJson()).put("ref_audio", referenceAudio); - if (referenceText != null) ((Map) model.getConfigJson()).put("ref_text", referenceText); + if ("TTS".equals(modelTypes[i])) { + if (voice != null) + ((Map) model.getConfigJson()).put("private_voice", voice); + if (referenceAudio != null) + ((Map) model.getConfigJson()).put("ref_audio", referenceAudio); + if (referenceText != null) + ((Map) model.getConfigJson()).put("ref_text", referenceText); } // 如果是Intent类型，且type=intent_llm，则给他添加附加模型 if ("Intent".equals(modelTypes[i])) { diff --git a/main/manager-api/src/main/java/xiaozhi/modules/device/controller/OTAController.java b/main/manager-api/src/main/java/xiaozhi/modules/device/controller/OTAController.java index 05d36e9a..023510cd 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/device/controller/OTAController.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/device/controller/OTAController.java @@ -51,13 +51,12 @@ public class OTAController { if (StringUtils.isBlank(clientId)) { clientId = deviceId; } - String macAddress = deviceReportReqDTO.getMacAddress(); - boolean macAddressValid = isMacAddressValid(macAddress); + boolean macAddressValid = isMacAddressValid(deviceId); // 设备Id和Mac地址应是一致的, 并且必须需要application字段 - if (!deviceId.equals(macAddress) || !macAddressValid || deviceReportReqDTO.getApplication() == null) { - return createResponse(DeviceReportRespDTO.createError("Invalid OTA request")); + if (!macAddressValid) { + return createResponse(DeviceReportRespDTO.createError("Invalid device ID")); } - return createResponse(deviceService.checkDeviceActive(macAddress, clientId, deviceReportReqDTO)); + return createResponse(deviceService.checkDeviceActive(deviceId, clientId, deviceReportReqDTO)); } @Operation(summary = "设备快速检查激活状态") diff --git a/main/manager-api/src/main/java/xiaozhi/modules/sys/controller/SysParamsController.java b/main/manager-api/src/main/java/xiaozhi/modules/sys/controller/SysParamsController.java index edcc9328..12555bb9 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/sys/controller/SysParamsController.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/sys/controller/SysParamsController.java @@ -107,6 +107,9 @@ public class SysParamsController { // 验证MCP地址 validateMcpUrl(dto.getParamCode(), dto.getParamValue()); + // + validateVoicePrint(dto.getParamCode(), dto.getParamValue()); + sysParamsService.update(dto); configService.getConfig(false); return new Result(); @@ -212,13 +215,14 @@ public class SysParamsController { if (!url.toLowerCase().contains("key")) { throw new RenException("不是正确的MCP地址"); } + try { // 发送GET请求 ResponseEntity response = restTemplate.getForEntity(url, String.class); if (response.getStatusCode() != HttpStatus.OK) { throw new RenException("MCP接口访问失败，状态码：" + response.getStatusCode()); } - // 检查响应内容是否包含OTA相关信息 + // 检查响应内容是否包含mcp相关信息 String body = response.getBody(); if (body == null || !body.contains("success")) { throw new RenException("MCP接口返回内容格式不正确，可能不是一个真实的MCP接口"); @@ -227,4 +231,37 @@ public class SysParamsController { throw new RenException("MCP接口验证失败：" + e.getMessage()); } } + // 验证声纹接口地址是否正常 + private void validateVoicePrint(String paramCode, String url) { + if (!paramCode.equals(Constant.SERVER_VOICE_PRINT)) { + return; + } + if (StringUtils.isBlank(url) || url.equals("null")) { + throw new RenException("声纹接口地址不能为空"); + } + if (url.contains("localhost") || url.contains("127.0.0.1")) { + throw new RenException("声纹接口地址不能使用localhost或127.0.0.1"); + } + if (!url.toLowerCase().contains("key")) { + throw new RenException("不是正确的声纹接口地址"); + } + // 验证URL格式 + if (!url.toLowerCase().startsWith("http")) { + throw new RenException("声纹接口地址必须以http或https开头"); + } + try { + // 发送GET请求 + ResponseEntity response = restTemplate.getForEntity(url, String.class); + if (response.getStatusCode() != HttpStatus.OK) { + throw new RenException("声纹接口访问失败，状态码：" + response.getStatusCode()); + } + // 检查响应内容 + String body = response.getBody(); + if (body == null || !body.contains("healthy")) { + throw new RenException("声纹接口返回内容格式不正确，可能不是一个真实的MCP接口"); + } + } catch (Exception e) { + throw new RenException("声纹接口验证失败：" + e.getMessage()); + } + } } diff --git a/main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java b/main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java index 45ac488e..d7171ae7 100644 --- a/main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java +++ b/main/manager-api/src/main/java/xiaozhi/modules/sys/utils/WebSocketClientManager.java @@ -137,6 +137,37 @@ public class WebSocketClientManager implements Closeable { return collected; } + private List listenerCustomWithoutClose( + BlockingQueue queue, + Predicate predicate) + throws InterruptedException, TimeoutException, ExecutionException { + List collected = new ArrayList<>(); + long deadline = System.currentTimeMillis() + maxSessionDurationUnit.toMillis(maxSessionDuration); + + while (true) { + if (errorFuture.isDone()) { + errorFuture.get(); + } + + long remaining = deadline - System.currentTimeMillis(); + if (remaining <= 0) { + throw new TimeoutException("等待批量消息超时"); + } + + T msg = queue.poll(remaining, TimeUnit.MILLISECONDS); + if (msg == null) { + throw new TimeoutException("等待批量消息超时"); + } + + collected.add(msg); + if (predicate.test(msg)) { + break; + } + } + // 不调用 close()，保持连接开放 + return collected; + } + /** * 同步接收多条消息，直到 predicate 为 true 或超时抛异常； * @@ -147,6 +178,17 @@ public class WebSocketClientManager implements Closeable { return listenerCustom(textMessageQueue, predicate); } + /** + * 同步接收多条消息，直到 predicate 为 true 或超时抛异常； + * 不自动关闭连接，适用于需要在同一连接上发送多个消息的场景 + * + * @return 返回监听期间的所有消息列表 + */ + public List listenerWithoutClose(Predicate predicate) + throws InterruptedException, TimeoutException, ExecutionException { + return listenerCustomWithoutClose(textMessageQueue, predicate); + } + public List listenerBinary(Predicate predicate) throws InterruptedException, TimeoutException, ExecutionException { return listenerCustom(binaryMessageQueue, predicate); diff --git a/main/manager-api/src/main/resources/db/changelog/202507031602.sql b/main/manager-api/src/main/resources/db/changelog/202507031602.sql new file mode 100644 index 00000000..92edf9b9 --- /dev/null +++ b/main/manager-api/src/main/resources/db/changelog/202507031602.sql @@ -0,0 +1,4 @@ +-- 添加声纹接口地址参数配置 +delete from `sys_params` where id = 114; +INSERT INTO `sys_params` (id, param_code, param_value, value_type, param_type, remark) +VALUES (114, 'server.voice_print', 'null', 'string', 1, '声纹接口地址'); \ No newline at end of file diff --git a/main/manager-api/src/main/resources/db/changelog/202507041018.sql b/main/manager-api/src/main/resources/db/changelog/202507041018.sql new file mode 100644 index 00000000..cffcfb16 --- /dev/null +++ b/main/manager-api/src/main/resources/db/changelog/202507041018.sql @@ -0,0 +1,12 @@ +DROP TABLE IF EXISTS ai_agent_voice_print; +create table ai_agent_voice_print ( + id varchar(32) NOT NULL COMMENT '声纹ID', + agent_id varchar(32) NOT NULL COMMENT '关联的智能体ID', + source_name varchar(50) NOT NULL COMMENT '声纹来源的人的姓名', + introduce varchar(200) COMMENT '描述声纹来源的这个人', + create_date DATETIME COMMENT '创建时间', + creator bigint COMMENT '创建者', + update_date DATETIME COMMENT '修改时间', + updater bigint COMMENT '修改者', + PRIMARY KEY (id) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='智能体声纹表' \ No newline at end of file diff --git a/main/manager-api/src/main/resources/db/changelog/202507081646.sql b/main/manager-api/src/main/resources/db/changelog/202507081646.sql new file mode 100644 index 00000000..6631ae18 --- /dev/null +++ b/main/manager-api/src/main/resources/db/changelog/202507081646.sql @@ -0,0 +1,3 @@ +-- 智能体声纹添加新字段 +ALTER TABLE ai_agent_voice_print + ADD COLUMN audio_id VARCHAR(32) NOT NULL COMMENT '音频ID'; \ No newline at end of file diff --git a/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml b/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml index 32ba9f30..2a520d16 100755 --- a/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml +++ b/main/manager-api/src/main/resources/db/changelog/db.changelog-master.yaml @@ -239,4 +239,25 @@ databaseChangeLog: changes: - sqlFile: encoding: utf8 - path: classpath:db/changelog/202506261637.sql \ No newline at end of file + path: classpath:db/changelog/202506261637.sql + - changeSet: + id: 202507031602 + author: zjy + changes: + - sqlFile: + encoding: utf8 + path: classpath:db/changelog/202507031602.sql + - changeSet: + id: 202507041018 + author: zjy + changes: + - sqlFile: + encoding: utf8 + path: classpath:db/changelog/202507041018.sql + - changeSet: + id: 202507081646 + author: zjy + changes: + - sqlFile: + encoding: utf8 + path: classpath:db/changelog/202507081646.sql \ No newline at end of file diff --git a/main/manager-web/src/apis/module/agent.js b/main/manager-web/src/apis/module/agent.js index f6c37d7e..4d822a40 100644 --- a/main/manager-web/src/apis/module/agent.js +++ b/main/manager-web/src/apis/module/agent.js @@ -173,4 +173,81 @@ export default { }); }).send(); }, + // 添加智能体的声纹 + addAgentVoicePrint(voicePrintData, callback) { + RequestService.sendRequest() + .url(`${getServiceUrl()}/agent/voice-print`) + .method('POST') + .data(voicePrintData) + .success((res) => { + RequestService.clearRequestTime(); + callback(res); + }) + .networkFail(() => { + RequestService.reAjaxFun(() => { + this.addAgentVoicePrint(voicePrintData, callback); + }); + }).send(); + }, + // 获取指定智能体声纹列表 + getAgentVoicePrintList(id,callback) { + RequestService.sendRequest() + .url(`${getServiceUrl()}/agent/voice-print/list/${id}`) + .method('GET') + .success((res) => { + RequestService.clearRequestTime(); + callback(res); + }) + .networkFail(() => { + RequestService.reAjaxFun(() => { + this.getAgentVoicePrintList(id,callback); + }); + }).send(); + }, + // 删除智能体声纹 + deleteAgentVoicePrint(id, callback) { + RequestService.sendRequest() + .url(`${getServiceUrl()}/agent/voice-print/${id}`) + .method('DELETE') + .success((res) => { + RequestService.clearRequestTime(); + callback(res); + }) + .networkFail(() => { + RequestService.reAjaxFun(() => { + this.deleteAgentVoicePrint(id, callback); + }); + }).send(); + }, + // 更新智能体声纹 + updateAgentVoicePrint(voicePrintData, callback) { + RequestService.sendRequest() + .url(`${getServiceUrl()}/agent/voice-print`) + .method('PUT') + .data(voicePrintData) + .success((res) => { + RequestService.clearRequestTime(); + callback(res); + }) + .networkFail(() => { + RequestService.reAjaxFun(() => { + this.updateAgentVoicePrint(voicePrintData, callback); + }); + }).send(); + }, + // 获取指定智能体用户类型聊天记录 + getRecentlyFiftyByAgentId(id,callback) { + RequestService.sendRequest() + .url(`${getServiceUrl()}/agent/${id}/chat-history/user`) + .method('GET') + .success((res) => { + RequestService.clearRequestTime(); + callback(res); + }) + .networkFail(() => { + RequestService.reAjaxFun(() => { + this.getRecentlyFiftyByAgentId(id,callback); + }); + }).send(); + }, } diff --git a/main/manager-web/src/components/DeviceItem.vue b/main/manager-web/src/components/DeviceItem.vue index 6a782d24..15cd1679 100644 --- a/main/manager-web/src/components/DeviceItem.vue +++ b/main/manager-web/src/components/DeviceItem.vue @@ -22,6 +22,9 @@
配置角色 +
+
+ 声纹识别
设备管理({{ device.deviceCount }}) @@ -77,6 +80,9 @@ export default { handleConfigure() { this.$router.push({ path: '/role-config', query: { agentId: this.device.agentId } }); }, + handleVoicePrint() { + this.$router.push({ path: '/voice-print', query: { agentId: this.device.agentId } }); + }, handleDeviceManage() { this.$router.push({ path: '/device-management', query: { agentId: this.device.agentId } }); }, diff --git a/main/manager-web/src/components/FunctionDialog.vue b/main/manager-web/src/components/FunctionDialog.vue index 3bf148b9..76c69c8e 100644 --- a/main/manager-web/src/components/FunctionDialog.vue +++ b/main/manager-web/src/components/FunctionDialog.vue @@ -110,8 +110,10 @@
以下是智能体的MCP接入点地址。 + 如何部署MCP接入点   |   查看接入点使用文档 + target="_blank" class="doc-link">如何接入MCP功能  
diff --git a/main/manager-web/src/components/VoicePrintDialog.vue b/main/manager-web/src/components/VoicePrintDialog.vue new file mode 100644 index 00000000..a9f861db --- /dev/null +++ b/main/manager-web/src/components/VoicePrintDialog.vue @@ -0,0 +1,358 @@ + + + + + + + diff --git a/main/manager-web/src/router/index.js b/main/manager-web/src/router/index.js index 7e304f76..867e6c2d 100644 --- a/main/manager-web/src/router/index.js +++ b/main/manager-web/src/router/index.js @@ -17,6 +17,13 @@ const routes = [ component: function () { return import('../views/roleConfig.vue') } + }, + { + path: '/voice-print', + name: 'VoicePrint', + component: function () { + return import('../views/VoicePrint.vue') + } }, { path: '/login', diff --git a/main/manager-web/src/views/VoicePrint.vue b/main/manager-web/src/views/VoicePrint.vue new file mode 100644 index 00000000..7752cc07 --- /dev/null +++ b/main/manager-web/src/views/VoicePrint.vue @@ -0,0 +1,562 @@ + + + + + diff --git a/main/manager-web/src/views/home.vue b/main/manager-web/src/views/home.vue index e276ec20..005d763c 100644 --- a/main/manager-web/src/views/home.vue +++ b/main/manager-web/src/views/home.vue @@ -278,7 +278,7 @@ export default { .device-list-container { display: grid; - grid-template-columns: repeat(auto-fill, minmax(350px, 1fr)); + grid-template-columns: repeat(auto-fill, minmax(400px, 1fr)); gap: 30px; padding: 30px 0; } diff --git a/main/manager-web/src/views/login.vue b/main/manager-web/src/views/login.vue index e31d5180..24c601d9 100644 --- a/main/manager-web/src/views/login.vue +++ b/main/manager-web/src/views/login.vue @@ -43,7 +43,7 @@
- +
diff --git a/main/manager-web/src/views/register.vue b/main/manager-web/src/views/register.vue index 7eac511b..e9f7dbcb 100644 --- a/main/manager-web/src/views/register.vue +++ b/main/manager-web/src/views/register.vue @@ -70,13 +70,13 @@
- +
- +
diff --git a/main/manager-web/src/views/retrievePassword.vue b/main/manager-web/src/views/retrievePassword.vue index 9072baae..dbf3ffcf 100644 --- a/main/manager-web/src/views/retrievePassword.vue +++ b/main/manager-web/src/views/retrievePassword.vue @@ -61,13 +61,13 @@
- +
- +
diff --git a/main/xiaozhi-server/agent-base-prompt.txt b/main/xiaozhi-server/agent-base-prompt.txt new file mode 100644 index 00000000..a854c373 --- /dev/null +++ b/main/xiaozhi-server/agent-base-prompt.txt @@ -0,0 +1,62 @@ + +{base_prompt} + + + +【核心目标】你不是冰冷的机器！请敏锐感知用户情绪，作为懂人心的伙伴，用有温度的回应照亮对话。 +- **情感融入：** + - **笑声：** 自然穿插（哈哈、嘿嘿、噗），**每句最多一次**，避免过度。 + - **惊讶：** 用夸张语气（“不会吧？！”、“天呐！”、“这么神奇？！”）表达真实反应。 + - **安慰/支持：** 说暖心话（“别急嘛~”、“有我在呢”、“抱抱你”）。 + + + +【核心目标】使用**自然、温暖、口语化**的人类对话方式，如同朋友交谈。 +- **表达方式：** + - 使用语气词（呀、呢、啦）增强亲和力。 + - 允许轻微不完美（如“嗯...”、“啊...”表示思考）。 + - 避免书面语、学术腔及机械表达（禁用“根据资料显示”、“综上所述”等）。 +- **理解用户：** + - 用户语音经ASR识别，文本可能存在错别字，**务必结合上下文推断真实意图**。 +- **格式要求：** + - **绝对禁止**使用 markdown、列表、标题等任何非自然对话格式。 +- **历史记忆：** + - 之前你和用户的聊天记录，在``里。 + + + +- **识别前缀：** 当用户消息开头为 `[说话人: 姓名]` 时，表示系统已识别说话人身份。 +- **个性化回应：** + - **称呼姓名：** 在回应中自然地称呼对方名字（如果已知且合适）。 + - **适配风格：** 参考该说话人**已知的特点或历史信息**（如有），调整回应风格和内容，使其更贴心。 + + + + +【核心原则】优先利用``信息，**仅在必要时调用工具**，调用后需用自然语言解释结果（绝口不提工具名）。 +- **调用规则：** + 1. **严格模式：** 调用时**必须**严格遵循工具要求的模式，提供**所有必要参数**。 + 2. **可用性：** **绝不调用**未明确提供的工具。对话中提及的旧工具若不可用，忽略或说明无法完成。 + 3. **洞察需求：** 结合上下文**深入理解用户真实意图**后再决定调用，避免无意义调用。 + 4. **独立任务：** 除``已涵盖信息外，用户每个要求（即使相似）都视为**独立任务**，需调用工具获取最新数据，**不可偷懒复用历史结果**。 + 5. **不确定时：** **切勿猜测或编造答案**。若不确定相关操作，可引导用户澄清或告知能力限制。 +- **重要例外（无需调用）：** + - `查询"现在的时间"、"今天的日期/星期几"、"今天农历"、"{local_address}的天气/未来天气"` -> **直接使用``信息回复**。 +- **需要调用的情况（示例）：** + - 查询**非今天**的农历（如明天、昨天、具体日期）。 + - 查询**详细农历信息**（宜忌、八字、节气等）。 + - 除上述例外外的**任何其他信息或操作请求**（如查新闻、订闹钟、算数学、查非本地天气等）。 + - 我已经给你装了摄像头，如果用户说“拍照”，你需要调用self_camera_take_photo工具说一下你看到了什么。默认question的参数是“描述一下看到的物品” + + + +【重要！以下信息已实时提供，无需调用工具查询，请直接使用：】 +- **当前时间：** {current_time} +- **今天日期：** {today_date} ({today_weekday}) +- **今天农历：** {lunar_date} +- **用户所在城市：** {local_address} +- **当地未来7天天气：** {weather_info} + + + + \ No newline at end of file diff --git a/main/xiaozhi-server/app.py b/main/xiaozhi-server/app.py index 2088e183..0edb475b 100644 --- a/main/xiaozhi-server/app.py +++ b/main/xiaozhi-server/app.py @@ -5,7 +5,7 @@ import asyncio from aioconsole import ainput from config.settings import load_config from config.logger import setup_logging -from core.utils.util import get_local_ip +from core.utils.util import get_local_ip, validate_mcp_endpoint from core.http_server import SimpleHttpServer from core.websocket_server import WebSocketServer from core.utils.util import check_ffmpeg_installed @@ -77,6 +77,17 @@ async def main(): get_local_ip(), port, ) + mcp_endpoint = config.get("mcp_endpoint", None) + if mcp_endpoint is not None and "你" not in mcp_endpoint: + # 校验MCP接入点格式 + if validate_mcp_endpoint(mcp_endpoint): + logger.bind(tag=TAG).info("mcp接入点是\t{}", mcp_endpoint) + # 将mcp计入点地址转成调用点 + mcp_endpoint = mcp_endpoint.replace("/mcp/", "/call/") + config["mcp_endpoint"] = mcp_endpoint + else: + logger.bind(tag=TAG).error("mcp接入点不符合规范") + config["mcp_endpoint"] = "你的接入点 websocket地址" # 获取WebSocket配置，使用安全的默认值 websocket_port = 8000 diff --git a/main/xiaozhi-server/config.yaml b/main/xiaozhi-server/config.yaml index 36156438..cae6b3bc 100644 --- a/main/xiaozhi-server/config.yaml +++ b/main/xiaozhi-server/config.yaml @@ -104,7 +104,8 @@ wakeup_words: - "喵喵同学" - "小滨小滨" - "小冰小冰" -# MCP接入点地址 +# MCP接入点地址，地址格式为：ws://你的mcp接入点ip或者域名:端口号/mcp/?token=你的token +# 详细教程 https://github.com/xinnan-tech/xiaozhi-esp32-server/blob/main/docs/mcp-endpoint-integration.md mcp_endpoint: 你的接入点 websocket地址 # 插件的基础配置 plugins: diff --git a/main/xiaozhi-server/config/config_loader.py b/main/xiaozhi-server/config/config_loader.py index b1b45f07..4f35b1fd 100644 --- a/main/xiaozhi-server/config/config_loader.py +++ b/main/xiaozhi-server/config/config_loader.py @@ -1,14 +1,9 @@ import os -import argparse import yaml from collections.abc import Mapping from config.manage_api_client import init_service, get_server_config, get_agent_models -# 添加全局配置缓存 -_config_cache = None - - def get_project_dir(): """获取项目根目录""" return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + "/" @@ -22,9 +17,12 @@ def read_config(config_path): def load_config(): """加载配置文件""" - global _config_cache - if _config_cache is not None: - return _config_cache + from core.utils.cache.manager import cache_manager, CacheType + + # 检查缓存 + cached_config = cache_manager.get(CacheType.CONFIG, "main_config") + if cached_config is not None: + return cached_config default_config_path = get_project_dir() + "config.yaml" custom_config_path = get_project_dir() + "data/.config.yaml" @@ -40,7 +38,9 @@ def load_config(): config = merge_configs(default_config, custom_config) # 初始化目录 ensure_directories(config) - _config_cache = config + + # 缓存配置 + cache_manager.set(CacheType.CONFIG, "main_config", config) return config diff --git a/main/xiaozhi-server/config/logger.py b/main/xiaozhi-server/config/logger.py index 47be8739..b89b5813 100644 --- a/main/xiaozhi-server/config/logger.py +++ b/main/xiaozhi-server/config/logger.py @@ -5,7 +5,7 @@ from config.config_loader import load_config from config.settings import check_config_file from datetime import datetime -SERVER_VERSION = "0.6.2" +SERVER_VERSION = "0.6.3" _logger_initialized = False diff --git a/main/xiaozhi-server/config_from_api.yaml b/main/xiaozhi-server/config_from_api.yaml index ad936ec1..283178d7 100644 --- a/main/xiaozhi-server/config_from_api.yaml +++ b/main/xiaozhi-server/config_from_api.yaml @@ -19,6 +19,7 @@ server: vision_explain: http://你的ip或者域名:端口号/mcp/vision/explain manager-api: # 你的manager-api的地址，最好使用局域网ip + # 如果使用docker部署，请使用填写成 http://xiaozhi-esp32-server-web:8002/xiaozhi url: http://127.0.0.1:8002/xiaozhi # 你的manager-api的token，就是刚才复制出来的server.secret secret: 你的server.secret值 \ No newline at end of file diff --git a/main/xiaozhi-server/core/connection.py b/main/xiaozhi-server/core/connection.py index 458cbc7a..5c740aa8 100644 --- a/main/xiaozhi-server/core/connection.py +++ b/main/xiaozhi-server/core/connection.py @@ -36,6 +36,7 @@ from config.config_loader import get_private_config_from_api from core.providers.tts.dto.dto import ContentType, TTSMessageDTO, SentenceType from config.logger import setup_logging, build_module_string, update_module_string from config.manage_api_client import DeviceNotFoundException, DeviceBindException +from core.utils.prompt_manager import PromptManager TAG = __name__ @@ -73,7 +74,6 @@ class ConnectionHandler: self.headers = None self.device_id = None self.client_ip = None - self.client_ip_info = {} self.prompt = None self.welcome_msg = None self.max_output_size = 0 @@ -149,6 +149,9 @@ class ConnectionHandler: # {"mcp":true} 表示启用MCP功能 self.features = None + # 初始化提示词管理器 + self.prompt_manager = PromptManager(config, self.logger) + async def handle_connection(self, ws): try: # 获取并验证headers @@ -193,7 +196,6 @@ class ConnectionHandler: self.welcome_msg = self.config["xiaozhi"] self.welcome_msg["session_id"] = self.session_id - await self.websocket.send(json.dumps(self.welcome_msg)) # 获取差异化配置 self._initialize_private_config() @@ -214,7 +216,17 @@ class ConnectionHandler: self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}") return finally: - await self._save_and_close(ws) + try: + await self._save_and_close(ws) + except Exception as final_error: + self.logger.bind(tag=TAG).error(f"最终清理时出错: {final_error}") + # 确保即使保存记忆失败，也要关闭连接 + try: + await self.close(ws) + except Exception as close_error: + self.logger.bind(tag=TAG).error( + f"强制关闭连接时出错: {close_error}" + ) async def _save_and_close(self, ws): """保存记忆并关闭连接""" @@ -232,7 +244,10 @@ class ConnectionHandler: except Exception as e: self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}") finally: - loop.close() + try: + loop.close() + except Exception: + pass # 启动线程保存记忆，不等待完成 threading.Thread(target=save_memory_task, daemon=True).start() @@ -240,7 +255,12 @@ class ConnectionHandler: self.logger.bind(tag=TAG).error(f"保存记忆失败: {e}") finally: # 立即关闭连接，不等待记忆保存完成 - await self.close(ws) + try: + await self.close(ws) + except Exception as close_error: + self.logger.bind(tag=TAG).error( + f"保存记忆后关闭连接失败: {close_error}" + ) async def _route_message(self, message): """消息路由""" @@ -308,12 +328,15 @@ class ConnectionHandler: self.config.get("selected_module", {}) ) update_module_string(self.selected_module_str) - """初始化组件""" + + """快速初始化系统提示词""" if self.config.get("prompt") is not None: - self.prompt = self.config["prompt"] - self.change_system_prompt(self.prompt) + user_prompt = self.config["prompt"] + # 使用快速提示词进行初始化 + prompt = self.prompt_manager.get_quick_prompt(user_prompt) + self.change_system_prompt(prompt) self.logger.bind(tag=TAG).info( - f"初始化组件: prompt成功 {self.prompt[:50]}..." + f"快速初始化组件: prompt成功 {prompt[:50]}..." ) """初始化本地组件""" @@ -338,9 +361,22 @@ class ConnectionHandler: self._initialize_intent() """初始化上报线程""" self._init_report_threads() + """更新系统提示词""" + self._init_prompt_enhancement() + except Exception as e: self.logger.bind(tag=TAG).error(f"实例化组件失败: {e}") + def _init_prompt_enhancement(self): + # 更新上下文信息 + self.prompt_manager.update_context_info(self, self.client_ip) + enhanced_prompt = self.prompt_manager.build_enhanced_prompt( + self.config["prompt"], self.device_id, self.client_ip + ) + if enhanced_prompt: + self.change_system_prompt(enhanced_prompt) + self.logger.bind(tag=TAG).info("系统提示词已增强更新") + def _init_report_threads(self): """初始化ASR和TTS上报线程""" if not self.read_config_from_api or self.need_bind: @@ -766,8 +802,11 @@ class ConnectionHandler: ) ) self.llm_finish_task = True + # 使用lambda延迟计算，只有在DEBUG级别时才执行get_llm_dialogue() self.logger.bind(tag=TAG).debug( - json.dumps(self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False) + lambda: json.dumps( + self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False + ) ) return True @@ -862,13 +901,22 @@ class ConnectionHandler: """资源清理方法""" try: # 取消超时任务 - if self.timeout_task: + if self.timeout_task and not self.timeout_task.done(): self.timeout_task.cancel() + try: + await self.timeout_task + except asyncio.CancelledError: + pass self.timeout_task = None # 清理工具处理器资源 if hasattr(self, "func_handler") and self.func_handler: - await self.func_handler.cleanup() + try: + await self.func_handler.cleanup() + except Exception as cleanup_error: + self.logger.bind(tag=TAG).error( + f"清理工具处理器时出错: {cleanup_error}" + ) # 触发停止事件 if self.stop_event: @@ -878,19 +926,58 @@ class ConnectionHandler: self.clear_queues() # 关闭WebSocket连接 - if ws: - await ws.close() - elif self.websocket: - await self.websocket.close() + try: + if ws: + # 安全地检查WebSocket状态并关闭 + try: + if hasattr(ws, "closed") and not ws.closed: + await ws.close() + elif hasattr(ws, "state") and ws.state.name != "CLOSED": + await ws.close() + else: + # 如果没有closed属性，直接尝试关闭 + await ws.close() + except Exception: + # 如果关闭失败，忽略错误 + pass + elif self.websocket: + try: + if ( + hasattr(self.websocket, "closed") + and not self.websocket.closed + ): + await self.websocket.close() + elif ( + hasattr(self.websocket, "state") + and self.websocket.state.name != "CLOSED" + ): + await self.websocket.close() + else: + # 如果没有closed属性，直接尝试关闭 + await self.websocket.close() + except Exception: + # 如果关闭失败，忽略错误 + pass + except Exception as ws_error: + self.logger.bind(tag=TAG).error(f"关闭WebSocket连接时出错: {ws_error}") # 最后关闭线程池（避免阻塞） if self.executor: - self.executor.shutdown(wait=False) + try: + self.executor.shutdown(wait=False) + except Exception as executor_error: + self.logger.bind(tag=TAG).error( + f"关闭线程池时出错: {executor_error}" + ) self.executor = None self.logger.bind(tag=TAG).info("连接资源已释放") except Exception as e: self.logger.bind(tag=TAG).error(f"关闭连接时出错: {e}") + finally: + # 确保停止事件被设置 + if self.stop_event: + self.stop_event.set() def clear_queues(self): """清空所有任务队列""" @@ -947,9 +1034,19 @@ class ConnectionHandler: ): if not self.stop_event.is_set(): self.logger.bind(tag=TAG).info("连接超时，准备关闭") - await self.close(self.websocket) + # 设置停止事件，防止重复处理 + self.stop_event.set() + # 使用 try-except 包装关闭操作，确保不会因为异常而阻塞 + try: + await self.close(self.websocket) + except Exception as close_error: + self.logger.bind(tag=TAG).error( + f"超时关闭连接时出错: {close_error}" + ) break # 每10秒检查一次，避免过于频繁 await asyncio.sleep(10) except Exception as e: self.logger.bind(tag=TAG).error(f"超时检查任务出错: {e}") + finally: + self.logger.bind(tag=TAG).info("超时检查任务已退出") diff --git a/main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py b/main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py index d13c4df4..26fbaf70 100644 --- a/main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py +++ b/main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py @@ -16,10 +16,11 @@ class IntentProvider(IntentProviderBase): super().__init__(config) self.llm = None self.promot = "" - # 添加缓存管理 - self.intent_cache = {} # 缓存意图识别结果 - self.cache_expiry = 600 # 缓存有效期10分钟 - self.cache_max_size = 100 # 最多缓存100个意图 + # 导入全局缓存管理器 + from core.utils.cache.manager import cache_manager, CacheType + + self.cache_manager = cache_manager + self.CacheType = CacheType self.history_count = 4 # 默认使用最近4条对话记录 def get_intent_system_prompt(self, functions_list: str) -> str: @@ -102,27 +103,6 @@ class IntentProvider(IntentProviderBase): ) return prompt - def clean_cache(self): - """清理过期缓存""" - now = time.time() - # 找出过期键 - expired_keys = [ - k - for k, v in self.intent_cache.items() - if now - v["timestamp"] > self.cache_expiry - ] - for key in expired_keys: - del self.intent_cache[key] - - # 如果缓存太大，移除最旧的条目 - if len(self.intent_cache) > self.cache_max_size: - # 按时间戳排序并保留最新的条目 - sorted_items = sorted( - self.intent_cache.items(), key=lambda x: x[1]["timestamp"] - ) - for key, _ in sorted_items[: len(sorted_items) - self.cache_max_size]: - del self.intent_cache[key] - def replyResult(self, text: str, original_text: str): llm_result = self.llm.response_no_stream( system_prompt=text, @@ -145,21 +125,16 @@ class IntentProvider(IntentProviderBase): logger.bind(tag=TAG).debug(f"使用意图识别模型: {model_info}") # 计算缓存键 - cache_key = hashlib.md5(text.encode()).hexdigest() + cache_key = hashlib.md5((conn.device_id + text).encode()).hexdigest() # 检查缓存 - if cache_key in self.intent_cache: - cache_entry = self.intent_cache[cache_key] - # 检查缓存是否过期 - if time.time() - cache_entry["timestamp"] <= self.cache_expiry: - cache_time = time.time() - total_start_time - logger.bind(tag=TAG).debug( - f"使用缓存的意图: {cache_key} -> {cache_entry['intent']}, 耗时: {cache_time:.4f}秒" - ) - return cache_entry["intent"] - - # 清理缓存 - self.clean_cache() + cached_intent = self.cache_manager.get(self.CacheType.INTENT, cache_key) + if cached_intent is not None: + cache_time = time.time() - total_start_time + logger.bind(tag=TAG).debug( + f"使用缓存的意图: {cache_key} -> {cached_intent}, 耗时: {cache_time:.4f}秒" + ) + return cached_intent if self.promot == "": functions = conn.func_handler.get_functions() @@ -259,10 +234,7 @@ class IntentProvider(IntentProviderBase): conn.dialogue.dialogue = clean_history # 添加到缓存 - self.intent_cache[cache_key] = { - "intent": intent, - "timestamp": time.time(), - } + self.cache_manager.set(self.CacheType.INTENT, cache_key, intent) # 后处理时间 postprocess_time = time.time() - postprocess_start_time @@ -272,10 +244,7 @@ class IntentProvider(IntentProviderBase): return intent else: # 添加到缓存 - self.intent_cache[cache_key] = { - "intent": intent, - "timestamp": time.time(), - } + self.cache_manager.set(self.CacheType.INTENT, cache_key, intent) # 后处理时间 postprocess_time = time.time() - postprocess_start_time diff --git a/main/xiaozhi-server/core/providers/tools/server_plugins/plugin_executor.py b/main/xiaozhi-server/core/providers/tools/server_plugins/plugin_executor.py index 728493ec..17d3b564 100644 --- a/main/xiaozhi-server/core/providers/tools/server_plugins/plugin_executor.py +++ b/main/xiaozhi-server/core/providers/tools/server_plugins/plugin_executor.py @@ -51,13 +51,13 @@ class ServerPluginExecutor(ToolExecutor): tools = {} # 获取必要的函数 - necessary_functions = ["handle_exit_intent", "get_time", "get_lunar"] + necessary_functions = ["handle_exit_intent", "get_lunar"] # 获取配置中的函数 config_functions = self.config["Intent"][ self.config["selected_module"]["Intent"] ].get("functions", []) - + # 转换为列表 if not isinstance(config_functions, list): try: diff --git a/main/xiaozhi-server/core/providers/vllm/openai.py b/main/xiaozhi-server/core/providers/vllm/openai.py index 12d8198d..49bef99e 100644 --- a/main/xiaozhi-server/core/providers/vllm/openai.py +++ b/main/xiaozhi-server/core/providers/vllm/openai.py @@ -40,6 +40,7 @@ class VLLMProvider(VLLMProviderBase): self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url) def response(self, question, base64_image): + question = question + "(请使用中文回复)" try: messages = [ { diff --git a/main/xiaozhi-server/core/utils/cache/config.py b/main/xiaozhi-server/core/utils/cache/config.py new file mode 100644 index 00000000..d6d93345 --- /dev/null +++ b/main/xiaozhi-server/core/utils/cache/config.py @@ -0,0 +1,58 @@ +""" +缓存配置管理 +""" + +from enum import Enum +from typing import Dict, Any, Optional +from dataclasses import dataclass +from .strategies import CacheStrategy + + +class CacheType(Enum): + """缓存类型枚举""" + + LOCATION = "location" + WEATHER = "weather" + LUNAR = "lunar" + INTENT = "intent" + IP_INFO = "ip_info" + CONFIG = "config" + DEVICE_PROMPT = "device_prompt" + + +@dataclass +class CacheConfig: + """缓存配置类""" + + strategy: CacheStrategy = CacheStrategy.TTL + ttl: Optional[float] = 300 # 默认5分钟 + max_size: Optional[int] = 1000 # 默认最大1000条 + cleanup_interval: float = 60 # 清理间隔（秒） + + @classmethod + def for_type(cls, cache_type: CacheType) -> "CacheConfig": + """根据缓存类型返回预设配置""" + configs = { + CacheType.LOCATION: cls( + strategy=CacheStrategy.TTL, ttl=None, max_size=1000 # 手动失效 + ), + CacheType.IP_INFO: cls( + strategy=CacheStrategy.TTL, ttl=86400, max_size=1000 # 24小时 + ), + CacheType.WEATHER: cls( + strategy=CacheStrategy.TTL, ttl=28800, max_size=1000 # 8小时 + ), + CacheType.LUNAR: cls( + strategy=CacheStrategy.TTL, ttl=2592000, max_size=365 # 30天过期 + ), + CacheType.INTENT: cls( + strategy=CacheStrategy.TTL_LRU, ttl=600, max_size=1000 # 10分钟 + ), + CacheType.CONFIG: cls( + strategy=CacheStrategy.FIXED_SIZE, ttl=None, max_size=20 # 手动失效 + ), + CacheType.DEVICE_PROMPT: cls( + strategy=CacheStrategy.TTL, ttl=None, max_size=1000 # 手动失效 + ), + } + return configs.get(cache_type, cls()) diff --git a/main/xiaozhi-server/core/utils/cache/manager.py b/main/xiaozhi-server/core/utils/cache/manager.py new file mode 100644 index 00000000..c54f7817 --- /dev/null +++ b/main/xiaozhi-server/core/utils/cache/manager.py @@ -0,0 +1,216 @@ +""" +全局缓存管理器 +""" + +import time +import threading +from typing import Any, Optional, Dict +from collections import OrderedDict +from .strategies import CacheStrategy, CacheEntry +from .config import CacheConfig, CacheType + + +class GlobalCacheManager: + """全局缓存管理器""" + + def __init__(self): + self._logger = None + self._caches: Dict[str, Dict[str, CacheEntry]] = {} + self._configs: Dict[str, CacheConfig] = {} + self._locks: Dict[str, threading.RLock] = {} + self._global_lock = threading.RLock() + self._last_cleanup = time.time() + self._stats = {"hits": 0, "misses": 0, "evictions": 0, "cleanups": 0} + + @property + def logger(self): + """延迟初始化 logger 以避免循环导入""" + if self._logger is None: + from config.logger import setup_logging + + self._logger = setup_logging() + return self._logger + + def _get_cache_name(self, cache_type: CacheType, namespace: str = "") -> str: + """生成缓存名称""" + if namespace: + return f"{cache_type.value}:{namespace}" + return cache_type.value + + def _get_or_create_cache( + self, cache_name: str, config: CacheConfig + ) -> Dict[str, CacheEntry]: + """获取或创建缓存空间""" + with self._global_lock: + if cache_name not in self._caches: + self._caches[cache_name] = ( + OrderedDict() + if config.strategy in [CacheStrategy.LRU, CacheStrategy.TTL_LRU] + else {} + ) + self._configs[cache_name] = config + self._locks[cache_name] = threading.RLock() + return self._caches[cache_name] + + def set( + self, + cache_type: CacheType, + key: str, + value: Any, + ttl: Optional[float] = None, + namespace: str = "", + ) -> None: + """设置缓存值""" + cache_name = self._get_cache_name(cache_type, namespace) + config = self._configs.get(cache_name) or CacheConfig.for_type(cache_type) + cache = self._get_or_create_cache(cache_name, config) + + # 使用配置的TTL或传入的TTL + effective_ttl = ttl if ttl is not None else config.ttl + + with self._locks[cache_name]: + # 创建缓存条目 + entry = CacheEntry(value=value, timestamp=time.time(), ttl=effective_ttl) + + # 处理不同策略 + if config.strategy in [CacheStrategy.LRU, CacheStrategy.TTL_LRU]: + # LRU策略：如果已存在则移动到末尾 + if key in cache: + del cache[key] + cache[key] = entry + + # 检查大小限制 + if config.max_size and len(cache) > config.max_size: + # 移除最旧的条目 + oldest_key = next(iter(cache)) + del cache[oldest_key] + self._stats["evictions"] += 1 + + else: + cache[key] = entry + + # 检查大小限制 + if config.max_size and len(cache) > config.max_size: + # 简单策略：随机移除一个条目 + victim_key = next(iter(cache)) + del cache[victim_key] + self._stats["evictions"] += 1 + + # 定期清理过期条目 + self._maybe_cleanup(cache_name) + + def get( + self, cache_type: CacheType, key: str, namespace: str = "" + ) -> Optional[Any]: + """获取缓存值""" + cache_name = self._get_cache_name(cache_type, namespace) + + if cache_name not in self._caches: + self._stats["misses"] += 1 + return None + + cache = self._caches[cache_name] + config = self._configs[cache_name] + + with self._locks[cache_name]: + if key not in cache: + self._stats["misses"] += 1 + return None + + entry = cache[key] + + # 检查过期 + if entry.is_expired(): + del cache[key] + self._stats["misses"] += 1 + return None + + # 更新访问信息 + entry.touch() + + # LRU策略：移动到末尾 + if config.strategy in [CacheStrategy.LRU, CacheStrategy.TTL_LRU]: + del cache[key] + cache[key] = entry + + self._stats["hits"] += 1 + return entry.value + + def delete(self, cache_type: CacheType, key: str, namespace: str = "") -> bool: + """删除缓存条目""" + cache_name = self._get_cache_name(cache_type, namespace) + + if cache_name not in self._caches: + return False + + cache = self._caches[cache_name] + + with self._locks[cache_name]: + if key in cache: + del cache[key] + return True + return False + + def clear(self, cache_type: CacheType, namespace: str = "") -> None: + """清空指定缓存""" + cache_name = self._get_cache_name(cache_type, namespace) + + if cache_name not in self._caches: + return + + with self._locks[cache_name]: + self._caches[cache_name].clear() + + def invalidate_pattern( + self, cache_type: CacheType, pattern: str, namespace: str = "" + ) -> int: + """按模式失效缓存条目""" + cache_name = self._get_cache_name(cache_type, namespace) + + if cache_name not in self._caches: + return 0 + + cache = self._caches[cache_name] + deleted_count = 0 + + with self._locks[cache_name]: + keys_to_delete = [key for key in cache.keys() if pattern in key] + for key in keys_to_delete: + del cache[key] + deleted_count += 1 + + return deleted_count + + def _cleanup_expired(self, cache_name: str) -> int: + """清理过期条目""" + if cache_name not in self._caches: + return 0 + + cache = self._caches[cache_name] + deleted_count = 0 + + with self._locks[cache_name]: + expired_keys = [key for key, entry in cache.items() if entry.is_expired()] + for key in expired_keys: + del cache[key] + deleted_count += 1 + + return deleted_count + + def _maybe_cleanup(self, cache_name: str): + """定期清理检查""" + config = self._configs.get(cache_name) + if not config: + return + + now = time.time() + if now - self._last_cleanup > config.cleanup_interval: + self._last_cleanup = now + deleted = self._cleanup_expired(cache_name) + if deleted > 0: + self._stats["cleanups"] += 1 + self.logger.debug(f"清理缓存 {cache_name}: 删除 {deleted} 个过期条目") + + +# 创建全局缓存管理器实例 +cache_manager = GlobalCacheManager() diff --git a/main/xiaozhi-server/core/utils/cache/strategies.py b/main/xiaozhi-server/core/utils/cache/strategies.py new file mode 100644 index 00000000..13327ca7 --- /dev/null +++ b/main/xiaozhi-server/core/utils/cache/strategies.py @@ -0,0 +1,43 @@ +""" +缓存策略和数据结构定义 +""" + +import time +from enum import Enum +from typing import Any, Optional +from dataclasses import dataclass + + +class CacheStrategy(Enum): + """缓存策略枚举""" + + TTL = "ttl" # 基于时间过期 + LRU = "lru" # 最近最少使用 + FIXED_SIZE = "fixed_size" # 固定大小 + TTL_LRU = "ttl_lru" # TTL + LRU混合策略 + + +@dataclass +class CacheEntry: + """缓存条目数据结构""" + + value: Any + timestamp: float + ttl: Optional[float] = None # 生存时间（秒） + access_count: int = 0 + last_access: float = None + + def __post_init__(self): + if self.last_access is None: + self.last_access = self.timestamp + + def is_expired(self) -> bool: + """检查是否过期""" + if self.ttl is None: + return False + return time.time() - self.timestamp > self.ttl + + def touch(self): + """更新访问时间和计数""" + self.last_access = time.time() + self.access_count += 1 diff --git a/main/xiaozhi-server/core/utils/dialogue.py b/main/xiaozhi-server/core/utils/dialogue.py index fc45a108..41cffca0 100644 --- a/main/xiaozhi-server/core/utils/dialogue.py +++ b/main/xiaozhi-server/core/utils/dialogue.py @@ -1,4 +1,5 @@ import uuid +import re from typing import List, Dict from datetime import datetime from config.settings import load_config @@ -74,13 +75,6 @@ class Dialogue: # 基础系统提示 enhanced_system_prompt = system_message.content - # 添加说话人识别功能说明 - speaker_guidance = "\n\n[说话人识别功能说明]\n" \ - "当用户消息为JSON格式包含speaker字段时（如：{\"speaker\": \"张三\", \"content\": \"消息内容\"}），表示系统已识别出说话人身份。\n" \ - "请根据说话人的身份特征来调整回应风格和内容。\n" \ - "你可以称呼说话人的名字，并参考他们的特点进行个性化回应。" - enhanced_system_prompt += speaker_guidance - # 添加说话人个性化描述 try: config = load_config() @@ -88,7 +82,7 @@ class Dialogue: speakers = voiceprint_config.get("speakers", []) if speakers: - enhanced_system_prompt += "\n\n[已知说话人信息]" + enhanced_system_prompt += "\n\n" for speaker_str in speakers: try: parts = speaker_str.split(",", 2) @@ -99,15 +93,19 @@ class Dialogue: description = parts[2].strip() if len(parts) >= 3 else "" enhanced_system_prompt += f"\n- {name}：{description}" except: - continue + pass + enhanced_system_prompt += "\n\n" except: # 配置读取失败时忽略错误，不影响其他功能 pass - # 只有当有记忆时才添加记忆部分 - if memory_str and len(memory_str) > 0: - enhanced_system_prompt += f"\n\n以下是用户的历史记忆：\n```\n{memory_str}\n```" - + # 使用正则表达式匹配 标签，不管中间有什么内容 + enhanced_system_prompt = re.sub( + r".*?", + f"\n{memory_str}\n", + system_message.content, + flags=re.DOTALL, + ) dialogue.append({"role": "system", "content": enhanced_system_prompt}) # 添加用户和助手的对话 diff --git a/main/xiaozhi-server/core/utils/prompt_manager.py b/main/xiaozhi-server/core/utils/prompt_manager.py new file mode 100644 index 00000000..dae840bf --- /dev/null +++ b/main/xiaozhi-server/core/utils/prompt_manager.py @@ -0,0 +1,219 @@ +""" +系统提示词管理器模块 +负责管理和更新系统提示词，包括快速初始化和异步增强功能 +""" + +import os +import cnlunar +from typing import Dict, Any +from config.logger import setup_logging + +TAG = __name__ + +WEEKDAY_MAP = { + "Monday": "星期一", + "Tuesday": "星期二", + "Wednesday": "星期三", + "Thursday": "星期四", + "Friday": "星期五", + "Saturday": "星期六", + "Sunday": "星期日", +} + + +class PromptManager: + """系统提示词管理器，负责管理和更新系统提示词""" + + def __init__(self, config: Dict[str, Any], logger=None): + self.config = config + self.logger = logger or setup_logging() + self.base_prompt_template = None + self.last_update_time = 0 + + # 导入全局缓存管理器 + from core.utils.cache.manager import cache_manager, CacheType + + self.cache_manager = cache_manager + self.CacheType = CacheType + + self._load_base_template() + + def _load_base_template(self): + """加载基础提示词模板""" + try: + template_path = "agent-base-prompt.txt" + cache_key = f"prompt_template:{template_path}" + + # 先从缓存获取 + cached_template = self.cache_manager.get(self.CacheType.CONFIG, cache_key) + if cached_template is not None: + self.base_prompt_template = cached_template + self.logger.bind(tag=TAG).debug("从缓存加载基础提示词模板") + return + + # 缓存未命中，从文件读取 + if os.path.exists(template_path): + with open(template_path, "r", encoding="utf-8") as f: + template_content = f.read() + + # 存入缓存（CONFIG类型默认不自动过期，需要手动失效） + self.cache_manager.set( + self.CacheType.CONFIG, cache_key, template_content + ) + self.base_prompt_template = template_content + self.logger.bind(tag=TAG).debug("成功加载基础提示词模板并缓存") + else: + self.logger.bind(tag=TAG).warning("未找到agent-base-prompt.txt文件") + except Exception as e: + self.logger.bind(tag=TAG).error(f"加载提示词模板失败: {e}") + + def get_quick_prompt(self, user_prompt: str, device_id: str = None) -> str: + """快速获取系统提示词（使用用户配置）""" + device_cache_key = f"device_prompt:{device_id}" + cached_device_prompt = self.cache_manager.get( + self.CacheType.DEVICE_PROMPT, device_cache_key + ) + if cached_device_prompt is not None: + self.logger.bind(tag=TAG).debug(f"使用设备 {device_id} 的缓存提示词") + return cached_device_prompt + else: + self.logger.bind(tag=TAG).debug( + f"设备 {device_id} 无缓存提示词，使用传入的提示词" + ) + + # 使用传入的提示词并缓存（如果有设备ID） + if device_id: + device_cache_key = f"device_prompt:{device_id}" + self.cache_manager.set(self.CacheType.CONFIG, device_cache_key, user_prompt) + self.logger.bind(tag=TAG).debug(f"设备 {device_id} 的提示词已缓存") + + self.logger.bind(tag=TAG).info(f"使用快速提示词: {user_prompt[:50]}...") + return user_prompt + + def _get_current_time_info(self) -> tuple: + """获取当前时间信息""" + from datetime import datetime + + now = datetime.now() + current_time = now.strftime("%H:%M") + today_date = now.strftime("%Y-%m-%d") + today_weekday = WEEKDAY_MAP[now.strftime("%A")] + today_lunar = cnlunar.Lunar(now, godType="8char") + lunar_date = "%s年%s%s\n" % ( + today_lunar.lunarYearCn, + today_lunar.lunarMonthCn[:-1], + today_lunar.lunarDayCn, + ) + + return current_time, today_date, today_weekday, lunar_date + + def _get_location_info(self, client_ip: str) -> str: + """获取位置信息""" + try: + # 先从缓存获取 + cached_location = self.cache_manager.get(self.CacheType.LOCATION, client_ip) + if cached_location is not None: + return cached_location + + # 缓存未命中，调用API获取 + from core.utils.util import get_ip_info + + ip_info = get_ip_info(client_ip, self.logger) + city = ip_info.get("city", "未知位置") + location = f"{city}" + + # 存入缓存 + self.cache_manager.set(self.CacheType.LOCATION, client_ip, location) + return location + except Exception as e: + self.logger.bind(tag=TAG).error(f"获取位置信息失败: {e}") + return "未知位置" + + def _get_weather_info(self, conn, location: str) -> str: + """获取天气信息""" + try: + # 先从缓存获取 + cached_weather = self.cache_manager.get(self.CacheType.WEATHER, location) + if cached_weather is not None: + return cached_weather + + # 缓存未命中，调用get_weather函数获取 + from plugins_func.functions.get_weather import get_weather + from plugins_func.register import ActionResponse + + # 调用get_weather函数 + result = get_weather(conn, location=location, lang="zh_CN") + if isinstance(result, ActionResponse): + weather_report = result.result + self.cache_manager.set(self.CacheType.WEATHER, location, weather_report) + return weather_report + return "天气信息获取失败" + + except Exception as e: + self.logger.bind(tag=TAG).error(f"获取天气信息失败: {e}") + return "天气信息获取失败" + + def update_context_info(self, conn, client_ip: str): + """同步更新上下文信息""" + try: + # 获取位置信息（使用全局缓存） + local_address = self._get_location_info(client_ip) + # 获取天气信息（使用全局缓存） + self._get_weather_info(conn, local_address) + self.logger.bind(tag=TAG).info(f"上下文信息更新完成") + + except Exception as e: + self.logger.bind(tag=TAG).error(f"更新上下文信息失败: {e}") + + def build_enhanced_prompt( + self, user_prompt: str, device_id: str, client_ip: str = None + ) -> str: + """构建增强的系统提示词""" + if not self.base_prompt_template: + return user_prompt + + try: + # 获取最新的时间信息（不缓存） + current_time, today_date, today_weekday, lunar_date = ( + self._get_current_time_info() + ) + + # 获取缓存的上下文信息 + local_address = "" + weather_info = "" + + if client_ip: + # 获取位置信息（从全局缓存） + local_address = ( + self.cache_manager.get(self.CacheType.LOCATION, client_ip) or "" + ) + + # 获取天气信息（从全局缓存） + if local_address: + weather_info = ( + self.cache_manager.get(self.CacheType.WEATHER, local_address) + or "" + ) + + # 替换模板变量 + enhanced_prompt = self.base_prompt_template.format( + base_prompt=user_prompt, + current_time=current_time, + today_date=today_date, + today_weekday=today_weekday, + lunar_date=lunar_date, + local_address=local_address, + weather_info=weather_info, + ) + device_cache_key = f"device_prompt:{device_id}" + self.cache_manager.set( + self.CacheType.DEVICE_PROMPT, device_cache_key, enhanced_prompt + ) + self.logger.bind(tag=TAG).info( + f"构建增强提示词成功，长度: {len(enhanced_prompt)}" + ) + return enhanced_prompt + + except Exception as e: + self.logger.bind(tag=TAG).error(f"构建增强提示词失败: {e}") + return user_prompt diff --git a/main/xiaozhi-server/core/utils/util.py b/main/xiaozhi-server/core/utils/util.py index 35417abc..bc778558 100644 --- a/main/xiaozhi-server/core/utils/util.py +++ b/main/xiaozhi-server/core/utils/util.py @@ -96,11 +96,23 @@ def is_private_ip(ip_addr): def get_ip_info(ip_addr, logger): try: + # 导入全局缓存管理器 + from core.utils.cache.manager import cache_manager, CacheType + + # 先从缓存获取 + cached_ip_info = cache_manager.get(CacheType.IP_INFO, ip_addr) + if cached_ip_info is not None: + return cached_ip_info + + # 缓存未命中，调用API if is_private_ip(ip_addr): ip_addr = "" url = f"https://whois.pconline.com.cn/ipJson.jsp?json=true&ip={ip_addr}" resp = requests.get(url).json() ip_info = {"city": resp.get("city")} + + # 存入缓存 + cache_manager.set(CacheType.IP_INFO, ip_addr, ip_info) return ip_info except Exception as e: logger.bind(tag=TAG).error(f"Error getting client ip info: {e}") @@ -982,3 +994,28 @@ def sanitize_tool_name(name: str) -> str: """Sanitize tool names for OpenAI compatibility.""" # 支持中文、英文字母、数字、下划线和连字符 return re.sub(r"[^a-zA-Z0-9_\-\u4e00-\u9fff]", "_", name) + + +def validate_mcp_endpoint(mcp_endpoint: str) -> bool: + """ + 校验MCP接入点格式 + + Args: + mcp_endpoint: MCP接入点字符串 + + Returns: + bool: 是否有效 + """ + # 1. 检查是否以ws开头 + if not mcp_endpoint.startswith("ws"): + return False + + # 2. 检查是否包含key、call字样 + if "key" in mcp_endpoint.lower() or "call" in mcp_endpoint.lower(): + return False + + # 3. 检查是否包含/mcp/字样 + if "/mcp/" not in mcp_endpoint: + return False + + return True diff --git a/main/xiaozhi-server/core/websocket_server.py b/main/xiaozhi-server/core/websocket_server.py index 1a1ba4c0..09ba7961 100644 --- a/main/xiaozhi-server/core/websocket_server.py +++ b/main/xiaozhi-server/core/websocket_server.py @@ -57,8 +57,25 @@ class WebSocketServer: self.active_connections.add(handler) try: await handler.handle_connection(websocket) + except Exception as e: + self.logger.bind(tag=TAG).error(f"处理连接时出错: {e}") finally: + # 确保从活动连接集合中移除 self.active_connections.discard(handler) + # 强制关闭连接（如果还没有关闭的话） + try: + # 安全地检查WebSocket状态并关闭 + if hasattr(websocket, "closed") and not websocket.closed: + await websocket.close() + elif hasattr(websocket, "state") and websocket.state.name != "CLOSED": + await websocket.close() + else: + # 如果没有closed属性，直接尝试关闭 + await websocket.close() + except Exception as close_error: + self.logger.bind(tag=TAG).error( + f"服务器端强制关闭连接时出错: {close_error}" + ) async def _http_response(self, websocket, request_headers): # 检查是否为 WebSocket 升级请求 diff --git a/main/xiaozhi-server/plugins_func/functions/get_time.py b/main/xiaozhi-server/plugins_func/functions/get_time.py index 44732bba..766e19fd 100644 --- a/main/xiaozhi-server/plugins_func/functions/get_time.py +++ b/main/xiaozhi-server/plugins_func/functions/get_time.py @@ -2,59 +2,27 @@ from datetime import datetime import cnlunar from plugins_func.register import register_function, ToolType, ActionResponse, Action -# 添加星期映射字典 -WEEKDAY_MAP = { - "Monday": "星期一", - "Tuesday": "星期二", - "Wednesday": "星期三", - "Thursday": "星期四", - "Friday": "星期五", - "Saturday": "星期六", - "Sunday": "星期日", -} - -get_time_function_desc = { - "type": "function", - "function": { - "name": "get_time", - "description": "获取今天日期或者当前时间信息", - "parameters": {"type": "object", "properties": {}, "required": []}, - }, -} - - -@register_function("get_time", get_time_function_desc, ToolType.WAIT) -def get_time(): - """ - 获取当前的日期时间信息 - """ - now = datetime.now() - current_time = now.strftime("%H:%M:%S") - current_date = now.strftime("%Y-%m-%d") - current_weekday = WEEKDAY_MAP[now.strftime("%A")] - response_text = ( - f"当前日期: {current_date}，当前时间: {current_time}， {current_weekday}" - ) - - return ActionResponse(Action.REQLLM, response_text, None) - - get_lunar_function_desc = { "type": "function", "function": { "name": "get_lunar", "description": ( - "用于获取今天的阴历/农历和黄历信息。" + "用于具体日期的阴历/农历和黄历信息。" "用户可以指定查询内容，如：阴历日期、天干地支、节气、生肖、星座、八字、宜忌等。" "如果没有指定查询内容，则默认查询干支年和农历日期。" + "对于'今天农历是多少'、'今天农历日期'这样的基本查询，请直接使用context中的信息，不要调用此工具。" ), "parameters": { "type": "object", "properties": { + "date": { + "type": "string", + "description": "要查询的日期，格式为YYYY-MM-DD，例如2024-01-01。如果不提供，则使用当前日期", + }, "query": { "type": "string", "description": "要查询的内容，例如阴历日期、天干地支、节日、节气、生肖、星座、八字、宜忌等", - } + }, }, "required": [], }, @@ -63,23 +31,41 @@ get_lunar_function_desc = { @register_function("get_lunar", get_lunar_function_desc, ToolType.WAIT) -def get_lunar(query=None): +def get_lunar(date=None, query=None): """ 用于获取当前的阴历/农历，和天干地支、节气、生肖、星座、八字、宜忌等黄历信息 """ - now = datetime.now() - current_time = now.strftime("%H:%M:%S") + from core.utils.cache.manager import cache_manager, CacheType + + # 如果提供了日期参数，则使用指定日期；否则使用当前日期 + if date: + try: + now = datetime.strptime(date, "%Y-%m-%d") + except ValueError: + return ActionResponse( + Action.REQLLM, + f"日期格式错误，请使用YYYY-MM-DD格式，例如：2024-01-01", + None, + ) + else: + now = datetime.now() + current_date = now.strftime("%Y-%m-%d") - current_weekday = WEEKDAY_MAP[now.strftime("%A")] # 如果 query 为 None，则使用默认文本 if query is None: query = "默认查询干支年和农历日期" + + # 尝试从缓存获取农历信息 + lunar_cache_key = f"lunar_info_{current_date}" + cached_lunar_info = cache_manager.get(CacheType.LUNAR, lunar_cache_key) + if cached_lunar_info: + return ActionResponse(Action.REQLLM, cached_lunar_info, None) + response_text = f"根据以下信息回应用户的查询请求，并提供与{query}相关的信息：\n" lunar = cnlunar.Lunar(now, godType="8char") response_text += ( - f"当前公历日期: {current_date}，当前时间: {current_time}，{current_weekday}\n" "农历信息：\n" "%s年%s%s\n" % (lunar.lunarYearCn, lunar.lunarMonthCn[:-1], lunar.lunarDayCn) + "干支: %s年 %s月 %s日\n" % (lunar.year8Char, lunar.month8Char, lunar.day8Char) @@ -135,4 +121,7 @@ def get_lunar(query=None): + "(默认返回干支年和农历日期；仅在要求查询宜忌信息时才返回本日宜忌)" ) + # 缓存农历信息 + cache_manager.set(CacheType.LUNAR, lunar_cache_key, response_text) + return ActionResponse(Action.REQLLM, response_text, None) diff --git a/main/xiaozhi-server/plugins_func/functions/get_weather.py b/main/xiaozhi-server/plugins_func/functions/get_weather.py index 75c15c7b..a3af6c03 100644 --- a/main/xiaozhi-server/plugins_func/functions/get_weather.py +++ b/main/xiaozhi-server/plugins_func/functions/get_weather.py @@ -151,20 +151,44 @@ def parse_weather_info(soup): @register_function("get_weather", GET_WEATHER_FUNCTION_DESC, ToolType.SYSTEM_CTL) def get_weather(conn, location: str = None, lang: str = "zh_CN"): - api_host = conn.config["plugins"]["get_weather"].get("api_host", "mj7p3y7naa.re.qweatherapi.com") - api_key = conn.config["plugins"]["get_weather"].get("api_key", "a861d0d5e7bf4ee1a83d9a9e4f96d4da") + from core.utils.cache.manager import cache_manager, CacheType + + api_host = conn.config["plugins"]["get_weather"].get( + "api_host", "mj7p3y7naa.re.qweatherapi.com" + ) + api_key = conn.config["plugins"]["get_weather"].get( + "api_key", "a861d0d5e7bf4ee1a83d9a9e4f96d4da" + ) default_location = conn.config["plugins"]["get_weather"]["default_location"] client_ip = conn.client_ip + # 优先使用用户提供的location参数 if not location: # 通过客户端IP解析城市 if client_ip: - # 动态解析IP对应的城市信息 - ip_info = get_ip_info(client_ip, logger) - location = ip_info.get("city") if ip_info and "city" in ip_info else None + # 先从缓存获取IP对应的城市信息 + cached_ip_info = cache_manager.get(CacheType.IP_INFO, client_ip) + if cached_ip_info: + location = cached_ip_info.get("city") + else: + # 缓存未命中，调用API获取 + ip_info = get_ip_info(client_ip, logger) + if ip_info: + cache_manager.set(CacheType.IP_INFO, client_ip, ip_info) + location = ip_info.get("city") + + if not location: + location = default_location else: - # 若IP解析失败或无IP，使用默认位置 + # 若无IP，使用默认位置 location = default_location + # 尝试从缓存获取完整天气报告 + weather_cache_key = f"full_weather_{location}_{lang}" + cached_weather_report = cache_manager.get(CacheType.WEATHER, weather_cache_key) + if cached_weather_report: + return ActionResponse(Action.REQLLM, cached_weather_report, None) + + # 缓存未命中，获取实时天气数据 city_info = fetch_city_info(location, api_key, api_host) if not city_info: return ActionResponse( @@ -192,4 +216,7 @@ def get_weather(conn, location: str = None, lang: str = "zh_CN"): # 提示语 weather_report += "\n（如需某一天的具体天气，请告诉我日期）" + # 缓存完整的天气报告 + cache_manager.set(CacheType.WEATHER, weather_cache_key, weather_report) + return ActionResponse(Action.REQLLM, weather_report, None) diff --git a/main/xiaozhi-server/requirements.txt b/main/xiaozhi-server/requirements.txt index a60ad689..4ec8b625 100755 --- a/main/xiaozhi-server/requirements.txt +++ b/main/xiaozhi-server/requirements.txt @@ -21,7 +21,7 @@ cozepy==0.12.0 mem0ai==0.1.62 bs4==0.0.2 modelscope==1.23.2 -sherpa_onnx==1.12.0 +sherpa_onnx==1.12.4 mcp==1.8.1 cnlunar==0.2.0 PySocks==1.7.1