fix:function call bug (#268)

* 优化function call消息处理

* fix:用户说话重复记录bug

* 2025-3-10-优化llm intent (#254)

Co-authored-by: 欣南科技 <huangrongzhuang@xin-nan.com>

* update:回复上版提示词，无需再二次识别歌曲名

---------

Co-authored-by: 玄凤科技 <eric230308@gmail.com>
Co-authored-by: hrz <1710360675@qq.com>
Co-authored-by: Jiao Haoyang <108573524+XuSenfeng@users.noreply.github.com>

.dockerignore

@@ -0,0 +1,7 @@
+.git
+__pycache__
+*.pyc
+.env
+Dockerfile
+tmp/
+data/

.github/workflows/docker-image.yml

@@ -7,30 +7,31 @@ on:
 
 jobs:
   release:
-    name: Release Docker image
+    name: Release Docker images
     runs-on: ubuntu-latest
     permissions:
       packages: write
       contents: write
       id-token: write
       issues: write
-
     steps:
       - name: Check Disk Space
         run: |
           df -h
           docker system df
+
       - name: Clean up Docker resources
         run: |
           docker system prune -af
           docker builder prune -af
-      - name: Check out the repo
+
+      - name: Checkout code
         uses: actions/checkout@v4
 
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
 
-      - name: Log in to the GitHub Container Registry
+      - name: Login to GitHub Container Registry
         uses: docker/login-action@v3
         with:
           registry: ghcr.io
@@ -42,13 +43,26 @@ jobs:
         run: |
           echo "VERSION=${GITHUB_REF#refs/tags/}" >> $GITHUB_ENV
 
-      - name: Build and push Docker image
+      # 构建 xiaozhi-server 镜像
-        id: build_push
+      - name: Build and push xiaozhi-server
         uses: docker/build-push-action@v6
         with:
           context: .
+          file: Dockerfile-server
           push: true
           tags: |
-            ghcr.io/${{ github.repository }}:${{ env.VERSION }}
+            ghcr.io/${{ github.repository }}:server_${{ env.VERSION }}
-            ghcr.io/${{ github.repository }}:latest
+            ghcr.io/${{ github.repository }}:server_latest
+          platforms: linux/amd64,linux/arm64
+
+      # 构建 manager-api 镜像
+      - name: Build and push manager-web
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: Dockerfile-web
+          push: true
+          tags: |
+            ghcr.io/${{ github.repository }}:web_${{ env.VERSION }}
+            ghcr.io/${{ github.repository }}:web_latest
           platforms: linux/amd64,linux/arm64

Dockerfile-server

@@ -23,7 +23,7 @@ RUN apt-get update && \
 COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
 
 # 复制应用代码
-COPY main/xiaozhi-server/ .
+COPY main/xiaozhi-server .
 
 # 启动应用
 CMD ["python", "app.py"]

Dockerfile-web

@@ -0,0 +1,40 @@
+# 第一阶段：构建Vue前端
+FROM node:18 as web-builder
+WORKDIR /app
+COPY main/manager-web/package*.json ./
+RUN npm install
+COPY main/manager-web .
+RUN npm run build
+
+# 第二阶段：构建Java后端
+FROM maven:3-eclipse-temurin-21-alpine as api-builder
+WORKDIR /app
+COPY main/manager-api/pom.xml .
+COPY main/manager-api/src ./src
+RUN mvn clean package -Dmaven.test.skip=true
+
+# 第三阶段：构建最终镜像
+FROM eclipse-temurin:21-jdk-jammy
+
+# 安装Nginx并清理缓存
+RUN apt-get update && \
+    apt-get install -y nginx && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# 配置Nginx
+COPY docs/docker/nginx.conf /etc/nginx/conf.d/default.conf
+
+# 复制前端构建产物
+COPY --from=web-builder /app/dist /usr/share/nginx/html
+
+# 复制Java后端JAR包
+COPY --from=api-builder /app/target/xiaozhi-esp32-api.jar /app/xiaozhi-esp32-api.jar
+
+# 暴露端口
+EXPOSE 8002
+
+# 启动脚本
+COPY docs/docker/start.sh /start.sh
+RUN chmod +x /start.sh
+CMD ["/start.sh"]

README.md

@@ -129,55 +129,67 @@ server:
   基于 `xiaozhi-esp32` 协议，通过 WebSocket 实现数据交互。
 - **对话交互**  
   支持唤醒对话、手动对话及实时打断。长时间无对话时自动休眠
+- **意图识别**  
+  支持使用LLM意图识别、function call函数调用，减少硬编码意图判断
 - **多语言识别**  
   支持国语、粤语、英语、日语、韩语（默认使用 FunASR）。
 - **LLM 模块**  
   支持灵活切换 LLM 模块，默认使用 ChatGLMLLM，也可选用阿里百炼、DeepSeek、Ollama 等接口。
 - **TTS 模块**  
   支持 EdgeTTS（默认）、火山引擎豆包 TTS 等多种 TTS 接口，满足语音合成需求。
+- **记忆功能**  
+  支持超长记忆、本地总结记忆、无记忆三种模式，满足不同场景需求。
 
 ### 正在开发 🚧
 
-- 对话记忆功能
 - 多种心情模式
 - 智控台webui
+- iot功能
 
 ![图片](docs/images/webui.png)
 ---
 
 ## 本项目支持的平台/组件列表 📋
 
-### LLM
+### LLM 语言模型
 
-| 类型  |        平台名称        |         使用方式          |   收费模式   |                                备注                                 |
+| 类型  |        平台名称        |         使用方式          |    收费模式     |                                                           备注                                                            |
-|:---:|:------------------:|:---------------------:|:--------:|:-----------------------------------------------------------------:|
+|:---:|:------------------:|:---------------------:|:-----------:|:-----------------------------------------------------------------------------------------------------------------------:|
-| LLM |   阿里百炼 (AliLLM)    |      openai 接口调用      | 消耗 token |  [点击申请密钥](https://bailian.console.aliyun.com/?apiKey=1#/api-key)  |
+| LLM |   阿里百炼 (AliLLM)    |      openai 接口调用      |  消耗 token   |                             [点击申请密钥](https://bailian.console.aliyun.com/?apiKey=1#/api-key)                             |
-| LLM | 深度求索 (DeepSeekLLM) |      openai 接口调用      | 消耗 token |             [点击申请密钥](https://platform.deepseek.com/)              |
+| LLM |     DoubaoLLM      |      openai 接口调用      |  消耗 token   | [点击申请密钥](https://console.volcengine.com/ark/region:ark+cn-beijing/model/detail?Id=doubao-pro-32k&projectName=undefined) |
-| LLM |   智谱（ChatGLMLLM）   |      openai 接口调用      |    免费    | 虽然免费，仍需[点击申请密钥](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) |
+| LLM | 深度求索 (DeepSeekLLM) |      openai 接口调用      |  消耗 token   |                                        [点击申请密钥](https://platform.deepseek.com/)                                         |
-| LLM |     OllamaLLM      |      ollama 接口调用      |  免费/自定义  |       需预先下载模型（`ollama pull`），服务地址：`http://localhost:11434`        |
+| LLM |   智谱（ChatGLMLLM）   |      openai 接口调用      |     免费      |                            虽然免费，仍需[点击申请密钥](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)                            |
-| LLM |      DifyLLM       |       dify 接口调用       | 消耗 token |                    本地化部署，注意配置提示词需在 Dify 控制台设置                     |
+| LLM |     OllamaLLM      |      ollama 接口调用      | 免费/消耗 token |                                  需预先下载模型（`ollama pull`），服务地址：`http://localhost:11434`                                   |
-| LLM |     GeminiLLM      |      gemini 接口调用      |    免费    |           [点击申请密钥](https://aistudio.google.com/apikey)            |
+| LLM |      DifyLLM       |       dify 接口调用       | 免费/消耗 token |                                               本地化部署，注意配置提示词需在 Dify 控制台设置                                                |
-| LLM |      CozeLLM       |       coze 接口调用       | 消耗 token |                     需提供 bot_id、user_id 及个人令牌                      |
+| LLM |     FastgptLLM     |     fastgpt 接口调用      | 免费/消耗 token |                                              本地化部署，注意配置提示词需在 Fastgpt 控制台设置                                              |
-| LLM |   Home Assistant   | homeassistant语音助手接口调用 |    免费    |                        需提供home assistant令牌                        |
+| LLM |     GeminiLLM      |      gemini 接口调用      |     免费      |                                      [点击申请密钥](https://aistudio.google.com/apikey)                                       |
+| LLM |      CozeLLM       |       coze 接口调用       |  消耗 token   |                                                需提供 bot_id、user_id 及个人令牌                                                 |
+| LLM |   Home Assistant   | homeassistant语音助手接口调用 |     免费      |                                                   需提供home assistant令牌                                                   |
 
 实际上，任何支持 openai 接口调用的 LLM 均可接入使用。
 
 ---
 
-### TTS
+### TTS 语音合成
 
 | 类型  |          平台名称          | 使用方式 |   收费模式   |                                    备注                                     |
 |:---:|:----------------------:|:----:|:--------:|:-------------------------------------------------------------------------:|
 | TTS |        EdgeTTS         | 接口调用 |    免费    |                             默认 TTS，基于微软语音合成技术                             |
 | TTS | 火山引擎豆包 TTS (DoubaoTTS) | 接口调用 | 消耗 token | [点击创建密钥](https://console.volcengine.com/speech/service/8)；建议使用付费版本以获得更高并发 |
+| TTS |       AliyunTTS        | 接口调用 | 消耗 token |          [点击创建密钥](https://nls-portal.console.aliyun.com/applist)          |
 | TTS |  CosyVoiceSiliconflow  | 接口调用 | 消耗 token |                         需申请硅基流动 API 密钥；输出格式为 wav                          |
+| TTS |        TTS302AI        | 接口调用 | 消耗 token |                  [点击创建密钥](https://dash.302.ai/apis/list)                  |
 | TTS |       CozeCnTTS        | 接口调用 | 消耗 token |                        需提供 Coze API key；输出格式为 wav                         |
+| TTS |        ACGNTTS         | 接口调用 | 消耗 token |                        [联系网站管理员购买密钥](www.ttson.cn)                        |
+| TTS |       OpenAITTS        | 接口调用 | 消耗 token |                                 境外使用，境外购买                                 |
 | TTS |       FishSpeech       | 接口调用 |  免费/自定义  |                         本地启动 TTS 服务；启动方法见配置文件内说明                          |
 | TTS |     GPT_SOVITS_V2      | 接口调用 |  免费/自定义  |                         本地启动 TTS 服务，适用于个性化语音合成场景                          |
+| TTS |     GPT_SOVITS_V3      | 接口调用 |  免费/自定义  |                         本地启动 TTS 服务，适用于个性化语音合成场景                          |
+| TTS |       MinimaxTTS       | 接口调用 |  免费/自定义  |                         本地启动 TTS 服务，适用于个性化语音合成场景                          |
 
 ---
 
-### VAD
+### VAD 语音活动检测
 
 | 类型  |   平台名称    | 使用方式 | 收费模式 | 备注 |
 |:---:|:---------:|:----:|:----:|:--:|
@@ -185,7 +197,7 @@ server:
 
 ---
 
-### ASR
+### ASR 语音识别
 
 | 类型  |   平台名称    | 使用方式 | 收费模式 | 备注 |
 |:---:|:---------:|:----:|:----:|:--:|
@@ -194,11 +206,21 @@ server:
 
 ---
 
-### Memory
+### Memory 记忆存储
 
-|   类型   |  平台名称  | 使用方式 | 收费模式 | 备注 |
+|   类型   |      平台名称       | 使用方式 |   收费模式   | 备注 |
-|:------:|:------:|:----:|:----:|:--:|
+|:------:|:---------------:|:----:|:--------:|:--:|
-| Memory | mem0ai | 接口调用 |  免费  |    |
+| Memory |     mem0ai      | 接口调用 | 100次/月额度 |    |
+| Memory | mem_local_short | 本地总结 |    免费    |    |
+
+---
+
+### Intent 意图识别
+
+|   类型   |     平台名称      | 使用方式 |  收费模式   |          备注           |
+|:------:|:-------------:|:----:|:-------:|:---------------------:|
+| Intent |  intent_llm   | 接口调用 | 根据LLM收费 |    通过大模型识别意图，通用性强     |
+| Intent | function_call | 接口调用 | 根据LLM收费 | 通过大模型函数调用完成意图，速度快，效果好 |
 
 ---
 
@@ -231,39 +253,18 @@ server:
 
 点这里查看[固件编译](./docs/firmware-build.md)的详细过程。
 
-编译成功且联网成功后，通过唤醒词唤醒小智，留意server端输出的控制台信息。
+烧录成功且联网成功后，通过唤醒词唤醒小智，留意server端输出的控制台信息。
 
 ---
 
 ## 常见问题 ❓
 
-### 1、TTS 经常失败，经常超时 ⏰
+### 1、为什么我说的话，小智识别出来很多韩文、日文、英文？🇰🇷
-
-建议：如果 `EdgeTTS` 经常失败，请先检查是否使用了代理（梯子）。如果使用了，请尝试关闭代理后再试；  
-如果用的是火山引擎的豆包 TTS，经常失败时建议使用付费版本，因为测试版本仅支持 2 个并发。
-
-### 2、我想通过小智控制电灯、空调、远程开关机等操作 💡
-
-建议：在配置文件中将 `LLM` 设置为 `HomeAssistant`，通过 调用`HomeAssistant`接口实现相关控制。
-
-### 3、我说话很慢，停顿时小智老是抢话 🗣️
-
-建议：在配置文件中找到如下部分，将 `min_silence_duration_ms` 的值调大（例如改为 `1000`）：
-
-```yaml
-VAD:
-  SileroVAD:
-    threshold: 0.5
-    model_dir: models/snakers4_silero-vad
-    min_silence_duration_ms: 700  # 如果说话停顿较长，可将此值调大
-```
-
-### 4、为什么我说的话，小智识别出来很多韩文、日文、英文？🇰🇷
 
 建议：检查一下`models/SenseVoiceSmall`是否已经有`model.pt`
 文件，如果没有就要下载，查看这里[下载语音识别模型文件](docs/Deployment.md#模型文件)
 
-### 5、为什么会出现“TTS 任务出错 文件不存在”？📁
+### 2、为什么会出现“TTS 任务出错 文件不存在”？📁
 
 建议：检查一下是否正确使用`conda` 安装了`libopus`和`ffmpeg`库。
 
@@ -274,7 +275,12 @@ conda install conda-forge::libopus
 conda install conda-forge::ffmpeg
 ```
 
-### 6、如何提高小智对话响应速度？ ⚡
+### 3、TTS 经常失败，经常超时 ⏰
+
+建议：如果 `EdgeTTS` 经常失败，请先检查是否使用了代理（梯子）。如果使用了，请尝试关闭代理后再试；  
+如果用的是火山引擎的豆包 TTS，经常失败时建议使用付费版本，因为测试版本仅支持 2 个并发。
+
+### 4、如何提高小智对话响应速度？ ⚡
 
 本项目默认配置为低成本方案，建议初学者先使用默认免费模型，解决“跑得动”的问题，再优化“跑得快”。  
 如需提升响应速度，可尝试更换各组件。以下为各组件的响应速度测试数据（仅供参考，不构成承诺）：
@@ -305,7 +311,6 @@ LLM 性能排行:
 |:-----------|:-----------|:--------|
 | AliLLM     | 0.547s     | 1.485s  |
 | ChatGLMLLM | 0.677s     | 3.057s  |
-| OllamaLLM  | 0.003s     | 0.003s  |
 
 TTS 性能排行:
 
@@ -332,6 +337,22 @@ TTS 性能排行:
 - LLM：`AliLLM`
 - TTS：`DoubaoTTS`
 
+### 5、我说话很慢，停顿时小智老是抢话 🗣️
+
+建议：在配置文件中找到如下部分，将 `min_silence_duration_ms` 的值调大（例如改为 `1000`）：
+
+```yaml
+VAD:
+  SileroVAD:
+    threshold: 0.5
+    model_dir: models/snakers4_silero-vad
+    min_silence_duration_ms: 700  # 如果说话停顿较长，可将此值调大
+```
+
+### 6、我想通过小智控制电灯、空调、远程开关机等操作 💡
+
+建议：在配置文件中将 `LLM` 设置为 `HomeAssistant`，通过 调用`HomeAssistant`接口实现相关控制。
+
 ### 7、更多问题，可联系我们反馈 💬
 
 我们的联系方式放在[百度网盘中,点击前往](https://pan.baidu.com/s/1x6USjvP1nTRsZ45XlJu65Q)，提取码是`223y`。

docs/Deployment.md

@@ -93,7 +93,7 @@ docker logs -f xiaozhi-esp32-server
 ```
 docker stop xiaozhi-esp32-server
 docker rm xiaozhi-esp32-server
-docker rmi ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:latest
+docker rmi ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:server_latest
 ```
 
 3、重新按docker方式部署
@@ -294,4 +294,19 @@ LLM:
 
 这个信息很有用的，后面`编译esp32固件`需要用到。
 
-接下来，你就可以开始 [编译esp32固件](firmware-build.md)了。
+接下来，你就可以开始 [编译esp32固件](firmware-build.md)了。
+
+
+以下是一些常见问题，供参考：
+
+[1、为什么我说的话，小智识别出来很多韩文、日文、英文](../README.md#1%E4%B8%BA%E4%BB%80%E4%B9%88%E6%88%91%E8%AF%B4%E7%9A%84%E8%AF%9D%E5%B0%8F%E6%99%BA%E8%AF%86%E5%88%AB%E5%87%BA%E6%9D%A5%E5%BE%88%E5%A4%9A%E9%9F%A9%E6%96%87%E6%97%A5%E6%96%87%E8%8B%B1%E6%96%87)
+
+[2、为什么会出现“TTS 任务出错 文件不存在”？](../README.md#1%E4%B8%BA%E4%BB%80%E4%B9%88%E6%88%91%E8%AF%B4%E7%9A%84%E8%AF%9D%E5%B0%8F%E6%99%BA%E8%AF%86%E5%88%AB%E5%87%BA%E6%9D%A5%E5%BE%88%E5%A4%9A%E9%9F%A9%E6%96%87%E6%97%A5%E6%96%87%E8%8B%B1%E6%96%87)
+
+[3、TTS 经常失败，经常超时](../README.md#1%E4%B8%BA%E4%BB%80%E4%B9%88%E6%88%91%E8%AF%B4%E7%9A%84%E8%AF%9D%E5%B0%8F%E6%99%BA%E8%AF%86%E5%88%AB%E5%87%BA%E6%9D%A5%E5%BE%88%E5%A4%9A%E9%9F%A9%E6%96%87%E6%97%A5%E6%96%87%E8%8B%B1%E6%96%87)
+
+[4、如何提高小智对话响应速度？](../README.md#1%E4%B8%BA%E4%BB%80%E4%B9%88%E6%88%91%E8%AF%B4%E7%9A%84%E8%AF%9D%E5%B0%8F%E6%99%BA%E8%AF%86%E5%88%AB%E5%87%BA%E6%9D%A5%E5%BE%88%E5%A4%9A%E9%9F%A9%E6%96%87%E6%97%A5%E6%96%87%E8%8B%B1%E6%96%87)
+
+[5、我说话很慢，停顿时小智老是抢话](../README.md#1%E4%B8%BA%E4%BB%80%E4%B9%88%E6%88%91%E8%AF%B4%E7%9A%84%E8%AF%9D%E5%B0%8F%E6%99%BA%E8%AF%86%E5%88%AB%E5%87%BA%E6%9D%A5%E5%BE%88%E5%A4%9A%E9%9F%A9%E6%96%87%E6%97%A5%E6%96%87%E8%8B%B1%E6%96%87)
+
+[6、我想通过小智控制电灯、空调、远程开关机等操作](../README.md#1%E4%B8%BA%E4%BB%80%E4%B9%88%E6%88%91%E8%AF%B4%E7%9A%84%E8%AF%9D%E5%B0%8F%E6%99%BA%E8%AF%86%E5%88%AB%E5%87%BA%E6%9D%A5%E5%BE%88%E5%A4%9A%E9%9F%A9%E6%96%87%E6%97%A5%E6%96%87%E8%8B%B1%E6%96%87)

docs/docker-build.md

@@ -8,41 +8,14 @@ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin
 ```
 2、编译docker镜像
 ```
-#进入xiaozhi-server目录
+#进入项目根目录
-cd main/xiaozhi-server/
+# 编译server
-# 普通编译
+docker build -t xiaozhi-esp32-server:server_latest -f ./Dockerfile-server .
-docker build -t xiaozhi-esp32-server:local -f ./Dockerfile-pip .
+# 编译web
-```
+docker build -t xiaozhi-esp32-server:web_latest -f ./Dockerfile-web .
-3、测试本地镜像
-```
-docker stop xiaozhi-esp32-server
-docker rm xiaozhi-esp32-server
-
-docker run -d --name xiaozhi-esp32-server --restart always -p 8000:8000 -v $(pwd)/data/.config.yaml:/opt/xiaozhi-esp32-server/config.yaml xiaozhi-esp32-server:local
-
-docker logs -f xiaozhi-esp32-server
 
+# 编译完成后，可以使用docker-compose启动项目
+# docker-compose.yml你需要修改成自己编译的镜像版本
+cd main/xiaozhi-server
+docker-compose up -d
 ```
-5、发布腾讯云镜像
-```
-# amd64
-docker tag xiaozhi-esp32-server:local ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest-amd64
-docker push ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest-amd64
-
-# arm64
-docker tag xiaozhi-esp32-server:local ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest-arm64
-docker push ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest-arm64
-
-# 推送最新版本
-docker manifest rm ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
-docker manifest create ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest-amd64 ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest-arm64 --amend
-docker manifest inspect ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
-docker manifest push ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
-
-```
-6、运行线上镜像
-```
-cd /Users/hrz/myworkspace/docker-java-env/thirddata/
-docker run -d --name xiaozhi-esp32-server --restart always -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-esp32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
-docker logs -f xiaozhi-esp32-server
-```

docs/docker/nginx.conf

@@ -0,0 +1,26 @@
+server {
+    listen 8002;
+    server_name localhost;
+
+    # 静态资源服务（Vue项目）
+    location / {
+        root /usr/share/nginx/html;
+        try_files $uri $uri/ /index.html;
+    }
+
+    # API反向代理（Java项目）
+    location /xiaozhi-esp32-api/ {
+        proxy_pass http://127.0.0.1:8003;
+        proxy_set_header   Host   $host;
+        proxy_cookie_path /api/ /;
+        proxy_set_header   Referer $http_referer;
+        proxy_set_header   Cookie $http_cookie;
+
+        proxy_connect_timeout 10;
+        proxy_send_timeout 10;
+        proxy_read_timeout 10;
+
+        proxy_set_header   X-Real-IP  $remote_addr;
+        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
+    }
+}

docs/docker/start.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+# 启动Java后端（docker内监听8003端口）
+java -jar /app/xiaozhi-esp32-api.jar \
+  --server.port=8003 \
+  --spring.datasource.druid.url=${SPRING_DATASOURCE_DRUID_URL} \
+  --spring.datasource.druid.username=${SPRING_DATASOURCE_DRUID_USERNAME} \
+  --spring.datasource.druid.password=${SPRING_DATASOURCE_DRUID_PASSWORD} \
+  --spring.data.redis.host=${SPRING_DATA_REDIS_HOST} \
+  --spring.data.redis.port=${SPRING_DATA_REDIS_PORT} &
+
+# 启动Nginx（前台运行保持容器存活）
+nginx -g 'daemon off;'

docs/firmware-build.md

@@ -87,3 +87,19 @@ https://espressif.github.io/esp-launchpad/
 
 打开这个教程，[Flash工具/Web端烧录固件（无IDF开发环境）](https://ccnphfhqs21z.feishu.cn/wiki/Zpz4wXBtdimBrLk25WdcXzxcnNS)。
 翻到：`方式二：ESP-Launchpad 浏览器WEB端烧录`，从`3. 烧录固件/下载到开发板`开始，按照教程操作。
+
+烧录成功且联网成功后，通过唤醒词唤醒小智，留意server端输出的控制台信息。
+
+以下是一些常见问题，供参考：
+
+[1、为什么我说的话，小智识别出来很多韩文、日文、英文](../README.md#1tts-%E7%BB%8F%E5%B8%B8%E5%A4%B1%E8%B4%A5%E7%BB%8F%E5%B8%B8%E8%B6%85%E6%97%B6-)
+
+[2、为什么会出现“TTS 任务出错 文件不存在”？](../README.md#1tts-%E7%BB%8F%E5%B8%B8%E5%A4%B1%E8%B4%A5%E7%BB%8F%E5%B8%B8%E8%B6%85%E6%97%B6-)
+
+[3、TTS 经常失败，经常超时](../README.md#1tts-%E7%BB%8F%E5%B8%B8%E5%A4%B1%E8%B4%A5%E7%BB%8F%E5%B8%B8%E8%B6%85%E6%97%B6-)
+
+[4、如何提高小智对话响应速度？](../README.md#1tts-%E7%BB%8F%E5%B8%B8%E5%A4%B1%E8%B4%A5%E7%BB%8F%E5%B8%B8%E8%B6%85%E6%97%B6-)
+
+[5、我说话很慢，停顿时小智老是抢话](../README.md#1tts-%E7%BB%8F%E5%B8%B8%E5%A4%B1%E8%B4%A5%E7%BB%8F%E5%B8%B8%E8%B6%85%E6%97%B6-)
+
+[6、我想通过小智控制电灯、空调、远程开关机等操作](../README.md#1tts-%E7%BB%8F%E5%B8%B8%E5%A4%B1%E8%B4%A5%E7%BB%8F%E5%B8%B8%E8%B6%85%E6%97%B6-)

main/manager-api/src/main/java/xiaozhi/common/exception/ErrorCode.java

@@ -35,8 +35,8 @@ public interface ErrorCode {
     int REDIS_ERROR = 10027;
     int JOB_ERROR = 10028;
     int INVALID_SYMBOL = 10029;
-
+    
-
+    // 密码相关错误码
     int PASSWORD_LENGTH_ERROR = 10030;
     int PASSWORD_WEAK_ERROR = 10031;
     int DEL_MYSELF_ERROR = 10032;

main/manager-web/src/apis/module/user.js

@@ -43,4 +43,19 @@ export default {
                 })
             }).send()
     },
+    // 解绑设备
+    unbindDevice(device_id, callback) {
+        RequestService.sendRequest()
+            .url(`${getServiceUrl()}/api/v1/user/device/unbind/${device_id}`)
+            .method('PUT')
+            .success((res) => {
+                RequestService.clearRequestTime();
+                callback(res);
+            })
+            .fail(() => {
+                RequestService.reAjaxFun(() => {
+                  this.unbindDevice(device_id, callback);
+                });
+              }).send()
+    },
 }

main/manager-web/src/utils/constant.js

@@ -1,6 +1,9 @@
 const HAVE_NO_RESULT = '暂无'
 export default {
     HAVE_NO_RESULT, // 项目的配置信息
+    PAGE: {
+        LOGIN: '/login',
+    },
     STORAGE_KEY: {
         TOKEN: 'TOKEN',
         PUBLIC_KEY: 'PUBLIC_KEY',

main/manager-web/src/views/home.vue

@@ -21,9 +21,9 @@
           </div>
           <div style="display: flex;align-items: center;gap: 8px;margin-top: 10px">
             <div class="serach-box">
-              <el-input placeholder="输入名称搜索.." v-model="serach" style="border: none; background: transparent;" />
+              <el-input placeholder="输入名称搜索.." v-model="serach" style="border: none; background: transparent;" @keyup.enter.native="handleSearch"  />
               <img src="@/assets/home/search.png" alt=""
-                style="width: 12px;height: 12px;margin-right: 11px;cursor: pointer;" />
+                style="width: 12px;height: 12px;margin-right: 11px;cursor: pointer;" @click="handleSearch" />
             </div>
             <img src="@/assets/home/avatar.png" alt="" style="width: 21px;height: 21px;" />
             <div class="user-info">
@@ -56,16 +56,16 @@
             </div>
           </div>
           <div
-            style="display: flex;flex-wrap: wrap;margin-top: 15px;gap: 15px;justify-content: space-between;box-sizing: border-box;">
+            style="display: flex;flex-wrap: wrap;margin-top: 15px;gap: 15px;justify-content: flex-start;box-sizing: border-box;">
-            <div class="device-item" v-for="(item,index) in deviceList" :key="index">
+            <div class="device-item" v-for="(item,index) in filteredDeviceList" :key="index">
-              <div style="display: flex;justify-content: space-between;">
+              <div style="display: flex;justify-content: space-between; align-items: center; ">
                 <div style="font-weight: 700;font-size: 18px;text-align: left;color: #3d4566;">
 <!--                  CC:ba:97:11:a6:ac-->
                   {{item.list[0]?.mac_address}}
                 </div>
-                <div>
+                <div style="display: flex;align-items: center;">
                   <img src="@/assets/home/delete.png" alt=""
-                    style="width: 18px;height: 18px;margin-right: 8px;" />
+                    style="width: 18px;height: 18px;margin-right: 8px;" @click="unbindDevice(item.list[0]?.id)" />
                   <img src="@/assets/home/info.png" alt="" style="width: 18px;height: 18px;" />
                 </div>
               </div>
@@ -229,7 +229,9 @@ export default {
   name: 'home',
   data() {
     return {
-      serach: '',
+      serach: '', // 搜索框输入内容
+      deviceList: [], // 原始设备列表
+      filteredDeviceList: [], // 过滤后的设备列表
       switchValue: false,
       addDeviceDialogVisible: false,
       deviceCode: "",
@@ -250,8 +252,7 @@ export default {
       }],
       userInfo: {
         mobile: '' // 初始化用户信息
-      },
+      }
-      deviceList:[]
     };
   },
   methods: {
@@ -270,14 +271,51 @@ export default {
     // 获取已绑设备
     getList(){
       Api.user.getHomeList(({data})=>{
-        console.log(data.data)
+        this.deviceList = data.data; // 保存原始设备列表
-        this.deviceList = data.data
+        this.filteredDeviceList = data.data; // 初始化过滤后的设备列表
       })
-    }
+    },
+    // 处理搜索
+    handleSearch() {
+      if (this.serach.trim() === '') {
+        // 如果搜索框为空，显示全部设备
+        this.filteredDeviceList = this.deviceList;
+      } else {
+        // 过滤设备列表
+        this.filteredDeviceList = this.deviceList.filter(device => {
+          return (
+            device.list[0]?.mac_address?.includes(this.serach) ||  // 匹配MAC地址
+            device.list[0]?.device_type?.includes(this.serach) ||  // 匹配设备型号
+            device.list[0]?.app_version?.includes(this.serach)     // 匹配APP版本
+          );
+        });
+      }
+    },
+    // 解绑设备
+    unbindDevice(device_id) {
+      this.$confirm('确定要解绑该设备吗？', '提示', {
+        confirmButtonText: '确定',
+        cancelButtonText: '取消',
+        type: 'warning'
+      }).then(() => {
+        // 调用解绑设备的接口
+        Api.user.unbindDevice(device_id, ({ data }) => {
+          if (data.code === 0) {
+            this.$message.success('解绑成功');
+            this.getList();
+          } else {
+            this.$message.error(data.msg || '解绑失败');
+          }
+        });
+      }).catch(() => {
+        // 用户取消操作
+        this.$message.info('已取消解绑');
+      });
+    },
   },
   mounted() {
     this.fetchUserInfo(); // 组件加载时获取用户信息
-    this.getList()
+    this.getList(); // 初始化设备列表
   }
 }
 </script>

main/xiaozhi-server/.dockerignore

@@ -1,13 +0,0 @@
-.git
-__pycache__
-*.pyc
-.env
-Dockerfile
-../docs/
-tmp/
-data/
-LICENSE
-README.md
-README_en.md
-manager/static
-manager/static/webui/

main/xiaozhi-server/Dockerfile

@@ -1,16 +0,0 @@
-# 第一阶段：构建 Python 依赖
-FROM kalicyh/poetry:v3.10_xiaozhi AS builder
-
-WORKDIR /opt/xiaozhi-esp32-server
-
-# 同时拷贝本地环境.venv
-COPY . .
-# 检查是否有缺失
-RUN poetry install --no-root
-
-# 设置虚拟环境路径
-ENV PATH="/app/.venv/bin:$PATH"
-
-# 启动应用
-ENTRYPOINT ["poetry", "run", "python"]
-CMD ["app.py"]

main/xiaozhi-server/config.yaml

@@ -37,7 +37,8 @@ log:
   data_dir: data
 iot:
   Speaker:
-    volume: 100
+    # 设置esp32的音量，范围0-100
+    volume: 80
 xiaozhi:
   type: hello
   version: 1
@@ -64,13 +65,34 @@ CMD_exit:
 
 # 具体处理时选择的模块(The module selected for specific processing)
 selected_module:
-  ASR: FunASR
+  # 语音活动检测模块，默认使用SileroVAD模型
   VAD: SileroVAD
+  # 语音识别模块，默认使用FunASR本地模型
+  ASR: FunASR
   # 将根据配置名称对应的type调用实际的LLM适配器
   LLM: ChatGLMLLM
   # TTS将根据配置名称对应的type调用实际的TTS适配器
   TTS: EdgeTTS
-  Memory: mem0ai
+  # 记忆模块，默认不开启记忆；如果想使用超长记忆，推荐使用mem0ai；如果注重隐私，请使用本地的mem_local_short
+  Memory: nomem
+  # 意图识别模块，默认不开启。开启后，可以播放音乐、控制音量、识别退出指令
+  # 意图识别使用intent_llm，优点：通用性强，缺点：增加串行前置意图识别模块，会增加处理时间
+  # 意图识别使用function_call，缺点：需要所选择的LLM支持function_call，优点：按需调用工具、速度快
+  # 如果意图识别设置成 function_call，建议把LLM设置成：DoubaoLLM，使用的具体model_name是：doubao-pro-32k-functioncall-241028
+  Intent: nointent
+
+# 意图识别，是用于理解用户意图的模块，例如：播放音乐
+Intent:
+  # 不使用意图识别
+  nointent:
+    # 不需要动
+    type: nointent
+  intent_llm:
+    # 不需要动
+    type: intent_llm
+  function_call:
+    # 不需要动
+    type: nointent
 
 Memory:
   mem0ai:
@@ -78,7 +100,13 @@ Memory:
     # https://app.mem0.ai/dashboard/api-keys
     # 每月有1000次免费调用
     api_key: 你的mem0ai api key
-    
+  nomem:
+    # 不想使用记忆功能，可以使用nomem 
+    type: nomem
+  mem_local_short:
+    # 本地记忆功能，通过selected_module的llm总结，数据保存在本地，不会上传到服务器 
+    type: mem_local_short
+
 ASR:
   FunASR:
     type: fun_local
@@ -105,6 +133,16 @@ LLM:
     base_url: https://dashscope.aliyuncs.com/compatible-mode/v1
     model_name: qwen-turbo
     api_key: 你的deepseek web key
+  DoubaoLLM:
+    # 定义LLM API类型
+    type: openai
+    # 先开通服务，打开以下网址，开通的服务搜索Doubao-pro-32k，开通它
+    # 开通改地址：https://console.volcengine.com/ark/region:ark+cn-beijing/openManagement?LLM=%7B%7D&OpenTokenDrawer=false
+    # 免费额度500000token
+    # 开通后，进入这里获取密钥：https://console.volcengine.com/ark/region:ark+cn-beijing/apiKey?apikey=%7B%7D
+    base_url: https://ark.cn-beijing.volces.com/api/v3
+    model_name: doubao-pro-32k-functioncall-241028
+    api_key: 你的doubao web key
   DeepSeekLLM:
     # 定义LLM API类型
     type: openai
@@ -258,6 +296,21 @@ TTS:
     parallel_infer: true
     repetition_penalty: 1.35
     aux_ref_audio_paths: []
+  GPT_SOVITS_V3:
+    type: gpt_sovits_v3
+    url: "http://127.0.0.1:9880/tts"
+    output_file: tmp/
+    text_lang: "auto"
+    ref_audio_path: "caixukun.wav"
+    prompt_lang: "zh"
+    prompt_text: ""
+    top_k: 5
+    top_p: 1
+    temperature: 1
+    sample_steps: 16
+    media_type: "wav"
+    streaming_mode: false
+    threshold: 30
   MinimaxTTS:
     # Minimax语音合成服务，需要先在minimax平台创建账户充值，并获取登录信息
     # 平台地址：https://platform.minimaxi.com/
@@ -321,7 +374,7 @@ TTS:
   TTS302AI:
     # 302AI语音合成服务，需要先在302平台创建账户充值，并获取密钥信息
     # 获取api_keyn路径：https://dash.302.ai/apis/list
-    # 价格，$35/百万字符。火山原版¥450元/万字符
+    # 价格，$35/百万字符。火山原版¥450元/百万字符
     type: doubao
     api_url: https://api.302ai.cn/doubao/tts_hd
     authorization: "Bearer "
@@ -367,18 +420,6 @@ module_test:
 
 # 本地音乐播放配置
 music:
-  music_commands:
-    - "来一首歌"
-    - "唱一首歌"
-    - "播放音乐"
-    - "来点音乐"
-    - "背景音乐"
-    - "放首歌"
-    - "播放歌曲"
-    - "来点背景音乐"
-    - "我想听歌"
-    - "我要听歌"
-    - "放点音乐"
   music_dir: "./music"  # 音乐文件存放路径，将从该目录及子目录下搜索音乐文件
   music_ext: # 音乐文件类型，p3格式效率最高
     - ".mp3"

main/xiaozhi-server/config/functionCallConfig.py

@@ -0,0 +1,36 @@
+FunctionCallConfig = [
+            {
+                "type": "function",
+                "function": {
+                    "name": "handle_exit_intent",
+                    "description": "当用户想结束对话或需要退出系统时调用",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "say_goodbye": {
+                                "type": "string",
+                                "description": "和用户友好结束对话的告别语"
+                            }
+                        },
+                        "required": []
+                    }
+                }
+            },
+            {
+                "type": "function",
+                "function": {
+                    "name": "play_music",
+                    "description": "唱歌、听歌、播放音乐方法。比如用户说播放音乐，参数为：random，比如用户说播放两只老虎，参数为：两只老虎",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "song_name": {
+                                "type": "string",
+                                "description": "歌曲名称，如果没有指定具体歌名则为'random'"
+                            }
+                        },
+                        "required": ["song_name"]
+                    }
+                }
+            }
+        ]

main/xiaozhi-server/core/connection.py

@@ -11,10 +11,11 @@ import websockets
 from typing import Dict, Any
 from core.utils.dialogue import Message, Dialogue
 from core.handle.textHandle import handleTextMessage
-from core.utils.util import get_string_no_punctuation_or_emoji
+from core.utils.util import get_string_no_punctuation_or_emoji, extract_json_from_string
 from concurrent.futures import ThreadPoolExecutor, TimeoutError
 from core.handle.sendAudioHandle import sendAudioMessage
 from core.handle.receiveAudioHandle import handleAudioMessage
+from core.handle.intentHandler import Action, get_functions, handle_llm_function_call
 from config.private_config import PrivateConfig
 from core.auth import AuthMiddleware, AuthenticationError
 from core.utils.auth_code_gen import AuthCodeGenerator
@@ -27,7 +28,7 @@ class TTSException(RuntimeError):
 
 
 class ConnectionHandler:
-    def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music, _memory):
+    def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music, _memory, _intent):
         self.config = config
         self.logger = setup_logging()
         self.auth = AuthMiddleware(config)
@@ -55,6 +56,7 @@ class ConnectionHandler:
         self.llm = _llm
         self.tts = _tts
         self.memory = _memory
+        self.intent = _intent
 
         # vad相关变量
         self.client_audio_buffer = bytes()
@@ -88,6 +90,12 @@ class ConnectionHandler:
         self.auth_code_gen = AuthCodeGenerator.get_instance()
         self.is_device_verified = False  # 添加设备验证状态标志
         self.music_handler = _music
+        self.close_after_chat = False  # 是否在聊天结束后关闭连接
+        self.use_function_call_mode = False
+        if self.config["selected_module"]["Intent"] == 'function_call':
+            self.use_function_call_mode = True
+
+        self.logger.bind(tag=TAG).info(f"use_function_call_mode:{self.use_function_call_mode}")
 
     async def handle_connection(self, ws):
         try:
@@ -101,7 +109,8 @@ class ConnectionHandler:
             await self.auth.authenticate(self.headers)
 
             device_id = self.headers.get("device-id", None)
-            self.memory.set_role_id(device_id)
+            self.memory.init_memory(device_id, self.llm)
+            self.intent.set_llm(self.llm)
 
             # Load private configuration if device_id is provided
             bUsePrivateConfig = self.config.get("use_private_config", False)
@@ -204,7 +213,6 @@ class ConnectionHandler:
             return False
         return not self.is_device_verified
 
-
     def chat(self, query):
         if self.isNeedAuth():
             self.llm_finish_task = True
@@ -213,6 +221,7 @@ class ConnectionHandler:
             return True
 
         self.dialogue.put(Message(role="user", content=query))
+
         response_message = []
         processed_chars = 0  # 跟踪已处理的字符位置
         try:
@@ -220,10 +229,10 @@ class ConnectionHandler:
             # 使用带记忆的对话
             future = asyncio.run_coroutine_threadsafe(self.memory.query_memory(query), self.loop)
             memory_str = future.result()
-            
+
-            self.logger.bind(tag=TAG).info(f"记忆内容: {memory_str}")
+            self.logger.bind(tag=TAG).debug(f"记忆内容: {memory_str}")
             llm_responses = self.llm.response(
-                self.session_id, 
+                self.session_id,
                 self.dialogue.get_llm_dialogue_with_memory(memory_str)
             )
         except Exception as e:
@@ -245,7 +254,7 @@ class ConnectionHandler:
             current_text = full_text[processed_chars:]  # 从未处理的位置开始
 
             # 查找最后一个有效标点
-            punctuations = ("。", "？", "！", "?", "!", ";", "；", ":", "：")
+            punctuations = ("。", "？", "！", "；", "：")
             last_punct_pos = -1
             for punct in punctuations:
                 pos = current_text.rfind(punct)
@@ -282,6 +291,154 @@ class ConnectionHandler:
         self.logger.bind(tag=TAG).debug(json.dumps(self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False))
         return True
 
+    def chat_with_function_calling(self, query):
+        self.logger.bind(tag=TAG).debug(f"Chat with function calling start: {query}")
+        """Chat with function calling for intent detection using streaming"""
+        if self.isNeedAuth():
+            self.llm_finish_task = True
+            future = asyncio.run_coroutine_threadsafe(self._check_and_broadcast_auth_code(), self.loop)
+            future.result()
+            return True
+
+        self.dialogue.put(Message(role="user", content=query))
+
+        # Define intent functions
+        functions = get_functions()
+
+        response_message = []
+        processed_chars = 0  # 跟踪已处理的字符位置
+   
+        try:
+            start_time = time.time()
+
+            # 使用带记忆的对话
+            future = asyncio.run_coroutine_threadsafe(self.memory.query_memory(query), self.loop)
+            memory_str = future.result()
+
+            #self.logger.bind(tag=TAG).info(f"对话记录: {self.dialogue.get_llm_dialogue_with_memory(memory_str)}")
+
+            # 使用支持functions的streaming接口
+            llm_responses = self.llm.response_with_functions(
+                self.session_id,
+                self.dialogue.get_llm_dialogue_with_memory(memory_str),
+                functions=functions
+            )
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"LLM 处理出错 {query}: {e}")
+            return None
+
+        self.llm_finish_task = False
+        text_index = 0
+
+        # 处理流式响应
+        tool_call_flag = False
+        function_name = None
+        function_id = None
+        function_arguments = ""
+        content_arguments = ""
+        for response in llm_responses:
+            content, tools_call = response
+            if content is not None and len(content)>0:
+                if len(response_message)<=0 and content=="```":
+                    tool_call_flag = True
+
+            if tools_call is not None:
+                tool_call_flag = True
+                if tools_call[0].id is not None:
+                    function_id = tools_call[0].id
+                if tools_call[0].function.name is not None:
+                    function_name = tools_call[0].function.name
+                if tools_call[0].function.arguments is not None:
+                    function_arguments += tools_call[0].function.arguments
+
+            if content is not None and len(content) > 0:
+                if tool_call_flag:
+                    content_arguments+=content
+                else:
+                    response_message.append(content)
+
+                    if self.client_abort:
+                        break
+
+                    end_time = time.time()
+                    self.logger.bind(tag=TAG).debug(f"大模型返回时间: {end_time - start_time} 秒, 生成token={content}")
+
+                    # 处理文本分段和TTS逻辑
+                    # 合并当前全部文本并处理未分割部分
+                    full_text = "".join(response_message)
+                    current_text = full_text[processed_chars:]  # 从未处理的位置开始
+
+                    # 查找最后一个有效标点
+                    punctuations = ("。", "？", "！", "；", "：")
+                    last_punct_pos = -1
+                    for punct in punctuations:
+                        pos = current_text.rfind(punct)
+                        if pos > last_punct_pos:
+                            last_punct_pos = pos
+
+                    # 找到分割点则处理
+                    if last_punct_pos != -1:
+                        segment_text_raw = current_text[:last_punct_pos + 1]
+                        segment_text = get_string_no_punctuation_or_emoji(segment_text_raw)
+                        if segment_text:
+                            text_index += 1
+                            self.recode_first_last_text(segment_text, text_index)
+                            future = self.executor.submit(self.speak_and_play, segment_text, text_index)
+                            self.tts_queue.put(future)
+                            processed_chars += len(segment_text_raw)  # 更新已处理字符位置
+
+        # 处理最后剩余的文本
+        full_text = "".join(response_message)
+        remaining_text = full_text[processed_chars:]
+        if remaining_text:
+            segment_text = get_string_no_punctuation_or_emoji(remaining_text)
+            if segment_text:
+                text_index += 1
+                self.recode_first_last_text(segment_text, text_index)
+                future = self.executor.submit(self.speak_and_play, segment_text, text_index)
+                self.tts_queue.put(future)
+
+        # 存储对话内容
+        if len(response_message)>0:
+            self.dialogue.put(Message(role="assistant", content="".join(response_message)))
+
+        # 处理function call
+        if tool_call_flag:
+            if function_id is None:
+                a = extract_json_from_string(content_arguments)
+                if a is not None:
+                    content_arguments_json = json.loads(a)
+                    function_name = content_arguments_json["function_name"]
+                    function_arguments = json.dumps(content_arguments_json["args"], ensure_ascii=False)
+                    function_id = str(uuid.uuid4().hex)
+                else:
+                    return []
+                function_arguments = json.loads(function_arguments)
+            self.logger.bind(tag=TAG).info(f"function_name={function_name}, function_id={function_id}, function_arguments={function_arguments}")
+            function_call_data = {
+                "name": function_name,
+                "id": function_id,
+                "arguments": function_arguments
+            }
+            result = handle_llm_function_call(self, function_call_data)
+            self._handle_function_result(result, function_call_data, text_index+1)
+
+        self.llm_finish_task = True
+        self.logger.bind(tag=TAG).debug(json.dumps(self.dialogue.get_llm_dialogue(), indent=4, ensure_ascii=False))
+
+        return True
+
+    def _handle_function_result(self, result, function_call_data, text_index):
+        if result.action == Action.RESPONSE: # 直接回复前端
+            text = result.response
+            self.recode_first_last_text(text, text_index)
+            future = self.executor.submit(self.speak_and_play, text, text_index)
+            self.tts_queue.put(future)
+            self.dialogue.put(Message(role="assistant", content=text))
+        if result.action == Action.REQLLM: # 调用函数后再请求llm生成回复
+            text = result.response
+        
+
     def _tts_priority_thread(self):
         while not self.stop_event.is_set():
             text = None
@@ -372,3 +529,14 @@ class ConnectionHandler:
         self.client_have_voice_last_time = 0
         self.client_voice_stop = False
         self.logger.bind(tag=TAG).debug("VAD states reset.")
+
+    def chat_and_close(self, text):
+        """Chat with the user and then close the connection"""
+        try:
+            # Use the existing chat method
+            self.chat(text)
+
+            # After chat is complete, close the connection
+            self.close_after_chat = True
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"Chat and close error: {str(e)}")

main/xiaozhi-server/core/handle/intentHandler.py

@@ -0,0 +1,168 @@
+from config.logger import setup_logging
+import json
+from core.handle.sendAudioHandle import send_stt_message
+from core.utils.dialogue import Message
+from config.functionCallConfig import FunctionCallConfig
+import asyncio
+from enum import Enum
+
+TAG = __name__
+logger = setup_logging()
+
+
+class Action(Enum):
+    NOTFOUND = (0, "没有找到函数")
+    NONE = (1, "啥也不干")
+    RESPONSE = (2, "直接回复")
+    REQLLM = (3, "调用函数后再请求llm生成回复")
+
+    def __init__(self, code, message):
+        self.code = code
+        self.message = message
+
+
+class ActionResponse:
+    def __init__(self, action: Action, result, response):
+        self.action = action  # 动作类型
+        self.result = result  # 动作产生的结果
+        self.response = response  # 直接回复的内容
+
+
+def get_functions():
+    """获取功能调用配置"""
+    return FunctionCallConfig
+
+
+def handle_llm_function_call(conn, function_call_data):
+    try:
+        function_name = function_call_data["name"]
+
+        if function_name == "handle_exit_intent":
+            # 处理退出意图
+            try:
+                say_goodbye = json.loads(function_call_data["arguments"]).get("say_goodbye", "再见")
+                conn.close_after_chat = True
+                logger.bind(tag=TAG).info(f"退出意图已处理:{say_goodbye}")
+                return ActionResponse(action=Action.RESPONSE, result="退出意图已处理", response=say_goodbye)
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"处理退出意图错误: {e}")
+
+        elif function_name == "play_music":
+            # 处理音乐播放意图
+            try:
+                song_name = "random"
+                arguments = function_call_data["arguments"]
+                if arguments is not None and len(arguments) > 0:
+                    args = json.loads(arguments)
+                    song_name = args.get("song_name", "random")
+                music_intent = f"播放音乐 {song_name}" if song_name != "random" else "随机播放音乐"
+
+                # 执行音乐播放命令
+                future = asyncio.run_coroutine_threadsafe(
+                    conn.music_handler.handle_music_command(conn, music_intent),
+                    conn.loop
+                )
+                future.result()
+                return ActionResponse(action=Action.RESPONSE, result="退出意图已处理", response="还想听什么歌？")
+            except Exception as e:
+                logger.bind(tag=TAG).error(f"处理音乐意图错误: {e}")
+        else:
+            return ActionResponse(action=Action.NOTFOUND, result="没有找到对应的函数", response="")
+    except Exception as e:
+        logger.bind(tag=TAG).error(f"处理function call错误: {e}")
+
+    return None
+
+
+async def handle_user_intent(conn, text):
+    """
+    Handle user intent before starting chat
+    
+    Args:
+        conn: Connection object
+        text: User's text input
+    
+    Returns:
+        bool: True if intent was handled, False if should proceed to chat
+    """
+    # 检查是否有明确的退出命令
+    if await check_direct_exit(conn, text):
+        return True
+
+    if conn.use_function_call_mode:
+        # 使用支持function calling的聊天方法,不再进行意图分析
+        return False
+
+    logger.bind(tag=TAG).info(f"分析用户意图: {text}")
+
+    # 使用LLM进行意图分析
+    intent = await analyze_intent_with_llm(conn, text)
+
+    if not intent:
+        return False
+
+    # 处理各种意图
+    return await process_intent_result(conn, intent, text)
+
+
+async def check_direct_exit(conn, text):
+    """检查是否有明确的退出命令"""
+    cmd_exit = conn.cmd_exit
+    for cmd in cmd_exit:
+        if text == cmd:
+            logger.bind(tag=TAG).info(f"识别到明确的退出命令: {text}")
+            await conn.close()
+            return True
+    return False
+
+
+async def analyze_intent_with_llm(conn, text):
+    """使用LLM分析用户意图"""
+    if not hasattr(conn, 'intent') or not conn.intent:
+        logger.bind(tag=TAG).warning("意图识别服务未初始化")
+        return None
+
+    # 对话历史记录
+    dialogue = conn.dialogue
+    try:
+        intent_result = await conn.intent.detect_intent(dialogue.dialogue, text)
+        logger.bind(tag=TAG).info(f"意图识别结果: {intent_result}")
+
+        # 尝试解析JSON结果
+        try:
+            intent_data = json.loads(intent_result)
+            if "intent" in intent_data:
+                return intent_data["intent"]
+        except json.JSONDecodeError:
+            # 如果不是JSON格式，尝试直接获取意图文本
+            return intent_result.strip()
+
+    except Exception as e:
+        logger.bind(tag=TAG).error(f"意图识别失败: {str(e)}")
+
+    return None
+
+
+async def process_intent_result(conn, intent, original_text):
+    """处理意图识别结果"""
+    # 处理退出意图
+    if "结束聊天" in intent:
+        logger.bind(tag=TAG).info(f"识别到退出意图: {intent}")
+
+        # 如果正在播放音乐，可以关了 TODO
+
+        # 如果是明确的离别意图，发送告别语并关闭连接
+        await send_stt_message(conn, original_text)
+        conn.executor.submit(conn.chat_and_close, original_text)
+        return True
+
+    # 处理播放音乐意图
+    if "播放音乐" in intent:
+        logger.bind(tag=TAG).info(f"识别到音乐播放意图: {intent}")
+        await conn.music_handler.handle_music_command(conn, intent)
+        return True
+
+    # 其他意图处理可以在这里扩展
+
+    # 默认返回False，表示继续常规聊天流程
+    return False

main/xiaozhi-server/core/handle/iotHandle.py

@@ -109,7 +109,48 @@ async def handleIotDescriptors(conn, descriptors):
         default_iot_volume = conn.config["iot"]["Speaker"]["volume"]
     logger.bind(tag=TAG).info(f"服务端设置音量为{default_iot_volume}")
     await send_iot_conn(conn, "Speaker", "SetVolume", {"volume": default_iot_volume})
+async def handleIotStatus(conn, states):
+    """
+    处理物联网状态
+    示例: [{
+        "name":"Speaker",
+        "state":{
+            "volume":100
+        }
+    }]
+    states: 状态列表
+    """
+    for state in states:
+        for key, value in conn.iot_descriptors.items():
+            if key == state["name"]:
+                for property_item in value.properties:
+                    # properties为字典列表, 记录各种属性
+                    for k, v in state["state"].items():
+                        # state为字典, 记录各种属性的值, 是需要记录的信息
+                        if property_item["name"] == k:
+                            # 检查一下属性是不是相同的
+                            if type(v) != type(property_item["value"]):
+                                logger.bind(tag=TAG).error(f"属性{property_item['name']}的值类型不匹配")
+                                break
+                            else:
+                                property_item["value"] = v
+                                logger.bind(tag=TAG).info(f"物联网状态更新: {key} , {property_item['name']} = {v}")
+                            break
+                break
 
+async def get_iot_status(conn, name, property_name):
+    """
+    获取物联网状态
+    name: 设备名称 "Speaker"
+    property_name: 属性名称 "volume"
+    返回值: 属性值, 实际的属性有int, bool和str三种类型
+    """
+    for key, value in conn.iot_descriptors.items():
+        if key == name:
+            for property_item in value.properties:
+                if property_item["name"] == property_name:
+                    return property_item["value"]
+    return None
 
 async def send_iot_conn(conn, name, method_name, parameters):
     """

main/xiaozhi-server/core/handle/musicHandler.py

@@ -15,7 +15,7 @@ logger = setup_logging()
 
 def _extract_song_name(text):
     """从用户输入中提取歌名"""
-    for keyword in ["听", "播放", "放", "唱"]:
+    for keyword in ["播放音乐"]:
         if keyword in text:
             parts = text.split(keyword)
             if len(parts) > 1:
@@ -36,6 +36,7 @@ def _find_best_match(potential_song, music_files):
             best_match = music_file
     return best_match
 
+
 class MusicManager:
     def __init__(self, music_dir, music_ext):
         self.music_dir = Path(music_dir)
@@ -55,23 +56,20 @@ class MusicManager:
                     music_files.append(str(file.relative_to(self.music_dir)))
         return music_files
 
+
 class MusicHandler:
     def __init__(self, config):
         self.config = config
-        self.music_related_keywords = []
 
         if "music" in self.config:
             self.music_config = self.config["music"]
             self.music_dir = os.path.abspath(
                 self.music_config.get("music_dir", "./music")  # 默认路径修改
             )
-            self.music_related_keywords = self.music_config.get("music_commands", [])
             self.music_ext = self.music_config.get("music_ext", (".mp3", ".wav", ".p3"))
             self.refresh_time = self.music_config.get("refresh_time", 60)
         else:
             self.music_dir = os.path.abspath("./music")
-            self.music_related_keywords = ["来一首歌", "唱一首歌", "播放音乐", "来点音乐", "背景音乐", "放首歌",
-                                           "播放歌曲", "来点背景音乐", "我想听歌", "我要听歌", "放点音乐"]
             self.music_ext = (".mp3", ".wav", ".p3")
             self.refresh_time = 60
 
@@ -100,13 +98,9 @@ class MusicHandler:
                     logger.bind(tag=TAG).info(f"找到最匹配的歌曲: {best_match}")
                     await self.play_local_music(conn, specific_file=best_match)
                     return True
-
         # 检查是否是通用播放音乐命令
-        if any(cmd in clean_text for cmd in self.music_related_keywords):
+        await self.play_local_music(conn)
-            await self.play_local_music(conn)
+        return True
-            return True
-
-        return False
 
     async def play_local_music(self, conn, specific_file=None):
         """播放本地音乐文件"""
@@ -117,26 +111,18 @@ class MusicHandler:
 
             # 确保路径正确性
             if specific_file:
-                music_path = os.path.join(self.music_dir, specific_file)
-                if not os.path.exists(music_path):
-                    logger.bind(tag=TAG).error(f"指定的音乐文件不存在: {music_path}")
-                    return
                 selected_music = specific_file
+                music_path = os.path.join(self.music_dir, specific_file)
             else:
-                if time.time() - self.scan_time > self.refresh_time:
-                    # 刷新音乐文件列表
-                    self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
-                    self.scan_time = time.time()
-                    logger.bind(tag=TAG).debug(f"刷新的音乐文件列表: {self.music_files}")
-
                 if not self.music_files:
                     logger.bind(tag=TAG).error("未找到MP3音乐文件")
                     return
                 selected_music = random.choice(self.music_files)
                 music_path = os.path.join(self.music_dir, selected_music)
-                if not os.path.exists(music_path):
+
-                    logger.bind(tag=TAG).error(f"选定的音乐文件不存在: {music_path}")
+            if not os.path.exists(music_path):
-                    return
+                logger.bind(tag=TAG).error(f"选定的音乐文件不存在: {music_path}")
+                return
             text = f"正在播放{selected_music}"
             await send_stt_message(conn, text)
             conn.tts_first_text_index = 0

main/xiaozhi-server/core/handle/receiveAudioHandle.py

@@ -2,6 +2,7 @@ from config.logger import setup_logging
 import time
 from core.utils.util import remove_punctuation_and_length
 from core.handle.sendAudioHandle import send_stt_message
+from core.handle.intentHandler import handle_user_intent
 
 TAG = __name__
 logger = setup_logging()
@@ -33,13 +34,7 @@ async def handleAudioMessage(conn, audio):
         else:
             text, file_path = await conn.asr.speech_to_text(conn.asr_audio, conn.session_id)
             logger.bind(tag=TAG).info(f"识别文本: {text}")
-            text_len, text_without_punctuation = remove_punctuation_and_length(text)
+            text_len, _ = remove_punctuation_and_length(text)
-            if await conn.music_handler.handle_music_command(conn, text_without_punctuation):
-                conn.asr_server_receive = True
-                conn.asr_audio.clear()
-                return
-            if text_len <= conn.max_cmd_length and await handleCMDMessage(conn, text_without_punctuation):
-                return
             if text_len > 0:
                 await startToChat(conn, text)
             else:
@@ -48,20 +43,22 @@ async def handleAudioMessage(conn, audio):
         conn.reset_vad_states()
 
 
-async def handleCMDMessage(conn, text):
-    cmd_exit = conn.cmd_exit
-    for cmd in cmd_exit:
-        if text == cmd:
-            logger.bind(tag=TAG).info("识别到明确的退出命令".format(text))
-            await conn.close()
-            return True
-    return False
-
-
 async def startToChat(conn, text):
-    # 异步发送 stt 信息
+    # 首先进行意图分析
+    intent_handled = await handle_user_intent(conn, text)
+    
+    if intent_handled:
+        # 如果意图已被处理，不再进行聊天
+        conn.asr_server_receive = True
+        return
+    
+    # 意图未被处理，继续常规聊天流程
     await send_stt_message(conn, text)
-    conn.executor.submit(conn.chat, text)
+    if conn.use_function_call_mode:
+        # 使用支持function calling的聊天方法
+        conn.executor.submit(conn.chat_with_function_calling, text)
+    else:
+        conn.executor.submit(conn.chat, text)
 
 
 async def no_voice_close_connect(conn):
@@ -70,8 +67,9 @@ async def no_voice_close_connect(conn):
     else:
         no_voice_time = time.time() * 1000 - conn.client_no_voice_last_time
         close_connection_no_voice_time = conn.config.get("close_connection_no_voice_time", 120)
-        if no_voice_time > 1000 * close_connection_no_voice_time:
+        if not conn.close_after_chat and no_voice_time > 1000 * close_connection_no_voice_time:
+            conn.close_after_chat = True
             conn.client_abort = False
             conn.asr_server_receive = False
-            prompt = "时间过得真快，我都好久没说话了。请你用十个字左右话跟我告别，以“再见”或“拜拜”为结尾"
+            prompt = "请你以“时间过得真快”未来头，用富有感情、依依不舍的话来结束这场对话吧。"
             await startToChat(conn, prompt)

main/xiaozhi-server/core/handle/sendAudioHandle.py

@@ -7,49 +7,49 @@ from core.utils.util import remove_punctuation_and_length, get_string_no_punctua
 TAG = __name__
 logger = setup_logging()
 
-
-async def isLLMWantToFinish(last_text):
-    _, last_text_without_punctuation = remove_punctuation_and_length(last_text)
-    if "再见" in last_text_without_punctuation or "拜拜" in last_text_without_punctuation:
-        return True
-    return False
-
-
 async def sendAudioMessage(conn, audios, text, text_index=0):
     # 发送句子开始消息
     if text_index == conn.tts_first_text_index:
         logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
     await send_tts_message(conn, "sentence_start", text)
 
-    # 初始化流控参数
+    # 流控参数优化
-    frame_duration = 60  # 毫秒
+    original_frame_duration = 60  # 原始帧时长（毫秒）
-    start_time = time.perf_counter()  # 使用高精度计时器
+    adjusted_frame_duration = int(original_frame_duration * 0.8)  # 缩短20%
-    play_position = 0  # 已播放的时长（毫秒）
+    total_frames = len(audios)  # 获取总帧数
+    compensation = total_frames * (original_frame_duration - adjusted_frame_duration) / 1000  # 补偿时间（秒）
+
+    start_time = time.perf_counter()
+    play_position = 0  # 已播放时长（毫秒）
 
     for opus_packet in audios:
         if conn.client_abort:
             return
 
-        # 计算当前包的预期发送时间
+        # 计算带加速因子的预期时间
         expected_time = start_time + (play_position / 1000)
         current_time = time.perf_counter()
 
-        # 等待直到预期时间
+        # 流控等待（使用加速后的帧时长）
         delay = expected_time - current_time
         if delay > 0:
             await asyncio.sleep(delay)
 
-        # 发送音频包
         await conn.websocket.send(opus_packet)
-        play_position += frame_duration  # 更新播放位置
+        play_position += adjusted_frame_duration  # 使用调整后的帧时长
+
+    # 补偿因加速损失的时长
+    if compensation > 0:
+        await asyncio.sleep(compensation)
+
     await send_tts_message(conn, "sentence_end", text)
+
     # 发送结束消息（如果是最后一个文本）
     if conn.llm_finish_task and text_index == conn.tts_last_text_index:
         await send_tts_message(conn, 'stop', None)
-        if await isLLMWantToFinish(text):
+        if conn.close_after_chat:
             await conn.close()
 
-
 async def send_tts_message(conn, state, text=None):
     """发送 TTS 状态消息"""
     message = {

main/xiaozhi-server/core/handle/textHandle.py

@@ -3,7 +3,7 @@ import json
 from core.handle.abortHandle import handleAbortMessage
 from core.handle.helloHandle import handleHelloMessage
 from core.handle.receiveAudioHandle import startToChat
-from core.handle.iotHandle import handleIotDescriptors
+from core.handle.iotHandle import handleIotDescriptors, handleIotStatus
 
 TAG = __name__
 logger = setup_logging()
@@ -40,5 +40,7 @@ async def handleTextMessage(conn, message):
         elif msg_json["type"] == "iot":
             if "descriptors" in msg_json:
                 await handleIotDescriptors(conn, msg_json["descriptors"])
+            if "states" in msg_json:
+                await handleIotStatus(conn, msg_json["states"])  
     except json.JSONDecodeError:
         await conn.websocket.send(message)

main/xiaozhi-server/core/providers/intent/base.py

@@ -0,0 +1,34 @@
+from abc import ABC, abstractmethod
+from typing import List, Dict
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class IntentProviderBase(ABC):
+    def __init__(self, config):
+        self.config = config
+        self.intent_options = config.get("intent_options", {
+            "continue_chat": "继续聊天",
+            "end_chat": "结束聊天",
+            "play_music": "播放音乐"
+        })
+
+    def set_llm(self, llm):
+        self.llm = llm
+        logger.bind(tag=TAG).debug("Set LLM for intent provider")
+
+    @abstractmethod
+    async def detect_intent(self, dialogue_history: List[Dict], text: str) -> str:
+        """
+        检测用户最后一句话的意图
+        Args:
+            dialogue_history: 对话历史记录列表，每条记录包含role和content
+        Returns:
+            返回识别出的意图，格式为:
+            - "继续聊天"
+            - "结束聊天" 
+            - "播放音乐 歌名" 或 "随机播放音乐"
+        """
+        pass

main/xiaozhi-server/core/providers/intent/intent_llm/intent_llm.py

@@ -0,0 +1,60 @@
+from typing import List, Dict
+from ..base import IntentProviderBase
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+
+class IntentProvider(IntentProviderBase):
+    def __init__(self, config):
+        super().__init__(config)
+        self.llm = None
+        self.promot = self.get_intent_system_prompt()
+
+    def get_intent_system_prompt(self) -> str:
+        """
+        根据配置的意图选项动态生成系统提示词
+        Returns:
+            格式化后的系统提示词
+        """
+        intent_list = []
+        for key, value in self.intent_options.items():
+            if key == "play_music":
+                intent_list.append(f"{value} [歌名]")
+            else:
+                intent_list.append(value)
+                
+        prompt = (
+            "你是一个意图识别助手。你需要根据和用户的对话记录，重点分析用户的最后一句话，判断用户意图属于以下哪一类：\n"
+            f"{', '.join(intent_list)}\n"
+            "如果是唱歌、听歌、播放音乐，请指定歌名，格式为'播放音乐 [识别出的歌名]'。\n"
+            "如果听不出具体歌名，可以返回'随机播放音乐'。\n"
+            "只需要返回意图结果的json，不要解释。"
+            "返回格式如下：\n"
+            "{intent: '用户意图'}"
+        )
+        return prompt
+    
+    async def detect_intent(self, dialogue_history: List[Dict], text:str) -> str:
+        if not self.llm:
+            raise ValueError("LLM provider not set")
+
+        # 构建用户最后一句话的提示
+        msgStr = ""
+        for msg in dialogue_history:
+            if msg.role == "user":
+                msgStr += f"User: {msg.content}\n"
+            elif msg.role== "assistant":
+                msgStr += f"Assistant: {msg.content}\n"
+        msgStr += f"User: {text}\n"
+        user_prompt = f"请分析用户的意图：\n{msgStr}"
+        # 使用LLM进行意图识别
+        intent = self.llm.response_no_stream(
+            system_prompt=self.promot,
+            user_prompt=user_prompt
+        )
+  
+        logger.bind(tag=TAG).info(f"Detected intent: {intent}")
+        return intent.strip()

main/xiaozhi-server/core/providers/intent/nointent/nointent.py

@@ -0,0 +1,20 @@
+from ..base import IntentProviderBase
+from typing import List, Dict
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+
+class IntentProvider(IntentProviderBase):
+    async def detect_intent(self, dialogue_history: List[Dict], text: str) -> str:
+        """
+        默认的意图识别实现，始终返回继续聊天
+        Args:
+            dialogue_history: 对话历史记录列表
+            text: 本次对话记录
+        Returns:
+            固定返回"继续聊天"
+        """
+        logger.bind(tag=TAG).debug("Using NoIntentProvider, always returning continue chat")
+        return self.intent_options["continue_chat"]

main/xiaozhi-server/core/providers/llm/base.py

@@ -1,8 +1,38 @@
 from abc import ABC, abstractmethod
+from config.logger import setup_logging
 
+TAG = __name__
+logger = setup_logging()
 
 class LLMProviderBase(ABC):
     @abstractmethod
     def response(self, session_id, dialogue):
         """LLM response generator"""
         pass
+
+    def response_no_stream(self, system_prompt, user_prompt):
+        try:
+            # 构造对话格式
+            dialogue = [
+                {"role": "system", "content": system_prompt},
+                {"role": "user", "content": user_prompt}
+            ]
+            result = ""
+            for part in self.response("", dialogue):
+                result += part
+            return result
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"Error in Ollama response generation: {e}")
+            return "【LLM服务响应异常】"
+    
+    def response_with_functions(self, session_id, dialogue, functions=None):
+        """
+        Default implementation for function calling (streaming)
+        This should be overridden by providers that support function calls
+        
+        Returns: generator that yields either text tokens or a special function call token
+        """
+        # For providers that don't support functions, just return regular response
+        for token in self.response(session_id, dialogue):
+            yield {"type": "content", "content": token}

main/xiaozhi-server/core/providers/llm/ollama/ollama.py

@@ -1,5 +1,6 @@
 from config.logger import setup_logging
-import requests, json
+from openai import OpenAI
+import json
 from core.providers.llm.base import LLMProviderBase
 
 TAG = __name__
@@ -8,39 +9,51 @@ logger = setup_logging()
 
 class LLMProvider(LLMProviderBase):
     def __init__(self, config):
-
         self.model_name = config.get("model_name")
         self.base_url = config.get("base_url", "http://localhost:11434")
+        # Initialize OpenAI client with Ollama base URL
+        #如果没有v1，增加v1
+        if not self.base_url.endswith("/v1"):
+            self.base_url = f"{self.base_url}/v1"
+            
+        self.client = OpenAI(
+            base_url=self.base_url,
+            api_key="ollama"  # Ollama doesn't need an API key but OpenAI client requires one
+        )
 
-    def response(self, session_id, dialogue):        
+    def response(self, session_id, dialogue):
         try:
-            # Convert dialogue format to Ollama format
+            responses = self.client.chat.completions.create(
-            prompt = ""
+                model=self.model_name,
-            for msg in dialogue:
+                messages=dialogue,
-                if msg["role"] == "system":
-                    prompt += f"System: {msg['content']}\n"
-                elif msg["role"] == "user":
-                    prompt += f"User: {msg['content']}\n"
-                elif msg["role"] == "assistant":
-                    prompt += f"Assistant: {msg['content']}\n"
-
-            # Make request to Ollama API
-            response = requests.post(
-                f"{self.base_url}/api/generate",
-                json={
-                    "model": self.model_name,
-                    "prompt": prompt,
-                    "stream": True
-                },
                 stream=True
             )
-
+            
-            for line in response.iter_lines():
+            for chunk in responses:
-                if line:
+                try:
-                    json_response = json.loads(line)
+                    delta = chunk.choices[0].delta if getattr(chunk, 'choices', None) else None
-                    if "response" in json_response:
+                    content = delta.content if hasattr(delta, 'content') else ''
-                        yield json_response["response"]
+                    if content:
+                        yield content
+                except Exception as e:
+                    logger.bind(tag=TAG).error(f"Error processing chunk: {e}")
 
         except Exception as e:
             logger.bind(tag=TAG).error(f"Error in Ollama response generation: {e}")
             yield "【Ollama服务响应异常】"
+
+    def response_with_functions(self, session_id, dialogue, functions=None):
+        try:
+            stream = self.client.chat.completions.create(
+                model=self.model_name,
+                messages=dialogue,
+                stream=True,
+                tools=functions,
+            )
+            
+            for chunk in stream:
+                yield chunk.choices[0].delta.content, chunk.choices[0].delta.tool_calls
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"Error in Ollama function call: {e}")
+            yield {"type": "content", "content": f"【Ollama服务响应异常: {str(e)}】"}

main/xiaozhi-server/core/providers/llm/openai/openai.py

@@ -43,3 +43,19 @@ class LLMProvider(LLMProviderBase):
 
         except Exception as e:
             logger.bind(tag=TAG).error(f"Error in response generation: {e}")
+
+    def response_with_functions(self, session_id, dialogue, functions=None):
+        try:
+            stream = self.client.chat.completions.create(
+                model=self.model_name,
+                messages=dialogue,
+                stream=True,
+                tools=functions,
+            )
+            
+            for chunk in stream:
+                yield chunk.choices[0].delta.content, chunk.choices[0].delta.tool_calls
+                
+        except Exception as e:
+            self.logger.bind(tag=TAG).error(f"Error in function call streaming: {e}")
+            yield {"type": "content", "content": f"【OpenAI服务响应异常: {e}】"}

main/xiaozhi-server/core/providers/memory/base.py

@@ -8,6 +8,7 @@ class MemoryProviderBase(ABC):
     def __init__(self, config):
         self.config = config
         self.role_id = None
+        self.llm = None
 
     @abstractmethod
     async def save_memory(self, msgs):
@@ -19,5 +20,6 @@ class MemoryProviderBase(ABC):
         """Query memories for specific role based on similarity"""
         return "please implement query method"
 
-    def set_role_id(self, role_id: str):
+    def init_memory(self, role_id, llm):
-        self.role_id = role_id
+        self.role_id = role_id    
+        self.llm = llm

main/xiaozhi-server/core/providers/memory/mem_local_short/mem_local_short.py

@@ -0,0 +1,156 @@
+from ..base import MemoryProviderBase, logger
+import time
+import json
+import os
+import yaml
+from core.utils.util import get_project_dir
+
+short_term_memory_prompt = """
+# 时空记忆编织者
+
+## 核心使命
+构建可生长的动态记忆网络，在有限空间内保留关键信息的同时，智能维护信息演变轨迹
+根据对话记录，总结user的重要信息，以便在未来的对话中提供更个性化的服务
+
+## 记忆法则
+### 1. 三维度记忆评估（每次更新必执行）
+| 维度       | 评估标准                  | 权重分 |
+|------------|---------------------------|--------|
+| 时效性     | 信息新鲜度（按对话轮次） | 40%    |
+| 情感强度   | 含💖标记/重复提及次数     | 35%    |
+| 关联密度   | 与其他信息的连接数量      | 25%    |
+
+### 2. 动态更新机制
+**名字变更处理示例：**
+原始记忆："曾用名": ["张三"], "现用名": "张三丰"
+触发条件：当检测到「我叫X」「称呼我Y」等命名信号时
+操作流程：
+1. 将旧名移入"曾用名"列表
+2. 记录命名时间轴："2024-02-15 14:32:启用张三丰"
+3. 在记忆立方追加：「从张三到张三丰的身份蜕变」
+
+### 3. 空间优化策略
+- **信息压缩术**：用符号体系提升密度
+  - ✅"张三丰[北/软工/🐱]"
+  - ❌"北京软件工程师，养猫"
+- **淘汰预警**：当总字数≥900时触发
+  1. 删除权重分<60且3轮未提及的信息
+  2. 合并相似条目（保留时间戳最近的）
+
+## 记忆结构
+输出格式必须为可解析的json字符串，不需要解释、注释和说明，保存记忆时仅从对话提取信息，不要混入示例内容
+```json
+{
+  "时空档案": {
+    "身份图谱": {
+      "现用名": "",
+      "特征标记": [] 
+    },
+    "记忆立方": [
+      {
+        "事件": "入职新公司",
+        "时间戳": "2024-03-20",
+        "情感值": 0.9,
+        "关联项": ["下午茶"],
+        "保鲜期": 30 
+      }
+    ]
+  },
+  "关系网络": {
+    "高频话题": {"职场": 12},
+    "暗线联系": [""]
+  },
+  "待响应": {
+    "紧急事项": ["需立即处理的任务"], 
+    "潜在关怀": ["可主动提供的帮助"]
+  },
+  "高光语录": [
+    "最打动人心的瞬间，强烈的情感表达，user的原话"
+  ]
+}
+```
+"""
+
+def extract_json_data(json_code):
+    start = json_code.find("```json")
+    # 从start开始找到下一个```结束
+    end = json_code.find("```", start+1)
+    #print("start:", start, "end:", end)
+    if start == -1 or end == -1:
+        try:
+            jsonData = json.loads(json_code)
+            return json_code
+        except Exception as e:
+            print("Error:", e)
+        return ""
+    jsonData = json_code[start+7:end]
+    return jsonData
+
+TAG = __name__
+
+class MemoryProvider(MemoryProviderBase):
+    def __init__(self, config):
+        super().__init__(config)
+        self.short_momery = ""
+        self.memory_path = get_project_dir() + 'data/.memory.yaml'
+        self.load_memory()
+
+    def init_memory(self, role_id, llm):
+        super().init_memory(role_id, llm)
+        self.load_memory()
+    
+    def load_memory(self):
+        all_memory = {}
+        if os.path.exists(self.memory_path):
+            with open(self.memory_path, 'r', encoding='utf-8') as f:
+                all_memory = yaml.safe_load(f) or {}
+        if self.role_id in all_memory:
+            self.short_momery = all_memory[self.role_id]
+    
+    def save_memory_to_file(self):
+        all_memory = {}
+        if os.path.exists(self.memory_path):
+              with open(self.memory_path, 'r', encoding='utf-8') as f:
+                  all_memory = yaml.safe_load(f) or {}
+        all_memory[self.role_id] = self.short_momery
+        with open(self.memory_path, 'w', encoding='utf-8') as f:
+            yaml.dump(all_memory, f, allow_unicode=True)
+        
+    async def save_memory(self, msgs):
+        if self.llm is None:
+            logger.bind(tag=TAG).error("LLM is not set for memory provider")
+            return None
+        
+        if len(msgs) < 2:
+            return None
+        
+        msgStr = ""
+        for msg in msgs:
+            if msg.role == "user":
+                msgStr += f"User: {msg.content}\n"
+            elif msg.role== "assistant":
+                msgStr += f"Assistant: {msg.content}\n"
+        if len(self.short_momery) > 0:
+            msgStr+="历史记忆：\n"
+            msgStr+=self.short_momery
+        
+        #当前时间
+        time_str = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
+        msgStr += f"当前时间：{time_str}"
+
+        result = self.llm.response_no_stream(short_term_memory_prompt, msgStr)
+ 
+        json_str = extract_json_data(result)
+        try:
+            json_data = json.loads(json_str) # 检查json格式是否正确
+            self.short_momery = json_str
+        except Exception as e:
+            print("Error:", e)
+        
+        self.save_memory_to_file()
+        logger.bind(tag=TAG).info(f"Save memory successful - Role: {self.role_id}")
+
+        return self.short_momery
+    
+    async def query_memory(self, query: str)-> str:
+        return self.short_momery

main/xiaozhi-server/core/providers/memory/nomem/nomem.py

@@ -0,0 +1,18 @@
+'''
+不使用记忆，可以选择此模块
+'''
+from ..base import MemoryProviderBase, logger
+
+TAG = __name__
+
+class MemoryProvider(MemoryProviderBase):
+    def __init__(self, config):
+        super().__init__(config)
+      
+    async def save_memory(self, msgs):
+        logger.bind(tag=TAG).debug("nomem mode: No memory saving is performed.")
+        return None
+
+    async def query_memory(self, query: str)-> str:
+        logger.bind(tag=TAG).debug("nomem mode: No memory query is performed.")
+        return ""

main/xiaozhi-server/core/providers/tts/gpt_sovits_v3.py

@@ -0,0 +1,52 @@
+import os
+import uuid
+import requests
+from config.logger import setup_logging
+from datetime import datetime
+from core.providers.tts.base import TTSProviderBase
+
+TAG = __name__
+logger = setup_logging()
+
+class TTSProvider(TTSProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__(config, delete_audio_file)
+        self.url = config.get("url")
+        self.text_lang = config.get("text_lang", "audo")
+        self.ref_audio_path = config.get("ref_audio_path")
+        self.prompt_lang = config.get("prompt_lang")
+        self.prompt_text = config.get("prompt_text")
+        self.top_k = config.get("top_k", 5)
+        self.top_p = config.get("top_p", 1)
+        self.temperature = config.get("temperature", 1)
+        self.sample_steps = config.get("sample_steps", 16)
+        self.media_type = config.get("media_type", "wav")
+        self.streaming_mode = config.get("streaming_mode", False)
+        self.threshold = config.get("threshold", 30)
+
+
+    def generate_filename(self, extension=".wav"):
+        return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
+
+    async def text_to_speak(self, text, output_file):
+        request_params = {
+            "text": text,
+            "text_lang": self.text_lang,
+            "ref_audio_path": self.ref_audio_path,
+            "prompt_lang": self.prompt_lang,
+            "prompt_text": self.prompt_text,
+            "top_k": self.top_k,
+            "top_p": self.top_p,
+            "temperature": self.temperature,
+            "sample_steps": self.sample_steps,
+            "media_type": self.media_type,
+            "streaming_mode": self.streaming_mode,
+            "threshold": self.threshold,
+        }
+
+        resp = requests.get(self.url, params=request_params)
+        if resp.status_code == 200:
+            with open(output_file, "wb") as file:
+                file.write(resp.content)
+        else:
+            logger.bind(tag=TAG).error(f"GPT_SoVITS_V3 TTS请求失败: {resp.status_code} - {resp.text}")

main/xiaozhi-server/core/utils/dialogue.py

@@ -26,6 +26,9 @@ class Dialogue:
         return dialogue
 
     def get_llm_dialogue_with_memory(self, memory_str: str = None) -> List[Dict[str, str]]:
+        if memory_str is None or len(memory_str) == 0:
+            return self.get_llm_dialogue()
+        
         # 构建带记忆的对话
         dialogue = []
         

main/xiaozhi-server/core/utils/intent.py

@@ -0,0 +1,17 @@
+import os
+import sys
+from config.logger import setup_logging
+import importlib
+
+logger = setup_logging()
+
+
+def create_instance(class_name, *args, **kwargs):
+    # 创建intent实例
+    if os.path.exists(os.path.join('core', 'providers', 'intent', class_name, f'{class_name}.py')):
+        lib_name = f'core.providers.intent.{class_name}.{class_name}'
+        if lib_name not in sys.modules:
+            sys.modules[lib_name] = importlib.import_module(f'{lib_name}')
+        return sys.modules[lib_name].IntentProvider(*args, **kwargs)
+
+    raise ValueError(f"不支持的intent类型: {class_name}，请检查该配置的type是否设置正确")

main/xiaozhi-server/core/utils/util.py

@@ -4,6 +4,7 @@ import yaml
 import socket
 import subprocess
 import logging
+import re
 
 
 def get_project_dir():
@@ -75,7 +76,7 @@ def get_string_no_punctuation_or_emoji(s):
 def remove_punctuation_and_length(text):
     # 全角符号和半角符号的Unicode范围
     full_width_punctuations = '！＂＃＄％＆＇（）＊＋，－。／：；＜＝＞？＠［＼］＾＿｀｛｜｝～'
-    half_width_punctuations = '!"#$%&\'()*+,-./:;<=>?@[\]^_`{|}~'
+    half_width_punctuations = r'!"#$%&\'()*+,-./:;<=>?@[\]^_`{|}~'
     space = ' '  # 半角空格
     full_width_space = '　'  # 全角空格
 
@@ -119,3 +120,11 @@ def check_ffmpeg_installed():
         error_msg += "1、按照项目的安装文档，正确进入conda环境\n"
         error_msg += "2、查阅安装文档，如何在conda环境中安装ffmpeg\n"
         raise ValueError(error_msg)
+    
+def extract_json_from_string(input_string):
+    """提取字符串中的 JSON 部分"""
+    pattern = r'(\{.*\})'
+    match = re.search(pattern, input_string)
+    if match:
+        return match.group(1)  # 返回提取的 JSON 字符串
+    return None

main/xiaozhi-server/core/websocket_server.py

@@ -4,7 +4,7 @@ from config.logger import setup_logging
 from core.connection import ConnectionHandler
 from core.handle.musicHandler import MusicHandler
 from core.utils.util import get_local_ip
-from core.utils import asr, vad, llm, tts, memory
+from core.utils import asr, vad, llm, tts, memory, intent
 
 TAG = __name__
 
@@ -13,11 +13,11 @@ class WebSocketServer:
     def __init__(self, config: dict):
         self.config = config
         self.logger = setup_logging()
-        self._vad, self._asr, self._llm, self._tts, self._music, self._memory = self._create_processing_instances()
+        self._vad, self._asr, self._llm, self._tts, self._music, self._memory, self.intent = self._create_processing_instances()
         self.active_connections = set()  # 添加全局连接记录
 
     def _create_processing_instances(self):
-        memory_cls_name = self.config["selected_module"].get("Memory", "mem0ai") # 默认使用mem0ai
+        memory_cls_name = self.config["selected_module"].get("Memory", "nomem") # 默认使用nomem
         has_memory_cfg = self.config.get("Memory") and memory_cls_name in self.config["Memory"]
         memory_cfg = self.config["Memory"][memory_cls_name] if has_memory_cfg else {}
 
@@ -52,6 +52,13 @@ class WebSocketServer:
             ),
             MusicHandler(self.config),
             memory.create_instance(memory_cls_name, memory_cfg),
+            intent.create_instance(
+                self.config["selected_module"]["Intent"]
+                if not 'type' in self.config["Intent"][self.config["selected_module"]["Intent"]]
+                else
+                self.config["Intent"][self.config["selected_module"]["Intent"]]["type"],
+                self.config["Intent"][self.config["selected_module"]["Intent"]]
+            ),
         )
 
     async def start(self):
@@ -71,7 +78,7 @@ class WebSocketServer:
     async def _handle_connection(self, websocket):
         """处理新连接，每次创建独立的ConnectionHandler"""
         # 创建ConnectionHandler时传入当前server实例
-        handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music, self._memory)
+        handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music, self._memory, self.intent)
         self.active_connections.add(handler)
         try:
             await handler.handle_connection(websocket)

main/xiaozhi-server/docker-compose.yml

@@ -1,7 +1,12 @@
+# 如果本机已经安装了MySQL，可以直接在数据库中创建名为`xiaozhi_esp32_server`的数据库。
+# 如果还没有MySQL，你可以通过docker安装mysql,执行以下一句话
+# docker run --name xiaozhi-esp32-server-db -e MYSQL_ROOT_PASSWORD=123456 -p 3306:3306 -e MYSQL_DATABASE=xiaozhi_esp32_server -e MYSQL_INITDB_ARGS="--character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci" -d mysql:latest
+# 如果你的mysql账号和密码有修改过，记得修改下方数据库的账号和密码
+# 记得修改下方数据库的IP，ip不能写127.0.0.1或localhost，否则容器无法访问，要写你电脑局域网ip
 version: '3'
 services:
   xiaozhi-esp32-server:
-    image: ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:latest
+    image: ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:server_latest
     container_name: xiaozhi-esp32-server
     restart: always
     security_opt:
@@ -11,10 +16,23 @@ services:
     ports:
       # ws服务端
       - "8000:8000"
-      # 管理后台
-      - "8002:8002"
     volumes:
       # 配置文件目录
       - ./data:/opt/xiaozhi-esp32-server/data
       # 模型文件挂接，很重要
-      - ./models/SenseVoiceSmall/model.pt:/opt/xiaozhi-esp32-server/models/SenseVoiceSmall/model.pt
+      - ./models/SenseVoiceSmall/model.pt:/opt/xiaozhi-esp32-server/models/SenseVoiceSmall/model.pt
+#  #智控台还没开发好，还不能完全使用，会报很多错误，如果是非技术人员，请不要启用智控台服务
+#  xiaozhi-esp32-server-web:
+#    image: ghcr.nju.edu.cn/xinnan-tech/xiaozhi-esp32-server:web_latest
+#    container_name: xiaozhi-esp32-server-web
+#    restart: always
+#    ports:
+#      - "8002:8002"
+#    environment:
+#      - TZ=Asia/Shanghai
+#      ##记得改mysql和redis IP 密码
+#      - SPRING_DATASOURCE_DRUID_URL=jdbc:mysql://192.168.1.20:3306/xiaozhi_esp32_server?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai
+#      - SPRING_DATASOURCE_DRUID_USERNAME=root
+#      - SPRING_DATASOURCE_DRUID_PASSWORD=123456
+#      - SPRING_DATA_REDIS_HOST=192.168.1.20
+#      - SPRING_DATA_REDIS_PORT=6379

main/xiaozhi-server/pyproject.toml

@@ -1,42 +0,0 @@
-[tool.poetry]
-name = "xiaozhi-esp32-server"
-version = "0.1.0"
-description = ""
-authors = ["kalicyh <34980061+kaliCYH@users.noreply.github.com>"]
-readme = "README.md"
-
-[tool.poetry.dependencies]
-python = "^3.10"
-pyyml = "0.0.2"
-torch = "2.2.2"
-silero-vad = "5.1.2"
-websockets = "14.2"
-numpy = "1.26.4"
-pydub = "0.25.1"
-funasr = "1.2.3"
-torchaudio = "2.2.2"
-openai = "1.61.0"
-google-generativeai = "0.8.4"
-edge-tts = "7.0.0"
-httpx = "0.27.2"
-aiohttp = "3.9.3"
-aiohttp-cors = "0.7.0"
-ormsgpack = "1.7.0"
-ruamel-yaml = "0.18.10"
-setuptools = "^75.8.0"
-loguru = "^0.7.3"
-opuslib-next = "^1.1.2"
-fastapi = {extras = ["all"], version = "^0.115.8"}
-uvicorn = "^0.34.0"
-pyjwt = "^2.10.1"
-python-jose = {extras = ["cryptography"], version = "^3.3.0"}
-bcrypt = "^4.2.1"
-sqlalchemy = "^2.0.38"
-pymysql = "^1.1.1"
-asyncpg = "^0.30.0"
-onnxruntime = "1.19.2"
-
-
-[build-system]
-requires = ["poetry-core"]
-build-backend = "poetry.core.masonry.api"