Mem0ai (#199)

* update:增加mem0ai包依赖

* update:文档增加mem0ai的描述

---------

Co-authored-by: hrz <1710360675@qq.com>

.github/workflows/docker-image.yml

@@ -16,6 +16,14 @@ jobs:
       issues: write
 
     steps:
+      - name: Check Disk Space
+        run: |
+          df -h
+          docker system df
+      - name: Clean up Docker resources
+        run: |
+          docker system prune -af
+          docker builder prune -af
       - name: Check out the repo
         uses: actions/checkout@v4
 
@@ -43,4 +51,4 @@ jobs:
           tags: |
             ghcr.io/${{ github.repository }}:${{ env.VERSION }}
             ghcr.io/${{ github.repository }}:latest
-          platforms: linux/amd64,linux/arm64
+          platforms: linux/amd64,linux/arm64

.gitignore

@@ -132,6 +132,9 @@ dmypy.json
 # Pyre type checker
 .pyre/
 
+# music directory
+music/
+
 # pytype static type analyzer
 .pytype/
 
@@ -146,3 +149,4 @@ tmp
 .private_config.yaml
 .env.development
 docker-compose.yml
+web/vue/node_modules

Dockerfile

@@ -1,20 +1,4 @@
-# 第一阶段：前端构建
+# 第一阶段：构建 Python 依赖
-
-FROM kalicyh/node:v18-alpine AS frontend-builder
-
-WORKDIR /app/ZhiKongTaiWeb
-
-# RUN corepack enable && yarn config set registry https://registry.npmmirror.com
-
-COPY ZhiKongTaiWeb/package.json ZhiKongTaiWeb/yarn.lock ./
-
-RUN yarn install --frozen-lockfile
-
-COPY ZhiKongTaiWeb . 
-RUN yarn build
-
-# 第二阶段：构建 Python 依赖
-
 FROM kalicyh/poetry:v3.10_xiaozhi AS builder
 
 WORKDIR /app
@@ -24,19 +8,6 @@ COPY . .
 # 检查是否有缺失
 RUN poetry install --no-root
 
-# 使用清华源加速apt安装，该镜像内置所以注释
-# RUN rm -rf /etc/apt/sources.list.d/* && \
-#     echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware" > /etc/apt/sources.list && \
-#     echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware" >> /etc/apt/sources.list && \
-#     echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware" >> /etc/apt/sources.list && \
-#     echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security main contrib non-free non-free-firmware" >> /etc/apt/sources.list && \
-#     apt-get update && \
-#     apt-get install -y --no-install-recommends libopus0 ffmpeg && \
-#     apt-get clean
-
-# 从构建阶段复制虚拟环境和前端构建产物
-COPY --from=frontend-builder /app/ZhiKongTaiWeb/dist /app/manager/static/webui
-
 # 设置虚拟环境路径
 ENV PATH="/app/.venv/bin:$PATH"
 

README.md

@@ -57,6 +57,22 @@
          </picture>
         </a>
     </td>
+  </tr>
+  <tr>
+    <td>
+        <a href="https://www.bilibili.com/video/BV1Vy96YCE3R" target="_blank">
+         <picture>
+           <img alt="自定义音色" src="docs/images/demo6.png" />
+         </picture>
+        </a>
+    </td>
+    <td>
+        <a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
+         <picture>
+           <img alt="播放音乐" src="docs/images/demo7.png" />
+         </picture>
+        </a>
+    </td>
     <td>
         <a href="https://www.bilibili.com/video/BV1kgA2eYEQ9" target="_blank">
          <picture>
@@ -66,7 +82,7 @@
     </td>
     <td>
     </td>
-    </tr>
+  </tr>
 </table>
 
 ---
@@ -75,16 +91,24 @@
 
 - **硬件**：一套兼容 `xiaozhi-esp32`
   的硬件设备（具体型号请参考 [此处](https://rcnv1t9vps13.feishu.cn/wiki/DdgIw4BUgivWDPkhMj1cGIYCnRf)）。
-- **服务器**：至少 4 核 CPU、8G 内存的电脑。
+
-- **固件编译**：请将后端服务的接口地址更新至 `xiaozhi-esp32` 项目中，再重新编译固件并烧录到设备上。
+- **电脑或服务器**：至少 4 核 CPU、8G 内存的电脑。
+- **固件编译**：请将本后端服务的接口地址更新至 `xiaozhi-esp32` 项目中，再重新编译`xiaozhi-esp32`固件并烧录到设备上。
+
+如果你没有esp32相关的硬件设备，但是非常想体验该项目，可以使用以下的项目让你的电脑、手机模拟成esp32设备。
+
+- [小智安卓端](https://github.com/TOM88812/xiaozhi-android-client)
+- [小智电脑端](https://github.com/Huang-junsen/py-xiaozhi)
 
 ---
 
 ## 警告 ⚠️
 
-本项目成立时间较短，还未通过网络安全测评，请勿在生产环境中使用。
+1、本项目为开源软件，本软件与对接的任何第三方API服务商（包括但不限于语音识别、大模型、语音合成等平台）均不存在商业合作关系，不为其服务质量及资金安全提供任何形式的担保。
+建议使用者优先选择持有相关业务牌照的服务商，并仔细阅读其服务协议及隐私政策。本软件不托管任何账户密钥、不参与资金流转、不承担充值资金损失风险。
 
-如果您在公网环境中部署学习本项目，请务必在配置文件 `config.yaml` 中开启防护：
+2、本项目成立时间较短，还未通过网络安全测评，请勿在生产环境中使用。 如果您在公网环境中部署学习本项目，请务必在配置文件
+`config.yaml` 中开启防护：
 
 ```yaml
 server:
@@ -170,6 +194,14 @@ server:
 
 ---
 
+### Memory
+
+|   类型   |  平台名称  | 使用方式 | 收费模式 | 备注 |
+|:------:|:------:|:----:|:----:|:--:|
+| Memory | mem0ai | 接口调用 |  免费  |    |
+
+---
+
 ## 使用方式 🚀
 
 ### 一、[部署文档](./docs/Deployment.md)
@@ -302,7 +334,13 @@ TTS 性能排行:
 
 ### 7、更多问题，可联系我们反馈 💬
 
-![图片](docs/images/wechat.jpg)
+我们的联系方式放在[百度网盘中,点击前往](https://pan.baidu.com/s/1x6USjvP1nTRsZ45XlJu65Q)，提取码是`223y`。
+
+网盘里有“硬件烧录QQ群”、“开源服务端交流群”、“产品建议联系人” 三张图片，请根据需要选择加入。
+
+- 硬件烧录QQ群：适用于硬件烧录问题
+- 开源服务端交流群：适用于服务端问题
+- 产品建议联系人：适用于产品功能、产品设计等建议
 
 ---
 

README_en.md

@@ -278,7 +278,7 @@ _As of February 19, 2025, if my computer were located in Haizhu District, Guangz
 
 ### 7. For more questions, feel free to contact us for feedback 💬
 
-![WeChat QR Code](docs/images/wechat.jpg)
+Our contact information is in [Baidu Netdisk](https://pan.baidu.com/s/1x6USjvP1nTRsZ45XlJu65Q)，The extraction code is`223y`。
 
 ---
 

app.py

@@ -4,13 +4,14 @@ from config.settings import load_config, check_config_file
 from core.websocket_server import WebSocketServer
 from manager.http_server import WebUI
 from aiohttp import web
-from core.utils.util import get_local_ip
+from core.utils.util import get_local_ip, check_ffmpeg_installed
 
 TAG = __name__
 
 
 async def main():
     check_config_file()
+    check_ffmpeg_installed()
     logger = setup_logging()
     config = load_config()
 

config.yaml

@@ -80,7 +80,15 @@ selected_module:
   LLM: ChatGLMLLM
   # TTS将根据配置名称对应的type调用实际的TTS适配器
   TTS: EdgeTTS
+  Memory: mem0ai
 
+Memory:
+  mem0ai:
+    type: mem0ai
+    # https://app.mem0.ai/dashboard/api-keys
+    # 每月有1000次免费调用
+    api_key: 你的mem0ai api key
+    
 ASR:
   FunASR:
     type: fun_local
@@ -147,7 +155,6 @@ LLM:
     type: coze
     bot_id: 你的bot_id
     user_id: 你的user_id
-    base_url: "https://api.coze.cn/open_api/v2/chat" # 服务地址
     personal_access_token: 你的coze个人令牌
   LMStudioLLM:
     # 定义LLM API类型
@@ -175,8 +182,10 @@ TTS:
     # 山引擎语音一定要购买花钱，起步价30元，就有100并发了。如果用免费的只有2个并发，会经常报tts错误
     # 购买服务后，购买免费的音色后，可能要等半小时左右，才能使用。
     # 地址：https://console.volcengine.com/speech/service/8
+    api_url: https://openspeech.bytedance.com/api/v1/tts
     voice: BV001_streaming
     output_file: tmp/
+    authorization: "Bearer;"
     appid: 你的火山引擎语音合成服务appid
     access_token: 你的火山引擎语音合成服务access_token
     cluster: volcano_tts
@@ -305,7 +314,18 @@ TTS:
     # volume: 50
     # speech_rate: 0
     # pitch_rate: 0
-
+    # 添加 302.ai TTS 配置
+    # token申请地址：https://dash.302.ai/
+  TTS302AI:
+    # 302AI语音合成服务，需要先在302平台创建账户充值，并获取密钥信息
+    # 获取api_keyn路径：https://dash.302.ai/apis/list
+    # 价格，$35/百万字符。火山原版¥450元/万字符
+    type: doubao
+    api_url: https://api.302ai.cn/doubao/tts_hd
+    authorization: "Bearer "
+    voice: "zh_female_wanwanxiaohe_moon_bigtts"
+    output_file: tmp/
+    access_token: "你的302API密钥"
 # 模块测试配置
 module_test:
   test_sentences:  # 自定义测试语句
@@ -327,4 +347,9 @@ music:
     - "我想听歌"
     - "我要听歌"
     - "放点音乐"
-  music_dir: "./music"  # 音乐文件存放路径
+  music_dir: "./music"  # 音乐文件存放路径，将从该目录及子目录下搜索音乐文件
+  music_ext: # 音乐文件类型，p3格式效率最高
+    - ".mp3"
+    - ".wav"
+    - ".p3"
+  refresh_time: 300 # 刷新音乐列表的时间间隔，单位为秒

core/connection.py

@@ -22,8 +22,12 @@ from core.utils.auth_code_gen import AuthCodeGenerator
 TAG = __name__
 
 
+class TTSException(RuntimeError):
+    pass
+
+
 class ConnectionHandler:
-    def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music):
+    def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music, _memory):
         self.config = config
         self.logger = setup_logging()
         self.auth = AuthMiddleware(config)
@@ -50,7 +54,7 @@ class ConnectionHandler:
         self.asr = _asr
         self.llm = _llm
         self.tts = _tts
-        self.dialogue = None
+        self.memory = _memory
 
         # vad相关变量
         self.client_audio_buffer = bytes()
@@ -68,10 +72,8 @@ class ConnectionHandler:
         self.dialogue = Dialogue()
 
         # tts相关变量
-        self.tts_first_text = None
+        self.tts_first_text_index = -1
-        self.tts_last_text = None
+        self.tts_last_text_index = -1
-        self.tts_start_speak_time = None
-        self.tts_duration = 0
 
         # iot相关变量
         self.iot_descriptors = {}
@@ -99,6 +101,7 @@ class ConnectionHandler:
             await self.auth.authenticate(self.headers)
 
             device_id = self.headers.get("device-id", None)
+            self.memory.set_role_id(device_id)
 
             # Load private configuration if device_id is provided
             bUsePrivateConfig = self.config.get("use_private_config", False)
@@ -161,6 +164,8 @@ class ConnectionHandler:
             self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}")
             await ws.close()
             return
+        finally:
+            await self.memory.save_memory(self.dialogue.dialogue)
 
     async def _route_message(self, message):
         """消息路由"""
@@ -199,15 +204,12 @@ class ConnectionHandler:
             return False
         return not self.is_device_verified
 
+
     def chat(self, query):
         if self.isNeedAuth():
             self.llm_finish_task = True
-            loop = asyncio.new_event_loop()
+            future = asyncio.run_coroutine_threadsafe(self._check_and_broadcast_auth_code(), self.loop)
-            asyncio.set_event_loop(loop)
+            future.result()
-            try:
-                loop.run_until_complete(self._check_and_broadcast_auth_code())
-            finally:
-                loop.close()
             return True
 
         self.dialogue.put(Message(role="user", content=query))
@@ -215,12 +217,21 @@ class ConnectionHandler:
         processed_chars = 0  # 跟踪已处理的字符位置
         try:
             start_time = time.time()
-            llm_responses = self.llm.response(self.session_id, self.dialogue.get_llm_dialogue())
+            # 使用带记忆的对话
+            future = asyncio.run_coroutine_threadsafe(self.memory.query_memory(query), self.loop)
+            memory_str = future.result()
+            
+            self.logger.bind(tag=TAG).info(f"记忆内容: {memory_str}")
+            llm_responses = self.llm.response(
+                self.session_id, 
+                self.dialogue.get_llm_dialogue_with_memory(memory_str)
+            )
         except Exception as e:
             self.logger.bind(tag=TAG).error(f"LLM 处理出错 {query}: {e}")
             return None
 
         self.llm_finish_task = False
+        text_index = 0
         for content in llm_responses:
             response_message.append(content)
             if self.client_abort:
@@ -234,7 +245,7 @@ class ConnectionHandler:
             current_text = full_text[processed_chars:]  # 从未处理的位置开始
 
             # 查找最后一个有效标点
-            punctuations = ("。", "？", "！", ".", "?", "!", ";", "；", ":", "：", "，", ",")
+            punctuations = ("。", "？", "！", "?", "!", ";", "；", ":", "：")
             last_punct_pos = -1
             for punct in punctuations:
                 pos = current_text.rfind(punct)
@@ -246,8 +257,12 @@ class ConnectionHandler:
                 segment_text_raw = current_text[:last_punct_pos + 1]
                 segment_text = get_string_no_punctuation_or_emoji(segment_text_raw)
                 if segment_text:
-                    self.recode_first_last_text(segment_text)
+                    # 强制设置空字符，测试TTS出错返回语音的健壮性
-                    future = self.executor.submit(self.speak_and_play, segment_text)
+                    # if text_index % 2 == 0:
+                    #     segment_text = " "
+                    text_index += 1
+                    self.recode_first_last_text(segment_text, text_index)
+                    future = self.executor.submit(self.speak_and_play, segment_text, text_index)
                     self.tts_queue.put(future)
                     processed_chars += len(segment_text_raw)  # 更新已处理字符位置
 
@@ -257,8 +272,9 @@ class ConnectionHandler:
         if remaining_text:
             segment_text = get_string_no_punctuation_or_emoji(remaining_text)
             if segment_text:
-                self.recode_first_last_text(segment_text)
+                text_index += 1
-                future = self.executor.submit(self.speak_and_play, segment_text)
+                self.recode_first_last_text(segment_text, text_index)
+                future = self.executor.submit(self.speak_and_play, segment_text, text_index)
                 self.tts_queue.put(future)
 
         self.llm_finish_task = True
@@ -274,30 +290,28 @@ class ConnectionHandler:
                 if future is None:
                     continue
                 text = None
+                opus_datas, text_index, tts_file = [], 0, None
                 try:
                     self.logger.bind(tag=TAG).debug("正在处理TTS任务...")
-                    tts_file, text = future.result(timeout=10)
+                    tts_file, text, text_index = future.result(timeout=10)
                     if text is None or len(text) <= 0:
-                        continue
+                        self.logger.bind(tag=TAG).error(f"TTS出错：{text_index}: tts text is empty")
-                    if tts_file is None:
+                    elif tts_file is None:
-                        self.logger.bind(tag=TAG).error(f"TTS文件生成失败: {text}")
+                        self.logger.bind(tag=TAG).error(f"TTS出错： file is empty: {text_index}: {text}")
-                        continue
-                    self.logger.bind(tag=TAG).debug(f"TTS文件生成完毕，文件路径: {tts_file}")
-                    if os.path.exists(tts_file):
-                        opus_datas, duration = self.tts.wav_to_opus_data(tts_file)
                     else:
-                        self.logger.bind(tag=TAG).error(f"TTS文件不存在: {tts_file}")
+                        self.logger.bind(tag=TAG).debug(f"TTS生成：文件路径: {tts_file}")
-                        opus_datas = []
+                        if os.path.exists(tts_file):
+                            opus_datas, duration = self.tts.wav_to_opus_data(tts_file)
+                        else:
+                            self.logger.bind(tag=TAG).error(f"TTS出错：文件不存在{tts_file}")
                 except TimeoutError:
-                    self.logger.bind(tag=TAG).error("TTS 任务超时")
+                    self.logger.bind(tag=TAG).error("TTS超时")
-                    continue
                 except Exception as e:
-                    self.logger.bind(tag=TAG).error(f"TTS 任务出错: {e}")
+                    self.logger.bind(tag=TAG).error(f"TTS出错: {e}")
-                    continue
                 if not self.client_abort:
                     # 如果没有中途打断就发送语音
-                    self.audio_play_queue.put((opus_datas, text))
+                    self.audio_play_queue.put((opus_datas, text, text_index))
-                if self.tts.delete_audio_file and os.path.exists(tts_file):
+                if self.tts.delete_audio_file and tts_file is not None and os.path.exists(tts_file):
                     os.remove(tts_file)
             except Exception as e:
                 self.logger.bind(tag=TAG).error(f"TTS任务处理错误: {e}")
@@ -306,42 +320,41 @@ class ConnectionHandler:
                     self.websocket.send(json.dumps({"type": "tts", "state": "stop", "session_id": self.session_id})),
                     self.loop
                 )
-                self.logger.bind(tag=TAG).error(f"tts_priority priority_thread: {text}{e}")
+                self.logger.bind(tag=TAG).error(f"tts_priority priority_thread: {text} {e}")
 
     def _audio_play_priority_thread(self):
         while not self.stop_event.is_set():
             text = None
             try:
-                opus_datas, text = self.audio_play_queue.get()
+                opus_datas, text, text_index = self.audio_play_queue.get()
-                future = asyncio.run_coroutine_threadsafe(sendAudioMessage(self, opus_datas, text), self.loop)
+                future = asyncio.run_coroutine_threadsafe(sendAudioMessage(self, opus_datas, text, text_index),
+                                                          self.loop)
                 future.result()
             except Exception as e:
-                self.logger.bind(tag=TAG).error(f"audio_play_priority priority_thread: {text}{e}")
+                self.logger.bind(tag=TAG).error(f"audio_play_priority priority_thread: {text} {e}")
 
-    def speak_and_play(self, text):
+    def speak_and_play(self, text, text_index=0):
         if text is None or len(text) <= 0:
             self.logger.bind(tag=TAG).info(f"无需tts转换，query为空，{text}")
-            return None, text
+            return None, text, text_index
         tts_file = self.tts.to_tts(text)
         if tts_file is None:
             self.logger.bind(tag=TAG).error(f"tts转换失败，{text}")
-            return None, text
+            return None, text, text_index
         self.logger.bind(tag=TAG).debug(f"TTS 文件生成完毕: {tts_file}")
-        return tts_file, text
+        return tts_file, text, text_index
 
     def clearSpeakStatus(self):
         self.logger.bind(tag=TAG).debug(f"清除服务端讲话状态")
         self.asr_server_receive = True
-        self.tts_last_text = None
+        self.tts_last_text_index = -1
-        self.tts_first_text = None
+        self.tts_first_text_index = -1
-        self.tts_duration = 0
-        self.tts_start_speak_time = None
 
-    def recode_first_last_text(self, text):
+    def recode_first_last_text(self, text, text_index=0):
-        if not self.tts_first_text:
+        if self.tts_first_text_index == -1:
             self.logger.bind(tag=TAG).info(f"大模型说出第一句话: {text}")
-            self.tts_first_text = text
+            self.tts_first_text_index = text_index
-        self.tts_last_text = text
+        self.tts_last_text_index = text_index
 
     async def close(self):
         """资源清理方法"""

core/handle/musicHandler.py

@@ -4,7 +4,10 @@ import random
 import difflib
 import re
 import traceback
+from pathlib import Path
+import time
 from core.handle.sendAudioHandle import send_stt_message
+from core.utils import p3
 
 TAG = __name__
 logger = setup_logging()
@@ -33,6 +36,24 @@ def _find_best_match(potential_song, music_files):
             best_match = music_file
     return best_match
 
+class MusicManager:
+    def __init__(self, music_dir, music_ext):
+        self.music_dir = Path(music_dir)
+        self.music_ext = music_ext
+
+    def get_music_files(self):
+        music_files = []
+        for file in self.music_dir.rglob("*"):
+            # 判断是否是文件
+            if file.is_file():
+                # 获取文件扩展名
+                ext = file.suffix.lower()
+                # 判断扩展名是否在列表中
+                if ext in self.music_ext:
+                    # music_files.append(str(file.resolve()))  # 添加绝对路径
+                    # 添加相对路径
+                    music_files.append(str(file.relative_to(self.music_dir)))
+        return music_files
 
 class MusicHandler:
     def __init__(self, config):
@@ -45,10 +66,19 @@ class MusicHandler:
                 self.music_config.get("music_dir", "./music")  # 默认路径修改
             )
             self.music_related_keywords = self.music_config.get("music_commands", [])
+            self.music_ext = self.music_config.get("music_ext", (".mp3", ".wav", ".p3"))
+            self.refresh_time = self.music_config.get("refresh_time", 60)
         else:
             self.music_dir = os.path.abspath("./music")
             self.music_related_keywords = ["来一首歌", "唱一首歌", "播放音乐", "来点音乐", "背景音乐", "放首歌",
                                            "播放歌曲", "来点背景音乐", "我想听歌", "我要听歌", "放点音乐"]
+            self.music_ext = (".mp3", ".wav", ".p3")
+            self.refresh_time = 60
+
+        # 获取音乐文件列表
+        self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
+        self.scan_time = time.time()
+        logger.bind(tag=TAG).debug(f"找到的音乐文件: {self.music_files}")
 
     async def handle_music_command(self, conn, text):
         """处理音乐播放指令"""
@@ -57,12 +87,15 @@ class MusicHandler:
 
         # 尝试匹配具体歌名
         if os.path.exists(self.music_dir):
-            music_files = [f for f in os.listdir(self.music_dir) if f.endswith('.mp3')]
+            if time.time() - self.scan_time > self.refresh_time:
-            logger.bind(tag=TAG).debug(f"找到的音乐文件: {music_files}")
+                # 刷新音乐文件列表
+                self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
+                self.scan_time = time.time()
+                logger.bind(tag=TAG).debug(f"刷新的音乐文件: {self.music_files}")
 
             potential_song = _extract_song_name(clean_text)
             if potential_song:
-                best_match = _find_best_match(potential_song, music_files)
+                best_match = _find_best_match(potential_song, self.music_files)
                 if best_match:
                     logger.bind(tag=TAG).info(f"找到最匹配的歌曲: {best_match}")
                     await self.play_local_music(conn, specific_file=best_match)
@@ -90,20 +123,30 @@ class MusicHandler:
                     return
                 selected_music = specific_file
             else:
-                music_files = [f for f in os.listdir(self.music_dir) if f.endswith('.mp3')]
+                if time.time() - self.scan_time > self.refresh_time:
-                if not music_files:
+                    # 刷新音乐文件列表
+                    self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
+                    self.scan_time = time.time()
+                    logger.bind(tag=TAG).debug(f"刷新的音乐文件列表: {self.music_files}")
+
+                if not self.music_files:
                     logger.bind(tag=TAG).error("未找到MP3音乐文件")
                     return
-                selected_music = random.choice(music_files)
+                selected_music = random.choice(self.music_files)
                 music_path = os.path.join(self.music_dir, selected_music)
+                if not os.path.exists(music_path):
+                    logger.bind(tag=TAG).error(f"选定的音乐文件不存在: {music_path}")
+                    return
             text = f"正在播放{selected_music}"
             await send_stt_message(conn, text)
-            conn.tts_first_text = selected_music
+            conn.tts_first_text_index = 0
-            conn.tts_last_text = selected_music
+            conn.tts_last_text_index = 0
             conn.llm_finish_task = True
-            opus_packets, duration = conn.tts.wav_to_opus_data(music_path)
+            if music_path.endswith(".p3"):
-
+                opus_packets, duration = p3.decode_opus_from_file(music_path)
-            conn.audio_play_queue.put((opus_packets, selected_music))
+            else:
+                opus_packets, duration = conn.tts.wav_to_opus_data(music_path)
+            conn.audio_play_queue.put((opus_packets, selected_music, 0))
 
         except Exception as e:
             logger.bind(tag=TAG).error(f"播放音乐失败: {str(e)}")

core/handle/sendAudioHandle.py

@@ -15,30 +15,37 @@ async def isLLMWantToFinish(last_text):
     return False
 
 
-async def sendAudioMessage(conn, audios, text):
+async def sendAudioMessage(conn, audios, text, text_index=0):
-    # 发送 tts.start
+    # 发送句子开始消息
-    if text == conn.tts_first_text:
+    if text_index == conn.tts_first_text_index:
         logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
-        conn.tts_start_speak_time = time.time()
     await send_tts_message(conn, "sentence_start", text)
 
-    # 发送音频数据
+    # 初始化流控参数
-    frame_duration = 60  # 初始帧持续时间（毫秒）
+    frame_duration = 60  # 毫秒
-    start_time = time.time()  # 记录开始时间
+    start_time = time.perf_counter()  # 使用高精度计时器
-    for idx, opus_packet in enumerate(audios):
+    play_position = 0  # 已播放的时长（毫秒）
+
+    for opus_packet in audios:
         if conn.client_abort:
             return
+
         # 计算当前包的预期发送时间
-        expected_time = start_time + idx * (frame_duration / 1000)
+        expected_time = start_time + (play_position / 1000)
-        current_time = time.time()
+        current_time = time.perf_counter()
-        # 如果未到预期时间则等待差值
+
-        if current_time < expected_time:
+        # 等待直到预期时间
-            await asyncio.sleep(expected_time - current_time)
+        delay = expected_time - current_time
+        if delay > 0:
+            await asyncio.sleep(delay)
+
         # 发送音频包
         await conn.websocket.send(opus_packet)
-
+        play_position += frame_duration  # 更新播放位置
-    if conn.llm_finish_task and text == conn.tts_last_text:
+    await send_tts_message(conn, "sentence_end", text)
-        await send_tts_message(conn, 'stop')
+    # 发送结束消息（如果是最后一个文本）
+    if conn.llm_finish_task and text_index == conn.tts_last_text_index:
+        await send_tts_message(conn, 'stop', None)
         if await isLLMWantToFinish(text):
             await conn.close()
 

core/providers/llm/coze/coze.py

@@ -3,94 +3,35 @@ import requests
 import json
 import re
 from core.providers.llm.base import LLMProviderBase
+import os
+# official coze sdk for Python [cozepy](https://github.com/coze-dev/coze-py)
+from cozepy import COZE_CN_BASE_URL
+from cozepy import Coze, TokenAuth, Message, ChatStatus, MessageContentType, ChatEventType  # noqa
 
 TAG = __name__
 logger = setup_logging()
 
-# 定义用于匹配中文标点符号的正则表达式（包括句号、感叹号、问号、分号）
-punctuation_pattern = re.compile(r'([。！？；])')
-
 class LLMProvider(LLMProviderBase):
     def __init__(self, config):
         self.personal_access_token = config.get("personal_access_token")
         self.bot_id = config.get("bot_id")
-        self.user_id = config.get("user_id")  # 默认用户 ID
+        self.user_id = config.get("user_id")
-        self.base_url = config.get("base_url")
 
     def response(self, session_id, dialogue):
-        try:
+        coze_api_token = self.personal_access_token
-            # 从对话中取出最新的用户消息
+        coze_api_base = COZE_CN_BASE_URL
-            last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
+
-            data = {
+        last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
-                "conversation_id": session_id,
+
-                "bot_id": self.bot_id,
+        coze = Coze(auth=TokenAuth(token=coze_api_token), base_url=coze_api_base)
-                "user": self.user_id,
+
-                "query": last_msg["content"],
+        for event in coze.chat.stream(
-                "stream": True
+            bot_id=self.bot_id,
-            }
+            user_id=self.user_id,
-            logger.bind(tag=TAG).info(f"发送到 Coze API 的请求数据: {json.dumps(data, ensure_ascii=False)}")
+            additional_messages=[
-            
+                Message.build_user_question_text(last_msg["content"]),
-            headers = {
+            ],
-                'Authorization': f'Bearer {self.personal_access_token}',
+        ):
-                'Content-Type': 'application/json',
+            if event.event == ChatEventType.CONVERSATION_MESSAGE_DELTA:
-                'Accept': '*/*',
+                print(event.message.content, end="", flush=True)
-                'Host': 'api.coze.cn',
+                yield event.message.content
-                'Connection': 'keep-alive'
-            }
-            
-            response = requests.post(
-                self.base_url,
-                headers=headers,
-                json=data,
-                stream=True
-            )
-            logger.bind(tag=TAG).info(f"请求状态: {response.status_code}")
-            
-            if response.status_code == 200:
-                # 对每一行流数据进行处理，不做跨块累积
-                for line_bytes in response.iter_lines(decode_unicode=False):
-                    if not line_bytes:
-                        continue
-                    try:
-                        # 使用 utf-8 解码，错误部分用替换符
-                        line = line_bytes.decode('utf-8', errors='replace')
-                    except Exception as e:
-                        logger.bind(tag=TAG).error(f"解码失败: {e}")
-                        continue
-                    if line.startswith("data:"):
-                        data_str = line[len("data:"):].strip()
-                        if data_str == "[DONE]":
-                            break
-                        try:
-                            data_chunk = json.loads(data_str)
-                        except json.JSONDecodeError as e:
-                            logger.bind(tag=TAG).error(f"JSON解析失败: {e} 数据: {line}")
-                            continue
-                        msg = data_chunk.get("message", {})
-                        if msg.get("role") == "assistant" and msg.get("type") == "answer":
-                            content = msg.get("content", "")
-                            # 如果返回内容中包含标点符号，则按标点拆分，立即返回每个片段
-                            if punctuation_pattern.search(content):
-                                # 利用 finditer 找到每个标点，并返回以标点结尾的片段
-                                start = 0
-                                for match in punctuation_pattern.finditer(content):
-                                    end = match.end()
-                                    sentence = content[start:end].strip()
-                                    if sentence:
-                                        yield sentence
-                                    start = end
-                                # 如果拆分后剩余内容也返回（不含标点），直接返回
-                                if start < len(content):
-                                    remainder = content[start:].strip()
-                                    if remainder:
-                                        yield remainder
-                            else:
-                                # 如果没有标点，则直接返回这块内容
-                                if content.strip():
-                                    yield content.strip()
-            else:
-                logger.bind(tag=TAG).error(f"请求失败，状态码: {response.status_code}")
-                yield f"【Coze服务响应异常：请求失败，状态码 {response.status_code}】"
-        except Exception as e:
-            logger.bind(tag=TAG).error(f"Error in Coze response generation: {e}")
-            yield "【Coze服务响应异常】"

core/providers/memory/base.py

@@ -0,0 +1,23 @@
+from abc import ABC, abstractmethod
+from config.logger import setup_logging
+
+TAG = __name__
+logger = setup_logging()
+
+class MemoryProviderBase(ABC):
+    def __init__(self, config):
+        self.config = config
+        self.role_id = None
+
+    @abstractmethod
+    async def save_memory(self, msgs):
+        """Save a new memory for specific role and return memory ID"""
+        print("this is base func", msgs)
+
+    @abstractmethod
+    async def query_memory(self, query: str) -> str:
+        """Query memories for specific role based on similarity"""
+        return "please implement query method"
+
+    def set_role_id(self, role_id: str):
+        self.role_id = role_id

core/providers/memory/mem0ai/mem0ai.py

@@ -0,0 +1,74 @@
+from ..base import MemoryProviderBase, logger
+from mem0 import MemoryClient
+
+TAG = __name__
+
+class MemoryProvider(MemoryProviderBase):
+    def __init__(self, config):
+        super().__init__(config)
+        self.api_key = config.get("api_key", "")
+        self.api_version = config.get("api_version", "v1.1")
+        if len(self.api_key) == 0 or "你" in self.api_key:
+            logger.bind(tag=TAG).error("你还没配置Mem0ai的密钥，请在配置文件中配置密钥，否则无法提供记忆服务")
+            self.use_mem0 = False
+            return
+        else:
+            self.use_mem0 = True
+        self.client = MemoryClient(api_key=self.api_key)
+
+    async def save_memory(self, msgs):
+        if not self.use_mem0:
+            return None
+        if len(msgs) < 2:
+            return None
+        
+        try:
+            # Format the content as a message list for mem0
+            messages = [
+                {"role": message.role, "content": message.content}
+                for message in msgs if message.role != "system"
+            ]
+            result = self.client.add(messages, user_id=self.role_id, output_format=self.api_version)
+            logger.bind(tag=TAG).debug(f"Save memory result: {result}")
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"保存记忆失败: {str(e)}")
+            return None
+
+    async def query_memory(self, query: str)-> str:
+        if not self.use_mem0:
+            return ""
+        try:
+            results = self.client.search(
+                query,
+                user_id=self.role_id,
+                output_format=self.api_version
+            )
+            if not results or 'results' not in results:
+                return ""
+                
+            # Format each memory entry with its update time up to minutes
+            memories = []
+            for entry in results['results']:
+                timestamp = entry.get('updated_at', '')
+                if timestamp:
+                    try:
+                        # Parse and reformat the timestamp
+                        dt = timestamp.split('.')[0]  # Remove milliseconds
+                        formatted_time = dt.replace('T', ' ')
+                    except:
+                        formatted_time = timestamp
+                memory = entry.get('memory', '')
+                if timestamp and memory:
+                    # Store tuple of (timestamp, formatted_string) for sorting
+                    memories.append((timestamp, f"[{formatted_time}] {memory}"))
+            
+            # Sort by timestamp in descending order (newest first)
+            memories.sort(key=lambda x: x[0], reverse=True)
+            
+            # Extract only the formatted strings
+            memories_str = "\n".join(f"- {memory[1]}" for memory in memories)
+            logger.bind(tag=TAG).debug(f"Query results: {memories_str}")
+            return memories_str
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"查询记忆失败: {str(e)}")
+            return ""

core/providers/tts/doubao.py

@@ -14,10 +14,9 @@ class TTSProvider(TTSProviderBase):
         self.access_token = config.get("access_token")
         self.cluster = config.get("cluster")
         self.voice = config.get("voice")
-
+        self.api_url = config.get("api_url")
-        self.host = "openspeech.bytedance.com"
+        self.authorization = config.get("authorization")
-        self.api_url = f"https://{self.host}/api/v1/tts"
+        self.header = {"Authorization": f"{self.authorization}{self.access_token}"}
-        self.header = {"Authorization": f"Bearer;{self.access_token}"}
 
     def generate_filename(self, extension=".wav"):
         return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")

core/utils/dialogue.py

@@ -24,3 +24,27 @@ class Dialogue:
         for m in self.dialogue:
             dialogue.append({"role": m.role, "content": m.content})
         return dialogue
+
+    def get_llm_dialogue_with_memory(self, memory_str: str = None) -> List[Dict[str, str]]:
+        # 构建带记忆的对话
+        dialogue = []
+        
+        # 添加系统提示和记忆
+        system_message = next(
+            (msg for msg in self.dialogue if msg.role == "system"), None
+        )
+
+
+        if system_message:
+            enhanced_system_prompt = (
+                f"{system_message.content}\n\n"
+                f"相关记忆：\n{memory_str}"
+            )
+            dialogue.append({"role": "system", "content": enhanced_system_prompt})
+
+        # 添加用户和助手的对话
+        for msg in self.dialogue:
+            if msg.role != "system":  # 跳过原始的系统消息
+                dialogue.append({"role": msg.role, "content": msg.content})
+
+        return dialogue

core/utils/memory.py

@@ -0,0 +1,17 @@
+import os
+import sys
+import importlib
+from config.logger import setup_logging
+from core.utils.util import read_config, get_project_dir
+
+logger = setup_logging()
+
+def create_instance(class_name, *args, **kwargs):
+    if os.path.exists(os.path.join('core', 'providers', 'memory', class_name, f'{class_name}.py')):
+        lib_name = f'core.providers.memory.{class_name}.{class_name}'
+        if lib_name not in sys.modules:
+            sys.modules[lib_name] = importlib.import_module(f'{lib_name}')
+        return sys.modules[lib_name].MemoryProvider(*args, **kwargs)
+
+    raise ValueError(f"不支持的记忆服务类型: {class_name}")
+

core/utils/p3.py

@@ -0,0 +1,33 @@
+import struct
+
+def decode_opus_from_file(input_file):
+    """
+    从p3文件中解码 Opus 数据，并返回一个 Opus 数据包的列表以及总时长。
+    """
+    opus_datas = []
+    total_frames = 0
+    sample_rate = 16000  # 文件采样率
+    frame_duration_ms = 60  # 帧时长
+    frame_size = int(sample_rate * frame_duration_ms / 1000)
+
+    with open(input_file, 'rb') as f:
+        while True:
+            # 读取头部（4字节）：[1字节类型，1字节保留，2字节长度]
+            header = f.read(4)
+            if not header:
+                break
+
+            # 解包头部信息
+            _, _, data_len = struct.unpack('>BBH', header)
+
+            # 根据头部指定的长度读取 Opus 数据
+            opus_data = f.read(data_len)
+            if len(opus_data) != data_len:
+                raise ValueError(f"Data length({len(opus_data)}) mismatch({data_len}) in the file.")
+
+            opus_datas.append(opus_data)
+            total_frames += 1
+
+    # 计算总时长
+    total_duration = (total_frames * frame_duration_ms) / 1000.0
+    return opus_datas, total_duration

core/utils/util.py

@@ -3,6 +3,7 @@ import re
 import json
 import yaml
 import socket
+import subprocess
 
 
 def get_project_dir():
@@ -114,3 +115,29 @@ def check_password(password):
 
     # 如果满足所有条件，则返回True
     return True
+
+def check_ffmpeg_installed():
+    ffmpeg_installed = False
+    try:
+        # 执行ffmpeg -version命令，并捕获输出
+        result = subprocess.run(
+            ['ffmpeg', '-version'],
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            text=True,
+            check=True  # 如果返回码非零则抛出异常
+        )
+        # 检查输出中是否包含版本信息（可选）
+        output = result.stdout + result.stderr
+        if 'ffmpeg version' in output.lower():
+            ffmpeg_installed = True
+        return False
+    except (subprocess.CalledProcessError, FileNotFoundError):
+        # 命令执行失败或未找到
+        ffmpeg_installed = False
+    if not ffmpeg_installed:
+        error_msg = "您的电脑还没正确安装ffmpeg\n"
+        error_msg += "\n建议您：\n"
+        error_msg += "1、按照项目的安装文档，正确进入conda环境\n"
+        error_msg += "2、查阅安装文档，如何在conda环境中安装ffmpeg\n"
+        raise ValueError(error_msg)

core/websocket_server.py

@@ -4,7 +4,7 @@ from config.logger import setup_logging
 from core.connection import ConnectionHandler
 from core.handle.musicHandler import MusicHandler
 from core.utils.util import get_local_ip
-from core.utils import asr, vad, llm, tts
+from core.utils import asr, vad, llm, tts, memory
 
 TAG = __name__
 
@@ -13,10 +13,14 @@ class WebSocketServer:
     def __init__(self, config: dict):
         self.config = config
         self.logger = setup_logging()
-        self._vad, self._asr, self._llm, self._tts, self._music = self._create_processing_instances()
+        self._vad, self._asr, self._llm, self._tts, self._music, self._memory = self._create_processing_instances()
         self.active_connections = set()  # 添加全局连接记录
 
     def _create_processing_instances(self):
+        memory_cls_name = self.config["selected_module"].get("Memory", "mem0ai") # 默认使用mem0ai
+        has_memory_cfg = self.config.get("Memory") and memory_cls_name in self.config["Memory"]
+        memory_cfg = self.config["Memory"][memory_cls_name] if has_memory_cfg else {}
+
         """创建处理模块实例"""
         return (
             vad.create_instance(
@@ -46,7 +50,8 @@ class WebSocketServer:
                 self.config["TTS"][self.config["selected_module"]["TTS"]],
                 self.config["delete_audio"]
             ),
-            MusicHandler(self.config)
+            MusicHandler(self.config),
+            memory.create_instance(memory_cls_name, memory_cfg),
         )
 
     async def start(self):
@@ -66,7 +71,7 @@ class WebSocketServer:
     async def _handle_connection(self, websocket):
         """处理新连接，每次创建独立的ConnectionHandler"""
         # 创建ConnectionHandler时传入当前server实例
-        handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music)
+        handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music, self._memory)
         self.active_connections.add(handler)
         try:
             await handler.handle_connection(websocket)

docs/Deployment.md

@@ -159,14 +159,31 @@ poetry run python app.py
 本项目使用`conda`管理依赖环境。如果不方便安装`conda`，需要根据实际的操作系统安装好`libopus`和`ffmpeg`。
 如果确定使用`conda`，则安装好后，开始执行以下命令。
 
+重要提示！windows 用户，可以通过安装`Anaconda`来管理环境。安装好`Anaconda`后，在`开始`那里搜索`anaconda`相关的关键词，
+找到`Anaconda Prpmpt`，使用管理员身份运行它。如下图。
+
+![conda_prompt](./images/conda_env_1.png)
+
+运行之后，如果你能看到命令行窗口前面有一个(base)字样，说明你成功进入了`conda`环境。那么你就可以执行以下命令了。
+
+![conda_env](./images/conda_env_2.png)
+
 ```
 conda remove -n xiaozhi-esp32-server --all -y
 conda create -n xiaozhi-esp32-server python=3.10 -y
 conda activate xiaozhi-esp32-server
-conda install conda-forge::libopus -y
+
-conda install conda-forge::ffmpeg -y
+# 添加清华源通道
+conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
+conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
+conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
+
+conda install libopus -y
+conda install ffmpeg -y
 ```
 
+请注意，以上命令，不是一股脑执行就成功的，你需要一步步执行，每一步执行完后，都检查一下输出的日志，查看是否成功。
+
 ## 2.安装本项目依赖
 
 你先要下载本项目源码，源码可以通过`git clone`命令下载，如果你不熟悉`git clone`命令。
@@ -179,7 +196,7 @@ conda install conda-forge::ffmpeg -y
 你需要把它重命名成`xiaozhi-esp32-server`，好了请记住这个目录，我们暂且称它为`项目目录`。
 
 ```
-# 使用dos或者终端，进入到你的项目目录，执行以下命令
+# 继续使用conda环境，进入到你的项目目录，执行以下命令
 conda activate xiaozhi-esp32-server
 pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
 pip install -r requirements.txt
@@ -214,9 +231,10 @@ python app.py
 如果你的`项目目录`目录没有`data`，你需要创建`data`目录。
 如果你的`data`下面没有`.config.yaml`文件，你可以把源码目录下的`config.yaml`文件复制一份，重命名为`.config.yaml`
 
-修改`项目目录`下`data`目录下的`.config.yaml`文件，配置本项目所需的各种参数。默认的LLM使用的是`ChatGLMLLM`
+修改`项目目录`下`data`目录下的`.config.yaml`文件，配置本项目必须的两个配置。
-，你需要配置密钥，因为他们的模型，虽然有免费的，但是仍要去[官网](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)注册密钥，才能启动。
+
-默认的TTS使用的是`EdgeTTS`，这个无需配置，如果你需要更换成`豆包TTS`，则需要配置密钥。
+- 默认的LLM使用的是`ChatGLMLLM`，你需要配置密钥，因为他们的模型，虽然有免费的，但是仍要去[官网](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)注册密钥，才能启动。
+- 默认的记忆层`mem0ai`，你需要配置密钥，因为他们的API，虽然有免费额度，但是仍要去[官网](https://app.mem0.ai/dashboard/api-keys)注册密钥，才能启动。
 
 配置说明：这里是各个功能使用的默认组件，例如LLM默认使用`ChatGLMLLM`模型。如果需要切换模型，就是改对应的名称。
 本项目的默认配置仅是成本最低配置（`glm-4-flash`和`EdgeTTS`都是免费的），如果需要更优的更快的搭配，需要自己结合部署环境切换各组件的使用。
@@ -250,7 +268,7 @@ LLM:
 ## 模型文件
 
 本项目语音识别模型，默认使用`SenseVoiceSmall`模型，进行语音转文字。因为模型较大，需要独立下载，下载后把`model.pt`
-文件放在`model/SenseVoiceSmall`
+文件放在`models/SenseVoiceSmall`
 目录下。下面两个下载路线任选一个。
 
 - 线路一：阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)

requirements.txt

@@ -16,4 +16,7 @@ aiohttp_cors==0.7.0
 ormsgpack==1.7.0
 ruamel.yaml==0.18.10
 loguru==0.7.3
-requests==2.32.3
+requests==2.32.3
+cozepy==0.12.0
+miniaudio==1.61
+mem0ai==0.1.62