Compare commits

...
17 Commits
Author SHA1 Message Date
8b151d07c2 Mem0ai (#199)
* update:增加mem0ai包依赖

* update:文档增加mem0ai的描述

---------

Co-authored-by: hrz <1710360675@qq.com>
2025-03-05 00:20:51 +08:00
香草味的纳西妲andGitHub e31e04639b add: 不跟踪music下的文件的更改 (#187) 2025-03-04 22:40:17 +08:00
TOM88812andGitHub 53479fafa5 Merge pull request #188 from xinnan-tech/mem0ai
Mem0ai分支合并到主分支,增加mem0ai模块,支持api储存和查询记忆
2025-03-04 20:29:25 +08:00
玄凤科技 93a94dea45 记忆对时间戳排序,便于梳理前后关系 2025-03-04 13:58:14 +08:00
玄凤科技 83911d9ad1 记忆增加时间,以便大模型判断先后关系 2025-03-04 10:28:38 +08:00
玄凤科技 cf7acb12d8 Merge branch 'main' into mem0ai 2025-03-04 09:13:14 +08:00
玄凤科技 343ecb3ad4 优化chat中异步调用方式 2025-03-04 08:54:27 +08:00
72fa13b0e7 update:更新demo (#180)
Co-authored-by: hrz <1710360675@qq.com>
2025-03-04 01:08:13 +08:00
bee0663040 TTS处理增加index信息,完善TTS出错时的处理 (#177)
* TTS处理增加index信息,完善TTS出错时的处理

* 第一句语音index处理,歌曲播放index处理

* update:去掉了无用变量,修复continuebug

---------

Co-authored-by: hrz <1710360675@qq.com>
2025-03-04 00:35:51 +08:00
8dd045d9fd Fix docker space (#179)
* update:调试docker编译缓存

* update:调试docker编译

* update:调试docker编译容量

* update:调试docker编译

---------

Co-authored-by: hrz <1710360675@qq.com>
2025-03-03 22:34:48 +08:00
HonestQiaoandGitHub 126aa6d4a7 Merge pull request #175 from strix214/main
修改了coze的实现方式,使用了v3接口,修复了只能对话一次的bug
2025-03-03 16:57:34 +08:00
玄凤科技 f2e68060de 使用mem0ai api实现记忆功能 2025-03-03 15:00:04 +08:00
strix214 6507a2ae3e 修改了coze的实现方式,使用了v3接口,修复了只能对话一次的bug 2025-03-03 14:19:30 +08:00
effd79b465 Tts302 (#170)
* feat: add 302.ai TTS integration (#166)

* update:将tts_302ai tts合并到doubao tts

---------

Co-authored-by: Junsen <66542771+Huang-junsen@users.noreply.github.com>
Co-authored-by: hrz <1710360675@qq.com>
2025-03-02 16:59:25 +08:00
bc531652e9 音乐播放功能添加p3文件格式支持,添加文件目录(含子目录)扫描和文件列表刷新功能 (#157)
Co-authored-by: hrz <1710360675@qq.com>
2025-03-01 17:57:43 +08:00
1c5581b765 Fix ffmpeg env (#163)
* update:ffmpeg依赖检查

* update:微信群更换发布方式

---------

Co-authored-by: hrz <1710360675@qq.com>
2025-03-01 17:34:14 +08:00
b486599ab8 update:ffmpeg依赖检查 (#162)
Co-authored-by: hrz <1710360675@qq.com>
2025-03-01 17:09:01 +08:00
26 changed files with 490 additions and 216 deletions
+9 -1
View File
@@ -16,6 +16,14 @@ jobs:
issues: write
steps:
- name: Check Disk Space
run: |
df -h
docker system df
- name: Clean up Docker resources
run: |
docker system prune -af
docker builder prune -af
- name: Check out the repo
uses: actions/checkout@v4
@@ -43,4 +51,4 @@ jobs:
tags: |
ghcr.io/${{ github.repository }}:${{ env.VERSION }}
ghcr.io/${{ github.repository }}:latest
platforms: linux/amd64,linux/arm64
platforms: linux/amd64,linux/arm64
+4
View File
@@ -132,6 +132,9 @@ dmypy.json
# Pyre type checker
.pyre/
# music directory
music/
# pytype static type analyzer
.pytype/
@@ -146,3 +149,4 @@ tmp
.private_config.yaml
.env.development
docker-compose.yml
web/vue/node_modules
+1 -30
View File
@@ -1,20 +1,4 @@
# 第一阶段:前端构建
FROM kalicyh/node:v18-alpine AS frontend-builder
WORKDIR /app/ZhiKongTaiWeb
# RUN corepack enable && yarn config set registry https://registry.npmmirror.com
COPY ZhiKongTaiWeb/package.json ZhiKongTaiWeb/yarn.lock ./
RUN yarn install --frozen-lockfile
COPY ZhiKongTaiWeb .
RUN yarn build
# 第二阶段:构建 Python 依赖
# 第一阶段:构建 Python 依赖
FROM kalicyh/poetry:v3.10_xiaozhi AS builder
WORKDIR /app
@@ -24,19 +8,6 @@ COPY . .
# 检查是否有缺失
RUN poetry install --no-root
# 使用清华源加速apt安装,该镜像内置所以注释
# RUN rm -rf /etc/apt/sources.list.d/* && \
# echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware" > /etc/apt/sources.list && \
# echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware" >> /etc/apt/sources.list && \
# echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware" >> /etc/apt/sources.list && \
# echo "deb https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security main contrib non-free non-free-firmware" >> /etc/apt/sources.list && \
# apt-get update && \
# apt-get install -y --no-install-recommends libopus0 ffmpeg && \
# apt-get clean
# 从构建阶段复制虚拟环境和前端构建产物
COPY --from=frontend-builder /app/ZhiKongTaiWeb/dist /app/manager/static/webui
# 设置虚拟环境路径
ENV PATH="/app/.venv/bin:$PATH"
+44 -6
View File
@@ -57,6 +57,22 @@
</picture>
</a>
</td>
</tr>
<tr>
<td>
<a href="https://www.bilibili.com/video/BV1Vy96YCE3R" target="_blank">
<picture>
<img alt="自定义音色" src="docs/images/demo6.png" />
</picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
<picture>
<img alt="播放音乐" src="docs/images/demo7.png" />
</picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1kgA2eYEQ9" target="_blank">
<picture>
@@ -66,7 +82,7 @@
</td>
<td>
</td>
</tr>
</tr>
</table>
---
@@ -75,16 +91,24 @@
- **硬件**:一套兼容 `xiaozhi-esp32`
的硬件设备(具体型号请参考 [此处](https://rcnv1t9vps13.feishu.cn/wiki/DdgIw4BUgivWDPkhMj1cGIYCnRf))。
- **服务器**:至少 4 核 CPU、8G 内存的电脑。
- **固件编译**:请将后端服务的接口地址更新至 `xiaozhi-esp32` 项目中,再重新编译固件并烧录到设备上
- **电脑或服务器**:至少 4 核 CPU、8G 内存的电脑
- **固件编译**:请将本后端服务的接口地址更新至 `xiaozhi-esp32` 项目中,再重新编译`xiaozhi-esp32`固件并烧录到设备上。
如果你没有esp32相关的硬件设备,但是非常想体验该项目,可以使用以下的项目让你的电脑、手机模拟成esp32设备。
- [小智安卓端](https://github.com/TOM88812/xiaozhi-android-client)
- [小智电脑端](https://github.com/Huang-junsen/py-xiaozhi)
---
## 警告 ⚠️
本项目成立时间较短,还未通过网络安全测评,请勿在生产环境中使用
1、本项目为开源软件,本软件与对接的任何第三方API服务商(包括但不限于语音识别、大模型、语音合成等平台)均不存在商业合作关系,不为其服务质量及资金安全提供任何形式的担保
建议使用者优先选择持有相关业务牌照的服务商,并仔细阅读其服务协议及隐私政策。本软件不托管任何账户密钥、不参与资金流转、不承担充值资金损失风险。
如果您在公网环境中部署学习本项目,请务必在配置文件 `config.yaml` 中开启防护:
2、本项目成立时间较短,还未通过网络安全测评,请勿在生产环境中使用。 如果您在公网环境中部署学习本项目,请务必在配置文件
`config.yaml` 中开启防护:
```yaml
server:
@@ -170,6 +194,14 @@ server:
---
### Memory
| 类型 | 平台名称 | 使用方式 | 收费模式 | 备注 |
|:------:|:------:|:----:|:----:|:--:|
| Memory | mem0ai | 接口调用 | 免费 | |
---
## 使用方式 🚀
### 一、[部署文档](./docs/Deployment.md)
@@ -302,7 +334,13 @@ TTS 性能排行:
### 7、更多问题,可联系我们反馈 💬
![图片](docs/images/wechat.jpg)
我们的联系方式放在[百度网盘中,点击前往](https://pan.baidu.com/s/1x6USjvP1nTRsZ45XlJu65Q),提取码是`223y`
网盘里有“硬件烧录QQ群”、“开源服务端交流群”、“产品建议联系人” 三张图片,请根据需要选择加入。
- 硬件烧录QQ群:适用于硬件烧录问题
- 开源服务端交流群:适用于服务端问题
- 产品建议联系人:适用于产品功能、产品设计等建议
---
+1 -1
View File
@@ -278,7 +278,7 @@ _As of February 19, 2025, if my computer were located in Haizhu District, Guangz
### 7. For more questions, feel free to contact us for feedback 💬
![WeChat QR Code](docs/images/wechat.jpg)
Our contact information is in [Baidu Netdisk](https://pan.baidu.com/s/1x6USjvP1nTRsZ45XlJu65Q)The extraction code is`223y`
---
+2 -1
View File
@@ -4,13 +4,14 @@ from config.settings import load_config, check_config_file
from core.websocket_server import WebSocketServer
from manager.http_server import WebUI
from aiohttp import web
from core.utils.util import get_local_ip
from core.utils.util import get_local_ip, check_ffmpeg_installed
TAG = __name__
async def main():
check_config_file()
check_ffmpeg_installed()
logger = setup_logging()
config = load_config()
+28 -3
View File
@@ -80,7 +80,15 @@ selected_module:
LLM: ChatGLMLLM
# TTS将根据配置名称对应的type调用实际的TTS适配器
TTS: EdgeTTS
Memory: mem0ai
Memory:
mem0ai:
type: mem0ai
# https://app.mem0.ai/dashboard/api-keys
# 每月有1000次免费调用
api_key: 你的mem0ai api key
ASR:
FunASR:
type: fun_local
@@ -147,7 +155,6 @@ LLM:
type: coze
bot_id: 你的bot_id
user_id: 你的user_id
base_url: "https://api.coze.cn/open_api/v2/chat" # 服务地址
personal_access_token: 你的coze个人令牌
LMStudioLLM:
# 定义LLM API类型
@@ -175,8 +182,10 @@ TTS:
# 山引擎语音一定要购买花钱,起步价30元,就有100并发了。如果用免费的只有2个并发,会经常报tts错误
# 购买服务后,购买免费的音色后,可能要等半小时左右,才能使用。
# 地址:https://console.volcengine.com/speech/service/8
api_url: https://openspeech.bytedance.com/api/v1/tts
voice: BV001_streaming
output_file: tmp/
authorization: "Bearer;"
appid: 你的火山引擎语音合成服务appid
access_token: 你的火山引擎语音合成服务access_token
cluster: volcano_tts
@@ -305,7 +314,18 @@ TTS:
# volume: 50
# speech_rate: 0
# pitch_rate: 0
# 添加 302.ai TTS 配置
# token申请地址:https://dash.302.ai/
TTS302AI:
# 302AI语音合成服务,需要先在302平台创建账户充值,并获取密钥信息
# 获取api_keyn路径:https://dash.302.ai/apis/list
# 价格,$35/百万字符。火山原版¥450元/万字符
type: doubao
api_url: https://api.302ai.cn/doubao/tts_hd
authorization: "Bearer "
voice: "zh_female_wanwanxiaohe_moon_bigtts"
output_file: tmp/
access_token: "你的302API密钥"
# 模块测试配置
module_test:
test_sentences: # 自定义测试语句
@@ -327,4 +347,9 @@ music:
- "我想听歌"
- "我要听歌"
- "放点音乐"
music_dir: "./music" # 音乐文件存放路径
music_dir: "./music" # 音乐文件存放路径,将从该目录及子目录下搜索音乐文件
music_ext: # 音乐文件类型,p3格式效率最高
- ".mp3"
- ".wav"
- ".p3"
refresh_time: 300 # 刷新音乐列表的时间间隔,单位为秒
+63 -50
View File
@@ -22,8 +22,12 @@ from core.utils.auth_code_gen import AuthCodeGenerator
TAG = __name__
class TTSException(RuntimeError):
pass
class ConnectionHandler:
def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music):
def __init__(self, config: Dict[str, Any], _vad, _asr, _llm, _tts, _music, _memory):
self.config = config
self.logger = setup_logging()
self.auth = AuthMiddleware(config)
@@ -50,7 +54,7 @@ class ConnectionHandler:
self.asr = _asr
self.llm = _llm
self.tts = _tts
self.dialogue = None
self.memory = _memory
# vad相关变量
self.client_audio_buffer = bytes()
@@ -68,10 +72,8 @@ class ConnectionHandler:
self.dialogue = Dialogue()
# tts相关变量
self.tts_first_text = None
self.tts_last_text = None
self.tts_start_speak_time = None
self.tts_duration = 0
self.tts_first_text_index = -1
self.tts_last_text_index = -1
# iot相关变量
self.iot_descriptors = {}
@@ -99,6 +101,7 @@ class ConnectionHandler:
await self.auth.authenticate(self.headers)
device_id = self.headers.get("device-id", None)
self.memory.set_role_id(device_id)
# Load private configuration if device_id is provided
bUsePrivateConfig = self.config.get("use_private_config", False)
@@ -161,6 +164,8 @@ class ConnectionHandler:
self.logger.bind(tag=TAG).error(f"Connection error: {str(e)}-{stack_trace}")
await ws.close()
return
finally:
await self.memory.save_memory(self.dialogue.dialogue)
async def _route_message(self, message):
"""消息路由"""
@@ -199,15 +204,12 @@ class ConnectionHandler:
return False
return not self.is_device_verified
def chat(self, query):
if self.isNeedAuth():
self.llm_finish_task = True
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(self._check_and_broadcast_auth_code())
finally:
loop.close()
future = asyncio.run_coroutine_threadsafe(self._check_and_broadcast_auth_code(), self.loop)
future.result()
return True
self.dialogue.put(Message(role="user", content=query))
@@ -215,12 +217,21 @@ class ConnectionHandler:
processed_chars = 0 # 跟踪已处理的字符位置
try:
start_time = time.time()
llm_responses = self.llm.response(self.session_id, self.dialogue.get_llm_dialogue())
# 使用带记忆的对话
future = asyncio.run_coroutine_threadsafe(self.memory.query_memory(query), self.loop)
memory_str = future.result()
self.logger.bind(tag=TAG).info(f"记忆内容: {memory_str}")
llm_responses = self.llm.response(
self.session_id,
self.dialogue.get_llm_dialogue_with_memory(memory_str)
)
except Exception as e:
self.logger.bind(tag=TAG).error(f"LLM 处理出错 {query}: {e}")
return None
self.llm_finish_task = False
text_index = 0
for content in llm_responses:
response_message.append(content)
if self.client_abort:
@@ -234,7 +245,7 @@ class ConnectionHandler:
current_text = full_text[processed_chars:] # 从未处理的位置开始
# 查找最后一个有效标点
punctuations = ("", "", "", ".", "?", "!", ";", "", ":", "", "", ",")
punctuations = ("", "", "", "?", "!", ";", "", ":", "")
last_punct_pos = -1
for punct in punctuations:
pos = current_text.rfind(punct)
@@ -246,8 +257,12 @@ class ConnectionHandler:
segment_text_raw = current_text[:last_punct_pos + 1]
segment_text = get_string_no_punctuation_or_emoji(segment_text_raw)
if segment_text:
self.recode_first_last_text(segment_text)
future = self.executor.submit(self.speak_and_play, segment_text)
# 强制设置空字符,测试TTS出错返回语音的健壮性
# if text_index % 2 == 0:
# segment_text = " "
text_index += 1
self.recode_first_last_text(segment_text, text_index)
future = self.executor.submit(self.speak_and_play, segment_text, text_index)
self.tts_queue.put(future)
processed_chars += len(segment_text_raw) # 更新已处理字符位置
@@ -257,8 +272,9 @@ class ConnectionHandler:
if remaining_text:
segment_text = get_string_no_punctuation_or_emoji(remaining_text)
if segment_text:
self.recode_first_last_text(segment_text)
future = self.executor.submit(self.speak_and_play, segment_text)
text_index += 1
self.recode_first_last_text(segment_text, text_index)
future = self.executor.submit(self.speak_and_play, segment_text, text_index)
self.tts_queue.put(future)
self.llm_finish_task = True
@@ -274,30 +290,28 @@ class ConnectionHandler:
if future is None:
continue
text = None
opus_datas, text_index, tts_file = [], 0, None
try:
self.logger.bind(tag=TAG).debug("正在处理TTS任务...")
tts_file, text = future.result(timeout=10)
tts_file, text, text_index = future.result(timeout=10)
if text is None or len(text) <= 0:
continue
if tts_file is None:
self.logger.bind(tag=TAG).error(f"TTS文件生成失败: {text}")
continue
self.logger.bind(tag=TAG).debug(f"TTS文件生成完毕,文件路径: {tts_file}")
if os.path.exists(tts_file):
opus_datas, duration = self.tts.wav_to_opus_data(tts_file)
self.logger.bind(tag=TAG).error(f"TTS出错:{text_index}: tts text is empty")
elif tts_file is None:
self.logger.bind(tag=TAG).error(f"TTS出错: file is empty: {text_index}: {text}")
else:
self.logger.bind(tag=TAG).error(f"TTS文件不存在: {tts_file}")
opus_datas = []
self.logger.bind(tag=TAG).debug(f"TTS生成:文件路径: {tts_file}")
if os.path.exists(tts_file):
opus_datas, duration = self.tts.wav_to_opus_data(tts_file)
else:
self.logger.bind(tag=TAG).error(f"TTS出错:文件不存在{tts_file}")
except TimeoutError:
self.logger.bind(tag=TAG).error("TTS 任务超时")
continue
self.logger.bind(tag=TAG).error("TTS超时")
except Exception as e:
self.logger.bind(tag=TAG).error(f"TTS 任务出错: {e}")
continue
self.logger.bind(tag=TAG).error(f"TTS出错: {e}")
if not self.client_abort:
# 如果没有中途打断就发送语音
self.audio_play_queue.put((opus_datas, text))
if self.tts.delete_audio_file and os.path.exists(tts_file):
self.audio_play_queue.put((opus_datas, text, text_index))
if self.tts.delete_audio_file and tts_file is not None and os.path.exists(tts_file):
os.remove(tts_file)
except Exception as e:
self.logger.bind(tag=TAG).error(f"TTS任务处理错误: {e}")
@@ -306,42 +320,41 @@ class ConnectionHandler:
self.websocket.send(json.dumps({"type": "tts", "state": "stop", "session_id": self.session_id})),
self.loop
)
self.logger.bind(tag=TAG).error(f"tts_priority priority_thread: {text}{e}")
self.logger.bind(tag=TAG).error(f"tts_priority priority_thread: {text} {e}")
def _audio_play_priority_thread(self):
while not self.stop_event.is_set():
text = None
try:
opus_datas, text = self.audio_play_queue.get()
future = asyncio.run_coroutine_threadsafe(sendAudioMessage(self, opus_datas, text), self.loop)
opus_datas, text, text_index = self.audio_play_queue.get()
future = asyncio.run_coroutine_threadsafe(sendAudioMessage(self, opus_datas, text, text_index),
self.loop)
future.result()
except Exception as e:
self.logger.bind(tag=TAG).error(f"audio_play_priority priority_thread: {text}{e}")
self.logger.bind(tag=TAG).error(f"audio_play_priority priority_thread: {text} {e}")
def speak_and_play(self, text):
def speak_and_play(self, text, text_index=0):
if text is None or len(text) <= 0:
self.logger.bind(tag=TAG).info(f"无需tts转换,query为空,{text}")
return None, text
return None, text, text_index
tts_file = self.tts.to_tts(text)
if tts_file is None:
self.logger.bind(tag=TAG).error(f"tts转换失败,{text}")
return None, text
return None, text, text_index
self.logger.bind(tag=TAG).debug(f"TTS 文件生成完毕: {tts_file}")
return tts_file, text
return tts_file, text, text_index
def clearSpeakStatus(self):
self.logger.bind(tag=TAG).debug(f"清除服务端讲话状态")
self.asr_server_receive = True
self.tts_last_text = None
self.tts_first_text = None
self.tts_duration = 0
self.tts_start_speak_time = None
self.tts_last_text_index = -1
self.tts_first_text_index = -1
def recode_first_last_text(self, text):
if not self.tts_first_text:
def recode_first_last_text(self, text, text_index=0):
if self.tts_first_text_index == -1:
self.logger.bind(tag=TAG).info(f"大模型说出第一句话: {text}")
self.tts_first_text = text
self.tts_last_text = text
self.tts_first_text_index = text_index
self.tts_last_text_index = text_index
async def close(self):
"""资源清理方法"""
+54 -11
View File
@@ -4,7 +4,10 @@ import random
import difflib
import re
import traceback
from pathlib import Path
import time
from core.handle.sendAudioHandle import send_stt_message
from core.utils import p3
TAG = __name__
logger = setup_logging()
@@ -33,6 +36,24 @@ def _find_best_match(potential_song, music_files):
best_match = music_file
return best_match
class MusicManager:
def __init__(self, music_dir, music_ext):
self.music_dir = Path(music_dir)
self.music_ext = music_ext
def get_music_files(self):
music_files = []
for file in self.music_dir.rglob("*"):
# 判断是否是文件
if file.is_file():
# 获取文件扩展名
ext = file.suffix.lower()
# 判断扩展名是否在列表中
if ext in self.music_ext:
# music_files.append(str(file.resolve())) # 添加绝对路径
# 添加相对路径
music_files.append(str(file.relative_to(self.music_dir)))
return music_files
class MusicHandler:
def __init__(self, config):
@@ -45,10 +66,19 @@ class MusicHandler:
self.music_config.get("music_dir", "./music") # 默认路径修改
)
self.music_related_keywords = self.music_config.get("music_commands", [])
self.music_ext = self.music_config.get("music_ext", (".mp3", ".wav", ".p3"))
self.refresh_time = self.music_config.get("refresh_time", 60)
else:
self.music_dir = os.path.abspath("./music")
self.music_related_keywords = ["来一首歌", "唱一首歌", "播放音乐", "来点音乐", "背景音乐", "放首歌",
"播放歌曲", "来点背景音乐", "我想听歌", "我要听歌", "放点音乐"]
self.music_ext = (".mp3", ".wav", ".p3")
self.refresh_time = 60
# 获取音乐文件列表
self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
self.scan_time = time.time()
logger.bind(tag=TAG).debug(f"找到的音乐文件: {self.music_files}")
async def handle_music_command(self, conn, text):
"""处理音乐播放指令"""
@@ -57,12 +87,15 @@ class MusicHandler:
# 尝试匹配具体歌名
if os.path.exists(self.music_dir):
music_files = [f for f in os.listdir(self.music_dir) if f.endswith('.mp3')]
logger.bind(tag=TAG).debug(f"找到的音乐文件: {music_files}")
if time.time() - self.scan_time > self.refresh_time:
# 刷新音乐文件列表
self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
self.scan_time = time.time()
logger.bind(tag=TAG).debug(f"刷新的音乐文件: {self.music_files}")
potential_song = _extract_song_name(clean_text)
if potential_song:
best_match = _find_best_match(potential_song, music_files)
best_match = _find_best_match(potential_song, self.music_files)
if best_match:
logger.bind(tag=TAG).info(f"找到最匹配的歌曲: {best_match}")
await self.play_local_music(conn, specific_file=best_match)
@@ -90,20 +123,30 @@ class MusicHandler:
return
selected_music = specific_file
else:
music_files = [f for f in os.listdir(self.music_dir) if f.endswith('.mp3')]
if not music_files:
if time.time() - self.scan_time > self.refresh_time:
# 刷新音乐文件列表
self.music_files = MusicManager(self.music_dir, self.music_ext).get_music_files()
self.scan_time = time.time()
logger.bind(tag=TAG).debug(f"刷新的音乐文件列表: {self.music_files}")
if not self.music_files:
logger.bind(tag=TAG).error("未找到MP3音乐文件")
return
selected_music = random.choice(music_files)
selected_music = random.choice(self.music_files)
music_path = os.path.join(self.music_dir, selected_music)
if not os.path.exists(music_path):
logger.bind(tag=TAG).error(f"选定的音乐文件不存在: {music_path}")
return
text = f"正在播放{selected_music}"
await send_stt_message(conn, text)
conn.tts_first_text = selected_music
conn.tts_last_text = selected_music
conn.tts_first_text_index = 0
conn.tts_last_text_index = 0
conn.llm_finish_task = True
opus_packets, duration = conn.tts.wav_to_opus_data(music_path)
conn.audio_play_queue.put((opus_packets, selected_music))
if music_path.endswith(".p3"):
opus_packets, duration = p3.decode_opus_from_file(music_path)
else:
opus_packets, duration = conn.tts.wav_to_opus_data(music_path)
conn.audio_play_queue.put((opus_packets, selected_music, 0))
except Exception as e:
logger.bind(tag=TAG).error(f"播放音乐失败: {str(e)}")
+23 -16
View File
@@ -15,30 +15,37 @@ async def isLLMWantToFinish(last_text):
return False
async def sendAudioMessage(conn, audios, text):
# 发送 tts.start
if text == conn.tts_first_text:
async def sendAudioMessage(conn, audios, text, text_index=0):
# 发送句子开始消息
if text_index == conn.tts_first_text_index:
logger.bind(tag=TAG).info(f"发送第一段语音: {text}")
conn.tts_start_speak_time = time.time()
await send_tts_message(conn, "sentence_start", text)
# 发送音频数据
frame_duration = 60 # 初始帧持续时间(毫秒
start_time = time.time() # 记录开始时间
for idx, opus_packet in enumerate(audios):
# 初始化流控参数
frame_duration = 60 # 毫秒
start_time = time.perf_counter() # 使用高精度计时器
play_position = 0 # 已播放的时长(毫秒)
for opus_packet in audios:
if conn.client_abort:
return
# 计算当前包的预期发送时间
expected_time = start_time + idx * (frame_duration / 1000)
current_time = time.time()
# 如果未到预期时间则等待差值
if current_time < expected_time:
await asyncio.sleep(expected_time - current_time)
expected_time = start_time + (play_position / 1000)
current_time = time.perf_counter()
# 等待直到预期时间
delay = expected_time - current_time
if delay > 0:
await asyncio.sleep(delay)
# 发送音频包
await conn.websocket.send(opus_packet)
if conn.llm_finish_task and text == conn.tts_last_text:
await send_tts_message(conn, 'stop')
play_position += frame_duration # 更新播放位置
await send_tts_message(conn, "sentence_end", text)
# 发送结束消息(如果是最后一个文本)
if conn.llm_finish_task and text_index == conn.tts_last_text_index:
await send_tts_message(conn, 'stop', None)
if await isLLMWantToFinish(text):
await conn.close()
+22 -81
View File
@@ -3,94 +3,35 @@ import requests
import json
import re
from core.providers.llm.base import LLMProviderBase
import os
# official coze sdk for Python [cozepy](https://github.com/coze-dev/coze-py)
from cozepy import COZE_CN_BASE_URL
from cozepy import Coze, TokenAuth, Message, ChatStatus, MessageContentType, ChatEventType # noqa
TAG = __name__
logger = setup_logging()
# 定义用于匹配中文标点符号的正则表达式(包括句号、感叹号、问号、分号)
punctuation_pattern = re.compile(r'([。!?;])')
class LLMProvider(LLMProviderBase):
def __init__(self, config):
self.personal_access_token = config.get("personal_access_token")
self.bot_id = config.get("bot_id")
self.user_id = config.get("user_id") # 默认用户 ID
self.base_url = config.get("base_url")
self.user_id = config.get("user_id")
def response(self, session_id, dialogue):
try:
# 从对话中取出最新的用户消息
last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
data = {
"conversation_id": session_id,
"bot_id": self.bot_id,
"user": self.user_id,
"query": last_msg["content"],
"stream": True
}
logger.bind(tag=TAG).info(f"发送到 Coze API 的请求数据: {json.dumps(data, ensure_ascii=False)}")
headers = {
'Authorization': f'Bearer {self.personal_access_token}',
'Content-Type': 'application/json',
'Accept': '*/*',
'Host': 'api.coze.cn',
'Connection': 'keep-alive'
}
response = requests.post(
self.base_url,
headers=headers,
json=data,
stream=True
)
logger.bind(tag=TAG).info(f"请求状态: {response.status_code}")
if response.status_code == 200:
# 对每一行流数据进行处理,不做跨块累积
for line_bytes in response.iter_lines(decode_unicode=False):
if not line_bytes:
continue
try:
# 使用 utf-8 解码,错误部分用替换符
line = line_bytes.decode('utf-8', errors='replace')
except Exception as e:
logger.bind(tag=TAG).error(f"解码失败: {e}")
continue
if line.startswith("data:"):
data_str = line[len("data:"):].strip()
if data_str == "[DONE]":
break
try:
data_chunk = json.loads(data_str)
except json.JSONDecodeError as e:
logger.bind(tag=TAG).error(f"JSON解析失败: {e} 数据: {line}")
continue
msg = data_chunk.get("message", {})
if msg.get("role") == "assistant" and msg.get("type") == "answer":
content = msg.get("content", "")
# 如果返回内容中包含标点符号,则按标点拆分,立即返回每个片段
if punctuation_pattern.search(content):
# 利用 finditer 找到每个标点,并返回以标点结尾的片段
start = 0
for match in punctuation_pattern.finditer(content):
end = match.end()
sentence = content[start:end].strip()
if sentence:
yield sentence
start = end
# 如果拆分后剩余内容也返回(不含标点),直接返回
if start < len(content):
remainder = content[start:].strip()
if remainder:
yield remainder
else:
# 如果没有标点,则直接返回这块内容
if content.strip():
yield content.strip()
else:
logger.bind(tag=TAG).error(f"请求失败,状态码: {response.status_code}")
yield f"【Coze服务响应异常:请求失败,状态码 {response.status_code}"
except Exception as e:
logger.bind(tag=TAG).error(f"Error in Coze response generation: {e}")
yield "【Coze服务响应异常】"
coze_api_token = self.personal_access_token
coze_api_base = COZE_CN_BASE_URL
last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
coze = Coze(auth=TokenAuth(token=coze_api_token), base_url=coze_api_base)
for event in coze.chat.stream(
bot_id=self.bot_id,
user_id=self.user_id,
additional_messages=[
Message.build_user_question_text(last_msg["content"]),
],
):
if event.event == ChatEventType.CONVERSATION_MESSAGE_DELTA:
print(event.message.content, end="", flush=True)
yield event.message.content
+23
View File
@@ -0,0 +1,23 @@
from abc import ABC, abstractmethod
from config.logger import setup_logging
TAG = __name__
logger = setup_logging()
class MemoryProviderBase(ABC):
def __init__(self, config):
self.config = config
self.role_id = None
@abstractmethod
async def save_memory(self, msgs):
"""Save a new memory for specific role and return memory ID"""
print("this is base func", msgs)
@abstractmethod
async def query_memory(self, query: str) -> str:
"""Query memories for specific role based on similarity"""
return "please implement query method"
def set_role_id(self, role_id: str):
self.role_id = role_id
+74
View File
@@ -0,0 +1,74 @@
from ..base import MemoryProviderBase, logger
from mem0 import MemoryClient
TAG = __name__
class MemoryProvider(MemoryProviderBase):
def __init__(self, config):
super().__init__(config)
self.api_key = config.get("api_key", "")
self.api_version = config.get("api_version", "v1.1")
if len(self.api_key) == 0 or "" in self.api_key:
logger.bind(tag=TAG).error("你还没配置Mem0ai的密钥,请在配置文件中配置密钥,否则无法提供记忆服务")
self.use_mem0 = False
return
else:
self.use_mem0 = True
self.client = MemoryClient(api_key=self.api_key)
async def save_memory(self, msgs):
if not self.use_mem0:
return None
if len(msgs) < 2:
return None
try:
# Format the content as a message list for mem0
messages = [
{"role": message.role, "content": message.content}
for message in msgs if message.role != "system"
]
result = self.client.add(messages, user_id=self.role_id, output_format=self.api_version)
logger.bind(tag=TAG).debug(f"Save memory result: {result}")
except Exception as e:
logger.bind(tag=TAG).error(f"保存记忆失败: {str(e)}")
return None
async def query_memory(self, query: str)-> str:
if not self.use_mem0:
return ""
try:
results = self.client.search(
query,
user_id=self.role_id,
output_format=self.api_version
)
if not results or 'results' not in results:
return ""
# Format each memory entry with its update time up to minutes
memories = []
for entry in results['results']:
timestamp = entry.get('updated_at', '')
if timestamp:
try:
# Parse and reformat the timestamp
dt = timestamp.split('.')[0] # Remove milliseconds
formatted_time = dt.replace('T', ' ')
except:
formatted_time = timestamp
memory = entry.get('memory', '')
if timestamp and memory:
# Store tuple of (timestamp, formatted_string) for sorting
memories.append((timestamp, f"[{formatted_time}] {memory}"))
# Sort by timestamp in descending order (newest first)
memories.sort(key=lambda x: x[0], reverse=True)
# Extract only the formatted strings
memories_str = "\n".join(f"- {memory[1]}" for memory in memories)
logger.bind(tag=TAG).debug(f"Query results: {memories_str}")
return memories_str
except Exception as e:
logger.bind(tag=TAG).error(f"查询记忆失败: {str(e)}")
return ""
+3 -4
View File
@@ -14,10 +14,9 @@ class TTSProvider(TTSProviderBase):
self.access_token = config.get("access_token")
self.cluster = config.get("cluster")
self.voice = config.get("voice")
self.host = "openspeech.bytedance.com"
self.api_url = f"https://{self.host}/api/v1/tts"
self.header = {"Authorization": f"Bearer;{self.access_token}"}
self.api_url = config.get("api_url")
self.authorization = config.get("authorization")
self.header = {"Authorization": f"{self.authorization}{self.access_token}"}
def generate_filename(self, extension=".wav"):
return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
+24
View File
@@ -24,3 +24,27 @@ class Dialogue:
for m in self.dialogue:
dialogue.append({"role": m.role, "content": m.content})
return dialogue
def get_llm_dialogue_with_memory(self, memory_str: str = None) -> List[Dict[str, str]]:
# 构建带记忆的对话
dialogue = []
# 添加系统提示和记忆
system_message = next(
(msg for msg in self.dialogue if msg.role == "system"), None
)
if system_message:
enhanced_system_prompt = (
f"{system_message.content}\n\n"
f"相关记忆:\n{memory_str}"
)
dialogue.append({"role": "system", "content": enhanced_system_prompt})
# 添加用户和助手的对话
for msg in self.dialogue:
if msg.role != "system": # 跳过原始的系统消息
dialogue.append({"role": msg.role, "content": msg.content})
return dialogue
+17
View File
@@ -0,0 +1,17 @@
import os
import sys
import importlib
from config.logger import setup_logging
from core.utils.util import read_config, get_project_dir
logger = setup_logging()
def create_instance(class_name, *args, **kwargs):
if os.path.exists(os.path.join('core', 'providers', 'memory', class_name, f'{class_name}.py')):
lib_name = f'core.providers.memory.{class_name}.{class_name}'
if lib_name not in sys.modules:
sys.modules[lib_name] = importlib.import_module(f'{lib_name}')
return sys.modules[lib_name].MemoryProvider(*args, **kwargs)
raise ValueError(f"不支持的记忆服务类型: {class_name}")
+33
View File
@@ -0,0 +1,33 @@
import struct
def decode_opus_from_file(input_file):
"""
从p3文件中解码 Opus 数据,并返回一个 Opus 数据包的列表以及总时长。
"""
opus_datas = []
total_frames = 0
sample_rate = 16000 # 文件采样率
frame_duration_ms = 60 # 帧时长
frame_size = int(sample_rate * frame_duration_ms / 1000)
with open(input_file, 'rb') as f:
while True:
# 读取头部(4字节):[1字节类型,1字节保留,2字节长度]
header = f.read(4)
if not header:
break
# 解包头部信息
_, _, data_len = struct.unpack('>BBH', header)
# 根据头部指定的长度读取 Opus 数据
opus_data = f.read(data_len)
if len(opus_data) != data_len:
raise ValueError(f"Data length({len(opus_data)}) mismatch({data_len}) in the file.")
opus_datas.append(opus_data)
total_frames += 1
# 计算总时长
total_duration = (total_frames * frame_duration_ms) / 1000.0
return opus_datas, total_duration
+27
View File
@@ -3,6 +3,7 @@ import re
import json
import yaml
import socket
import subprocess
def get_project_dir():
@@ -114,3 +115,29 @@ def check_password(password):
# 如果满足所有条件,则返回True
return True
def check_ffmpeg_installed():
ffmpeg_installed = False
try:
# 执行ffmpeg -version命令,并捕获输出
result = subprocess.run(
['ffmpeg', '-version'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
check=True # 如果返回码非零则抛出异常
)
# 检查输出中是否包含版本信息(可选)
output = result.stdout + result.stderr
if 'ffmpeg version' in output.lower():
ffmpeg_installed = True
return False
except (subprocess.CalledProcessError, FileNotFoundError):
# 命令执行失败或未找到
ffmpeg_installed = False
if not ffmpeg_installed:
error_msg = "您的电脑还没正确安装ffmpeg\n"
error_msg += "\n建议您:\n"
error_msg += "1、按照项目的安装文档,正确进入conda环境\n"
error_msg += "2、查阅安装文档,如何在conda环境中安装ffmpeg\n"
raise ValueError(error_msg)
+9 -4
View File
@@ -4,7 +4,7 @@ from config.logger import setup_logging
from core.connection import ConnectionHandler
from core.handle.musicHandler import MusicHandler
from core.utils.util import get_local_ip
from core.utils import asr, vad, llm, tts
from core.utils import asr, vad, llm, tts, memory
TAG = __name__
@@ -13,10 +13,14 @@ class WebSocketServer:
def __init__(self, config: dict):
self.config = config
self.logger = setup_logging()
self._vad, self._asr, self._llm, self._tts, self._music = self._create_processing_instances()
self._vad, self._asr, self._llm, self._tts, self._music, self._memory = self._create_processing_instances()
self.active_connections = set() # 添加全局连接记录
def _create_processing_instances(self):
memory_cls_name = self.config["selected_module"].get("Memory", "mem0ai") # 默认使用mem0ai
has_memory_cfg = self.config.get("Memory") and memory_cls_name in self.config["Memory"]
memory_cfg = self.config["Memory"][memory_cls_name] if has_memory_cfg else {}
"""创建处理模块实例"""
return (
vad.create_instance(
@@ -46,7 +50,8 @@ class WebSocketServer:
self.config["TTS"][self.config["selected_module"]["TTS"]],
self.config["delete_audio"]
),
MusicHandler(self.config)
MusicHandler(self.config),
memory.create_instance(memory_cls_name, memory_cfg),
)
async def start(self):
@@ -66,7 +71,7 @@ class WebSocketServer:
async def _handle_connection(self, websocket):
"""处理新连接,每次创建独立的ConnectionHandler"""
# 创建ConnectionHandler时传入当前server实例
handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music)
handler = ConnectionHandler(self.config, self._vad, self._asr, self._llm, self._tts, self._music, self._memory)
self.active_connections.add(handler)
try:
await handler.handle_connection(websocket)
+25 -7
View File
@@ -159,14 +159,31 @@ poetry run python app.py
本项目使用`conda`管理依赖环境。如果不方便安装`conda`,需要根据实际的操作系统安装好`libopus``ffmpeg`
如果确定使用`conda`,则安装好后,开始执行以下命令。
重要提示!windows 用户,可以通过安装`Anaconda`来管理环境。安装好`Anaconda`后,在`开始`那里搜索`anaconda`相关的关键词,
找到`Anaconda Prpmpt`,使用管理员身份运行它。如下图。
![conda_prompt](./images/conda_env_1.png)
运行之后,如果你能看到命令行窗口前面有一个(base)字样,说明你成功进入了`conda`环境。那么你就可以执行以下命令了。
![conda_env](./images/conda_env_2.png)
```
conda remove -n xiaozhi-esp32-server --all -y
conda create -n xiaozhi-esp32-server python=3.10 -y
conda activate xiaozhi-esp32-server
conda install conda-forge::libopus -y
conda install conda-forge::ffmpeg -y
# 添加清华源通道
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
conda install libopus -y
conda install ffmpeg -y
```
请注意,以上命令,不是一股脑执行就成功的,你需要一步步执行,每一步执行完后,都检查一下输出的日志,查看是否成功。
## 2.安装本项目依赖
你先要下载本项目源码,源码可以通过`git clone`命令下载,如果你不熟悉`git clone`命令。
@@ -179,7 +196,7 @@ conda install conda-forge::ffmpeg -y
你需要把它重命名成`xiaozhi-esp32-server`,好了请记住这个目录,我们暂且称它为`项目目录`
```
# 使用dos或者终端,进入到你的项目目录,执行以下命令
# 继续使用conda环境,进入到你的项目目录,执行以下命令
conda activate xiaozhi-esp32-server
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
pip install -r requirements.txt
@@ -214,9 +231,10 @@ python app.py
如果你的`项目目录`目录没有`data`,你需要创建`data`目录。
如果你的`data`下面没有`.config.yaml`文件,你可以把源码目录下的`config.yaml`文件复制一份,重命名为`.config.yaml`
修改`项目目录``data`目录下的`.config.yaml`文件,配置本项目所需的各种参数。默认的LLM使用的是`ChatGLMLLM`
,你需要配置密钥,因为他们的模型,虽然有免费的,但是仍要去[官网](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)注册密钥,才能启动。
默认的TTS使用的是`EdgeTTS`,这个无需配置,如果你需要更换成`豆包TTS`,则需要配置密钥
修改`项目目录``data`目录下的`.config.yaml`文件,配置本项目必须的两个配置。
- 默认的LLM使用的是`ChatGLMLLM`,你需要配置密钥,因为他们的模型,虽然有免费的,但是仍要去[官网](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)注册密钥,才能启动
- 默认的记忆层`mem0ai`,你需要配置密钥,因为他们的API,虽然有免费额度,但是仍要去[官网](https://app.mem0.ai/dashboard/api-keys)注册密钥,才能启动。
配置说明:这里是各个功能使用的默认组件,例如LLM默认使用`ChatGLMLLM`模型。如果需要切换模型,就是改对应的名称。
本项目的默认配置仅是成本最低配置(`glm-4-flash``EdgeTTS`都是免费的),如果需要更优的更快的搭配,需要自己结合部署环境切换各组件的使用。
@@ -250,7 +268,7 @@ LLM:
## 模型文件
本项目语音识别模型,默认使用`SenseVoiceSmall`模型,进行语音转文字。因为模型较大,需要独立下载,下载后把`model.pt`
文件放在`model/SenseVoiceSmall`
文件放在`models/SenseVoiceSmall`
目录下。下面两个下载路线任选一个。
- 线路一:阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
Binary file not shown.

After

Width:  |  Height:  |  Size: 781 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 588 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 306 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 325 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 363 KiB

+4 -1
View File
@@ -16,4 +16,7 @@ aiohttp_cors==0.7.0
ormsgpack==1.7.0
ruamel.yaml==0.18.10
loguru==0.7.3
requests==2.32.3
requests==2.32.3
cozepy==0.12.0
miniaudio==1.61
mem0ai==0.1.62