mirror of
https://github.com/xinnan-tech/xiaozhi-esp32-server.git
synced 2026-07-22 07:03:53 +08:00
Merge branch 'main' into main
This commit is contained in:
欣南科技
+2
-1
@@ -7,4 +7,5 @@ docs/
|
||||
tmp/
|
||||
LICENSE
|
||||
README.md
|
||||
README_en.md
|
||||
README_en.md
|
||||
.config.yaml
|
||||
@@ -139,3 +139,5 @@ cython_debug/
|
||||
*.iml
|
||||
model.pt
|
||||
tmp
|
||||
.DS_Store
|
||||
.config.yaml
|
||||
|
||||
+1
-1
@@ -21,7 +21,7 @@ RUN pip install --no-cache-dir -r requirements.txt -i https://mirrors.aliyun.com
|
||||
# 第二阶段：生产镜像
|
||||
FROM python:3.10-slim
|
||||
|
||||
WORKDIR /opt/xiaozhi-es32-server
|
||||
WORKDIR /opt/xiaozhi-esp32-server
|
||||
|
||||
# 从构建阶段复制虚拟环境
|
||||
COPY --from=builder /opt/venv /opt/venv
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
- `xiaozhi-esp32` 通信 WebSocket 协议
|
||||
- 支持唤醒对话、手动对话、实时打断对话
|
||||
- 支持国语、粤语、英语、日语、韩语 5 种语言识别（FunASR（默认））
|
||||
- 自由更换 LLM（支持ChatGLM（默认）、阿里百炼、Dify、DeepSeek）
|
||||
- 自由更换 LLM（openai接口支持ChatGLM（默认）、阿里百炼、DeepSeek等，dify接口支持Dify官方及私有化部署）
|
||||
- 自由更换 TTS（支持EdgeTTS（默认）、火山引擎豆包TTS）
|
||||
|
||||
## 正在实现
|
||||
@@ -52,18 +52,28 @@
|
||||
|
||||
| 类型 | 平台名称 | 使用方式 | 收费模式 | 备注 |
|
||||
|:----|:----------|:----:|:--------|:----------------------------------------------------------------|
|
||||
| LLM | 阿里百炼 | 接口调用 | 消耗token | [点击申请密钥](https://bailian.console.aliyun.com/?apiKey=1#/api-key) |
|
||||
| LLM | 深度求索 | 接口调用 | 消耗token | [点击申请密钥](https://platform.deepseek.com/) |
|
||||
| LLM | Dify | 接口调用 | 消耗token | 本地化部署 |
|
||||
| LLM | 智谱 | 接口调用 | 免费 | [点击创建密钥](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) |
|
||||
| LLM | 阿里百炼 | openai接口调用 | 消耗token | [点击申请密钥](https://bailian.console.aliyun.com/?apiKey=1#/api-key) |
|
||||
| LLM | 深度求索 | openai接口调用 | 消耗token | [点击申请密钥](https://platform.deepseek.com/) |
|
||||
| LLM | 智谱 | openai接口调用 | 免费 | [点击创建密钥](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) |
|
||||
| LLM | Dify | dify接口调用 | 消耗token | 本地化部署 |
|
||||
| TTS | 火山引擎 | 接口调用 | 消耗token | [点击创建密钥](https://console.volcengine.com/speech/service/8) |
|
||||
| TTS | EdgeTTS | 接口调用 | 免费 | |
|
||||
| VAD | SileroVAD | 本地使用 | 免费 | |
|
||||
| ASR | FunASR | 本地使用 | 免费 | |
|
||||
|
||||
实际上，任何支持openai接口调用的LLM，都可以接入使用。
|
||||
|
||||
# 部署方式
|
||||
|
||||
本项目支持docker快速部署和本地源码运行。如果您主要是想快速体验，推荐使用docker部署。如果想深入了解本项目，推荐本地源码运行。
|
||||
本项目支持docker快速部署、借助docker环境运行部署和本地源码运行三种方式。三种方式怎么选，每个人的情况不一样，以下是一些推荐。
|
||||
|
||||
如果您主要是想快速体验，推荐使用`方式一：docker快速部署`。
|
||||
|
||||
如果您已经安装docker，不想折腾环境，又想对代码进行DIY修改，推荐`方式二：借助docker环境运行部署`。
|
||||
|
||||
如果您对conda的使用得心应手，或想从零开始学习搭建环境，推荐`方式三：本地源码运行`。
|
||||
|
||||
如果你对小智的回答速度非常的敏感，不希望有丝丝的性能损失，推荐`方式三：本地源码运行`。
|
||||
|
||||
## 方式一：docker快速部署
|
||||
|
||||
@@ -86,7 +96,7 @@ docker镜像已支持x86架构、arm64架构的CPU，支持在国产操作系统
|
||||
|
||||
### 4. 修改配置文件
|
||||
|
||||
修改刚才你下载的`config.yaml`文件，配置本项目所需的各种参数。默认的LLM使用的是`ChatGLMLLM`，你需要配置密钥，才能启动。
|
||||
修改刚才你下载的`config.yaml`文件，配置本项目所需的各种参数。默认的LLM使用的是`ChatGLMLLM`，你需要配置密钥，因为他们的模型，虽然有免费的，但是仍要去[官网](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)注册密钥，才能启动。
|
||||
|
||||
默认的TTS使用的是`EdgeTTS`，这个无需配置，如果你需要更换成`豆包TTS`，则需要配置密钥。
|
||||
|
||||
@@ -102,17 +112,23 @@ selected_module:
|
||||
TTS: EdgeTTS
|
||||
```
|
||||
|
||||
比如修改`LLM`使用的组件，就看本项目支持哪些`LLM`，如下就是支持`DeepSeekLLM`、`ChatGLMLLM`。你们在`selected_module`修改成对应的LLM
|
||||
比如修改`LLM`使用的组件，就看本项目支持哪些`LLM` API接口，当前支持的是`openai`、`dify`。欢迎验证和支持更多LLM平台的接口。
|
||||
|
||||
使用时，在`selected_module`修改成对应的如下LLM配置的名称：
|
||||
|
||||
```
|
||||
LLM:
|
||||
AliLLM:
|
||||
type: openai
|
||||
...
|
||||
DeepSeekLLM:
|
||||
type: openai
|
||||
...
|
||||
ChatGLMLLM:
|
||||
type: openai
|
||||
...
|
||||
DifyLLM:
|
||||
type: dify
|
||||
...
|
||||
```
|
||||
|
||||
@@ -132,7 +148,7 @@ dir
|
||||
如果你能看到`config.yaml`文件，确确实实进入到了`项目目录`，接着执行以下命令：
|
||||
|
||||
```
|
||||
docker run -d --name xiaozhi-esp32-server --restart always --security-opt seccomp:unconfined -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-es32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
docker run -d --name xiaozhi-esp32-server --restart always --security-opt seccomp:unconfined -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-esp32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
```
|
||||
|
||||
如果首次执行，可能需要几分钟时间，你要耐心等待他完成拉取。正常拉取完成后，你可以在命令行执行以下命令查看服务是否启动成功
|
||||
@@ -158,6 +174,9 @@ docker logs -f xiaozhi-esp32-server
|
||||
正常来说，假设你的ip是`192.168.1.25`，那么你的接口地址就是：`ws://192.168.1.25:8000`。这个信息很有用的，后面`编译esp32固件`
|
||||
需要用到。
|
||||
|
||||
请注意，你的接口地址是`websocket`协议的地址，你可以使用`apifox`等工具调试。但是不能直接用浏览器打开访问，如果用浏览器打开，日志会显示错误，会让你怀疑是否部署成功了。
|
||||
|
||||
|
||||
后期如果想升级版本，可以这么操作
|
||||
|
||||
1、备份好`config.yaml`文件，一些关键的配置到时复制到新的`config.yaml`文件里。
|
||||
@@ -172,7 +191,58 @@ docker rmi ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
|
||||
3.按本教程重新来一遍
|
||||
|
||||
## 方式二：本地源码运行
|
||||
## 方式二：借助docker环境运行部署
|
||||
|
||||
这个方法的原理，其实是和第一种方式类似。区别在于，第一种方式只要下载一个配置文件，而本方式要下载项目源码。
|
||||
|
||||
优点就是源码在你主机躺着，你可以使用本机的代码编辑器加载项目和修改源码。每次修改完项目，想要看效果，那就要重启docker。
|
||||
|
||||
缺点就要下载本项目的代码，还要下载模型文件，如果这个项目不常用，确实会占用您电脑的空间。
|
||||
|
||||
下载源码后，需要下载模型文件。 默认使用`SenseVoiceSmall`模型，进行语音转文字。因为模型较大，需要独立下载，下载后把`model.pt`文件放在`model/SenseVoiceSmall`
|
||||
目录下。下面两个下载路线任选一个。
|
||||
|
||||
- 线路一：阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
|
||||
- 线路二：百度网盘下载[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna) 提取码:
|
||||
`qvna`
|
||||
|
||||
下载模型后，需要按照`方式一：docker快速部署`修改配置文件`config.yaml`
|
||||
|
||||
修改完配置后，打开命令行工具，`cd`进入到你的项目目录下，执行以下命令
|
||||
|
||||
```
|
||||
docker run -d --name xiaozhi-esp32-server --restart always --security-opt seccomp:unconfined \
|
||||
-p 8000:8000 \
|
||||
-v $(pwd):/opt/xiaozhi-esp32-server \
|
||||
ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
```
|
||||
|
||||
```
|
||||
docker logs -f xiaozhi-esp32-server
|
||||
```
|
||||
|
||||
如果你能看到，类似以下日志,则是本项目服务启动成功的标志。
|
||||
|
||||
```
|
||||
2025-xx-xx xx:51:59,492 - core.server - INFO - Server is running at ws://xx.xx.xx.xxx:8000
|
||||
2025-xx-xx xx:51:59,516 - websockets.server - INFO - server listening on 0.0.0.0:8000
|
||||
```
|
||||
|
||||
接下来，你就可以开始 `编译esp32固件`了，请往下翻，翻到编译`esp32固件`相关章节。那么由于你是用docker部署，你要自己查看自己本机电脑的ip是多少。
|
||||
正常来说，假设你的ip是`192.168.1.25`，那么你的接口地址就是：`ws://192.168.1.25:8000`。这个信息很有用的，后面`编译esp32固件`
|
||||
需要用到。
|
||||
|
||||
请注意，你的接口地址是`websocket`协议的地址，你可以使用`apifox`等工具调试。但是不能直接用浏览器打开访问，如果用浏览器打开，日志会显示错误，会让你怀疑是否部署成功了。
|
||||
|
||||
后期，如果你修改了代码，想让新代码跑起来，可以用以下命令让docker容器重启：
|
||||
|
||||
```
|
||||
docker restart xiaozhi-esp32-server
|
||||
# 查看日志输出
|
||||
docker logs -f xiaozhi-esp32-server
|
||||
```
|
||||
|
||||
## 方式三：本地源码运行
|
||||
|
||||
### 1.安装基础环境
|
||||
|
||||
@@ -214,14 +284,19 @@ pip install -r requirements.txt
|
||||
目录下。下面两个下载路线任选一个。
|
||||
|
||||
- 线路一：阿里魔塔下载[SenseVoiceSmall](https://modelscope.cn/models/iic/SenseVoiceSmall/resolve/master/model.pt)
|
||||
- 线路二：百度网盘下载[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna) 提取码:
|
||||
`qvna`
|
||||
- 线路二：百度网盘下载[SenseVoiceSmall](https://pan.baidu.com/share/init?surl=QlgM58FHhYv1tFnUT_A8Sg&pwd=qvna) 提取码: `qvna`
|
||||
|
||||
### 4.配置项目
|
||||
|
||||
修改`config.yaml`文件，配置本项目所需的各种参数。默认的LLM使用的是`ChatGLMLLM`，你需要配置密钥，才能启动。
|
||||
修改`config.yaml`文件，配置本项目所需的各种参数。默认的LLM使用的是`ChatGLMLLM`，你需要配置密钥，因为他们的模型，虽然有免费的，但是仍要去[官网](https://bigmodel.cn/usercenter/proj-mgmt/apikeys)注册密钥，才能启动。
|
||||
默认的TTS使用的是`EdgeTTS`，这个无需配置，如果你需要更换成`豆包TTS`，则需要配置密钥。
|
||||
|
||||
```
|
||||
# 如果您是一名开发者，建议阅读以下内容。如果不是开发者，可以忽略这部分内容。
|
||||
在开发中，建议将【config.yaml】复制一份，改成【.config.yaml】。 系统会优先读取【.config.yaml】文件的配置。
|
||||
这样做，可以避免在提交代码的时候，错误地提交密钥信息，保护您的密钥安全。
|
||||
```
|
||||
|
||||
配置说明：这里是各个功能使用的默认组件，例如LLM默认使用`ChatGLMLLM`模型。如果需要切换模型，就是改对应的名称。
|
||||
|
||||
本项目的默认配置仅是成本最低配置（`glm-4-flash`和`EdgeTTS`都是免费的），如果需要更优的更快的搭配，需要自己结合部署环境切换各组件的使用。
|
||||
@@ -234,15 +309,19 @@ selected_module:
|
||||
TTS: EdgeTTS
|
||||
```
|
||||
|
||||
比如修改`LLM`使用的组件，就看本项目支持哪些`LLM`，如下就是支持`DeepSeekLLM`、`ChatGLMLLM`。你们在`selected_module`修改成对应的LLM
|
||||
比如修改`LLM`使用的组件，就看本项目支持哪些`LLM` API接口，当前支持的是`openai`、`dify`。欢迎验证和支持更多LLM平台的接口。
|
||||
使用时，在`selected_module`修改成对应的如下LLM配置的名称：
|
||||
|
||||
```
|
||||
LLM:
|
||||
DeepSeekLLM:
|
||||
type: openai
|
||||
...
|
||||
ChatGLMLLM:
|
||||
type: openai
|
||||
...
|
||||
DifyLLM:
|
||||
type: dify
|
||||
...
|
||||
```
|
||||
|
||||
@@ -267,6 +346,8 @@ python app.py
|
||||
|
||||
其中上面的`ws://192.168.1.25:8000`就是本项目提供的接口地址了，当然你自己的机器和我的是不一样的，记得要找到自己的地址。
|
||||
|
||||
请注意，你的接口地址是`websocket`协议的地址，你可以使用`apifox`等工具调试。但是不能直接用浏览器打开访问，如果用浏览器打开，日志会显示错误，会让你怀疑是否部署成功了。
|
||||
|
||||
# 编译esp32固件
|
||||
|
||||
1. 下载`xiaozhi-esp32`
|
||||
@@ -297,6 +378,13 @@ config WEBSOCKET_URL
|
||||
Communication with the server through websocket after wake up.
|
||||
```
|
||||
|
||||
注意：你的地址是`ws://`开头，不是`wss://`开头，一定不要写错了。
|
||||
|
||||
注意：你的地址是`ws://`开头，不是`wss://`开头，一定不要写错了。
|
||||
|
||||
注意：你的地址是`ws://`开头，不是`wss://`开头，一定不要写错了。
|
||||
|
||||
|
||||
3. 设置编译参数
|
||||
|
||||
```
|
||||
@@ -332,6 +420,8 @@ python release.py
|
||||
编译成功后，会在项目根目录下的`build`目录下生成固件文件`merged-binary.bin`。
|
||||
这个`merged-binary.bin`就是要烧录到硬件上的固件文件。
|
||||
|
||||
注意：如果执行到第二命令后，报了“zip”相关的错误，请忽略这个错误，只要`build`目录下生成固件文件`merged-binary.bin`，对你没有太大影响，请继续。
|
||||
|
||||
6. 烧录固件
|
||||
将esp32设备连接电脑，使用chrome浏览器，打开以下网址
|
||||
|
||||
|
||||
+22
-10
@@ -44,7 +44,7 @@ use it in the production environment.
|
||||
- `xiaozhi-esp32` WebSocket communication protocol
|
||||
- Supports wake-word initiated dialogue, manual dialogue, and real-time interruption of dialogue.
|
||||
- Support for 5 languages: Mandarin, Cantonese, English, Japanese, Korean (FunASR - default)
|
||||
- Flexible LLM switching (ChatGLM - default, Aliyun, Dify, DeepSeek)
|
||||
- Flexible LLM switching (openai:ChatGLM - default, Aliyun, DeepSeek; dify:Dify)
|
||||
- Flexible TTS switching (EdgeTTS - default, ByteDance Doubao TTS)
|
||||
|
||||
## In Progress
|
||||
@@ -57,15 +57,17 @@ use it in the production environment.
|
||||
|
||||
| Type | Service | Usage | Pricing Model | Notes |
|
||||
|:-----|:-----------|:--------:|:---------------|:---------------------------------------------------------------------------|
|
||||
| LLM | Aliyun | API call | Token-based | [Apply for API Key](https://bailian.console.aliyun.com/?apiKey=1#/api-key) |
|
||||
| LLM | DeepSeek | API call | Token-based | [Apply for API Key](https://platform.deepseek.com/) |
|
||||
| LLM | Dify | API call | Token-based | Self-hosted |
|
||||
| LLM | Bigmodel | API call | Free | [Create API Key](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) |
|
||||
| LLM | Aliyun | openai API call | Token-based | [Apply for API Key](https://bailian.console.aliyun.com/?apiKey=1#/api-key) |
|
||||
| LLM | DeepSeek | openai API call | Token-based | [Apply for API Key](https://platform.deepseek.com/) |
|
||||
| LLM | Bigmodel | openai API call | Free | [Create API Key](https://bigmodel.cn/usercenter/proj-mgmt/apikeys) |
|
||||
| LLM | Dify | dify API call | Token-based | Self-hosted |
|
||||
| TTS | HuoshanTTS | API call | Token-based | [Create API Key](https://console.volcengine.com/speech/service/8) |
|
||||
| TTS | EdgeTTS | API call | Free | |
|
||||
| VAD | SileroVAD | Local | Free | |
|
||||
| ASR | FunASR | Local | Free | |
|
||||
|
||||
In fact, any LLM that supports OpenAI API calls can be integrated and used.
|
||||
|
||||
# Deployment
|
||||
|
||||
This project supports rapid deployment of docker and local source code operation. If you want to have a quick
|
||||
@@ -117,18 +119,22 @@ selected_module:
|
||||
TTS: EdgeTTS
|
||||
```
|
||||
|
||||
For example, modify the components used by `llm`, depending on which` llm` supports this project, as follows, it
|
||||
supports `Deepseekllm` and` Chatglmllm`. You are modified to the corresponding LLM in `selectd_module`
|
||||
For example, to modify the components used by the `LLM`, it depends on which `LLM` API interfaces are supported by this project. Currently, the supported ones are `openai` and `dify`. We welcome validation and support for more LLM platforms' interfaces.
|
||||
When using it, change the `selected_module` to the corresponding name of the following LLM configurations:
|
||||
|
||||
```
|
||||
LLM:
|
||||
AliLLM:
|
||||
type: openai
|
||||
...
|
||||
DeepSeekLLM:
|
||||
type: openai
|
||||
...
|
||||
ChatGLMLLM:
|
||||
type: openai
|
||||
...
|
||||
DifyLLM:
|
||||
type: openai
|
||||
...
|
||||
```
|
||||
|
||||
@@ -150,7 +156,7 @@ If you can see the `config.yaml` file, you have indeed entered the `project dire
|
||||
command:
|
||||
|
||||
```
|
||||
docker run -d --name xiaozhi-esp32-server --restart always --security-opt seccomp:unconfined -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-es32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
docker run -d --name xiaozhi-esp32-server --restart always --security-opt seccomp:unconfined -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-esp32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
```
|
||||
|
||||
If executed for the first time, it may take several minutes, and you have to be patient to wait for it to complete the
|
||||
@@ -254,16 +260,22 @@ selected_module:
|
||||
TTS: EdgeTTS
|
||||
```
|
||||
|
||||
For example, modify the components used by `llm`, depending on which` llm` supports this project, as follows, it
|
||||
supports `Deepseekllm` and` Chatglmllm`. You are modified to the corresponding LLM in `selectd_module`
|
||||
For example, to modify the components used by the `LLM`, it depends on which `LLM` API interfaces are supported by this project. Currently, the supported ones are `openai` and `dify`. We welcome validation and support for more LLM platforms' interfaces.
|
||||
When using it, change the `selected_module` to the corresponding name of the following LLM configurations:
|
||||
|
||||
```
|
||||
LLM:
|
||||
AliLLM:
|
||||
type: openai
|
||||
...
|
||||
DeepSeekLLM:
|
||||
type: openai
|
||||
...
|
||||
ChatGLMLLM:
|
||||
type: openai
|
||||
...
|
||||
DifyLLM:
|
||||
type: openai
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
+18
-1
@@ -38,7 +38,9 @@ delete_audio: true
|
||||
selected_module:
|
||||
ASR: FunASR
|
||||
VAD: SileroVAD
|
||||
LLM: DifyLLM
|
||||
# 将根据配置名称对应的type调用实际的LLM适配器
|
||||
LLM: ChatGLMLLM
|
||||
# TTS将根据配置名称对应的type调用实际的TTS适配器
|
||||
TTS: EdgeTTS
|
||||
|
||||
ASR:
|
||||
@@ -53,31 +55,46 @@ VAD:
|
||||
min_silence_duration_ms: 700 # 如果说话停顿比较长，可以把这个值设置大一些
|
||||
|
||||
LLM:
|
||||
# 当前支持的type为openai、dify，可自行适配
|
||||
AliLLM:
|
||||
# 定义LLM API类型
|
||||
type: openai
|
||||
# 可在这里找到你的 api_key https://bailian.console.aliyun.com/?apiKey=1#/api-key
|
||||
base_url: https://dashscope.aliyuncs.com/compatible-mode/v1
|
||||
model_name: qwen-turbo
|
||||
api_key: 你的deepseek api key
|
||||
DeepSeekLLM:
|
||||
# 定义LLM API类型
|
||||
type: openai
|
||||
# 可在这里找到你的api key https://platform.deepseek.com/
|
||||
model_name: deepseek-chat
|
||||
url: https://api.deepseek.com
|
||||
api_key: 你的deepseek api key
|
||||
ChatGLMLLM:
|
||||
# 定义LLM API类型
|
||||
type: openai
|
||||
# glm-4-flash 是免费的，但是还是需要注册填写api_key的
|
||||
# 可在这里找到你的api key https://bigmodel.cn/usercenter/proj-mgmt/apikeys
|
||||
model_name: glm-4-flash
|
||||
url: https://open.bigmodel.cn/api/paas/v4/
|
||||
api_key: 你的ChatGLMLLM api key
|
||||
DifyLLM:
|
||||
# 定义LLM API类型
|
||||
type: dify
|
||||
# 建议使用本地部署的dify接口，国内部分区域访问dify公有云接口可能会受限
|
||||
# 如果使用DifyLLM，配置文件里prompt(提示词)是无效的，需要在dify控制台设置提示词
|
||||
base_url: https://api.dify.cn/v1
|
||||
api_key: 你的DifyLLM api key
|
||||
TTS:
|
||||
# 当前支持的type为edge、doubao，可自行适配
|
||||
EdgeTTS:
|
||||
# 定义TTS API类型
|
||||
type: edge
|
||||
voice: zh-CN-XiaoxiaoNeural
|
||||
output_file: tmp/
|
||||
DoubaoTTS:
|
||||
# 定义TTS API类型
|
||||
type: doubao
|
||||
# 火山引擎语音合成服务，需要先在火山引擎控制台创建应用并获取appid和access_token
|
||||
# 山引擎语音一定要购买花钱，起步价30元，就有100并发了。如果用免费的只有2个并发，会经常报tts错误
|
||||
# 购买服务后，购买免费的音色后，可能要等半小时左右，才能使用。
|
||||
|
||||
+7
-2
@@ -1,10 +1,15 @@
|
||||
import os
|
||||
import argparse
|
||||
from core.utils.util import read_config
|
||||
from core.utils.util import read_config, get_project_dir
|
||||
|
||||
|
||||
def load_config():
|
||||
"""加载配置文件"""
|
||||
parser = argparse.ArgumentParser(description="Server configuration")
|
||||
parser.add_argument("--config_path", type=str, default="config.yaml")
|
||||
default_config_file = "config.yaml"
|
||||
# 判断是否存在私有的配置文件
|
||||
if os.path.exists(get_project_dir() + "." + default_config_file):
|
||||
default_config_file = "." + default_config_file
|
||||
parser.add_argument("--config_path", type=str, default=default_config_file)
|
||||
args = parser.parse_args()
|
||||
return read_config(args.config_path)
|
||||
|
||||
@@ -35,11 +35,13 @@ async def handleAudioMessage(conn, audio):
|
||||
conn.asr_audio.clear()
|
||||
conn.reset_vad_states()
|
||||
|
||||
|
||||
async def startToChat(conn, text):
|
||||
# 异步发送 stt 信息
|
||||
asyncio.create_task(
|
||||
stt_task = asyncio.create_task(
|
||||
schedule_with_interrupt(0, send_stt_message(conn, text))
|
||||
)
|
||||
conn.scheduled_tasks.append(stt_task)
|
||||
conn.executor.submit(conn.chat, text)
|
||||
|
||||
|
||||
@@ -52,7 +54,10 @@ async def sendAudioMessage(conn, audios, duration, text):
|
||||
conn.tts_start_speak_time = time.time()
|
||||
|
||||
# 发送 sentence_start（每个音频文件之前发送一次）
|
||||
await send_tts_message(conn, "sentence_start", text)
|
||||
sentence_task = asyncio.create_task(
|
||||
schedule_with_interrupt(base_delay, send_tts_message(conn, "sentence_start", text))
|
||||
)
|
||||
conn.scheduled_tasks.append(sentence_task)
|
||||
|
||||
conn.tts_duration += duration
|
||||
|
||||
@@ -60,10 +65,6 @@ async def sendAudioMessage(conn, audios, duration, text):
|
||||
for idx, opus_packet in enumerate(audios):
|
||||
await conn.websocket.send(opus_packet)
|
||||
|
||||
# 每个音频文件发送结束时，发送 sentence_end
|
||||
if idx == len(audios) - 1:
|
||||
await send_tts_message(conn, "sentence_end", text)
|
||||
|
||||
if conn.llm_finish_task and text == conn.tts_last_text:
|
||||
stop_duration = conn.tts_duration - (time.time() - conn.tts_start_speak_time)
|
||||
stop_task = asyncio.create_task(
|
||||
@@ -71,6 +72,7 @@ async def sendAudioMessage(conn, audios, duration, text):
|
||||
)
|
||||
conn.scheduled_tasks.append(stop_task)
|
||||
|
||||
|
||||
async def send_tts_message(conn, state, text=None):
|
||||
"""发送 TTS 状态消息"""
|
||||
message = {
|
||||
@@ -85,6 +87,7 @@ async def send_tts_message(conn, state, text=None):
|
||||
if state == "stop":
|
||||
conn.clearSpeakStatus()
|
||||
|
||||
|
||||
async def send_stt_message(conn, text):
|
||||
"""发送 STT 状态消息"""
|
||||
stt_text = get_string_no_punctuation_or_emoji(text)
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
|
||||
class LLMProviderBase(ABC):
|
||||
@abstractmethod
|
||||
def response(self, session_id, dialogue):
|
||||
"""LLM response generator"""
|
||||
pass
|
||||
@@ -0,0 +1,39 @@
|
||||
import json
|
||||
import logging
|
||||
import requests
|
||||
from core.providers.llm.base import LLMProviderBase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMProvider(LLMProviderBase):
|
||||
def __init__(self, config):
|
||||
self.api_key = config["api_key"]
|
||||
self.base_url = config.get("base_url", "https://api.dify.ai/v1").rstrip('/')
|
||||
|
||||
def response(self, session_id, dialogue):
|
||||
try:
|
||||
# 取最后一条用户消息
|
||||
last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
|
||||
|
||||
# 发起流式请求
|
||||
with requests.post(
|
||||
f"{self.base_url}/chat-messages",
|
||||
headers={"Authorization": f"Bearer {self.api_key}"},
|
||||
json={
|
||||
"query": last_msg["content"],
|
||||
"response_mode": "streaming",
|
||||
"user": session_id,
|
||||
"inputs": {}
|
||||
},
|
||||
stream=True
|
||||
) as r:
|
||||
for line in r.iter_lines():
|
||||
if line.startswith(b'data: '):
|
||||
event = json.loads(line[6:])
|
||||
if event.get('answer'):
|
||||
yield event['answer']
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error in response generation: {e}")
|
||||
yield "【服务响应异常】"
|
||||
@@ -0,0 +1,35 @@
|
||||
import logging
|
||||
import openai
|
||||
from core.providers.llm.base import LLMProviderBase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLMProvider(LLMProviderBase):
|
||||
def __init__(self, config):
|
||||
self.model_name = config.get("model_name")
|
||||
self.api_key = config.get("api_key")
|
||||
if 'base_url' in config:
|
||||
self.base_url = config.get("base_url")
|
||||
else:
|
||||
self.base_url = config.get("url")
|
||||
if "你" in self.api_key:
|
||||
logger.error("你还没配置LLM的密钥，请在配置文件中配置密钥，否则无法正常工作")
|
||||
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
|
||||
|
||||
def response(self, session_id, dialogue):
|
||||
try:
|
||||
responses = self.client.chat.completions.create(
|
||||
model=self.model_name,
|
||||
messages=dialogue,
|
||||
stream=True
|
||||
)
|
||||
for chunk in responses:
|
||||
# 检查是否存在有效的choice且content不为空
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
content = getattr(delta, 'content', '')
|
||||
if content: # 仅在content非空时生成
|
||||
yield content
|
||||
except Exception as e:
|
||||
logger.error(f"Error in response generation: {e}")
|
||||
@@ -0,0 +1,83 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import numpy as np
|
||||
import opuslib
|
||||
from pydub import AudioSegment
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TTSProviderBase(ABC):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
self.delete_audio_file = delete_audio_file
|
||||
self.output_file = config.get("output_file")
|
||||
|
||||
@abstractmethod
|
||||
def generate_filename(self):
|
||||
pass
|
||||
|
||||
def to_tts(self, text):
|
||||
tmp_file = self.generate_filename()
|
||||
try:
|
||||
max_repeat_time = 5
|
||||
while not os.path.exists(tmp_file) and max_repeat_time > 0:
|
||||
asyncio.run(self.text_to_speak(text, tmp_file))
|
||||
if not os.path.exists(tmp_file):
|
||||
max_repeat_time = max_repeat_time - 1
|
||||
logger.error(f"语音生成失败: {text}:{tmp_file}，再试{max_repeat_time}次")
|
||||
|
||||
if max_repeat_time > 0:
|
||||
logger.info(f"语音生成成功: {text}:{tmp_file}，重试{5 - max_repeat_time}次")
|
||||
|
||||
return tmp_file
|
||||
except Exception as e:
|
||||
logger.info(f"Failed to generate TTS file: {e}")
|
||||
return None
|
||||
|
||||
@abstractmethod
|
||||
async def text_to_speak(self, text, output_file):
|
||||
pass
|
||||
|
||||
def wav_to_opus_data(self, wav_file_path):
|
||||
# 使用pydub加载PCM文件
|
||||
# 获取文件后缀名
|
||||
file_type = os.path.splitext(wav_file_path)[1]
|
||||
if file_type:
|
||||
file_type = file_type.lstrip('.')
|
||||
audio = AudioSegment.from_file(wav_file_path, format=file_type)
|
||||
|
||||
duration = len(audio) / 1000.0
|
||||
|
||||
# 转换为单声道和16kHz采样率（确保与编码器匹配）
|
||||
audio = audio.set_channels(1).set_frame_rate(16000)
|
||||
|
||||
# 获取原始PCM数据（16位小端）
|
||||
raw_data = audio.raw_data
|
||||
|
||||
# 初始化Opus编码器
|
||||
encoder = opuslib.Encoder(16000, 1, opuslib.APPLICATION_AUDIO)
|
||||
|
||||
# 编码参数
|
||||
frame_duration = 60 # 60ms per frame
|
||||
frame_size = int(16000 * frame_duration / 1000) # 960 samples/frame
|
||||
|
||||
opus_datas = []
|
||||
# 按帧处理所有音频数据（包括最后一帧可能补零）
|
||||
for i in range(0, len(raw_data), frame_size * 2): # 16bit=2bytes/sample
|
||||
# 获取当前帧的二进制数据
|
||||
chunk = raw_data[i:i + frame_size * 2]
|
||||
|
||||
# 如果最后一帧不足，补零
|
||||
if len(chunk) < frame_size * 2:
|
||||
chunk += b'\x00' * (frame_size * 2 - len(chunk))
|
||||
|
||||
# 转换为numpy数组处理
|
||||
np_frame = np.frombuffer(chunk, dtype=np.int16)
|
||||
|
||||
# 编码Opus数据
|
||||
opus_data = encoder.encode(np_frame.tobytes(), frame_size)
|
||||
opus_datas.append(opus_data)
|
||||
|
||||
return opus_datas, duration
|
||||
@@ -0,0 +1,56 @@
|
||||
import os
|
||||
import uuid
|
||||
import json
|
||||
import base64
|
||||
import requests
|
||||
from datetime import datetime
|
||||
from core.providers.tts.base import TTSProviderBase
|
||||
|
||||
|
||||
class TTSProvider(TTSProviderBase):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
super().__init__(config, delete_audio_file)
|
||||
self.appid = config.get("appid")
|
||||
self.access_token = config.get("access_token")
|
||||
self.cluster = config.get("cluster")
|
||||
self.voice = config.get("voice")
|
||||
|
||||
self.host = "openspeech.bytedance.com"
|
||||
self.api_url = f"https://{self.host}/api/v1/tts"
|
||||
self.header = {"Authorization": f"Bearer;{self.access_token}"}
|
||||
|
||||
def generate_filename(self, extension=".wav"):
|
||||
return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
|
||||
|
||||
async def text_to_speak(self, text, output_file):
|
||||
request_json = {
|
||||
"app": {
|
||||
"appid": self.appid,
|
||||
"token": "access_token",
|
||||
"cluster": self.cluster
|
||||
},
|
||||
"user": {
|
||||
"uid": "1"
|
||||
},
|
||||
"audio": {
|
||||
"voice_type": self.voice,
|
||||
"encoding": "wav",
|
||||
"speed_ratio": 1.0,
|
||||
"volume_ratio": 1.0,
|
||||
"pitch_ratio": 1.0,
|
||||
},
|
||||
"request": {
|
||||
"reqid": str(uuid.uuid4()),
|
||||
"text": text,
|
||||
"text_type": "plain",
|
||||
"operation": "query",
|
||||
"with_frontend": 1,
|
||||
"frontend_type": "unitTson"
|
||||
}
|
||||
}
|
||||
|
||||
resp = requests.post(self.api_url, json.dumps(request_json), headers=self.header)
|
||||
if "data" in resp.json():
|
||||
data = resp.json()["data"]
|
||||
file_to_save = open(output_file, "wb")
|
||||
file_to_save.write(base64.b64decode(data))
|
||||
@@ -0,0 +1,18 @@
|
||||
import os
|
||||
import uuid
|
||||
import edge_tts
|
||||
from datetime import datetime
|
||||
from core.providers.tts.base import TTSProviderBase
|
||||
|
||||
|
||||
class TTSProvider(TTSProviderBase):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
super().__init__(config, delete_audio_file)
|
||||
self.voice = config.get("voice")
|
||||
|
||||
def generate_filename(self, extension=".mp3"):
|
||||
return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
|
||||
|
||||
async def text_to_speak(self, text, output_file):
|
||||
communicate = edge_tts.Communicate(text, voice=self.voice) # Use your preferred voice
|
||||
await communicate.save(output_file)
|
||||
+8
-2
@@ -25,11 +25,17 @@ class WebSocketServer:
|
||||
self.config["delete_audio"]
|
||||
),
|
||||
llm.create_instance(
|
||||
self.config["selected_module"]["LLM"],
|
||||
self.config["selected_module"]["LLM"]
|
||||
if not 'type' in self.config["LLM"][self.config["selected_module"]["LLM"]]
|
||||
else
|
||||
self.config["LLM"][self.config["selected_module"]["LLM"]]['type'],
|
||||
self.config["LLM"][self.config["selected_module"]["LLM"]],
|
||||
),
|
||||
tts.create_instance(
|
||||
self.config["selected_module"]["TTS"],
|
||||
self.config["selected_module"]["TTS"]
|
||||
if not 'type' in self.config["TTS"][self.config["selected_module"]["TTS"]]
|
||||
else
|
||||
self.config["TTS"][self.config["selected_module"]["TTS"]]["type"],
|
||||
self.config["TTS"][self.config["selected_module"]["TTS"]],
|
||||
self.config["delete_audio"]
|
||||
)
|
||||
|
||||
+14
-125
@@ -1,7 +1,10 @@
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import logging
|
||||
import openai
|
||||
import requests
|
||||
import importlib
|
||||
from datetime import datetime
|
||||
from core.utils.util import is_segment
|
||||
from core.utils.util import get_string_no_punctuation_or_emoji
|
||||
@@ -11,132 +14,15 @@ from abc import ABC, abstractmethod
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class LLM(ABC):
|
||||
@abstractmethod
|
||||
def response(self, session_id, dialogue):
|
||||
"""LLM response generator"""
|
||||
pass
|
||||
|
||||
|
||||
class DeepSeekLLM(LLM):
|
||||
def __init__(self, config):
|
||||
self.model_name = config.get("model_name")
|
||||
self.api_key = config.get("api_key")
|
||||
self.base_url = config.get("url")
|
||||
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
|
||||
|
||||
def response(self, session_id, dialogue):
|
||||
logger.info(f"Generating response using {dialogue}")
|
||||
try:
|
||||
responses = self.client.chat.completions.create(
|
||||
model=self.model_name,
|
||||
messages=dialogue,
|
||||
stream=True
|
||||
)
|
||||
for chunk in responses:
|
||||
# 检查是否存在有效的choice且content不为空
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
content = getattr(delta, 'content', '')
|
||||
if content: # 仅在content非空时生成
|
||||
yield content
|
||||
except Exception as e:
|
||||
logger.error(f"Error in response generation: {e}")
|
||||
|
||||
|
||||
class ChatGLMLLM(LLM):
|
||||
def __init__(self, config):
|
||||
self.model_name = config.get("model_name")
|
||||
self.api_key = config.get("api_key")
|
||||
self.base_url = config.get("url")
|
||||
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
|
||||
|
||||
def response(self, session_id, dialogue):
|
||||
try:
|
||||
responses = self.client.chat.completions.create(
|
||||
model=self.model_name,
|
||||
messages=dialogue,
|
||||
stream=True
|
||||
)
|
||||
for chunk in responses:
|
||||
# 检查是否存在有效的choice且content不为空
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
content = getattr(delta, 'content', '')
|
||||
if content: # 仅在content非空时生成
|
||||
yield content
|
||||
except Exception as e:
|
||||
logger.error(f"Error in response generation: {e}")
|
||||
|
||||
class AliLLM(LLM):
|
||||
def __init__(self, config):
|
||||
self.model_name = config.get("model_name")
|
||||
self.api_key = config.get("api_key")
|
||||
self.base_url = config.get("base_url")
|
||||
self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
|
||||
|
||||
def response(self, session_id, dialogue):
|
||||
try:
|
||||
responses = self.client.chat.completions.create(
|
||||
model=self.model_name,
|
||||
messages=dialogue,
|
||||
stream=True
|
||||
)
|
||||
for chunk in responses:
|
||||
# 检查是否存在有效的choice且content不为空
|
||||
if chunk.choices and len(chunk.choices) > 0:
|
||||
delta = chunk.choices[0].delta
|
||||
content = getattr(delta, 'content', '')
|
||||
if content: # 仅在content非空时生成
|
||||
yield content
|
||||
except Exception as e:
|
||||
logger.error(f"Error in response generation: {e}")
|
||||
|
||||
class DifyLLM(LLM):
|
||||
def __init__(self, config):
|
||||
self.api_key = config["api_key"]
|
||||
self.base_url = config.get("base_url", "https://api.dify.ai/v1").rstrip('/')
|
||||
|
||||
def response(self, session_id, dialogue):
|
||||
try:
|
||||
# 取最后一条用户消息
|
||||
last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
|
||||
|
||||
# 发起流式请求
|
||||
with requests.post(
|
||||
f"{self.base_url}/chat-messages",
|
||||
headers={"Authorization": f"Bearer {self.api_key}"},
|
||||
json={
|
||||
"query": last_msg["content"],
|
||||
"response_mode": "streaming",
|
||||
"user": session_id,
|
||||
"inputs": {}
|
||||
},
|
||||
stream=True
|
||||
) as r:
|
||||
for line in r.iter_lines():
|
||||
if line.startswith(b'data: '):
|
||||
event = json.loads(line[6:])
|
||||
if event.get('answer'):
|
||||
yield event['answer']
|
||||
|
||||
except Exception:
|
||||
yield "【服务响应异常】"
|
||||
|
||||
|
||||
def create_instance(class_name, *args, **kwargs):
|
||||
# 获取类对象
|
||||
cls_map = {
|
||||
"DeepSeekLLM": DeepSeekLLM,
|
||||
"ChatGLMLLM": ChatGLMLLM,
|
||||
"DifyLLM": DifyLLM,
|
||||
"AliLLM": AliLLM,
|
||||
# 可扩展其他LLM实现
|
||||
}
|
||||
# 创建LLM实例
|
||||
if os.path.exists(os.path.join('core', 'providers', 'llm', class_name, f'{class_name}.py')):
|
||||
lib_name = f'core.providers.llm.{class_name}.{class_name}'
|
||||
if lib_name not in sys.modules:
|
||||
sys.modules[lib_name] = importlib.import_module(f'{lib_name}')
|
||||
return sys.modules[lib_name].LLMProvider(*args, **kwargs)
|
||||
|
||||
if cls := cls_map.get(class_name):
|
||||
return cls(*args, **kwargs)
|
||||
raise ValueError(f"不支持的LLM类型: {class_name}")
|
||||
raise ValueError(f"不支持的LLM类型: {class_name}，请检查该配置的type是否设置正确")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -145,7 +31,10 @@ if __name__ == "__main__":
|
||||
"""
|
||||
config = read_config(get_project_dir() + "config.yaml")
|
||||
llm = create_instance(
|
||||
config["selected_module"]["LLM"],
|
||||
config["selected_module"]["LLM"]
|
||||
if not "type" in config["LLM"][config["selected_module"]["LLM"]]
|
||||
else
|
||||
config["LLM"][config["selected_module"]["LLM"]]["type"],
|
||||
config["LLM"][config["selected_module"]["LLM"]]
|
||||
)
|
||||
|
||||
|
||||
+14
-154
@@ -1,165 +1,22 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
import uuid
|
||||
import base64
|
||||
import sys
|
||||
import logging
|
||||
import importlib
|
||||
from datetime import datetime
|
||||
import edge_tts
|
||||
import numpy as np
|
||||
import opuslib
|
||||
import requests
|
||||
from core.utils.util import read_config, get_project_dir
|
||||
from pydub import AudioSegment
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TTS(ABC):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
self.delete_audio_file = delete_audio_file
|
||||
self.output_file = config.get("output_file")
|
||||
|
||||
@abstractmethod
|
||||
def generate_filename(self):
|
||||
pass
|
||||
|
||||
def to_tts(self, text):
|
||||
tmp_file = self.generate_filename()
|
||||
try:
|
||||
max_repeat_time = 5
|
||||
while not os.path.exists(tmp_file) and max_repeat_time > 0:
|
||||
asyncio.run(self.text_to_speak(text, tmp_file))
|
||||
if not os.path.exists(tmp_file):
|
||||
max_repeat_time = max_repeat_time - 1
|
||||
logger.error(f"语音生成失败: {text}:{tmp_file}，再试{max_repeat_time}次")
|
||||
|
||||
return tmp_file
|
||||
except Exception as e:
|
||||
logger.info(f"Failed to generate TTS file: {e}")
|
||||
return None
|
||||
|
||||
@abstractmethod
|
||||
async def text_to_speak(self, text, output_file):
|
||||
pass
|
||||
|
||||
def wav_to_opus_data(self, wav_file_path):
|
||||
# 使用pydub加载PCM文件
|
||||
# 获取文件后缀名
|
||||
file_type = os.path.splitext(wav_file_path)[1]
|
||||
if file_type:
|
||||
file_type = file_type.lstrip('.')
|
||||
audio = AudioSegment.from_file(wav_file_path, format=file_type)
|
||||
|
||||
duration = len(audio) / 1000.0
|
||||
|
||||
# 转换为单声道和16kHz采样率（确保与编码器匹配）
|
||||
audio = audio.set_channels(1).set_frame_rate(16000)
|
||||
|
||||
# 获取原始PCM数据（16位小端）
|
||||
raw_data = audio.raw_data
|
||||
|
||||
# 初始化Opus编码器
|
||||
encoder = opuslib.Encoder(16000, 1, opuslib.APPLICATION_AUDIO)
|
||||
|
||||
# 编码参数
|
||||
frame_duration = 60 # 60ms per frame
|
||||
frame_size = int(16000 * frame_duration / 1000) # 960 samples/frame
|
||||
|
||||
opus_datas = []
|
||||
# 按帧处理所有音频数据（包括最后一帧可能补零）
|
||||
for i in range(0, len(raw_data), frame_size * 2): # 16bit=2bytes/sample
|
||||
# 获取当前帧的二进制数据
|
||||
chunk = raw_data[i:i + frame_size * 2]
|
||||
|
||||
# 如果最后一帧不足，补零
|
||||
if len(chunk) < frame_size * 2:
|
||||
chunk += b'\x00' * (frame_size * 2 - len(chunk))
|
||||
|
||||
# 转换为numpy数组处理
|
||||
np_frame = np.frombuffer(chunk, dtype=np.int16)
|
||||
|
||||
# 编码Opus数据
|
||||
opus_data = encoder.encode(np_frame.tobytes(), frame_size)
|
||||
opus_datas.append(opus_data)
|
||||
|
||||
return opus_datas, duration
|
||||
|
||||
|
||||
class EdgeTTS(TTS):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
super().__init__(config, delete_audio_file)
|
||||
self.voice = config.get("voice")
|
||||
|
||||
def generate_filename(self, extension=".mp3"):
|
||||
return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
|
||||
|
||||
async def text_to_speak(self, text, output_file):
|
||||
communicate = edge_tts.Communicate(text, voice=self.voice) # Use your preferred voice
|
||||
await communicate.save(output_file)
|
||||
|
||||
|
||||
class DoubaoTTS(TTS):
|
||||
def __init__(self, config, delete_audio_file):
|
||||
super().__init__(config, delete_audio_file)
|
||||
self.appid = config.get("appid")
|
||||
self.access_token = config.get("access_token")
|
||||
self.cluster = config.get("cluster")
|
||||
self.voice = config.get("voice")
|
||||
|
||||
self.host = "openspeech.bytedance.com"
|
||||
self.api_url = f"https://{self.host}/api/v1/tts"
|
||||
self.header = {"Authorization": f"Bearer;{self.access_token}"}
|
||||
|
||||
def generate_filename(self, extension=".wav"):
|
||||
return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
|
||||
|
||||
async def text_to_speak(self, text, output_file):
|
||||
request_json = {
|
||||
"app": {
|
||||
"appid": self.appid,
|
||||
"token": "access_token",
|
||||
"cluster": self.cluster
|
||||
},
|
||||
"user": {
|
||||
"uid": "1"
|
||||
},
|
||||
"audio": {
|
||||
"voice_type": self.voice,
|
||||
"encoding": "wav",
|
||||
"speed_ratio": 1.0,
|
||||
"volume_ratio": 1.0,
|
||||
"pitch_ratio": 1.0,
|
||||
},
|
||||
"request": {
|
||||
"reqid": str(uuid.uuid4()),
|
||||
"text": text,
|
||||
"text_type": "plain",
|
||||
"operation": "query",
|
||||
"with_frontend": 1,
|
||||
"frontend_type": "unitTson"
|
||||
}
|
||||
}
|
||||
|
||||
resp = requests.post(self.api_url, json.dumps(request_json), headers=self.header)
|
||||
if "data" in resp.json():
|
||||
data = resp.json()["data"]
|
||||
file_to_save = open(output_file, "wb")
|
||||
file_to_save.write(base64.b64decode(data))
|
||||
|
||||
|
||||
def create_instance(class_name, *args, **kwargs):
|
||||
# 获取类对象
|
||||
cls_map = {
|
||||
"DoubaoTTS": DoubaoTTS,
|
||||
"EdgeTTS": EdgeTTS,
|
||||
# 可扩展其他TTS实现
|
||||
}
|
||||
# 创建TTS实例
|
||||
if os.path.exists(os.path.join('core', 'providers', 'tts', f'{class_name}.py')):
|
||||
lib_name = f'core.providers.tts.{class_name}'
|
||||
if lib_name not in sys.modules:
|
||||
sys.modules[lib_name] = importlib.import_module(f'{lib_name}')
|
||||
return sys.modules[lib_name].TTSProvider(*args, **kwargs)
|
||||
|
||||
if cls := cls_map.get(class_name):
|
||||
return cls(*args, **kwargs)
|
||||
raise ValueError(f"不支持的TTS类型: {class_name}")
|
||||
raise ValueError(f"不支持的TTS类型: {class_name}，请检查该配置的type是否设置正确")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
@@ -168,7 +25,10 @@ if __name__ == "__main__":
|
||||
"""
|
||||
config = read_config(get_project_dir() + "config.yaml")
|
||||
tts = create_instance(
|
||||
config["selected_module"]["TTS"],
|
||||
config["selected_module"]["TTS"]
|
||||
if not 'type' in config["TTS"][config["selected_module"]["TTS"]]
|
||||
else
|
||||
config["TTS"][config["selected_module"]["TTS"]]["type"],
|
||||
config["TTS"][config["selected_module"]["TTS"]],
|
||||
config["delete_audio"]
|
||||
)
|
||||
|
||||
+7
-5
@@ -1,8 +1,7 @@
|
||||
import yaml
|
||||
import unicodedata
|
||||
import socket
|
||||
import os
|
||||
import json
|
||||
import yaml
|
||||
import socket
|
||||
|
||||
|
||||
def get_project_dir():
|
||||
@@ -41,6 +40,7 @@ def is_segment(tokens):
|
||||
else:
|
||||
return False
|
||||
|
||||
|
||||
def is_punctuation_or_emoji(char):
|
||||
"""检查字符是否为空格、指定标点或表情符号"""
|
||||
# 定义需要去除的中英文标点（包括全角/半角）
|
||||
@@ -49,7 +49,7 @@ def is_punctuation_or_emoji(char):
|
||||
'。', '.', # 中文句号 + 英文句号
|
||||
'！', '!', # 中文感叹号 + 英文感叹号
|
||||
'-', '－', # 英文连字符 + 中文全角横线
|
||||
'、' # 中文顿号
|
||||
'、' # 中文顿号
|
||||
}
|
||||
if char.isspace() or char in punctuation_set:
|
||||
return True
|
||||
@@ -63,6 +63,7 @@ def is_punctuation_or_emoji(char):
|
||||
]
|
||||
return any(start <= code_point <= end for start, end in emoji_ranges)
|
||||
|
||||
|
||||
def get_string_no_punctuation_or_emoji(s):
|
||||
"""去除字符串首尾的空格、标点符号和表情符号"""
|
||||
chars = list(s)
|
||||
@@ -74,7 +75,8 @@ def get_string_no_punctuation_or_emoji(s):
|
||||
end = len(chars) - 1
|
||||
while end >= start and is_punctuation_or_emoji(chars[end]):
|
||||
end -= 1
|
||||
return ''.join(chars[start:end+1])
|
||||
return ''.join(chars[start:end + 1])
|
||||
|
||||
|
||||
def remove_punctuation_and_length(text):
|
||||
# 全角符号和半角符号的Unicode范围
|
||||
|
||||
@@ -13,9 +13,7 @@ docker build -t xiaozhi-esp32-server:local -f ./Dockerfile .
|
||||
docker stop xiaozhi-esp32-server
|
||||
docker rm xiaozhi-esp32-server
|
||||
|
||||
cd /Users/hrz/myworkspace/docker-java-env/thirddata/
|
||||
cp /Users/hrz/myworkspace/esp32/xiaozhi-esp32-server/config.yaml ./
|
||||
docker run -d --name xiaozhi-esp32-server --restart always -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-es32-server/config.yaml xiaozhi-esp32-server:local
|
||||
docker run -d --name xiaozhi-esp32-server --restart always -p 8000:8000 -v $(pwd)/.config.yaml:/opt/xiaozhi-esp32-server/config.yaml xiaozhi-esp32-server:local
|
||||
|
||||
docker logs -f xiaozhi-esp32-server
|
||||
|
||||
@@ -40,6 +38,6 @@ docker manifest push ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
6、运行线上镜像
|
||||
```
|
||||
cd /Users/hrz/myworkspace/docker-java-env/thirddata/
|
||||
docker run -d --name xiaozhi-esp32-server --restart always -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-es32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
docker run -d --name xiaozhi-esp32-server --restart always -p 8000:8000 -v $(pwd)/config.yaml:/opt/xiaozhi-esp32-server/config.yaml ccr.ccs.tencentyun.com/xinnan/xiaozhi-esp32-server:latest
|
||||
docker logs -f xiaozhi-esp32-server
|
||||
```
|
||||
Reference in New Issue
Block a user