Hass/xiaozhi-esp32-server
1
0
Fork 0
mirror of https://github.com/xinnan-tech/xiaozhi-esp32-server.git synced 2026-07-25 00:23:53 +08:00
Code Issues Packages Projects Releases Wiki Activity

udpate:更新文档

Browse Source
This commit is contained in:
hrz
2025-06-03 10:27:13 +08:00
parent 3e68667323
commit a6bc910b0f
3 changed files with 242 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README_en.md
+77 -103
View File
@@ -6,14 +6,14 @@
This project provides backend services for the open-source smart hardware project
<a href="https://github.com/78/xiaozhi-esp32">xiaozhi-esp32</a><br/>
Implemented using Python, Java, and Vue according to the <a href="https://ccnphfhqs21z.feishu.cn/wiki/M0XiwldO9iJwHikpXD5cEx71nKh">Xiaozhi Communication Protocol</a><br/>
Helping you quickly set up your Xiaozhi server
Helps you quickly set up your Xiaozhi server
</p>
<p align="center">
<a href="./README.md">中文</a>
· <a href="./docs/FAQ.md">FAQ</a>
· <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/issues">Report Issues</a>
· <a href="./README_ed.md#deployment-documentation">Deployment Guide</a>
· <a href="./README_en.md#deployment-documentation">Deployment Guide</a>
· <a href="https://github.com/xinnan-tech/xiaozhi-esp32-server/releases">Release Notes</a>
</p>
<p align="center">
@@ -50,7 +50,7 @@ Want to see it in action? Check out these videos 🎥
<td>
<a href="https://www.bilibili.com/video/BV1FMFyejExX" target="_blank">
<picture>
<img alt="Xiaozhi esp32 connecting to custom backend model" src="docs/images/demo1.png" />
<img alt="Xiaozhi esp32 connecting to own backend model" src="docs/images/demo1.png" />
</picture>
</a>
</td>
@@ -64,14 +64,14 @@ Want to see it in action? Check out these videos 🎥
<td>
<a href="https://www.bilibili.com/video/BV12yA2egEaC" target="_blank">
<picture>
<img alt="Cantonese communication" src="docs/images/demo3.png" />
<img alt="Using Cantonese" src="docs/images/demo3.png" />
</picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1pNXWYGEx1" target="_blank">
<picture>
<img alt="Home appliance control" src="docs/images/demo5.png" />
<img alt="Control home appliances" src="docs/images/demo5.png" />
</picture>
</a>
</td>
@@ -94,7 +94,7 @@ Want to see it in action? Check out these videos 🎥
<td>
<a href="https://www.bilibili.com/video/BV1VC96Y5EMH" target="_blank">
<picture>
<img alt="Music playback" src="docs/images/demo7.png" />
<img alt="Play music" src="docs/images/demo7.png" />
</picture>
</a>
</td>
@@ -108,7 +108,7 @@ Want to see it in action? Check out these videos 🎥
<td>
<a href="https://www.bilibili.com/video/BV178XuYfEpi" target="_blank">
<picture>
<img alt="IOT device control" src="docs/images/demo9.png" />
<img alt="IOT command control" src="docs/images/demo9.png" />
</picture>
</a>
</td>
@@ -120,6 +120,33 @@ Want to see it in action? Check out these videos 🎥
</a>
</td>
</tr>
<tr>
<td>
<a href="https://www.bilibili.com/video/BV12J7WzBEaH" target="_blank">
<picture>
<img alt="Real-time interruption" src="docs/images/demo10.png" />
</picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1Co76z7EvK" target="_blank">
<picture>
<img alt="Photo recognition" src="docs/images/demo12.png" />
</picture>
</a>
</td>
<td>
<a href="https://www.bilibili.com/video/BV1TJ7WzzEo6" target="_blank">
<picture>
<img alt="Multi-command tasks" src="docs/images/demo11.png" />
</picture>
</a>
</td>
<td>
</td>
<td>
</td>
</tr>
</table>
---
@@ -137,137 +164,84 @@ It is recommended that users prioritize service providers with relevant business
![Banners](docs/images/banner2.png)
This project offers two deployment methods. Please choose based on your specific needs:
This project provides two deployment methods. Please choose according to your specific needs:
#### 🚀 Deployment Method Selection
| Deployment Method | Features | Suitable Scenarios | Deployment Guide | Requirements | Video Tutorial |
|---------|------|---------|---------|---------|---------|
| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database needed | [Docker Version](./docs/Deployment.md#method-1-docker-server-only) / [Source Code Deployment](./docs/Deployment.md#method-2-local-source-code-server-only) | 2 cores 4G if using `FunASR`, 2 cores 2G if using all APIs | - |
| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience | [Docker Version](./docs/Deployment_all.md#method-1-docker-full-modules) / [Source Code Deployment](./docs/Deployment_all.md#method-2-local-source-code-full-modules) | 4 cores 8G if using `FunASR`, 2 cores 4G if using all APIs | [Local Source Code Startup Video Tutorial](https://www.bilibili.com/video/BV1wBJhz4Ewe) / [Local Source Code Auto-Update Tutorial](./docs/dev-ops-integration.md) |
| Deployment Method | Features | Use Case | Docker Deployment Guide | Source Code Deployment Guide |
|---------|------|---------|---------|---------|
| **Simplified Installation** | Smart dialogue, IOT functionality, data stored in configuration files | Low-configuration environment, no database required | [Docker Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E5%8F%AA%E8%BF%90%E8%A1%8Cserver) | [Local Source Code Server Only](./docs/Deployment.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E5%8F%AA%E8%BF%90%E8%A1%8Cserver)|
| **Full Module Installation** | Smart dialogue, IOT, OTA, Control Panel, data stored in database | Complete functionality experience |[Docker Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%B8%80docker%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) | [Local Source Code Full Module](./docs/Deployment_all.md#%E6%96%B9%E5%BC%8F%E4%BA%8C%E6%9C%AC%E5%9C%B0%E6%BA%90%E7%A0%81%E8%BF%90%E8%A1%8C%E5%85%A8%E6%A8%A1%E5%9D%97) |
> 💡 Note: Below is the test platform deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data cleared daily
> 💡 Note: Below are the test platforms deployed with the latest code. You can flash and test if needed. Concurrent users: 6, data will be cleared daily
```
Control Panel: https://2662r3426b.vicp.fun
Control Panel Address: https://2662r3426b.vicp.fun
Service Test Tool: https://2662r3426b.vicp.fun/test/
OTA Interface: https://2662r3426b.vicp.fun/xiaozhi/ota/
Websocket Interface: wss://2662r3426b.vicp.fun/xiaozhi/v1/
OTA Interface Address: https://2662r3426b.vicp.fun/xiaozhi/ota/
Websocket Interface Address: wss://2662r3426b.vicp.fun/xiaozhi/v1/
```
#### 🚩 Configuration Description and Recommendations
> [!Note]
> The default configuration of this project is `Entry Level Free` settings. For better results, we recommend using `Full Streaming Configuration`.
>
> Since version `0.5.2`, this project supports full streaming throughout the entire lifecycle. Compared to versions before `0.5`, response speed has improved by approximately `2.5 seconds`
| Module Name | Entry Level Free Settings | Full Streaming Configuration |
|---------|---------|------|
| ASR(Speech Recognition) | FunASR(Local) | ✅DoubaoASR(Volcano Streaming Speech Recognition) |
| LLM(Large Language Model) | ChatGLMLLM(Zhipu glm-4-flash) | ✅DoubaoLLM(Volcano doubao-1-5-pro-32k-250115) |
| VLLM(Vision Large Model) | ChatGLMVLLM(Zhipu glm-4v-flash) | ✅ChatGLMVLLM(Zhipu glm-4v-flash) |
| TTS(Speech Synthesis) | EdgeTTS(Microsoft Speech) | ✅HuoshanDoubleStreamTTS(Volcano Double Streaming Speech Synthesis) |
| Intent(Intent Recognition) | function_call(Function Call) | ✅function_call(Function Call) |
| Memory(Memory Function) | mem_local_short(Local Short-term Memory) | ✅mem_local_short(Local Short-term Memory) |
---
## Feature List ✨
### Implemented ✅
| Feature Module | Description |
|---------|------|
| Communication Protocol | Based on `xiaozhi-esp32` protocol, implements data interaction through WebSocket |
| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of inactivity |
| Dialogue Interaction | Supports wake-up dialogue, manual dialogue, and real-time interruption. Auto-sleep after long periods of no dialogue |
| Intent Recognition | Supports LLM intent recognition, function call, reducing hard-coded intent judgment |
| Multi-language Recognition | Supports Mandarin, Cantonese, English, Japanese, Korean (default using FunASR) |
| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, also supports Ali Bailing, DeepSeek, Ollama, etc. |
| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces for speech synthesis |
| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes for different scenarios |
| IOT Function | Supports managing registered device IOT functionality, intelligent IoT control based on dialogue context |
| Control Panel | Provides web management interface, supports agent management, user management, system configuration, etc. |
| LLM Module | Supports flexible LLM module switching, default using ChatGLMLLM, can also use Ali Bailian, DeepSeek, Ollama, etc. |
| TTS Module | Supports EdgeTTS (default), Volcano Engine Doubao TTS, and other TTS interfaces |
| Memory Function | Supports ultra-long memory, local summary memory, and no memory modes |
| IOT Function | Supports managing registered device IOT functionality, supports smart IoT control based on dialogue context |
| Control Panel | Provides Web management interface, supports agent management, user management, system configuration, etc. |
### In Development 🚧
To learn about specific development progress, [click here](https://github.com/users/xinnan-tech/projects/3)
If you're a software developer, here's an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
If you are a software developer, here is an [Open Letter to Developers](docs/contributor_open_letter.md). Welcome to join!
---
## Product Ecosystem 👬
Xiaozhi is an ecosystem. When using this product, you might want to check out other excellent projects in this ecosystem:
Xiaozhi is an ecosystem. When using this product, you might also want to check out other excellent projects in this ecosystem
| Project Name | Project Link | Description |
| Project Name | Project Address | Project Description |
|:---------------------|:--------|:--------|
| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue |
| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | A Python-based AI client that allows you to experience Xiaozhi AI functionality through code without physical hardware |
| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | A Java-based open-source project providing complete backend service solutions |
| Xiaozhi Android Client | [xiaozhi-android-client](https://github.com/TOM88812/xiaozhi-android-client) | A Flutter-based Android and iOS voice dialogue application supporting real-time voice interaction and text dialogue. |
| Xiaozhi PC Client | [py-xiaozhi](https://github.com/Huang-junsen/py-xiaozhi) | This project provides a Python-based Xiaozhi AI client, allowing you to experience Xiaozhi AI's functionality through code even without physical hardware. |
| Xiaozhi Java Server | [xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) | The Java version of Xiaozhi open-source backend service is a Java-based open-source project.<br/>It includes both frontend and backend services, aiming to provide users with a complete backend service solution. |
---
## Supported Platforms/Components 📋
## Supported Platforms/Components List 📋
### LLM Language Models
| Usage Method | Supported Platforms | Free Platforms |
|:---:|:---:|:---:|
| openai API | Ali Bailing, Volcano Engine Doubao, DeepSeek, ChatGLM, Gemini | ChatGLM, Gemini |
| ollama API | Ollama | - |
| dify API | Dify | - |
| fastgpt API | Fastgpt | - |
| coze API | Coze | - |
| openai interface call | Ali Bailian, Volcano Engine Doubao, DeepSeek, Zhipu ChatGLM, Gemini | Zhipu ChatGLM, Gemini |
| ollama interface call | Ollama | - |
| dify interface call | Dify | - |
| fastgpt interface call | Fastgpt | - |
| coze interface call | Coze | - |
Actually, any LLM supporting openai API calls can be integrated.
---
### TTS Speech Synthesis
| Usage Method | Supported Platforms | Free Platforms |
|:---:|:---:|:---:|
| API Calls | EdgeTTS, Volcano Engine Doubao TTS, Tencent Cloud, Aliyun TTS, CosyVoiceSiliconflow, TTS302AI, CozeCnTTS, GizwitsTTS, ACGNTTS, OpenAITTS | EdgeTTS, CosyVoiceSiliconflow(partial) |
| Local Service | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS | FishSpeech, GPT_SOVITS_V2, GPT_SOVITS_V3, MinimaxTTS |
---
### VAD Voice Activity Detection
| Type | Platform Name | Usage Method | Pricing | Notes |
|:---:|:---------:|:----:|:----:|:--:|
| VAD | SileroVAD | Local Use | Free | |
---
### ASR Speech Recognition
| Usage Method | Supported Platforms | Free Platforms |
|:---:|:---:|:---:|
| Local Use | FunASR, SherpaASR | FunASR, SherpaASR |
| API Calls | DoubaoASR, FunASRServer, TencentASR, AliyunASR | FunASRServer |
---
### Memory Storage
| Type | Platform Name | Usage Method | Pricing | Notes |
|:------:|:---------------:|:----:|:---------:|:--:|
| Memory | mem0ai | API Calls | 1000 calls/month quota | |
| Memory | mem_local_short | Local Summary | Free | |
---
### Intent Recognition
| Type | Platform Name | Usage Method | Pricing | Notes |
|:------:|:-------------:|:----:|:-------:|:---------------------:|
| Intent | intent_llm | API Calls | Based on LLM pricing | Uses large model for intent recognition, highly versatile |
| Intent | function_call | API Calls | Based on LLM pricing | Uses large model function calls for intent, fast and effective |
---
## Acknowledgments 🙏
| Logo | Project/Company | Description |
|:---:|:---:|:---|
| <img src="./docs/images/logo_bailing.png" width="160"> | [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) | This project was inspired by [Bailing Voice Dialogue Robot](https://github.com/wwbin2017/bailing) and implemented based on it |
| <img src="./docs/images/logo_tenclass.png" width="160"> | [Tenclass](https://www.tenclass.com/) | Thanks to [Tenclass](https://www.tenclass.com/) for developing standard communication protocols, multi-device compatibility solutions, and high-concurrency scenario practices for the Xiaozhi ecosystem; providing comprehensive technical documentation support for this project |
| <img src="./docs/images/logo_xuanfeng.png" width="160"> | [Xuanfeng Technology](https://github.com/Eric0308) | Thanks to [Xuanfeng Technology](https://github.com/Eric0308) for contributing function call framework, MCP communication protocol, and plugin call mechanism implementation code, significantly improving front-end device (IoT) interaction efficiency and functional extensibility through standardized instruction scheduling system and dynamic expansion capabilities |
| <img src="./docs/images/logo_huiyuan.png" width="160"> | [Huiyuan Design](http://ui.kwd988.net/) | Thanks to [Huiyuan Design](http://ui.kwd988.net/) for providing professional visual solutions for this project, empowering product user experience with their design experience serving over a thousand enterprises |
| <img src="./docs/images/logo_qinren.png" width="160"> | [Xi'an Qinren Information Technology](https://www.029app.com/) | Thanks to [Xi'an Qinren Information Technology](https://www.029app.com/) for deepening this project's visual system, ensuring consistency and extensibility of overall design style in multi-scenario applications |
<a href="https://star-history.com/#xinnan-tech/xiaozhi-esp32-server&Date">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date&theme=dark" />
<source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
<img alt="Star History Chart" src="https://api.star-history.com/svg?repos=xinnan-tech/xiaozhi-esp32-server&type=Date" />
</picture>
</a>
In fact, any LLM that supports openai interface calls can be integrated and used.