mirror of
https://github.com/xinnan-tech/xiaozhi-esp32-server.git
synced 2026-07-21 22:53:56 +08:00
update：补充
This commit is contained in:
3030332422
+58
-1
@@ -9,6 +9,7 @@
|
||||
* [3.2. `manager-api` (Management Backend - Java Spring Boot Implementation)](#32-manager-api-management-backend---java-spring-boot-implementation)
|
||||
* [3.3. `manager-web` (Web Management Frontend - Vue.js Implementation)](#33-manager-web-web-management-frontend---vuejs-implementation)
|
||||
* [3.4. `manager-mobile` (Mobile Management Console - uni-app Implementation)](#34-manager-mobile-mobile-management-console---uniapp-implementation)
|
||||
* [3.5. `digital-human` (Digital Human Test Module - Python+Web Implementation)](#35-digital-human-digital-human-test-module---pythonweb-implementation)
|
||||
4. [Data Flow and Interaction Mechanisms](#4-data-flow-and-interaction-mechanisms)
|
||||
5. [Key Features Summary](#5-key-features-summary)
|
||||
6. [Deployment and Configuration Overview](#6-deployment-and-configuration-overview)
|
||||
@@ -60,6 +61,21 @@ The `xiaozhi-esp32-server` system adopts a **distributed, multi-component collab
|
||||
* (Potential functionality) Monitoring system operation status, viewing logs, troubleshooting, etc.
|
||||
* Comprehensive interaction with all backend management functions provided by `manager-api`.
|
||||
|
||||
5. **`manager-mobile` (Mobile Management Console - uni-app Implementation):**
|
||||
This is a cross-platform mobile management application based on uni-app v3 + Vue 3 + Vite, supporting App (Android & iOS) and WeChat Mini Program. Its main capabilities include:
|
||||
* Providing a convenient management interface on mobile devices, similar in functionality to manager-web but optimized for mobile platforms.
|
||||
* Supporting core functions such as user login, device management, and AI service configuration.
|
||||
* Cross-platform adaptation, allowing a single codebase to run on iOS, Android, and WeChat Mini Programs.
|
||||
* Integrating with `manager-api` through alova + `@alova/adapter-uniapp`.
|
||||
* Using pinia for state management to ensure data consistency.
|
||||
|
||||
6. **`digital-human` (Digital Human Test Module - Python+Web Implementation):**
|
||||
This is an independent digital human test module that provides a local test page, frontend interaction resources, a wake word runtime, and an event bridge for end-to-end debugging of the digital human interaction flow. Its main capabilities include:
|
||||
* Providing a browser-based digital human test page for validating audio playback, audio reception, and interaction flow.
|
||||
* Integrating a local wake word runtime with keyword detection based on Sherpa-ONNX.
|
||||
* Bridging page state and the local runtime through an event bridge to simplify wake word debugging.
|
||||
* Existing as a standalone module alongside `xiaozhi-server`, `manager-web`, and `manager-api`, making independent development and deployment easier.
|
||||
|
||||
**High-Level Interaction Flow Overview:**
|
||||
|
||||
* **Voice Interaction Main Line:** After the **ESP32 device** captures user voice, it transmits audio data in real-time to **`xiaozhi-server`** through **WebSocket**. After `xiaozhi-server` completes a series of AI processing (VAD, ASR, LLM interaction, TTS), it sends the synthesized voice response back to the ESP32 device for playback through WebSocket. All real-time interactions directly related to voice are completed in this link.
|
||||
@@ -73,7 +89,8 @@ xiaozhi-esp32-server
|
||||
├─ xiaozhi-server Port 8000 Python development Responsible for ESP32 communication
|
||||
├─ manager-web Port 8001 Node.js+Vue development Responsible for providing web interface for console
|
||||
├─ manager-api Port 8002 Java development Responsible for providing console API
|
||||
└─ manager-mobile Cross-platform mobile application uni-app+Vue3 development Responsible for providing mobile console management
|
||||
├─ manager-mobile Cross-platform mobile application uni-app+Vue3 development Responsible for providing mobile console management
|
||||
└─ digital-human Digital human test module Python+Web implementation Responsible for local test page, wake word runtime, and event bridge
|
||||
```
|
||||
|
||||
---
|
||||
@@ -257,6 +274,46 @@ The `manager-mobile` component is a cross-platform mobile management application
|
||||
|
||||
`manager-mobile` provides users with a fully functional, smooth mobile management tool through the application of these technologies, allowing administrators to perform system management and configuration anytime, anywhere.
|
||||
|
||||
### 3.5. `digital-human` (Digital Human Test Module - Python+Web Implementation)
|
||||
|
||||
The `digital-human` component is an independent digital human test module responsible for providing a local test page, page resources, a wake word runtime, and an event bridge. It is primarily used to debug the complete digital human interaction flow and to validate frontend interaction, audio capability, and local wake word behavior.
|
||||
|
||||
* **Core Objectives:**
|
||||
* Provide an independent local test page for validating digital human interaction effects.
|
||||
* Provide a local wake word runtime for keyword detection and event reporting.
|
||||
* Provide an event bridge between the frontend page and the local runtime to connect the test flow.
|
||||
* Exist as a standalone module for easier independent development, debugging, and deployment.
|
||||
|
||||
* **Core Technologies:**
|
||||
* **Python 3:** Used for the local test runtime, service startup, wake word lifecycle management, and event bridge coordination.
|
||||
* **Native HTML/CSS/JavaScript:** Used to build the digital human test page and interaction logic.
|
||||
* **Sherpa-ONNX:** Used for local wake word keyword detection.
|
||||
* **ThreadingHTTPServer / WebSocket bridge:** Used to serve the local page, health check endpoint, and event communication.
|
||||
|
||||
* **Key Implementation Aspects:**
|
||||
|
||||
1. **Module Entry and Page Resources:**
|
||||
* `start.py` is the module entry point responsible for starting the local test runtime.
|
||||
* `index.html` is the entry page for digital human testing.
|
||||
* The `js/`, `css/`, `images/`, and `resources/` directories provide frontend scripts, styles, and static resources.
|
||||
|
||||
2. **Local Test Runtime:**
|
||||
* The `wakeword_runtime/` directory contains the local wake word runtime implementation.
|
||||
* The runtime is responsible for configuration loading, logging initialization, audio capture, keyword detection, and lifecycle management.
|
||||
* The module exposes a local page URL, event bridge URL, and health check endpoint through a local HTTP service.
|
||||
|
||||
3. **Wake Word Detection Flow:**
|
||||
* Uses Sherpa-ONNX keyword detection models to support local wake word triggering.
|
||||
* Model files and keyword configurations are stored in `wakeword_runtime/models/` and related configuration files.
|
||||
* The page can observe runtime state and trigger results through the event bridge.
|
||||
|
||||
4. **Debugging and Integration Positioning:**
|
||||
* After startup, developers can open the local page in a browser to validate playback, reception, and interaction logic.
|
||||
* The event bridge synchronizes wake word state to the page, which helps with local debugging.
|
||||
* Detailed model download, runtime configuration, and usage instructions are documented in `docs/digital-human-wakeword.md`.
|
||||
|
||||
The `digital-human` module allows digital human related capabilities to be validated independently from the main service, reducing the complexity of page debugging, wake word integration, and local environment setup.
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Flow and Interaction Mechanisms
|
||||
|
||||
@@ -4,8 +4,8 @@
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>小智服务器测试页面</title>
|
||||
<link rel="stylesheet" href="css/test_page.css?v=0206">
|
||||
<title>小智数字人页面</title>
|
||||
<link rel="stylesheet" href="css/index.css?v=0206">
|
||||
<script>
|
||||
// 检测是否使用file://协议打开
|
||||
if (window.location.protocol === 'file:') {
|
||||
@@ -21,11 +21,11 @@
|
||||
warningDiv.innerHTML = `
|
||||
<h2>⚠️ 警告：请使用HTTP服务器打开此页面</h2>
|
||||
<p>您当前使用的是本地文件方式打开页面（file://协议），这可能导致页面功能异常。</p>
|
||||
<p>您可以使用nginx映射启动测试页面，也可以请按照以下步骤启动测试运行时：</p>
|
||||
<p>您可以使用nginx映射启动小智数字人页面，也可以请按照以下步骤启动测试运行时：</p>
|
||||
<ol>
|
||||
<li>打开命令行终端</li>
|
||||
<li>命令行进入到 digital-human 目录</li>
|
||||
<li>执行以下命令启动测试页面运行时：</li>
|
||||
<li>执行以下命令启动小智数字人页面运行时：</li>
|
||||
</ol>
|
||||
<pre>python start.py</pre>
|
||||
<p>然后在浏览器中访问：<strong>http://localhost:8006/index.html</strong></p>
|
||||
|
||||
Reference in New Issue
Block a user