From 2aa1a8d590a105bc5693ef09ce3c5122a38258cf Mon Sep 17 00:00:00 2001 From: 3030332422 <3030332422@qq.com> Date: Tue, 12 May 2026 10:24:19 +0800 Subject: [PATCH] =?UTF-8?q?update=EF=BC=9A=E8=A1=A5=E5=85=85?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- main/README_en.md | 59 ++++++++++++++++++- .../css/{test_page.css => index.css} | 0 main/digital-human/index.html | 8 +-- 3 files changed, 62 insertions(+), 5 deletions(-) rename main/digital-human/css/{test_page.css => index.css} (100%) diff --git a/main/README_en.md b/main/README_en.md index 281eafd6..9b0a1fdf 100644 --- a/main/README_en.md +++ b/main/README_en.md @@ -9,6 +9,7 @@ * [3.2. `manager-api` (Management Backend - Java Spring Boot Implementation)](#32-manager-api-management-backend---java-spring-boot-implementation) * [3.3. `manager-web` (Web Management Frontend - Vue.js Implementation)](#33-manager-web-web-management-frontend---vuejs-implementation) * [3.4. `manager-mobile` (Mobile Management Console - uni-app Implementation)](#34-manager-mobile-mobile-management-console---uniapp-implementation) + * [3.5. `digital-human` (Digital Human Test Module - Python+Web Implementation)](#35-digital-human-digital-human-test-module---pythonweb-implementation) 4. [Data Flow and Interaction Mechanisms](#4-data-flow-and-interaction-mechanisms) 5. [Key Features Summary](#5-key-features-summary) 6. [Deployment and Configuration Overview](#6-deployment-and-configuration-overview) @@ -60,6 +61,21 @@ The `xiaozhi-esp32-server` system adopts a **distributed, multi-component collab * (Potential functionality) Monitoring system operation status, viewing logs, troubleshooting, etc. * Comprehensive interaction with all backend management functions provided by `manager-api`. +5. **`manager-mobile` (Mobile Management Console - uni-app Implementation):** + This is a cross-platform mobile management application based on uni-app v3 + Vue 3 + Vite, supporting App (Android & iOS) and WeChat Mini Program. Its main capabilities include: + * Providing a convenient management interface on mobile devices, similar in functionality to manager-web but optimized for mobile platforms. + * Supporting core functions such as user login, device management, and AI service configuration. + * Cross-platform adaptation, allowing a single codebase to run on iOS, Android, and WeChat Mini Programs. + * Integrating with `manager-api` through alova + `@alova/adapter-uniapp`. + * Using pinia for state management to ensure data consistency. + +6. **`digital-human` (Digital Human Test Module - Python+Web Implementation):** + This is an independent digital human test module that provides a local test page, frontend interaction resources, a wake word runtime, and an event bridge for end-to-end debugging of the digital human interaction flow. Its main capabilities include: + * Providing a browser-based digital human test page for validating audio playback, audio reception, and interaction flow. + * Integrating a local wake word runtime with keyword detection based on Sherpa-ONNX. + * Bridging page state and the local runtime through an event bridge to simplify wake word debugging. + * Existing as a standalone module alongside `xiaozhi-server`, `manager-web`, and `manager-api`, making independent development and deployment easier. + **High-Level Interaction Flow Overview:** * **Voice Interaction Main Line:** After the **ESP32 device** captures user voice, it transmits audio data in real-time to **`xiaozhi-server`** through **WebSocket**. After `xiaozhi-server` completes a series of AI processing (VAD, ASR, LLM interaction, TTS), it sends the synthesized voice response back to the ESP32 device for playback through WebSocket. All real-time interactions directly related to voice are completed in this link. @@ -73,7 +89,8 @@ xiaozhi-esp32-server ├─ xiaozhi-server Port 8000 Python development Responsible for ESP32 communication ├─ manager-web Port 8001 Node.js+Vue development Responsible for providing web interface for console ├─ manager-api Port 8002 Java development Responsible for providing console API - └─ manager-mobile Cross-platform mobile application uni-app+Vue3 development Responsible for providing mobile console management + ├─ manager-mobile Cross-platform mobile application uni-app+Vue3 development Responsible for providing mobile console management + └─ digital-human Digital human test module Python+Web implementation Responsible for local test page, wake word runtime, and event bridge ``` --- @@ -257,6 +274,46 @@ The `manager-mobile` component is a cross-platform mobile management application `manager-mobile` provides users with a fully functional, smooth mobile management tool through the application of these technologies, allowing administrators to perform system management and configuration anytime, anywhere. +### 3.5. `digital-human` (Digital Human Test Module - Python+Web Implementation) + +The `digital-human` component is an independent digital human test module responsible for providing a local test page, page resources, a wake word runtime, and an event bridge. It is primarily used to debug the complete digital human interaction flow and to validate frontend interaction, audio capability, and local wake word behavior. + +* **Core Objectives:** + * Provide an independent local test page for validating digital human interaction effects. + * Provide a local wake word runtime for keyword detection and event reporting. + * Provide an event bridge between the frontend page and the local runtime to connect the test flow. + * Exist as a standalone module for easier independent development, debugging, and deployment. + +* **Core Technologies:** + * **Python 3:** Used for the local test runtime, service startup, wake word lifecycle management, and event bridge coordination. + * **Native HTML/CSS/JavaScript:** Used to build the digital human test page and interaction logic. + * **Sherpa-ONNX:** Used for local wake word keyword detection. + * **ThreadingHTTPServer / WebSocket bridge:** Used to serve the local page, health check endpoint, and event communication. + +* **Key Implementation Aspects:** + + 1. **Module Entry and Page Resources:** + * `start.py` is the module entry point responsible for starting the local test runtime. + * `index.html` is the entry page for digital human testing. + * The `js/`, `css/`, `images/`, and `resources/` directories provide frontend scripts, styles, and static resources. + + 2. **Local Test Runtime:** + * The `wakeword_runtime/` directory contains the local wake word runtime implementation. + * The runtime is responsible for configuration loading, logging initialization, audio capture, keyword detection, and lifecycle management. + * The module exposes a local page URL, event bridge URL, and health check endpoint through a local HTTP service. + + 3. **Wake Word Detection Flow:** + * Uses Sherpa-ONNX keyword detection models to support local wake word triggering. + * Model files and keyword configurations are stored in `wakeword_runtime/models/` and related configuration files. + * The page can observe runtime state and trigger results through the event bridge. + + 4. **Debugging and Integration Positioning:** + * After startup, developers can open the local page in a browser to validate playback, reception, and interaction logic. + * The event bridge synchronizes wake word state to the page, which helps with local debugging. + * Detailed model download, runtime configuration, and usage instructions are documented in `docs/digital-human-wakeword.md`. + +The `digital-human` module allows digital human related capabilities to be validated independently from the main service, reducing the complexity of page debugging, wake word integration, and local environment setup. + --- ## 4. Data Flow and Interaction Mechanisms diff --git a/main/digital-human/css/test_page.css b/main/digital-human/css/index.css similarity index 100% rename from main/digital-human/css/test_page.css rename to main/digital-human/css/index.css diff --git a/main/digital-human/index.html b/main/digital-human/index.html index 1339ea16..3d844f32 100644 --- a/main/digital-human/index.html +++ b/main/digital-human/index.html @@ -4,8 +4,8 @@ - 小智服务器测试页面 - + 小智数字人页面 +