@@ -9,6 +9,7 @@
|
||||
* [3.2. `manager-api` (管理后端 - Java Spring Boot实现)](#32-manager-api-管理后端---java-spring-boot实现)
|
||||
* [3.3. `manager-web` (Web管理前端 - Vue.js实现)](#33-manager-web-web管理前端---vuejs实现)
|
||||
* [3.4. `manager-mobile` (移动管理端 - uni-app+Vue3实现)](#34-manager-mobile-移动管理端---uni-appvue3实现)
|
||||
* [3.5. `digital-human` (数字人测试模块 - Python+Web实现)](#35-digital-human-数字人测试模块---pythonweb实现)
|
||||
4. [数据流与交互机制](#4-数据流与交互机制)
|
||||
5. [核心功能概要](#5-核心功能概要)
|
||||
6. [部署与配置概述](#6-部署与配置概述)
|
||||
@@ -67,6 +68,13 @@
|
||||
* 基于alova + @alova/adapter-uniapp实现网络请求,与manager-api无缝集成。
|
||||
* 使用pinia进行状态管理,确保数据一致性。
|
||||
|
||||
6. **`digital-human` (数字人测试模块 - Python+Web实现):**
|
||||
这是一个独立的数字人测试模块,提供本地测试页面、前端交互资源、唤醒词运行时与事件桥能力,用于联调整个数字人交互链路。其主要能力包括:
|
||||
* 提供浏览器端的数字人测试页面,用于验证音频播放、接收和交互流程。
|
||||
* 集成本地唤醒词运行时,支持基于Sherpa-ONNX的关键词检测。
|
||||
* 通过事件桥将页面状态与本地运行时打通,便于调试唤醒词触发与交互效果。
|
||||
* 作为独立模块与`xiaozhi-server`、`manager-web`、`manager-api`并列,便于单独开发和部署。
|
||||
|
||||
**高层交互流程概述:**
|
||||
|
||||
* **语音交互主线:** **ESP32设备**捕捉到用户语音后,通过**WebSocket**将音频数据实时传输给**`xiaozhi-server`**。`xiaozhi-server`完成一系列AI处理(VAD、ASR、LLM交互、TTS)后,再通过WebSocket将合成的语音回复发送回ESP32设备进行播放。所有与语音直接相关的实时交互均在此链路完成。
|
||||
@@ -80,7 +88,8 @@ xiaozhi-esp32-server
|
||||
├─ xiaozhi-server 8000 端口 Python语言开发 负责与esp32通信
|
||||
├─ manager-web 8001 端口 Node.js+Vue开发 负责提供控制台的web界面
|
||||
├─ manager-api 8002 端口 Java语言开发 负责提供控制台的api
|
||||
└─ manager-mobile 跨平台移动应用 uni-app+Vue3开发 负责提供移动版智控台管理
|
||||
├─ manager-mobile 跨平台移动应用 uni-app+Vue3开发 负责提供移动版智控台管理
|
||||
└─ digital-human 数字人测试模块 Python+Web实现 负责本地测试页面、唤醒词运行时与事件桥
|
||||
```
|
||||
|
||||
---
|
||||
@@ -366,6 +375,48 @@ xiaozhi-esp32-server
|
||||
|
||||
---
|
||||
|
||||
### 3.5. `digital-human` (数字人测试模块 - Python+Web实现)
|
||||
|
||||
`digital-human` 组件是一个独立的数字人测试模块,负责提供本地测试页面、页面资源、唤醒词运行时和事件桥能力。它主要用于联调整个数字人交互链路,帮助开发者验证页面交互、音频能力和本地唤醒词流程。
|
||||
|
||||
* **核心目标:**
|
||||
* 提供独立的本地测试页面,用于验证数字人交互效果。
|
||||
* 提供本地唤醒词运行时,支持关键词检测与事件上报。
|
||||
* 为前端页面和本地运行时提供事件桥,打通测试链路。
|
||||
* 以独立模块形式存在,便于单独开发、调试和部署。
|
||||
|
||||
* **核心技术栈:**
|
||||
* **Python 3:** 作为测试运行时的主要语言,负责启动本地服务、管理唤醒词运行时和事件桥。
|
||||
* **原生HTML/CSS/JavaScript:** 用于构建数字人测试页面与交互逻辑。
|
||||
* **Sherpa-ONNX:** 用于本地唤醒词检测。
|
||||
* **ThreadingHTTPServer / WebSocket桥接:** 用于承载测试页面、健康检查和本地事件通信。
|
||||
|
||||
* **关键实现细节:**
|
||||
|
||||
1. **模块入口与页面资源:**
|
||||
* `start.py` 是模块启动入口,负责初始化测试运行时。
|
||||
* `index.html` 是数字人测试页面入口。
|
||||
* `js/`、`css/`、`images/`、`resources/` 目录提供页面脚本、样式和资源文件。
|
||||
|
||||
2. **本地测试运行时:**
|
||||
* `wakeword_runtime/` 目录承载本地唤醒词运行时实现。
|
||||
* 运行时负责配置加载、日志初始化、音频采集、关键词检测和服务生命周期管理。
|
||||
* 模块默认通过本地HTTP服务暴露页面地址、事件桥地址和健康检查接口。
|
||||
|
||||
3. **唤醒词检测链路:**
|
||||
* 基于Sherpa-ONNX关键词检测模型,支持本地唤醒词触发。
|
||||
* 模型文件和关键词配置位于 `wakeword_runtime/models/` 及相应配置文件中。
|
||||
* 页面侧可结合事件桥观察唤醒词服务状态和触发结果。
|
||||
|
||||
4. **调试与联调定位:**
|
||||
* 启动后可通过浏览器访问本地页面,验证音频播放、接收和交互逻辑。
|
||||
* 通过事件桥可以将唤醒词状态同步到页面,便于排查本地链路问题。
|
||||
* 详细的模型下载、运行时配置和使用说明已整理在 `docs/digital-human-wakeword.md` 中。
|
||||
|
||||
`digital-human` 的存在使得数字人相关能力可以脱离主服务独立验证,降低了页面调试、唤醒词联调和本地环境搭建的复杂度。
|
||||
|
||||
---
|
||||
|
||||
## 4. 数据流与交互机制
|
||||
|
||||
`xiaozhi-esp32-server` 系统通过各组件间定义清晰的数据流和交互协议来协同工作。主要的通信方式依赖于针对实时交互优化的WebSocket协议和适用于客户端-服务器请求的RESTful API。
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
* [3.2. `manager-api` (Management Backend - Java Spring Boot Implementation)](#32-manager-api-management-backend---java-spring-boot-implementation)
|
||||
* [3.3. `manager-web` (Web Management Frontend - Vue.js Implementation)](#33-manager-web-web-management-frontend---vuejs-implementation)
|
||||
* [3.4. `manager-mobile` (Mobile Management Console - uni-app Implementation)](#34-manager-mobile-mobile-management-console---uniapp-implementation)
|
||||
* [3.5. `digital-human` (Digital Human Test Module - Python+Web Implementation)](#35-digital-human-digital-human-test-module---pythonweb-implementation)
|
||||
4. [Data Flow and Interaction Mechanisms](#4-data-flow-and-interaction-mechanisms)
|
||||
5. [Key Features Summary](#5-key-features-summary)
|
||||
6. [Deployment and Configuration Overview](#6-deployment-and-configuration-overview)
|
||||
@@ -60,6 +61,21 @@ The `xiaozhi-esp32-server` system adopts a **distributed, multi-component collab
|
||||
* (Potential functionality) Monitoring system operation status, viewing logs, troubleshooting, etc.
|
||||
* Comprehensive interaction with all backend management functions provided by `manager-api`.
|
||||
|
||||
5. **`manager-mobile` (Mobile Management Console - uni-app Implementation):**
|
||||
This is a cross-platform mobile management application based on uni-app v3 + Vue 3 + Vite, supporting App (Android & iOS) and WeChat Mini Program. Its main capabilities include:
|
||||
* Providing a convenient management interface on mobile devices, similar in functionality to manager-web but optimized for mobile platforms.
|
||||
* Supporting core functions such as user login, device management, and AI service configuration.
|
||||
* Cross-platform adaptation, allowing a single codebase to run on iOS, Android, and WeChat Mini Programs.
|
||||
* Integrating with `manager-api` through alova + `@alova/adapter-uniapp`.
|
||||
* Using pinia for state management to ensure data consistency.
|
||||
|
||||
6. **`digital-human` (Digital Human Test Module - Python+Web Implementation):**
|
||||
This is an independent digital human test module that provides a local test page, frontend interaction resources, a wake word runtime, and an event bridge for end-to-end debugging of the digital human interaction flow. Its main capabilities include:
|
||||
* Providing a browser-based digital human test page for validating audio playback, audio reception, and interaction flow.
|
||||
* Integrating a local wake word runtime with keyword detection based on Sherpa-ONNX.
|
||||
* Bridging page state and the local runtime through an event bridge to simplify wake word debugging.
|
||||
* Existing as a standalone module alongside `xiaozhi-server`, `manager-web`, and `manager-api`, making independent development and deployment easier.
|
||||
|
||||
**High-Level Interaction Flow Overview:**
|
||||
|
||||
* **Voice Interaction Main Line:** After the **ESP32 device** captures user voice, it transmits audio data in real-time to **`xiaozhi-server`** through **WebSocket**. After `xiaozhi-server` completes a series of AI processing (VAD, ASR, LLM interaction, TTS), it sends the synthesized voice response back to the ESP32 device for playback through WebSocket. All real-time interactions directly related to voice are completed in this link.
|
||||
@@ -73,7 +89,8 @@ xiaozhi-esp32-server
|
||||
├─ xiaozhi-server Port 8000 Python development Responsible for ESP32 communication
|
||||
├─ manager-web Port 8001 Node.js+Vue development Responsible for providing web interface for console
|
||||
├─ manager-api Port 8002 Java development Responsible for providing console API
|
||||
└─ manager-mobile Cross-platform mobile application uni-app+Vue3 development Responsible for providing mobile console management
|
||||
├─ manager-mobile Cross-platform mobile application uni-app+Vue3 development Responsible for providing mobile console management
|
||||
└─ digital-human Digital human test module Python+Web implementation Responsible for local test page, wake word runtime, and event bridge
|
||||
```
|
||||
|
||||
---
|
||||
@@ -257,6 +274,46 @@ The `manager-mobile` component is a cross-platform mobile management application
|
||||
|
||||
`manager-mobile` provides users with a fully functional, smooth mobile management tool through the application of these technologies, allowing administrators to perform system management and configuration anytime, anywhere.
|
||||
|
||||
### 3.5. `digital-human` (Digital Human Test Module - Python+Web Implementation)
|
||||
|
||||
The `digital-human` component is an independent digital human test module responsible for providing a local test page, page resources, a wake word runtime, and an event bridge. It is primarily used to debug the complete digital human interaction flow and to validate frontend interaction, audio capability, and local wake word behavior.
|
||||
|
||||
* **Core Objectives:**
|
||||
* Provide an independent local test page for validating digital human interaction effects.
|
||||
* Provide a local wake word runtime for keyword detection and event reporting.
|
||||
* Provide an event bridge between the frontend page and the local runtime to connect the test flow.
|
||||
* Exist as a standalone module for easier independent development, debugging, and deployment.
|
||||
|
||||
* **Core Technologies:**
|
||||
* **Python 3:** Used for the local test runtime, service startup, wake word lifecycle management, and event bridge coordination.
|
||||
* **Native HTML/CSS/JavaScript:** Used to build the digital human test page and interaction logic.
|
||||
* **Sherpa-ONNX:** Used for local wake word keyword detection.
|
||||
* **ThreadingHTTPServer / WebSocket bridge:** Used to serve the local page, health check endpoint, and event communication.
|
||||
|
||||
* **Key Implementation Aspects:**
|
||||
|
||||
1. **Module Entry and Page Resources:**
|
||||
* `start.py` is the module entry point responsible for starting the local test runtime.
|
||||
* `index.html` is the entry page for digital human testing.
|
||||
* The `js/`, `css/`, `images/`, and `resources/` directories provide frontend scripts, styles, and static resources.
|
||||
|
||||
2. **Local Test Runtime:**
|
||||
* The `wakeword_runtime/` directory contains the local wake word runtime implementation.
|
||||
* The runtime is responsible for configuration loading, logging initialization, audio capture, keyword detection, and lifecycle management.
|
||||
* The module exposes a local page URL, event bridge URL, and health check endpoint through a local HTTP service.
|
||||
|
||||
3. **Wake Word Detection Flow:**
|
||||
* Uses Sherpa-ONNX keyword detection models to support local wake word triggering.
|
||||
* Model files and keyword configurations are stored in `wakeword_runtime/models/` and related configuration files.
|
||||
* The page can observe runtime state and trigger results through the event bridge.
|
||||
|
||||
4. **Debugging and Integration Positioning:**
|
||||
* After startup, developers can open the local page in a browser to validate playback, reception, and interaction logic.
|
||||
* The event bridge synchronizes wake word state to the page, which helps with local debugging.
|
||||
* Detailed model download, runtime configuration, and usage instructions are documented in `docs/digital-human-wakeword.md`.
|
||||
|
||||
The `digital-human` module allows digital human related capabilities to be validated independently from the main service, reducing the complexity of page debugging, wake word integration, and local environment setup.
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Flow and Interaction Mechanisms
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
本文档是开发类文档,如需部署小智服务端,[点击这里查看部署教程](../../README.md#%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3)
|
||||
|
||||
如需查看一体机数字人落地部署、Kiosk 全屏启动和系统环境配置,[点击这里查看一体机部署指南](../../docs/all-in-one-digital-human-setup.md)
|
||||
|
||||
如需查看唤醒词模型下载、运行时配置和详细使用说明,[点击这里查看唤醒词专题文档](../../docs/digital-human-wakeword.md)
|
||||
|
||||
# 项目介绍
|
||||
|
||||
digital-human 是独立的数字人测试模块,负责提供本地测试页面、前端交互资源、唤醒词运行时和事件桥能力,用于联调整个数字人交互链路。
|
||||
|
||||
# 快速启动
|
||||
|
||||
安装依赖:
|
||||
|
||||
```bash
|
||||
pip install -r wakeword_runtime/requirements.txt
|
||||
```
|
||||
|
||||
启动模块:
|
||||
|
||||
```bash
|
||||
python start.py
|
||||
```
|
||||
|
||||
# 访问地址
|
||||
|
||||
启动后可访问:
|
||||
|
||||
- 页面地址:http://127.0.0.1:8006/index.html
|
||||
- 事件桥地址:ws://127.0.0.1:8006/wakeword-ws
|
||||
- 健康检查:http://127.0.0.1:8006/health
|
||||
|
||||
# 目录说明
|
||||
|
||||
- `start.py`:模块启动入口
|
||||
- `index.html`:数字人测试页面入口
|
||||
- `wakeword_runtime`:本地唤醒词运行时与配置目录
|
||||
- `js`、`css`:页面前端脚本与样式
|
||||
- `images`、`resources`:页面资源文件
|
||||
|
||||
# 相关文档
|
||||
|
||||
- 一体机部署指南:适用于 x86 设备整机落地部署、Kiosk 展示和开机自启动配置
|
||||
[../../docs/all-in-one-digital-human-setup.md](../../docs/all-in-one-digital-human-setup.md)
|
||||
- 唤醒词专题文档:适用于唤醒词模型下载、运行时配置和本地调试说明
|
||||
[../../docs/digital-human-wakeword.md](../../docs/digital-human-wakeword.md)
|
||||
|
Before Width: | Height: | Size: 185 B After Width: | Height: | Size: 185 B |
@@ -812,6 +812,20 @@ body {
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.config-item textarea {
|
||||
width: 100%;
|
||||
padding: 12px;
|
||||
border: 1px solid #40444b;
|
||||
border-radius: 6px;
|
||||
background: #40444b;
|
||||
color: white;
|
||||
min-height: 100px;
|
||||
resize: vertical;
|
||||
box-sizing: border-box;
|
||||
font-family: inherit;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.model-select {
|
||||
width: 100%;
|
||||
padding: 10px 40px 10px 14px;
|
||||
|
Before Width: | Height: | Size: 4.2 KiB After Width: | Height: | Size: 4.2 KiB |
|
Before Width: | Height: | Size: 136 KiB After Width: | Height: | Size: 136 KiB |
|
Before Width: | Height: | Size: 321 KiB After Width: | Height: | Size: 321 KiB |
|
Before Width: | Height: | Size: 290 KiB After Width: | Height: | Size: 290 KiB |
@@ -4,8 +4,8 @@
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>小智服务器测试页面</title>
|
||||
<link rel="stylesheet" href="css/test_page.css?v=0205">
|
||||
<title>小智数字人页面</title>
|
||||
<link rel="stylesheet" href="css/index.css?v=0206">
|
||||
<script>
|
||||
// 检测是否使用file://协议打开
|
||||
if (window.location.protocol === 'file:') {
|
||||
@@ -21,14 +21,14 @@
|
||||
warningDiv.innerHTML = `
|
||||
<h2>⚠️ 警告:请使用HTTP服务器打开此页面</h2>
|
||||
<p>您当前使用的是本地文件方式打开页面(file://协议),这可能导致页面功能异常。</p>
|
||||
<p>您可以使用nginx映射启动测试页面,也可以请按照以下步骤使用python启动测试http服务:</p>
|
||||
<p>您可以使用nginx映射启动小智数字人页面,也可以请按照以下步骤启动测试运行时:</p>
|
||||
<ol>
|
||||
<li>打开命令行终端</li>
|
||||
<li>命令行进入到 xiaozhi-server/test 目录</li>
|
||||
<li>执行以下命令启动HTTP服务器:</li>
|
||||
<li>命令行进入到 digital-human 目录</li>
|
||||
<li>执行以下命令启动小智数字人页面运行时:</li>
|
||||
</ol>
|
||||
<pre>python -m http.server 8006</pre>
|
||||
<p>然后在浏览器中访问:<strong>http://localhost:8006/test_page.html</strong></p>
|
||||
<pre>python start.py</pre>
|
||||
<p>然后在浏览器中访问:<strong>http://localhost:8006/index.html</strong></p>
|
||||
`;
|
||||
document.body.appendChild(warningDiv);
|
||||
});
|
||||
@@ -137,6 +137,7 @@
|
||||
<div class="modal-body">
|
||||
<div class="settings-tabs">
|
||||
<button class="tab-btn active" data-tab="device">设备配置</button>
|
||||
<button class="tab-btn" data-tab="wakeword">唤醒词</button>
|
||||
<button class="tab-btn" data-tab="mcp">MCP工具</button>
|
||||
<button class="tab-btn" data-tab="other">数字人皮肤</button>
|
||||
</div>
|
||||
@@ -162,6 +163,15 @@
|
||||
placeholder="deviceName">
|
||||
</div>
|
||||
</div>
|
||||
<div class="config-row">
|
||||
<div class="config-item">
|
||||
<label for="emojiEnabled">情绪表情:</label>
|
||||
<select id="emojiEnabled" class="model-select">
|
||||
<option value="true">启用</option>
|
||||
<option value="false">禁用</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -184,6 +194,37 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="wakewordTab">
|
||||
<div class="config-panel">
|
||||
<div class="control-panel">
|
||||
<div class="config-row">
|
||||
<div class="config-item">
|
||||
<label for="wakewordEnabled">启用本地唤醒词:</label>
|
||||
<select id="wakewordEnabled" class="model-select">
|
||||
<option value="true">启用</option>
|
||||
<option value="false">禁用</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<div class="config-row">
|
||||
<div class="config-item">
|
||||
<label for="wakewordWsUrl">唤醒词服务地址:</label>
|
||||
<input type="text" id="wakewordWsUrl" placeholder="ws://127.0.0.1:8006/wakeword-ws">
|
||||
</div>
|
||||
</div>
|
||||
<div class="config-row">
|
||||
<div class="config-item">
|
||||
<label for="wakewordList">唤醒词配置 (一行一个):</label>
|
||||
<textarea id="wakewordList" placeholder="例如: 小智小智 你好小智"></textarea>
|
||||
</div>
|
||||
</div>
|
||||
<div class="modal-actions">
|
||||
<button type="button" class="btn-primary" id="applyWakewordBtn">应用唤醒词</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="tab-content" id="mcpTab">
|
||||
<!-- MCP 工具管理区域 -->
|
||||
<div class="mcp-tools-container">
|
||||
@@ -3,6 +3,7 @@ import { checkOpusLoaded, initOpusEncoder } from './core/audio/opus-codec.js?v=0
|
||||
import { getAudioPlayer } from './core/audio/player.js?v=0205';
|
||||
import { checkMicrophoneAvailability, isHttpNonLocalhost } from './core/audio/recorder.js?v=0205';
|
||||
import { initMcpTools } from './core/mcp/tools.js?v=0205';
|
||||
import { startWakewordBridgeListener } from './core/network/wakeword-bridge.js?v=0205';
|
||||
import { uiController } from './ui/controller.js?v=0205';
|
||||
import { log } from './utils/logger.js?v=0205';
|
||||
|
||||
@@ -43,6 +44,8 @@ class App {
|
||||
await this.audioPlayer.start();
|
||||
// 初始化MCP工具
|
||||
initMcpTools();
|
||||
// 初始化本地唤醒事件监听
|
||||
startWakewordBridgeListener();
|
||||
// 检查麦克风可用性
|
||||
await this.checkMicrophoneAvailability();
|
||||
// 检查摄像头可用性
|
||||
@@ -1,5 +1,8 @@
|
||||
// 配置管理模块
|
||||
|
||||
// 默认唤醒词列表
|
||||
export const DEFAULT_WAKE_WORDS = '你好小智\n你好小志\n小爱同学\n你好小鑫\n你好小新\n小美同学\n小龙小龙\n喵喵同学\n小滨小滨\n小冰小冰\n嘿你好呀';
|
||||
|
||||
// 生成随机MAC地址
|
||||
function generateRandomMac() {
|
||||
const hexDigits = '0123456789ABCDEF';
|
||||
@@ -19,6 +22,9 @@ export function loadConfig() {
|
||||
const deviceNameInput = document.getElementById('deviceName');
|
||||
const clientIdInput = document.getElementById('clientId');
|
||||
const otaUrlInput = document.getElementById('otaUrl');
|
||||
const wakewordWsUrlInput = document.getElementById('wakewordWsUrl');
|
||||
const wakewordEnabledInput = document.getElementById('wakewordEnabled');
|
||||
const wakewordListInput = document.getElementById('wakewordList');
|
||||
|
||||
// 从localStorage加载MAC地址,如果没有则生成新的
|
||||
let savedMac = localStorage.getItem('xz_tester_deviceMac');
|
||||
@@ -43,6 +49,29 @@ export function loadConfig() {
|
||||
if (savedOtaUrl) {
|
||||
otaUrlInput.value = savedOtaUrl;
|
||||
}
|
||||
|
||||
const savedWakewordWsUrl = localStorage.getItem('xz_tester_wakewordWsUrl');
|
||||
if (savedWakewordWsUrl !== null && wakewordWsUrlInput) {
|
||||
wakewordWsUrlInput.value = savedWakewordWsUrl;
|
||||
}
|
||||
|
||||
const savedWakewordEnabled = localStorage.getItem('xz_tester_wakewordEnabled');
|
||||
if (savedWakewordEnabled !== null && wakewordEnabledInput) {
|
||||
wakewordEnabledInput.value = savedWakewordEnabled;
|
||||
}
|
||||
|
||||
const savedWakewordList = localStorage.getItem('xz_tester_wakewordList');
|
||||
if (savedWakewordList !== null && wakewordListInput) {
|
||||
wakewordListInput.value = savedWakewordList;
|
||||
} else if (wakewordListInput) {
|
||||
wakewordListInput.value = DEFAULT_WAKE_WORDS;
|
||||
}
|
||||
|
||||
const emojiEnabledInput = document.getElementById('emojiEnabled');
|
||||
const savedEmojiEnabled = localStorage.getItem('xz_tester_emojiEnabled');
|
||||
if (savedEmojiEnabled !== null && emojiEnabledInput) {
|
||||
emojiEnabledInput.value = savedEmojiEnabled;
|
||||
}
|
||||
}
|
||||
|
||||
// 保存配置
|
||||
@@ -50,10 +79,26 @@ export function saveConfig() {
|
||||
const deviceMacInput = document.getElementById('deviceMac');
|
||||
const deviceNameInput = document.getElementById('deviceName');
|
||||
const clientIdInput = document.getElementById('clientId');
|
||||
const wakewordWsUrlInput = document.getElementById('wakewordWsUrl');
|
||||
const wakewordEnabledInput = document.getElementById('wakewordEnabled');
|
||||
const wakewordListInput = document.getElementById('wakewordList');
|
||||
|
||||
localStorage.setItem('xz_tester_deviceMac', deviceMacInput.value);
|
||||
localStorage.setItem('xz_tester_deviceName', deviceNameInput.value);
|
||||
localStorage.setItem('xz_tester_clientId', clientIdInput.value);
|
||||
const emojiEnabledInput = document.getElementById('emojiEnabled');
|
||||
if (emojiEnabledInput) {
|
||||
localStorage.setItem('xz_tester_emojiEnabled', emojiEnabledInput.value);
|
||||
}
|
||||
if (wakewordEnabledInput) {
|
||||
localStorage.setItem('xz_tester_wakewordEnabled', wakewordEnabledInput.value);
|
||||
}
|
||||
if (wakewordListInput) {
|
||||
localStorage.setItem('xz_tester_wakewordList', wakewordListInput.value);
|
||||
}
|
||||
if (wakewordWsUrlInput && wakewordWsUrlInput.value.trim()) {
|
||||
localStorage.setItem('xz_tester_wakewordWsUrl', wakewordWsUrlInput.value.trim());
|
||||
}
|
||||
}
|
||||
|
||||
// 获取配置值
|
||||
@@ -62,12 +107,14 @@ export function getConfig() {
|
||||
const deviceMac = document.getElementById('deviceMac')?.value.trim() || '';
|
||||
const deviceName = document.getElementById('deviceName')?.value.trim() || '';
|
||||
const clientId = document.getElementById('clientId')?.value.trim() || '';
|
||||
const emojiEnabled = document.getElementById('emojiEnabled')?.value !== 'false';
|
||||
|
||||
return {
|
||||
deviceId: deviceMac, // 使用MAC地址作为deviceId
|
||||
deviceName,
|
||||
deviceMac,
|
||||
clientId
|
||||
clientId,
|
||||
emojiEnabled
|
||||
};
|
||||
}
|
||||
|
||||
@@ -203,17 +203,6 @@ export class AudioRecorder {
|
||||
async start() {
|
||||
if (this.isRecording) return false;
|
||||
try {
|
||||
// Check if WebSocketHandler instance exists
|
||||
const { getWebSocketHandler } = await import('../network/websocket.js?v=0205');
|
||||
const wsHandler = getWebSocketHandler();
|
||||
// If machine is speaking, send abort message
|
||||
if (wsHandler && wsHandler.isRemoteSpeaking && wsHandler.currentSessionId) {
|
||||
const abortMessage = { session_id: wsHandler.currentSessionId, type: 'abort', reason: 'wake_word_detected' };
|
||||
if (this.websocket && this.websocket.readyState === WebSocket.OPEN) {
|
||||
this.websocket.send(JSON.stringify(abortMessage));
|
||||
log('已发送中止消息', 'info');
|
||||
}
|
||||
}
|
||||
if (!this.initEncoder()) {
|
||||
log('无法开始录音: Opus编码器初始化失败', 'error');
|
||||
return false;
|
||||
@@ -0,0 +1,225 @@
|
||||
import { uiController } from '../../ui/controller.js?v=0205';
|
||||
import { log } from '../../utils/logger.js?v=0205';
|
||||
|
||||
let wakewordSocket = null;
|
||||
let reconnectTimer = null;
|
||||
let reconnectAttempts = 0;
|
||||
let shouldReconnect = true;
|
||||
let wakewordRequestSeq = 0;
|
||||
let onNextBridgeConnectedCallback = null;
|
||||
|
||||
const pendingWakewordRequests = new Map();
|
||||
|
||||
export function startWakewordBridgeListener() {
|
||||
if (wakewordSocket) {
|
||||
return wakewordSocket;
|
||||
}
|
||||
|
||||
shouldReconnect = true;
|
||||
log('正在连接本地唤醒事件桥...', 'info');
|
||||
tryConnect();
|
||||
return wakewordSocket;
|
||||
}
|
||||
|
||||
function tryConnect() {
|
||||
const bridgeUrl = buildWakewordBridgeUrl();
|
||||
|
||||
try {
|
||||
wakewordSocket = new WebSocket(bridgeUrl);
|
||||
wakewordSocket.onopen = () => {
|
||||
reconnectAttempts = 0;
|
||||
log(`本地唤醒事件桥已连接: ${bridgeUrl}`, 'success');
|
||||
// 连接成功后自动保存地址,刷新后仍能记住
|
||||
localStorage.setItem('xz_tester_wakewordWsUrl', bridgeUrl);
|
||||
const urlInput = document.getElementById('wakewordWsUrl');
|
||||
if (urlInput) urlInput.value = bridgeUrl;
|
||||
};
|
||||
|
||||
wakewordSocket.onerror = () => {
|
||||
log(`本地唤醒事件桥连接失败: ${bridgeUrl}`, 'error');
|
||||
};
|
||||
|
||||
wakewordSocket.onmessage = async (event) => {
|
||||
try {
|
||||
const message = parseWakewordBridgeMessage(event.data);
|
||||
if (message.requestId && pendingWakewordRequests.has(message.requestId)) {
|
||||
settleWakewordRequest(message);
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.success === false) {
|
||||
log(`本地唤醒事件桥返回错误: ${message.error || '未知错误'}`, 'error');
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'bridge_connected') {
|
||||
log('本地唤醒监听已就绪', 'info');
|
||||
if (onNextBridgeConnectedCallback) {
|
||||
const cb = onNextBridgeConnectedCallback;
|
||||
onNextBridgeConnectedCallback = null;
|
||||
cb(message);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'service_ready') {
|
||||
log('本地唤醒服务已启动', 'info');
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'wakeword_config') {
|
||||
uiController.applyWakewordConfig(message.payload || {});
|
||||
log('已同步本地唤醒词配置', 'info');
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'service_stopping') {
|
||||
log('本地唤醒服务正在停止', 'warning');
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'wake_word_detected') {
|
||||
const wakeWord = message.payload?.wake_word || '唤醒词';
|
||||
log(`检测到本地唤醒事件: ${wakeWord}`, 'info');
|
||||
await uiController.triggerWakewordDial(wakeWord);
|
||||
}
|
||||
} catch (error) {
|
||||
log(`解析本地唤醒事件失败: ${error.message}`, 'error');
|
||||
}
|
||||
};
|
||||
|
||||
wakewordSocket.onclose = () => {
|
||||
if (wakewordSocket) {
|
||||
wakewordSocket = null;
|
||||
}
|
||||
|
||||
rejectAllWakewordRequests('本地唤醒事件桥已断开');
|
||||
|
||||
if (!shouldReconnect) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (reconnectTimer) {
|
||||
return;
|
||||
}
|
||||
|
||||
reconnectAttempts += 1;
|
||||
const delay = Math.min(1000 * reconnectAttempts, 5000);
|
||||
log(`本地唤醒事件桥将在 ${delay}ms 后重连: ${bridgeUrl}`, 'warning');
|
||||
reconnectTimer = window.setTimeout(() => {
|
||||
reconnectTimer = null;
|
||||
tryConnect();
|
||||
}, delay);
|
||||
};
|
||||
|
||||
return wakewordSocket;
|
||||
} catch (error) {
|
||||
log(`启动本地唤醒监听失败: ${error.message}`, 'error');
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export function stopWakewordBridgeListener() {
|
||||
shouldReconnect = false;
|
||||
|
||||
if (reconnectTimer) {
|
||||
window.clearTimeout(reconnectTimer);
|
||||
reconnectTimer = null;
|
||||
}
|
||||
|
||||
if (!wakewordSocket) {
|
||||
return;
|
||||
}
|
||||
|
||||
wakewordSocket.onclose = null;
|
||||
wakewordSocket.close();
|
||||
wakewordSocket = null;
|
||||
}
|
||||
|
||||
export function sendWakewordBridgeMessage(type, payload = {}, requestId = null) {
|
||||
if (!wakewordSocket || wakewordSocket.readyState !== WebSocket.OPEN) {
|
||||
log('本地唤醒事件桥未连接,无法发送消息', 'warning');
|
||||
return false;
|
||||
}
|
||||
|
||||
wakewordSocket.send(JSON.stringify({
|
||||
type,
|
||||
requestId,
|
||||
payload,
|
||||
}));
|
||||
return true;
|
||||
}
|
||||
|
||||
export function requestWakewordBridge(type, payload = {}, timeout = 5000) {
|
||||
const requestId = `wakeword-${Date.now()}-${++wakewordRequestSeq}`;
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
const timer = window.setTimeout(() => {
|
||||
pendingWakewordRequests.delete(requestId);
|
||||
reject(new Error('本地唤醒服务响应超时'));
|
||||
}, timeout);
|
||||
|
||||
pendingWakewordRequests.set(requestId, { resolve, reject, timer });
|
||||
|
||||
if (!sendWakewordBridgeMessage(type, payload, requestId)) {
|
||||
window.clearTimeout(timer);
|
||||
pendingWakewordRequests.delete(requestId);
|
||||
reject(new Error('本地唤醒事件桥未连接'));
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
export function getWakewordBridgeUrl() {
|
||||
if (wakewordSocket && wakewordSocket.url) {
|
||||
return wakewordSocket.url;
|
||||
}
|
||||
return buildWakewordBridgeUrl();
|
||||
}
|
||||
|
||||
export function onNextBridgeConnected(callback) {
|
||||
onNextBridgeConnectedCallback = callback;
|
||||
}
|
||||
|
||||
function buildWakewordBridgeUrl() {
|
||||
const configured = localStorage.getItem('xz_tester_wakewordWsUrl');
|
||||
if (configured && configured.trim()) {
|
||||
return configured.trim();
|
||||
}
|
||||
return 'ws://127.0.0.1:8006/wakeword-ws';
|
||||
}
|
||||
|
||||
function parseWakewordBridgeMessage(rawData) {
|
||||
const message = JSON.parse(rawData);
|
||||
return {
|
||||
type: message.type || '',
|
||||
requestId: message.requestId || null,
|
||||
success: message.success !== false,
|
||||
payload: message.payload || {},
|
||||
error: message.error || null,
|
||||
};
|
||||
}
|
||||
|
||||
function settleWakewordRequest(message) {
|
||||
const pendingRequest = pendingWakewordRequests.get(message.requestId);
|
||||
if (!pendingRequest) {
|
||||
return;
|
||||
}
|
||||
|
||||
window.clearTimeout(pendingRequest.timer);
|
||||
pendingWakewordRequests.delete(message.requestId);
|
||||
|
||||
if (message.success === false) {
|
||||
pendingRequest.reject(new Error(message.error || '本地唤醒服务返回失败'));
|
||||
return;
|
||||
}
|
||||
|
||||
pendingRequest.resolve(message);
|
||||
}
|
||||
|
||||
function rejectAllWakewordRequests(errorMessage) {
|
||||
pendingWakewordRequests.forEach((pendingRequest) => {
|
||||
window.clearTimeout(pendingRequest.timer);
|
||||
pendingRequest.reject(new Error(errorMessage));
|
||||
});
|
||||
pendingWakewordRequests.clear();
|
||||
}
|
||||
@@ -34,7 +34,8 @@ export class WebSocketHandler {
|
||||
device_mac: config.deviceMac,
|
||||
token: config.token,
|
||||
features: {
|
||||
mcp: true
|
||||
mcp: true,
|
||||
emoji: config.emojiEnabled
|
||||
}
|
||||
};
|
||||
|
||||
@@ -70,6 +71,28 @@ export class WebSocketHandler {
|
||||
}
|
||||
}
|
||||
|
||||
_sendWakeupMessages(sessionId) {
|
||||
if (!this.websocket || this.websocket.readyState !== WebSocket.OPEN) return;
|
||||
|
||||
// listen detect
|
||||
this.websocket.send(JSON.stringify({
|
||||
session_id: sessionId,
|
||||
type: 'listen',
|
||||
state: 'detect',
|
||||
text: '嘿,你好呀'
|
||||
}));
|
||||
log('发送listen detect消息,唤醒词: 嘿,你好呀', 'info');
|
||||
|
||||
// listen start:开始监听
|
||||
this.websocket.send(JSON.stringify({
|
||||
session_id: sessionId,
|
||||
type: 'listen',
|
||||
state: 'start',
|
||||
mode: 'auto'
|
||||
}));
|
||||
log('发送listen start消息', 'info');
|
||||
}
|
||||
|
||||
// 处理文本消息
|
||||
handleTextMessage(message) {
|
||||
if (message.type === 'hello') {
|
||||
@@ -77,6 +100,9 @@ export class WebSocketHandler {
|
||||
window.cameraAvailable = true;
|
||||
log('连接成功,摄像头已可用', 'success');
|
||||
uiController.updateDialButton(true);
|
||||
|
||||
this._sendWakeupMessages(message.session_id);
|
||||
|
||||
uiController.startAIChatSession();
|
||||
} else if (message.type === 'tts') {
|
||||
this.handleTTSMessage(message);
|
||||
@@ -2,7 +2,9 @@
|
||||
import { loadConfig, saveConfig } from '../config/manager.js?v=0205';
|
||||
import { getAudioPlayer } from '../core/audio/player.js?v=0205';
|
||||
import { getAudioRecorder } from '../core/audio/recorder.js?v=0205';
|
||||
import { requestWakewordBridge, stopWakewordBridgeListener, startWakewordBridgeListener, getWakewordBridgeUrl, onNextBridgeConnected } from '../core/network/wakeword-bridge.js?v=0205';
|
||||
import { getWebSocketHandler } from '../core/network/websocket.js?v=0205';
|
||||
import { log } from '../utils/logger.js?v=0205';
|
||||
|
||||
// UI controller class
|
||||
class UIController {
|
||||
@@ -14,6 +16,8 @@ class UIController {
|
||||
this.currentBackgroundIndex = localStorage.getItem('backgroundIndex') ? parseInt(localStorage.getItem('backgroundIndex')) : 0;
|
||||
this.backgroundImages = ['1.png', '2.png', '3.png'];
|
||||
this.dialBtnDisabled = false;
|
||||
this.isConnecting = false;
|
||||
this.lastWakewordDialTime = 0;
|
||||
|
||||
// Bind methods
|
||||
this.init = this.init.bind(this);
|
||||
@@ -25,6 +29,9 @@ class UIController {
|
||||
this.showModal = this.showModal.bind(this);
|
||||
this.hideModal = this.hideModal.bind(this);
|
||||
this.switchTab = this.switchTab.bind(this);
|
||||
this.applyWakewordConfig = this.applyWakewordConfig.bind(this);
|
||||
this.handleApplyWakeword = this.handleApplyWakeword.bind(this);
|
||||
this.triggerWakewordDial = this.triggerWakewordDial.bind(this);
|
||||
}
|
||||
|
||||
// Initialize
|
||||
@@ -262,6 +269,11 @@ class UIController {
|
||||
});
|
||||
});
|
||||
|
||||
const applyWakewordBtn = document.getElementById('applyWakewordBtn');
|
||||
if (applyWakewordBtn) {
|
||||
applyWakewordBtn.addEventListener('click', this.handleApplyWakeword);
|
||||
}
|
||||
|
||||
// 点击模态框背景关闭(仅对特定模态框禁用此功能)
|
||||
const modals = document.querySelectorAll('.modal');
|
||||
modals.forEach(modal => {
|
||||
@@ -512,6 +524,121 @@ class UIController {
|
||||
}
|
||||
}
|
||||
|
||||
applyWakewordConfig(config = {}) {
|
||||
const wakewordEnabledInput = document.getElementById('wakewordEnabled');
|
||||
const wakewordListInput = document.getElementById('wakewordList');
|
||||
|
||||
if (!wakewordEnabledInput || !wakewordListInput) {
|
||||
return;
|
||||
}
|
||||
|
||||
const wakeWords = Array.isArray(config.wakeWords)
|
||||
? config.wakeWords.filter(item => typeof item === 'string' && item.trim())
|
||||
: [];
|
||||
|
||||
wakewordEnabledInput.value = config.enabled === false ? 'false' : 'true';
|
||||
wakewordListInput.value = wakeWords.join('\n');
|
||||
saveConfig();
|
||||
}
|
||||
|
||||
async handleApplyWakeword() {
|
||||
const wakewordEnabledInput = document.getElementById('wakewordEnabled');
|
||||
const wakewordListInput = document.getElementById('wakewordList');
|
||||
const wakewordWsUrlInput = document.getElementById('wakewordWsUrl');
|
||||
if (!wakewordEnabledInput || !wakewordListInput) {
|
||||
return;
|
||||
}
|
||||
|
||||
const wakeWords = wakewordListInput.value
|
||||
.split(/\r?\n/u)
|
||||
.map(item => item.trim())
|
||||
.filter(Boolean)
|
||||
.filter((item, index, items) => items.indexOf(item) === index);
|
||||
|
||||
const payload = {
|
||||
enabled: wakewordEnabledInput.value !== 'false',
|
||||
wakeWords,
|
||||
};
|
||||
|
||||
if (payload.enabled && payload.wakeWords.length === 0) {
|
||||
this.addChatMessage('启用唤醒词时,至少需要填写一个唤醒词。', false);
|
||||
return;
|
||||
}
|
||||
|
||||
const applyWakewordBtn = document.getElementById('applyWakewordBtn');
|
||||
if (applyWakewordBtn) {
|
||||
applyWakewordBtn.disabled = true;
|
||||
applyWakewordBtn.textContent = '应用中...';
|
||||
}
|
||||
|
||||
try {
|
||||
// 保存地址到 localStorage
|
||||
if (wakewordWsUrlInput && wakewordWsUrlInput.value.trim()) {
|
||||
localStorage.setItem('xz_tester_wakewordWsUrl', wakewordWsUrlInput.value.trim());
|
||||
}
|
||||
|
||||
// 比较新地址和当前连接地址
|
||||
const newWsUrl = localStorage.getItem('xz_tester_wakewordWsUrl');
|
||||
const currentWsUrl = getWakewordBridgeUrl();
|
||||
const urlChanged = newWsUrl !== currentWsUrl;
|
||||
|
||||
if (urlChanged) {
|
||||
// 地址变了,先确认
|
||||
const shouldRestart = window.confirm('地址已变更,是否继续?(将断开旧连接并重新连接)');
|
||||
if (!shouldRestart) {
|
||||
// 恢复 localStorage 里的旧地址
|
||||
localStorage.setItem('xz_tester_wakewordWsUrl', currentWsUrl);
|
||||
this.addChatMessage('已取消地址变更。', false);
|
||||
return;
|
||||
}
|
||||
|
||||
// 断开旧连接
|
||||
stopWakewordBridgeListener();
|
||||
|
||||
// 启动新连接(自动用新地址)
|
||||
startWakewordBridgeListener();
|
||||
|
||||
// 等 bridge_connected
|
||||
await new Promise((resolve, reject) => {
|
||||
const timeout = setTimeout(() => {
|
||||
reject(new Error('连接新服务器超时'));
|
||||
}, 5000);
|
||||
|
||||
onNextBridgeConnected(() => {
|
||||
clearTimeout(timeout);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
|
||||
// 发配置和重启
|
||||
await requestWakewordBridge('set_wakeword_config', payload);
|
||||
this.applyWakewordConfig(payload);
|
||||
await requestWakewordBridge('restart_wakeword_service');
|
||||
this.addChatMessage('唤醒词配置已保存,唤醒词服务正在重启。', false);
|
||||
} else {
|
||||
// 地址没变:直接在当前连接操作
|
||||
const response = await requestWakewordBridge('set_wakeword_config', payload);
|
||||
this.applyWakewordConfig(response.payload || payload);
|
||||
|
||||
const shouldRestart = window.confirm('唤醒词已保存。是否现在重启唤醒词服务以立即生效?');
|
||||
if (!shouldRestart) {
|
||||
this.addChatMessage('唤醒词配置已保存,可稍后手动重启服务后生效。', false);
|
||||
return;
|
||||
}
|
||||
|
||||
await requestWakewordBridge('restart_wakeword_service');
|
||||
this.addChatMessage('唤醒词配置已保存,唤醒词服务正在重启。', false);
|
||||
}
|
||||
} catch (error) {
|
||||
this.addChatMessage(`应用唤醒词失败: ${error.message}`, false);
|
||||
} finally {
|
||||
if (applyWakewordBtn) {
|
||||
applyWakewordBtn.disabled = false;
|
||||
applyWakewordBtn.textContent = '应用唤醒词';
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Start AI chat session after connection
|
||||
startAIChatSession() {
|
||||
this.addChatMessage('连接成功,开始聊天吧~😊', false);
|
||||
@@ -550,47 +677,51 @@ class UIController {
|
||||
|
||||
// Handle connect button click
|
||||
async handleConnect() {
|
||||
console.log('handleConnect called');
|
||||
|
||||
// Switch to device settings tab
|
||||
this.switchTab('device');
|
||||
|
||||
// Wait for DOM update
|
||||
await new Promise(resolve => setTimeout(resolve, 50));
|
||||
|
||||
const otaUrlInput = document.getElementById('otaUrl');
|
||||
|
||||
console.log('otaUrl element:', otaUrlInput);
|
||||
|
||||
if (!otaUrlInput || !otaUrlInput.value) {
|
||||
this.addChatMessage('请输入OTA服务器地址', false);
|
||||
const wsHandler = getWebSocketHandler();
|
||||
if (this.isConnecting || (wsHandler && wsHandler.isConnected())) {
|
||||
log('连接已存在或正在进行,忽略本次拨号请求', 'info');
|
||||
return;
|
||||
}
|
||||
|
||||
const otaUrl = otaUrlInput.value;
|
||||
console.log('otaUrl value:', otaUrl);
|
||||
|
||||
// Update dial button state to connecting
|
||||
const dialBtn = document.getElementById('dialBtn');
|
||||
if (dialBtn) {
|
||||
dialBtn.classList.add('dial-active');
|
||||
dialBtn.querySelector('.btn-text').textContent = '连接中...';
|
||||
dialBtn.disabled = true;
|
||||
}
|
||||
|
||||
// Show connecting message
|
||||
this.addChatMessage('正在连接服务器...', false);
|
||||
|
||||
const chatIpt = document.getElementById('chatIpt');
|
||||
if (chatIpt) {
|
||||
chatIpt.style.display = 'flex';
|
||||
}
|
||||
this.isConnecting = true;
|
||||
console.log('handleConnect called');
|
||||
|
||||
try {
|
||||
// Switch to device settings tab
|
||||
this.switchTab('device');
|
||||
|
||||
// Wait for DOM update
|
||||
await new Promise(resolve => setTimeout(resolve, 50));
|
||||
|
||||
const otaUrlInput = document.getElementById('otaUrl');
|
||||
|
||||
console.log('otaUrl element:', otaUrlInput);
|
||||
|
||||
if (!otaUrlInput || !otaUrlInput.value) {
|
||||
this.addChatMessage('请输入OTA服务器地址', false);
|
||||
return;
|
||||
}
|
||||
|
||||
const otaUrl = otaUrlInput.value;
|
||||
console.log('otaUrl value:', otaUrl);
|
||||
|
||||
// Update dial button state to connecting
|
||||
const dialBtn = document.getElementById('dialBtn');
|
||||
if (dialBtn) {
|
||||
dialBtn.classList.add('dial-active');
|
||||
dialBtn.querySelector('.btn-text').textContent = '连接中...';
|
||||
dialBtn.disabled = true;
|
||||
}
|
||||
|
||||
// Show connecting message
|
||||
this.addChatMessage('正在连接服务器...', false);
|
||||
|
||||
const chatIpt = document.getElementById('chatIpt');
|
||||
if (chatIpt) {
|
||||
chatIpt.style.display = 'flex';
|
||||
}
|
||||
|
||||
// Get WebSocket handler instance
|
||||
const wsHandler = getWebSocketHandler();
|
||||
|
||||
// Register connection state callback BEFORE connecting
|
||||
wsHandler.onConnectionStateChange = (isConnected) => {
|
||||
this.updateConnectionUI(isConnected);
|
||||
@@ -670,9 +801,36 @@ class UIController {
|
||||
dialBtn.classList.remove('dial-active');
|
||||
console.log('Dial button state restored successfully');
|
||||
}
|
||||
} finally {
|
||||
this.isConnecting = false;
|
||||
}
|
||||
}
|
||||
|
||||
async triggerWakewordDial(wakeWord = '唤醒词') {
|
||||
const wsHandler = getWebSocketHandler();
|
||||
const now = Date.now();
|
||||
|
||||
if (wsHandler && wsHandler.isConnected()) {
|
||||
log('页面已连接,忽略自动拨号', 'info');
|
||||
return false;
|
||||
}
|
||||
|
||||
if (this.isConnecting || this.dialBtnDisabled) {
|
||||
log('页面正在连接中,忽略重复唤醒', 'info');
|
||||
return false;
|
||||
}
|
||||
|
||||
if (now - this.lastWakewordDialTime < 3000) {
|
||||
log('唤醒触发过于频繁,忽略本次自动拨号', 'warning');
|
||||
return false;
|
||||
}
|
||||
|
||||
this.lastWakewordDialTime = now;
|
||||
this.addChatMessage(`检测到唤醒词“${wakeWord}”,准备连接服务器...`, false);
|
||||
await this.handleConnect();
|
||||
return true;
|
||||
}
|
||||
|
||||
// Add MCP tool
|
||||
addMCPTool() {
|
||||
const mcpToolsList = document.getElementById('mcpToolsList');
|
||||
|
Before Width: | Height: | Size: 1.7 MiB After Width: | Height: | Size: 1.7 MiB |
|
Before Width: | Height: | Size: 2.4 MiB After Width: | Height: | Size: 2.4 MiB |
|
Before Width: | Height: | Size: 6.6 MiB After Width: | Height: | Size: 6.6 MiB |
@@ -0,0 +1,55 @@
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
from wakeword_runtime.config import load_config, setup_logging
|
||||
from wakeword_runtime.runtime import TestRuntimeApplication, TestRuntimeHttpServer
|
||||
|
||||
|
||||
def main() -> int:
|
||||
test_root = Path(__file__).resolve().parent
|
||||
runtime_root = test_root / "wakeword_runtime"
|
||||
|
||||
config = load_config(runtime_root)
|
||||
setup_logging(config.log_file, config.log_level)
|
||||
http_server = TestRuntimeHttpServer(test_root)
|
||||
app_lock = threading.RLock()
|
||||
app = TestRuntimeApplication(config, http_server.event_bridge)
|
||||
|
||||
def restart_runtime() -> None:
|
||||
nonlocal app
|
||||
with app_lock:
|
||||
app.shutdown()
|
||||
next_config = load_config(runtime_root)
|
||||
setup_logging(next_config.log_file, next_config.log_level)
|
||||
next_app = TestRuntimeApplication(next_config, http_server.event_bridge)
|
||||
next_app.setup()
|
||||
next_app.start()
|
||||
app = next_app
|
||||
|
||||
http_server.set_restart_handler(restart_runtime)
|
||||
|
||||
print(f"test runtime started: {http_server.page_url}")
|
||||
print(f"wakeword bridge websocket: {http_server.bridge_url}")
|
||||
print(f"wakeword enabled: {config.wakeword_enabled}")
|
||||
print("press Ctrl+C to stop")
|
||||
|
||||
try:
|
||||
with app_lock:
|
||||
try:
|
||||
app.setup()
|
||||
app.start()
|
||||
except Exception as e:
|
||||
print(f"警告: 唤醒词服务启动失败({e}),测试页面仍可正常使用")
|
||||
http_server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("test runtime stopped")
|
||||
finally:
|
||||
with app_lock:
|
||||
app.shutdown()
|
||||
http_server.shutdown()
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1 @@
|
||||
"""Test-side wakeword runtime package."""
|
||||
@@ -0,0 +1 @@
|
||||
from .event_bridge import WakewordEventBridge
|
||||
@@ -0,0 +1,106 @@
|
||||
import json
|
||||
import logging
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WakewordEventBridge:
|
||||
def __init__(self) -> None:
|
||||
self._clients: list[queue.Queue[str]] = []
|
||||
self._clients_lock = threading.Lock()
|
||||
self._running = True
|
||||
|
||||
@property
|
||||
def is_running(self) -> bool:
|
||||
return self._running
|
||||
|
||||
def build_ready_message(self) -> str:
|
||||
return self.build_message(
|
||||
"bridge_connected",
|
||||
{"status": "ready"},
|
||||
)
|
||||
|
||||
def publish_detected(self, wake_word: str) -> None:
|
||||
self.publish(
|
||||
"wake_word_detected",
|
||||
{
|
||||
"wake_word": wake_word,
|
||||
"timestamp": time.time(),
|
||||
},
|
||||
)
|
||||
|
||||
def publish(self, event_type: str, payload: dict[str, Any] | None = None) -> None:
|
||||
if not self._running:
|
||||
return
|
||||
|
||||
message = self.build_message(event_type, payload or {})
|
||||
with self._clients_lock:
|
||||
clients = list(self._clients)
|
||||
|
||||
stale_clients: list[queue.Queue[str]] = []
|
||||
for client_queue in clients:
|
||||
try:
|
||||
client_queue.put_nowait(message)
|
||||
except queue.Full:
|
||||
stale_clients.append(client_queue)
|
||||
|
||||
if stale_clients:
|
||||
with self._clients_lock:
|
||||
for client_queue in stale_clients:
|
||||
if client_queue in self._clients:
|
||||
self._clients.remove(client_queue)
|
||||
|
||||
def add_client(self) -> queue.Queue[str]:
|
||||
client_queue: queue.Queue[str] = queue.Queue(maxsize=16)
|
||||
with self._clients_lock:
|
||||
self._clients.append(client_queue)
|
||||
return client_queue
|
||||
|
||||
def build_message(
|
||||
self,
|
||||
event_type: str,
|
||||
payload: dict[str, Any] | None = None,
|
||||
request_id: str | None = None,
|
||||
success: bool = True,
|
||||
error: str | None = None,
|
||||
) -> str:
|
||||
message: dict[str, Any] = {
|
||||
"type": event_type,
|
||||
"requestId": request_id,
|
||||
"success": success,
|
||||
"payload": payload or {},
|
||||
}
|
||||
if error:
|
||||
message["error"] = error
|
||||
return json.dumps(message, ensure_ascii=False)
|
||||
|
||||
def remove_client(self, client_queue: queue.Queue[str]) -> None:
|
||||
with self._clients_lock:
|
||||
if client_queue in self._clients:
|
||||
self._clients.remove(client_queue)
|
||||
|
||||
def publish_service_ready(self) -> None:
|
||||
self.publish("service_ready", {"status": "ready"})
|
||||
|
||||
def publish_service_stopping(self) -> None:
|
||||
self.publish("service_stopping", {"status": "stopping"})
|
||||
|
||||
def close(self) -> None:
|
||||
if not self._running:
|
||||
return
|
||||
|
||||
self.publish_service_stopping()
|
||||
self._running = False
|
||||
with self._clients_lock:
|
||||
clients = list(self._clients)
|
||||
self._clients.clear()
|
||||
|
||||
for client_queue in clients:
|
||||
try:
|
||||
client_queue.put_nowait("__bridge_closed__")
|
||||
except queue.Full:
|
||||
pass
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"wakeword": {
|
||||
"enabled": true
|
||||
},
|
||||
"model_dir": "models",
|
||||
"audio": {
|
||||
"input_device": null,
|
||||
"sample_rate": 16000,
|
||||
"channels": 1
|
||||
},
|
||||
"detector": {
|
||||
"num_threads": 4,
|
||||
"provider": "cpu",
|
||||
"max_active_paths": 2,
|
||||
"keywords_score": 1.8,
|
||||
"keywords_threshold": 0.1,
|
||||
"num_trailing_blanks": 1,
|
||||
"cooldown_seconds": 1.5
|
||||
},
|
||||
"logging": {
|
||||
"level": "INFO",
|
||||
"dir": "logs",
|
||||
"file": "wakeword-runtime.log"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
from .config_loader import RuntimeConfig, load_config
|
||||
from .logging_setup import setup_logging
|
||||
@@ -0,0 +1,153 @@
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class WakewordSettings:
|
||||
enabled: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioSettings:
|
||||
input_device: str | int | None = None
|
||||
sample_rate: int = 16000
|
||||
channels: int = 1
|
||||
|
||||
|
||||
@dataclass
|
||||
class DetectorSettings:
|
||||
num_threads: int = 4
|
||||
provider: str = "cpu"
|
||||
max_active_paths: int = 2
|
||||
keywords_score: float = 1.8
|
||||
keywords_threshold: float = 0.2
|
||||
num_trailing_blanks: int = 1
|
||||
cooldown_seconds: float = 1.5
|
||||
|
||||
|
||||
@dataclass
|
||||
class LoggingSettings:
|
||||
level: str = "INFO"
|
||||
directory: str = "logs"
|
||||
file_name: str = "wakeword-runtime.log"
|
||||
|
||||
|
||||
@dataclass
|
||||
class RuntimeConfig:
|
||||
runtime_root: Path
|
||||
wakeword: WakewordSettings
|
||||
wake_words: list[str]
|
||||
model_dir: Path
|
||||
audio: AudioSettings
|
||||
detector: DetectorSettings
|
||||
logging: LoggingSettings
|
||||
|
||||
def validate(self) -> None:
|
||||
if self.wakeword.enabled and not self.wake_words:
|
||||
raise ValueError("keywords.txt cannot be empty when wakeword is enabled")
|
||||
|
||||
if self.audio.sample_rate <= 0:
|
||||
raise ValueError("audio.sample_rate must be greater than 0")
|
||||
|
||||
if self.audio.channels <= 0:
|
||||
raise ValueError("audio.channels must be greater than 0")
|
||||
|
||||
if self.detector.num_threads <= 0:
|
||||
raise ValueError("detector.num_threads must be greater than 0")
|
||||
|
||||
if self.detector.cooldown_seconds < 0:
|
||||
raise ValueError("detector.cooldown_seconds cannot be negative")
|
||||
|
||||
if not self.logging.level:
|
||||
raise ValueError("logging.level cannot be empty")
|
||||
|
||||
@property
|
||||
def wakeword_enabled(self) -> bool:
|
||||
return self.wakeword.enabled
|
||||
|
||||
@property
|
||||
def log_dir(self) -> Path:
|
||||
return (self.runtime_root / self.logging.directory).resolve()
|
||||
|
||||
@property
|
||||
def log_file(self) -> Path:
|
||||
return self.log_dir / self.logging.file_name
|
||||
|
||||
@property
|
||||
def log_level(self) -> str:
|
||||
return self.logging.level.upper()
|
||||
|
||||
|
||||
def load_config(runtime_root: Path) -> RuntimeConfig:
|
||||
config_path = runtime_root / "config.json"
|
||||
raw = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
wakeword_cfg = dict(raw.get("wakeword", {}))
|
||||
raw_model_dir = Path(str(raw.get("model_dir", "models")))
|
||||
model_dir = raw_model_dir.resolve() if raw_model_dir.is_absolute() else (runtime_root / raw_model_dir).resolve()
|
||||
wake_words = _load_wake_words_from_keywords_file(model_dir)
|
||||
|
||||
audio_cfg = dict(raw.get("audio", {}))
|
||||
detector_cfg = dict(raw.get("detector", {}))
|
||||
logging_cfg = dict(raw.get("logging", {}))
|
||||
|
||||
config = RuntimeConfig(
|
||||
runtime_root=runtime_root,
|
||||
wakeword=WakewordSettings(enabled=bool(wakeword_cfg.get("enabled", False))),
|
||||
wake_words=wake_words,
|
||||
model_dir=model_dir,
|
||||
audio=AudioSettings(
|
||||
input_device=audio_cfg.get("input_device"),
|
||||
sample_rate=int(audio_cfg.get("sample_rate", 16000)),
|
||||
channels=int(audio_cfg.get("channels", 1)),
|
||||
),
|
||||
detector=DetectorSettings(
|
||||
num_threads=int(detector_cfg.get("num_threads", 4)),
|
||||
provider=str(detector_cfg.get("provider", "cpu")),
|
||||
max_active_paths=int(detector_cfg.get("max_active_paths", 2)),
|
||||
keywords_score=float(detector_cfg.get("keywords_score", 1.8)),
|
||||
keywords_threshold=float(detector_cfg.get("keywords_threshold", 0.2)),
|
||||
num_trailing_blanks=int(detector_cfg.get("num_trailing_blanks", 1)),
|
||||
cooldown_seconds=float(detector_cfg.get("cooldown_seconds", 1.5)),
|
||||
),
|
||||
logging=LoggingSettings(
|
||||
level=str(logging_cfg.get("level", "INFO")).upper(),
|
||||
directory=str(logging_cfg.get("dir", "logs")),
|
||||
file_name=str(logging_cfg.get("file", "wakeword-runtime.log")),
|
||||
),
|
||||
)
|
||||
config.validate()
|
||||
return config
|
||||
|
||||
|
||||
DEFAULT_WAKE_WORDS = [
|
||||
"你好小智",
|
||||
"你好小志",
|
||||
"小爱同学",
|
||||
"你好小鑫",
|
||||
"你好小新",
|
||||
"小美同学",
|
||||
"小龙小龙",
|
||||
"喵喵同学",
|
||||
"小滨小滨",
|
||||
"小冰小冰",
|
||||
"嘿你好呀",
|
||||
]
|
||||
|
||||
|
||||
def _load_wake_words_from_keywords_file(model_dir: Path) -> list[str]:
|
||||
keywords_file = model_dir / "keywords.txt"
|
||||
if not keywords_file.exists():
|
||||
return DEFAULT_WAKE_WORDS.copy()
|
||||
|
||||
wake_words: list[str] = []
|
||||
for line in keywords_file.read_text(encoding="utf-8").splitlines():
|
||||
text = line.strip()
|
||||
if not text or text.startswith("#") or "@" not in text:
|
||||
continue
|
||||
|
||||
wake_word = text.split("@", 1)[1].strip()
|
||||
if wake_word:
|
||||
wake_words.append(wake_word)
|
||||
|
||||
return wake_words if wake_words else DEFAULT_WAKE_WORDS.copy()
|
||||
@@ -0,0 +1,23 @@
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def setup_logging(log_file: Path, level: str = "INFO") -> None:
|
||||
log_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
logger = logging.getLogger()
|
||||
logger.setLevel(getattr(logging, level.upper(), logging.INFO))
|
||||
logger.handlers.clear()
|
||||
|
||||
formatter = logging.Formatter(
|
||||
"%(asctime)s [%(levelname)s] %(name)s - %(message)s"
|
||||
)
|
||||
|
||||
stream_handler = logging.StreamHandler()
|
||||
stream_handler.setFormatter(formatter)
|
||||
|
||||
file_handler = logging.FileHandler(log_file, encoding="utf-8")
|
||||
file_handler.setFormatter(formatter)
|
||||
|
||||
logger.addHandler(stream_handler)
|
||||
logger.addHandler(file_handler)
|
||||
@@ -0,0 +1,3 @@
|
||||
from .detector_assets import DetectorAssets, DetectorAssetsBuilder
|
||||
from .detector import WakewordDetector
|
||||
from .microphone import MicrophoneListener
|
||||
@@ -0,0 +1,191 @@
|
||||
import queue
|
||||
import threading
|
||||
import logging
|
||||
import time
|
||||
from typing import Any, Callable
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ..config import RuntimeConfig
|
||||
from .detector_assets import DetectorAssetsBuilder
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WakewordDetector:
|
||||
def __init__(self, config: RuntimeConfig) -> None:
|
||||
self.config = config
|
||||
self.enabled = config.wakeword_enabled
|
||||
self.assets_builder = DetectorAssetsBuilder(config)
|
||||
self.audio_source = None
|
||||
self.keyword_spotter: Any | None = None
|
||||
self.stream: Any | None = None
|
||||
self.is_running_flag = False
|
||||
self.paused = False
|
||||
self.on_detected_callback: Callable[[str, str], None] | None = None
|
||||
self.on_error: Callable[[Exception], None] | None = None
|
||||
self._audio_queue: queue.Queue[np.ndarray] = queue.Queue(maxsize=100)
|
||||
self._worker_thread: threading.Thread | None = None
|
||||
self.last_detection_time = 0.0
|
||||
self.detection_cooldown = self.config.detector.cooldown_seconds
|
||||
|
||||
def initialize(self) -> None:
|
||||
if not self.enabled:
|
||||
raise RuntimeError("wakeword detector is disabled")
|
||||
|
||||
try:
|
||||
import sherpa_onnx
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"Missing dependency: sherpa-onnx. Install runtime dependencies before initializing detector."
|
||||
) from exc
|
||||
|
||||
assets = self.assets_builder.prepare()
|
||||
detector_cfg = self.config.detector
|
||||
|
||||
self.keyword_spotter = sherpa_onnx.KeywordSpotter(
|
||||
tokens=str(assets.tokens_file),
|
||||
encoder=str(assets.encoder_file),
|
||||
decoder=str(assets.decoder_file),
|
||||
joiner=str(assets.joiner_file),
|
||||
keywords_file=str(assets.keywords_file),
|
||||
num_threads=detector_cfg.num_threads,
|
||||
sample_rate=self.config.audio.sample_rate,
|
||||
feature_dim=80,
|
||||
max_active_paths=detector_cfg.max_active_paths,
|
||||
keywords_score=detector_cfg.keywords_score,
|
||||
keywords_threshold=detector_cfg.keywords_threshold,
|
||||
num_trailing_blanks=detector_cfg.num_trailing_blanks,
|
||||
provider=detector_cfg.provider,
|
||||
)
|
||||
self.stream = self.keyword_spotter.create_stream()
|
||||
logger.info("detector initialized")
|
||||
logger.info("detector model root: %s", assets.model_root)
|
||||
logger.info("detector keywords file: %s", assets.keywords_file)
|
||||
|
||||
def on_detected(self, callback: Callable[[str, str], None]) -> None:
|
||||
self.on_detected_callback = callback
|
||||
|
||||
def on_audio_data(self, audio_data: np.ndarray) -> None:
|
||||
if not self.enabled or not self.is_running_flag or self.paused:
|
||||
return
|
||||
|
||||
try:
|
||||
self._audio_queue.put_nowait(audio_data.copy())
|
||||
except queue.Full:
|
||||
try:
|
||||
self._audio_queue.get_nowait()
|
||||
self._audio_queue.put_nowait(audio_data.copy())
|
||||
except queue.Empty:
|
||||
pass
|
||||
except Exception as exc:
|
||||
logger.debug("audio data enqueue failed: %s", exc)
|
||||
|
||||
def start(self, audio_source) -> None:
|
||||
if not self.enabled:
|
||||
logger.info("wakeword detector disabled")
|
||||
return
|
||||
|
||||
if self.keyword_spotter is None or self.stream is None:
|
||||
self.initialize()
|
||||
|
||||
if self.is_running_flag:
|
||||
return
|
||||
|
||||
self.audio_source = audio_source
|
||||
self.audio_source.add_audio_listener(self)
|
||||
self.is_running_flag = True
|
||||
self.paused = False
|
||||
self._worker_thread = threading.Thread(
|
||||
target=self._detection_loop,
|
||||
name="wakeword-detector",
|
||||
daemon=True,
|
||||
)
|
||||
self._worker_thread.start()
|
||||
logger.info("wakeword detector started")
|
||||
|
||||
def stop(self) -> None:
|
||||
if not self.is_running_flag and self.audio_source is None and self._worker_thread is None:
|
||||
return
|
||||
|
||||
self.is_running_flag = False
|
||||
|
||||
if self.audio_source is not None:
|
||||
self.audio_source.remove_audio_listener(self)
|
||||
self.audio_source = None
|
||||
|
||||
if self._worker_thread is not None:
|
||||
self._worker_thread.join(timeout=1.0)
|
||||
self._worker_thread = None
|
||||
|
||||
while not self._audio_queue.empty():
|
||||
try:
|
||||
self._audio_queue.get_nowait()
|
||||
except queue.Empty:
|
||||
break
|
||||
|
||||
logger.info("wakeword detector stopped")
|
||||
|
||||
def _detection_loop(self) -> None:
|
||||
error_count = 0
|
||||
max_errors = 5
|
||||
|
||||
while self.is_running_flag:
|
||||
try:
|
||||
if self.paused:
|
||||
time.sleep(0.1)
|
||||
continue
|
||||
|
||||
audio_data = self._audio_queue.get(timeout=0.1)
|
||||
result = self.process_audio_chunk(audio_data)
|
||||
if result and self.on_detected_callback is not None:
|
||||
self.on_detected_callback(result, result)
|
||||
error_count = 0
|
||||
except queue.Empty:
|
||||
continue
|
||||
except Exception as exc:
|
||||
error_count += 1
|
||||
logger.error("wakeword detection loop error(%s/%s): %s", error_count, max_errors, exc)
|
||||
if self.on_error is not None:
|
||||
try:
|
||||
self.on_error(exc)
|
||||
except Exception:
|
||||
logger.exception("wakeword error callback failed")
|
||||
if error_count >= max_errors:
|
||||
logger.critical("too many wakeword detection errors, stopping detector")
|
||||
break
|
||||
time.sleep(1)
|
||||
|
||||
self.is_running_flag = False
|
||||
|
||||
def process_audio_chunk(self, audio_data: np.ndarray) -> str | None:
|
||||
if self.keyword_spotter is None or self.stream is None:
|
||||
raise RuntimeError("detector is not initialized")
|
||||
|
||||
if audio_data is None or len(audio_data) == 0:
|
||||
return None
|
||||
|
||||
if audio_data.dtype == np.int16:
|
||||
samples = audio_data.astype(np.float32) / 32768.0
|
||||
else:
|
||||
samples = audio_data.astype(np.float32)
|
||||
|
||||
sample_rate = self.config.audio.sample_rate
|
||||
self.stream.accept_waveform(sample_rate=sample_rate, waveform=samples)
|
||||
|
||||
if not self.keyword_spotter.is_ready(self.stream):
|
||||
return None
|
||||
|
||||
self.keyword_spotter.decode_stream(self.stream)
|
||||
result = self.keyword_spotter.get_result(self.stream)
|
||||
if not result:
|
||||
return None
|
||||
|
||||
self.keyword_spotter.reset_stream(self.stream)
|
||||
|
||||
current_time = time.time()
|
||||
if current_time - self.last_detection_time < self.detection_cooldown:
|
||||
return None
|
||||
|
||||
self.last_detection_time = current_time
|
||||
return str(result)
|
||||
@@ -0,0 +1,94 @@
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from ..config import RuntimeConfig
|
||||
|
||||
REQUIRED_MODEL_FILES = (
|
||||
"encoder.onnx",
|
||||
"decoder.onnx",
|
||||
"joiner.onnx",
|
||||
"tokens.txt",
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DetectorAssets:
|
||||
model_root: Path
|
||||
tokens_file: Path
|
||||
encoder_file: Path
|
||||
decoder_file: Path
|
||||
joiner_file: Path
|
||||
keywords_file: Path
|
||||
|
||||
|
||||
class DetectorAssetsBuilder:
|
||||
def __init__(self, config: RuntimeConfig) -> None:
|
||||
self.config = config
|
||||
|
||||
def prepare(self) -> DetectorAssets:
|
||||
model_root = self._resolve_model_root()
|
||||
keywords_file = self._write_keywords_file(model_root)
|
||||
return DetectorAssets(
|
||||
model_root=model_root,
|
||||
tokens_file=model_root / "tokens.txt",
|
||||
encoder_file=model_root / "encoder.onnx",
|
||||
decoder_file=model_root / "decoder.onnx",
|
||||
joiner_file=model_root / "joiner.onnx",
|
||||
keywords_file=keywords_file,
|
||||
)
|
||||
|
||||
def _resolve_model_root(self) -> Path:
|
||||
preferred = self.config.model_dir
|
||||
if self._has_required_files(preferred):
|
||||
return preferred
|
||||
|
||||
raise FileNotFoundError(
|
||||
"No valid model directory found. Expected configured model files to exist."
|
||||
)
|
||||
|
||||
def _write_keywords_file(self, model_root: Path) -> Path:
|
||||
try:
|
||||
from pypinyin import Style, pinyin
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"Missing dependency: pypinyin. Install runtime dependencies before generating keywords."
|
||||
) from exc
|
||||
|
||||
wake_words = self.config.wake_words
|
||||
if not wake_words:
|
||||
raise ValueError("keywords.txt cannot be empty")
|
||||
|
||||
keywords_path = model_root / "keywords.txt"
|
||||
lines: list[str] = []
|
||||
for keyword_text in wake_words:
|
||||
initials = pinyin(keyword_text, style=Style.INITIALS, strict=False)
|
||||
finals = pinyin(
|
||||
keyword_text,
|
||||
style=Style.FINALS_TONE,
|
||||
strict=False,
|
||||
neutral_tone_with_five=True,
|
||||
)
|
||||
|
||||
tokens: list[str] = []
|
||||
for initial_parts, final_parts in zip(initials, finals):
|
||||
initial = initial_parts[0].strip()
|
||||
final = final_parts[0].strip()
|
||||
if initial:
|
||||
tokens.append(initial)
|
||||
if final:
|
||||
tokens.append(final)
|
||||
|
||||
if not tokens:
|
||||
raise ValueError(
|
||||
f"failed to generate pinyin tokens for wake word: {keyword_text}"
|
||||
)
|
||||
|
||||
lines.append(f"{' '.join(tokens)} @{keyword_text}")
|
||||
|
||||
keywords_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
return keywords_path
|
||||
|
||||
def _has_required_files(self, directory: Path) -> bool:
|
||||
if not directory.exists():
|
||||
return False
|
||||
return all((directory / file_name).exists() for file_name in REQUIRED_MODEL_FILES)
|
||||
@@ -0,0 +1,91 @@
|
||||
import logging
|
||||
from typing import Protocol
|
||||
|
||||
import numpy as np
|
||||
|
||||
from ..config import RuntimeConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioListener(Protocol):
|
||||
def on_audio_data(self, audio_data: np.ndarray) -> None:
|
||||
...
|
||||
|
||||
|
||||
class MicrophoneListener:
|
||||
|
||||
def __init__(self, config: RuntimeConfig) -> None:
|
||||
self.config = config
|
||||
self._stream = None
|
||||
self._running = False
|
||||
self._listeners: list[AudioListener] = []
|
||||
self._sample_rate = self.config.audio.sample_rate
|
||||
self._channels = self.config.audio.channels
|
||||
self._device = self.config.audio.input_device
|
||||
self._block_duration_ms = 30
|
||||
self._block_size = int(self._sample_rate * (self._block_duration_ms / 1000))
|
||||
|
||||
def add_audio_listener(self, listener: AudioListener) -> None:
|
||||
if listener not in self._listeners:
|
||||
self._listeners.append(listener)
|
||||
|
||||
def remove_audio_listener(self, listener: AudioListener) -> None:
|
||||
if listener in self._listeners:
|
||||
self._listeners.remove(listener)
|
||||
|
||||
def start(self) -> None:
|
||||
try:
|
||||
import sounddevice as sd
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"Missing dependency: sounddevice. Install runtime dependencies before starting microphone listener."
|
||||
) from exc
|
||||
|
||||
self._stream = sd.InputStream(
|
||||
device=self._device,
|
||||
samplerate=self._sample_rate,
|
||||
channels=self._channels,
|
||||
dtype="int16",
|
||||
blocksize=self._block_size,
|
||||
callback=self._input_callback,
|
||||
latency="low",
|
||||
)
|
||||
self._stream.start()
|
||||
self._running = True
|
||||
|
||||
logger.info("microphone listener started")
|
||||
logger.info("microphone sample rate: %s", self._sample_rate)
|
||||
logger.info("microphone channels: %s", self._channels)
|
||||
logger.info("microphone device: %s", self._device if self._device is not None else "default")
|
||||
logger.info("microphone block size: %s", self._block_size)
|
||||
|
||||
def stop(self) -> None:
|
||||
if not self._running and self._stream is None:
|
||||
return
|
||||
|
||||
self._running = False
|
||||
if self._stream is not None:
|
||||
try:
|
||||
self._stream.stop()
|
||||
finally:
|
||||
self._stream.close()
|
||||
self._stream = None
|
||||
logger.info("microphone listener stopped")
|
||||
|
||||
def _input_callback(self, indata, frames, time_info, status) -> None:
|
||||
_ = frames, time_info
|
||||
if status and "overflow" not in str(status).lower():
|
||||
logger.warning("microphone status: %s", status)
|
||||
|
||||
audio = np.copy(indata)
|
||||
if audio.ndim > 1:
|
||||
audio = audio[:, 0]
|
||||
else:
|
||||
audio = audio.reshape(-1)
|
||||
|
||||
for listener in list(self._listeners):
|
||||
try:
|
||||
listener.on_audio_data(audio)
|
||||
except Exception:
|
||||
logger.exception("audio listener callback failed")
|
||||
@@ -0,0 +1,4 @@
|
||||
from .audio import AudioPlugin
|
||||
from .base import Plugin
|
||||
from .manager import PluginManager
|
||||
from .wake_word import WakeWordPlugin
|
||||
@@ -0,0 +1,37 @@
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from ..core import MicrophoneListener
|
||||
from .base import Plugin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioPlugin(Plugin):
|
||||
name = "audio"
|
||||
priority = 10
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.app = None
|
||||
self.source: MicrophoneListener | None = None
|
||||
|
||||
def setup(self, app: Any) -> None:
|
||||
self.app = app
|
||||
self.source = MicrophoneListener(app.config)
|
||||
self.app.audio_source = self.source
|
||||
|
||||
def start(self) -> None:
|
||||
if self.source is None:
|
||||
logger.warning("audio source not initialized")
|
||||
return
|
||||
self.source.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.source is not None:
|
||||
self.source.stop()
|
||||
|
||||
def shutdown(self) -> None:
|
||||
if self.app is not None:
|
||||
self.app.audio_source = None
|
||||
self.source = None
|
||||
self.app = None
|
||||
@@ -0,0 +1,18 @@
|
||||
from typing import Any
|
||||
|
||||
|
||||
class Plugin:
|
||||
name: str = "plugin"
|
||||
priority: int = 50
|
||||
|
||||
def setup(self, app: Any) -> None:
|
||||
return None
|
||||
|
||||
def start(self) -> None:
|
||||
return None
|
||||
|
||||
def stop(self) -> None:
|
||||
return None
|
||||
|
||||
def shutdown(self) -> None:
|
||||
return None
|
||||
@@ -0,0 +1,37 @@
|
||||
from typing import Any
|
||||
|
||||
from .base import Plugin
|
||||
|
||||
|
||||
class PluginManager:
|
||||
def __init__(self) -> None:
|
||||
self._plugins: list[Plugin] = []
|
||||
self._by_name: dict[str, Plugin] = {}
|
||||
|
||||
def register(self, *plugins: Plugin) -> None:
|
||||
for plugin in sorted(plugins, key=lambda item: getattr(item, "priority", 50)):
|
||||
if plugin in self._plugins:
|
||||
continue
|
||||
self._plugins.append(plugin)
|
||||
name = getattr(plugin, "name", "")
|
||||
if isinstance(name, str) and name:
|
||||
self._by_name[name] = plugin
|
||||
|
||||
def get_plugin(self, name: str) -> Plugin | None:
|
||||
return self._by_name.get(name)
|
||||
|
||||
def setup_all(self, app: Any) -> None:
|
||||
for plugin in list(self._plugins):
|
||||
plugin.setup(app)
|
||||
|
||||
def start_all(self) -> None:
|
||||
for plugin in list(self._plugins):
|
||||
plugin.start()
|
||||
|
||||
def stop_all(self) -> None:
|
||||
for plugin in reversed(self._plugins):
|
||||
plugin.stop()
|
||||
|
||||
def shutdown_all(self) -> None:
|
||||
for plugin in reversed(self._plugins):
|
||||
plugin.shutdown()
|
||||
@@ -0,0 +1,53 @@
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from ..core import WakewordDetector
|
||||
from .base import Plugin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WakeWordPlugin(Plugin):
|
||||
name = "wake_word"
|
||||
priority = 30
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.app = None
|
||||
self.detector: WakewordDetector | None = None
|
||||
|
||||
def setup(self, app: Any) -> None:
|
||||
self.app = app
|
||||
self.detector = WakewordDetector(app.config)
|
||||
if not self.detector.enabled:
|
||||
self.detector = None
|
||||
return
|
||||
self.detector.on_detected(self._on_detected)
|
||||
self.detector.on_error = self._on_error
|
||||
|
||||
def start(self) -> None:
|
||||
if self.detector is None:
|
||||
return
|
||||
|
||||
audio_plugin = self.app.plugins.get_plugin("audio") if self.app else None
|
||||
audio_source = getattr(audio_plugin, "source", None)
|
||||
if audio_source is None:
|
||||
logger.warning("audio source unavailable, wakeword plugin not started")
|
||||
return
|
||||
|
||||
self.detector.start(audio_source)
|
||||
|
||||
def stop(self) -> None:
|
||||
if self.detector is not None:
|
||||
self.detector.stop()
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self.detector = None
|
||||
self.app = None
|
||||
|
||||
def _on_detected(self, wake_word: str, full_text: str) -> None:
|
||||
if self.app is None:
|
||||
return
|
||||
self.app.handle_wake_word_detected(wake_word, full_text)
|
||||
|
||||
def _on_error(self, error: Exception) -> None:
|
||||
logger.error("wakeword detection error: %s", error)
|
||||
@@ -0,0 +1,3 @@
|
||||
sherpa-onnx==1.12.29
|
||||
sounddevice>=0.4.4
|
||||
pypinyin==0.55.0
|
||||
@@ -0,0 +1,2 @@
|
||||
from .app import TestRuntimeApplication
|
||||
from .http_server import TestRuntimeHttpServer
|
||||
@@ -0,0 +1,70 @@
|
||||
import logging
|
||||
from typing import Protocol
|
||||
|
||||
from ..config import RuntimeConfig
|
||||
from ..plugins import AudioPlugin, PluginManager, WakeWordPlugin
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EventPublisher(Protocol):
|
||||
def publish_service_ready(self) -> None:
|
||||
...
|
||||
|
||||
def publish_service_stopping(self) -> None:
|
||||
...
|
||||
|
||||
def publish_detected(self, wake_word: str) -> None:
|
||||
...
|
||||
|
||||
|
||||
class TestRuntimeApplication:
|
||||
def __init__(self, config: RuntimeConfig, event_publisher: EventPublisher) -> None:
|
||||
self.config = config
|
||||
self.event_bridge = event_publisher
|
||||
self.plugins = PluginManager()
|
||||
self.audio_source = None
|
||||
self._is_setup = False
|
||||
self._is_running = False
|
||||
|
||||
def setup(self) -> None:
|
||||
if self._is_setup:
|
||||
return
|
||||
|
||||
if self.config.wakeword_enabled:
|
||||
self.plugins.register(
|
||||
AudioPlugin(),
|
||||
WakeWordPlugin(),
|
||||
)
|
||||
self.plugins.setup_all(self)
|
||||
self._is_setup = True
|
||||
|
||||
def start(self) -> None:
|
||||
if self._is_running:
|
||||
return
|
||||
|
||||
self.setup()
|
||||
self.plugins.start_all()
|
||||
if self.config.wakeword_enabled:
|
||||
self.event_bridge.publish_service_ready()
|
||||
self._is_running = True
|
||||
logger.info("test runtime application started")
|
||||
|
||||
def stop(self) -> None:
|
||||
if not self._is_running:
|
||||
return
|
||||
|
||||
self.event_bridge.publish_service_stopping()
|
||||
self.plugins.stop_all()
|
||||
self._is_running = False
|
||||
logger.info("test runtime application stopped")
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self.stop()
|
||||
if self._is_setup:
|
||||
self.plugins.shutdown_all()
|
||||
|
||||
def handle_wake_word_detected(self, wake_word: str, full_text: str) -> None:
|
||||
_ = full_text
|
||||
logger.info("wake word detected: %s", wake_word)
|
||||
self.event_bridge.publish_detected(wake_word)
|
||||
@@ -0,0 +1,352 @@
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import queue
|
||||
import socket
|
||||
import threading
|
||||
from http import HTTPStatus
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Callable
|
||||
|
||||
from ..bridge import WakewordEventBridge
|
||||
from ..config.config_loader import load_config
|
||||
|
||||
|
||||
class TestRuntimeHttpServer:
|
||||
def __init__(self, test_root: Path, host: str = "0.0.0.0", port: int = 8006) -> None:
|
||||
self.test_root = test_root
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.event_bridge = WakewordEventBridge()
|
||||
self._restart_handler: Callable[[], None] | None = None
|
||||
self._restart_lock = threading.Lock()
|
||||
self._server = self._build_server()
|
||||
|
||||
@property
|
||||
def page_url(self) -> str:
|
||||
return f"http://127.0.0.1:{self.port}/index.html"
|
||||
|
||||
@property
|
||||
def bridge_url(self) -> str:
|
||||
return f"ws://127.0.0.1:{self.port}/wakeword-ws"
|
||||
|
||||
def serve_forever(self) -> None:
|
||||
self._server.serve_forever()
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self._server.shutdown()
|
||||
self.event_bridge.close()
|
||||
self._server.server_close()
|
||||
|
||||
def set_restart_handler(self, handler: Callable[[], None]) -> None:
|
||||
self._restart_handler = handler
|
||||
|
||||
def request_runtime_restart(self) -> None:
|
||||
with self._restart_lock:
|
||||
handler = self._restart_handler
|
||||
|
||||
if handler is None:
|
||||
raise RuntimeError("restart handler is not configured")
|
||||
|
||||
threading.Thread(
|
||||
target=self._run_restart_handler,
|
||||
name="test-runtime-restart",
|
||||
daemon=True,
|
||||
).start()
|
||||
|
||||
def _run_restart_handler(self) -> None:
|
||||
handler = self._restart_handler
|
||||
if handler is None:
|
||||
return
|
||||
handler()
|
||||
|
||||
def _build_server(self) -> ThreadingHTTPServer:
|
||||
test_root = self.test_root
|
||||
event_bridge = self.event_bridge
|
||||
schedule_restart = self.request_runtime_restart
|
||||
|
||||
class TestRuntimeHandler(SimpleHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__(*args, directory=str(test_root), **kwargs)
|
||||
|
||||
def handle(self) -> None:
|
||||
try:
|
||||
super().handle()
|
||||
except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError):
|
||||
pass
|
||||
|
||||
def do_GET(self) -> None:
|
||||
if self.path == "/wakeword-ws":
|
||||
self._handle_websocket(event_bridge)
|
||||
return
|
||||
|
||||
if self.path == "/health":
|
||||
body = json.dumps({"status": "ok"}).encode("utf-8")
|
||||
self.send_response(HTTPStatus.OK)
|
||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
||||
self.send_header("Cache-Control", "no-cache")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
return
|
||||
|
||||
super().do_GET()
|
||||
|
||||
def log_message(self, format: str, *args) -> None:
|
||||
return
|
||||
|
||||
def _handle_websocket(self, bridge: WakewordEventBridge) -> None:
|
||||
if self.headers.get("Upgrade", "").lower() != "websocket":
|
||||
self.send_error(HTTPStatus.BAD_REQUEST, "expected websocket upgrade")
|
||||
return
|
||||
|
||||
websocket_key = self.headers.get("Sec-WebSocket-Key")
|
||||
if not websocket_key:
|
||||
self.send_error(HTTPStatus.BAD_REQUEST, "missing Sec-WebSocket-Key")
|
||||
return
|
||||
|
||||
accept_source = websocket_key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
|
||||
accept_value = base64.b64encode(
|
||||
hashlib.sha1(accept_source.encode("utf-8")).digest()
|
||||
).decode("ascii")
|
||||
|
||||
client_queue = bridge.add_client()
|
||||
self.send_response(HTTPStatus.SWITCHING_PROTOCOLS)
|
||||
self.send_header("Upgrade", "websocket")
|
||||
self.send_header("Connection", "Upgrade")
|
||||
self.send_header("Sec-WebSocket-Accept", accept_value)
|
||||
self.end_headers()
|
||||
|
||||
try:
|
||||
self.connection.settimeout(0.2)
|
||||
self._send_websocket_text(bridge.build_ready_message())
|
||||
self._send_websocket_text(self._build_wakeword_config_message(bridge))
|
||||
|
||||
while bridge.is_running:
|
||||
inbound_message = self._receive_websocket_message()
|
||||
if inbound_message is not None:
|
||||
response_message = self._handle_bridge_request(bridge, inbound_message)
|
||||
if response_message:
|
||||
self._send_websocket_text(response_message)
|
||||
|
||||
try:
|
||||
message = client_queue.get(timeout=0.2)
|
||||
if message == "__bridge_closed__":
|
||||
break
|
||||
self._send_websocket_text(message)
|
||||
except queue.Empty:
|
||||
if not bridge.is_running:
|
||||
break
|
||||
continue
|
||||
except socket.timeout:
|
||||
pass
|
||||
except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError):
|
||||
pass
|
||||
finally:
|
||||
bridge.remove_client(client_queue)
|
||||
|
||||
def _build_wakeword_config_message(self, bridge: WakewordEventBridge) -> str:
|
||||
try:
|
||||
runtime_root = test_root / "wakeword_runtime"
|
||||
config = load_config(runtime_root)
|
||||
payload = {
|
||||
"enabled": config.wakeword_enabled,
|
||||
"wakeWords": config.wake_words,
|
||||
}
|
||||
return bridge.build_message("wakeword_config", payload)
|
||||
except Exception as exc:
|
||||
return bridge.build_message(
|
||||
"wakeword_config",
|
||||
{},
|
||||
success=False,
|
||||
error=f"读取唤醒词配置失败: {exc}",
|
||||
)
|
||||
|
||||
def _handle_bridge_request(self, bridge: WakewordEventBridge, raw_message: str) -> str | None:
|
||||
try:
|
||||
message = json.loads(raw_message)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
message_type = str(message.get("type", "")).strip()
|
||||
request_id = message.get("requestId")
|
||||
payload = message.get("payload") or {}
|
||||
result_type = f"{message_type}_result" if message_type else "bridge_request_result"
|
||||
|
||||
if message_type == "set_wakeword_config":
|
||||
try:
|
||||
result_payload = self._save_wakeword_config(payload)
|
||||
bridge.publish("wakeword_config", result_payload)
|
||||
return bridge.build_message(
|
||||
"set_wakeword_config_result",
|
||||
result_payload,
|
||||
request_id=request_id,
|
||||
)
|
||||
except Exception as exc:
|
||||
return bridge.build_message(
|
||||
"set_wakeword_config_result",
|
||||
{},
|
||||
request_id=request_id,
|
||||
success=False,
|
||||
error=f"保存唤醒词配置失败: {exc}",
|
||||
)
|
||||
|
||||
if message_type == "restart_wakeword_service":
|
||||
schedule_restart()
|
||||
return bridge.build_message(
|
||||
"restart_wakeword_service_result",
|
||||
{"restarting": True},
|
||||
request_id=request_id,
|
||||
)
|
||||
|
||||
return bridge.build_message(
|
||||
result_type,
|
||||
{},
|
||||
request_id=request_id,
|
||||
success=False,
|
||||
error=f"unsupported message type: {message_type}",
|
||||
)
|
||||
|
||||
def _save_wakeword_config(self, payload: dict) -> dict:
|
||||
runtime_root = test_root / "wakeword_runtime"
|
||||
config_path = runtime_root / "config.json"
|
||||
model_root = runtime_root / "models"
|
||||
keywords_path = model_root / "keywords.txt"
|
||||
|
||||
enabled = bool(payload.get("enabled", True))
|
||||
wake_words = payload.get("wakeWords") or []
|
||||
normalized_wake_words = []
|
||||
for item in wake_words:
|
||||
if not isinstance(item, str):
|
||||
continue
|
||||
text = item.strip()
|
||||
if text and text not in normalized_wake_words:
|
||||
normalized_wake_words.append(text)
|
||||
|
||||
if enabled and not normalized_wake_words:
|
||||
raise ValueError("wakeWords cannot be empty when wakeword is enabled")
|
||||
|
||||
raw_config = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
raw_config.setdefault("wakeword", {})["enabled"] = enabled
|
||||
config_path.write_text(
|
||||
json.dumps(raw_config, indent=2, ensure_ascii=False),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
keywords_lines = [self._build_keyword_line(item) for item in normalized_wake_words]
|
||||
keywords_path.write_text(
|
||||
("\n".join(keywords_lines) + "\n") if keywords_lines else "",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
return {
|
||||
"enabled": enabled,
|
||||
"wakeWords": normalized_wake_words,
|
||||
}
|
||||
|
||||
def _build_keyword_line(self, keyword_text: str) -> str:
|
||||
from pypinyin import Style, pinyin
|
||||
|
||||
initials = pinyin(keyword_text, style=Style.INITIALS, strict=False)
|
||||
finals = pinyin(
|
||||
keyword_text,
|
||||
style=Style.FINALS_TONE,
|
||||
strict=False,
|
||||
neutral_tone_with_five=True,
|
||||
)
|
||||
|
||||
tokens: list[str] = []
|
||||
for initial_parts, final_parts in zip(initials, finals):
|
||||
initial = initial_parts[0].strip()
|
||||
final = final_parts[0].strip()
|
||||
if initial:
|
||||
tokens.append(initial)
|
||||
if final:
|
||||
tokens.append(final)
|
||||
|
||||
if not tokens:
|
||||
raise ValueError(f"failed to generate pinyin tokens for wake word: {keyword_text}")
|
||||
|
||||
return f"{' '.join(tokens)} @{keyword_text}"
|
||||
|
||||
def _receive_websocket_message(self) -> str | None:
|
||||
try:
|
||||
header = self._read_exact(2)
|
||||
except socket.timeout:
|
||||
return None
|
||||
|
||||
if not header:
|
||||
return None
|
||||
|
||||
first_byte, second_byte = header[0], header[1]
|
||||
opcode = first_byte & 0x0F
|
||||
masked = (second_byte & 0x80) != 0
|
||||
payload_length = second_byte & 0x7F
|
||||
|
||||
if payload_length == 126:
|
||||
payload_length = int.from_bytes(self._read_exact(2), "big")
|
||||
elif payload_length == 127:
|
||||
payload_length = int.from_bytes(self._read_exact(8), "big")
|
||||
|
||||
masking_key = self._read_exact(4) if masked else b""
|
||||
payload = self._read_exact(payload_length) if payload_length else b""
|
||||
|
||||
if masked and payload:
|
||||
payload = bytes(
|
||||
byte ^ masking_key[index % 4]
|
||||
for index, byte in enumerate(payload)
|
||||
)
|
||||
|
||||
if opcode == 0x8:
|
||||
raise ConnectionAbortedError("websocket closed by client")
|
||||
|
||||
if opcode == 0x9:
|
||||
self._send_websocket_frame(0xA, payload)
|
||||
return None
|
||||
|
||||
if opcode == 0xA:
|
||||
return None
|
||||
|
||||
if opcode != 0x1:
|
||||
return None
|
||||
|
||||
return payload.decode("utf-8")
|
||||
|
||||
def _read_exact(self, size: int) -> bytes:
|
||||
if size <= 0:
|
||||
return b""
|
||||
|
||||
chunks = bytearray()
|
||||
while len(chunks) < size:
|
||||
chunk = self.connection.recv(size - len(chunks))
|
||||
if not chunk:
|
||||
raise ConnectionResetError("websocket connection closed")
|
||||
chunks.extend(chunk)
|
||||
return bytes(chunks)
|
||||
|
||||
def _send_websocket_text(self, message: str) -> None:
|
||||
self._send_websocket_frame(0x1, message.encode("utf-8"))
|
||||
|
||||
def _send_websocket_frame(self, opcode: int, payload: bytes) -> None:
|
||||
header = bytearray()
|
||||
header.append(0x80 | opcode)
|
||||
|
||||
payload_length = len(payload)
|
||||
if payload_length < 126:
|
||||
header.append(payload_length)
|
||||
elif payload_length < 65536:
|
||||
header.append(126)
|
||||
header.extend(payload_length.to_bytes(2, "big"))
|
||||
else:
|
||||
header.append(127)
|
||||
header.extend(payload_length.to_bytes(8, "big"))
|
||||
|
||||
self.wfile.write(bytes(header) + payload)
|
||||
self.wfile.flush()
|
||||
|
||||
server = ThreadingHTTPServer((self.host, self.port), TestRuntimeHandler)
|
||||
server.daemon_threads = True
|
||||
return server
|
||||