Native MQTT/UDP integration

The server can accept device MQTT control messages and encrypted UDP audio without running xiaozhi-mqtt-gateway. Native MQTT is optional. The existing Gateway path and direct WebSocket path remain supported.

Compatibility modes

Mode	Device control	Device audio	Server business runtime
Direct WebSocket	WebSocket JSON	WebSocket binary	Shared runtime
MQTT Gateway	MQTT via Gateway WebSocket	UDP via Gateway WebSocket	Shared runtime
Native MQTT	MQTT 3.1.1 QoS 0	AES-128-CTR UDP	Shared runtime

Native MQTT keeps the physical MQTT connection alive between conversations. Each device hello creates a new logical conversation and UDP encryption session. goodbye ends only that logical conversation; MQTT is closed only by MQTT disconnect, duplicate client takeover, keepAlive expiry, server shutdown, or a protocol/authentication error.

Enable Native MQTT

Native MQTT is disabled by default. Both the protocol and server switches must be enabled:

protocols:
  enabled_protocols: [websocket, mqtt]
  websocket_enabled: true
  mqtt_enabled: true

mqtt_server:
  enabled: true
  host: 0.0.0.0
  port: 1883
  udp_port: 1883
  udp_bind_host: ""
  public_endpoint: mqtt.example.com
  signature_key: replace-with-a-strong-secret
  max_connections: 1000
  max_pending_connections: 128
  heartbeat_interval: 30
  max_payload_size: 8192
  message_queue_size: 128
  business_ready_timeout: 30
  goodbye_timeout: 1
  close_timeout: 2

# Includes the common model and any Agent-specific local ASR variants.
shared_asr_max_models: 3

host is the local MQTT listen address. public_endpoint must be reachable by the device and may be either host or host:port; manager-api normalizes it and does not append the port twice. When host is a wildcard and public_endpoint is a local IPv4 address, the UDP socket binds that address so reply datagrams use the same source IP advertised to the device. Set udp_bind_host only when the local UDP bind address must differ from the advertised endpoint (for example, behind NAT); leaving it empty enables the automatic behavior. Open the configured MQTT TCP and UDP ports in the firewall.

When configuration is read from manager-api, configure the equivalent protocols.* and mqtt_server.* system parameters. The OTA response prefers a valid Native endpoint when Native is enabled. If Native is disabled or invalid, manager-api falls back to server.mqtt_gateway. If neither is configured, OTA returns only WebSocket information.

The current firmware persists the OTA mqtt object. The UDP server, key and nonce used by Native mode are negotiated in the MQTT hello response, so manager-api does not send a separate top-level OTA udp object.

Long-lived Native connections refresh Agent-private components at logical hello boundaries. Local ASR variants are shared by effective configuration instead of loaded once per device. shared_asr_max_models bounds resident models; when capacity is exhausted, the replacement runtime is rejected and the previous healthy runtime remains active. The default capacity reserves one transition slot so a single Agent can replace an active private local model without dropping its healthy runtime first. Idle variants are cached and evicted before another model is loaded; resident models never exceed this limit.

MQTT application packets are dispatched in order outside the socket read loop, so PINGREQ remains responsive while a logical Hello waits for private runtime refresh. business_ready_timeout bounds that wait and closes an unusable connection instead of leaving the device in a half-negotiated session. max_connections limits authenticated client identities, while max_pending_connections separately bounds sockets that have not completed CONNECT. A connection using an existing client id may therefore replace its previous owner even when active capacity is full. The server sends the success CONNACK only after the previous owner has been reclaimed. goodbye_timeout and close_timeout bound best-effort device reset and physical resource cleanup so a stalled peer cannot block heartbeat or shutdown indefinitely.

Authentication and topics

manager-api generates the device credentials and topics:

client id: <group>@@@<mac_with_underscores>@@@<mac_with_underscores>
publish topic: device-server
subscribe topic: devices/p2p/<mac_with_underscores>
username: Base64-encoded JSON metadata
password: Base64(HMAC-SHA256(client_id + "|" + username, signature key))

Use the same secret for manager-api mqtt_server.signature_key and xiaozhi-server mqtt_server.signature_key. The legacy server.mqtt_signature_key remains a fallback for Gateway compatibility.

Native MQTT accepts MQTT 3.1.1 CONNECT, SUBSCRIBE, PUBLISH QoS 0, PING and DISCONNECT. Unsupported protocol versions, QoS levels, malformed packets, or invalid credentials close the physical connection.

Session and UDP flow

Device connects and subscribes over MQTT.
Server validates CONNECT credentials and replaces any older connection with the same client id.
Device publishes protocol version 3 hello.
Server waits until private configuration and business components are ready.
Server creates a logical session and returns the UDP server, AES key, nonce, output audio parameters and session id.
The first valid UDP packet from the MQTT peer binds the UDP source tuple for that logical session. Source changes are rejected until the next hello.
MQTT JSON and decrypted Opus frames enter the same message processors used by WebSocket and Gateway connections.
Server goodbye returns the device to Idle and finalizes conversation tasks while preserving the MQTT connection.

The 16-byte UDP header is also the AES-CTR nonce:

Bytes	Field
0	packet type (`1`)
1	reserved
2..3	encrypted payload length, big endian
4..7	connection id, big endian
8..11	timestamp, big endian
12..15	sequence, big endian

Datagrams with an invalid type, header, payload length, connection id, source address or stale sequence are discarded. Sequence gaps are tolerated because UDP can lose packets; late and duplicate packets are discarded. AES-CTR provides confidentiality but not integrity, so Native MQTT and UDP should be exposed only with a strong signature key and appropriate network controls.

Verification

From main/xiaozhi-server:

python -m compileall -q .

From main/manager-api with JDK 21:

mvn clean package

Before deployment, verify Native and Gateway separately with real hardware: multiple conversations, TTS interruption, exit to Idle, duplicate reconnect, server restart, short network loss and a 30-60 minute keepAlive soak.

6.7 KiB Raw Blame History