The Virtual Controller (virtual_controller) resolves a command string against the keyboard controller policy assigned to a digital twin and publishes the matching actuation on cyberwave/twin/{uuid}/command. The MQTT payload matches keyboard teleoperation (KeyboardTeleoperationController / useMQTTTwin.publishCommand) — nothing is hardcoded per robot; bindings come from the twin’s policy at runtime.
Palette category: Actuation. Runs on cloud (Celery MQTT) and edge (worker client.mqtt.publish). Locomotion uses the MQTT command topic, not WebRTC; media streaming stays on separate channels.
Voice-to-actuation pipeline
audio_track
→ audio_assistant (optional VAD)
→ wake_word_engine (optional)
→ call_model (STT)
→ fuzzy_matcher ← twin (control_actuations / control_labels)
→ conditional (optional, branch on matched)
→ virtual_controller
→ cyberwave/twin/{uuid}/command
| Step | Node | Key wiring |
|---|
| 1 | Twin | Select target twin → wire control_actuations → Fuzzy Matcher candidates |
| 2 | Call Model | STT result → Fuzzy Matcher query |
| 3 | Fuzzy Matcher | match_string → Virtual Controller command (when match is true) |
| 4 | Virtual Controller | Same twin as Twin node → publishes resolved actuation |
| Field | Type | Required | Description |
|---|
twin_uuid | string | Yes | Twin whose controller policy resolves commands. Set in inspector (Twin combobox). |
command | string | Yes | Label or actuation to dispatch. Wire from Fuzzy Matcher match_string or another upstream string. |
Command matching
The node accepts either form (normalization is shared with Fuzzy Matcher):
| Input | Resolves to actuation |
|---|
camera_right | camera_right |
Camera Right | camera_right |
W (single key) | binding for that key |
- Exact actuation (case / punctuation insensitive)
- Exact human label
- Keyboard key
- Substring containment (label inside longer STT phrase)
Bindings are collected from:
metadata.keyboard_bindings (actuation, label, key, optional playground data)metadata.display_config.widgets (button command, slider command)- Universal keyboard policies with
supports_locomotion and no bindings → default WASD locomotion actuations (move_forward, turn_left, …)
Inspector
- Digital Twin — combobox; policy is never hardcoded in the node.
- command — wire from upstream (typically Fuzzy Matcher
match_string).
Inspector preview
When a twin is selected, the panel shows:
| Section | Contents |
|---|
| Twin | Name, slug, UUID, environment, asset UUID |
| Controller | Policy name, type (teleop), control mode (command / velocity / joint_control), input device, asset slug, policy UUID |
| Forwardable commands | Scrollable list: actuation slug, label, optional key |
MQTT payload
Published to cyberwave/twin/{twin_uuid}/command:
{
"source_type": "tele",
"command": "move_forward",
"data": {},
"timestamp": 1710000000.0
}
| Field | Cloud workflow | Edge worker |
|---|
source_type | tele (live) or sim_tele (simulation execution target) | edge |
command | Resolved actuation slug (not the raw input label) | |
data | From binding playground when present, else {} | |
Outputs
| Field | Type | Description |
|---|
command | string | Resolved actuation that was last sent on MQTT |
available_commands | array | {actuation, label, key} for every forwardable binding |
twin_name | string | Twin display name |
controller_name | string | Policy name |
controller_type | string | e.g. teleop |
matched | boolean | true when resolution succeeded |
sent | boolean | Cloud: MQTT queued; edge: publish attempted |
Execution targets
| Target | Behavior |
|---|
| Cloud | run_virtual_controller_node → send_update_to_mqtt.delay |
| Edge | Emitter caches policy per twin (_CW_VC_POLICY_CACHE) via API on first use; client.mqtt.publish with QoS 0 |
entries) in process memory so repeat commands on the same twin do not re-fetch the policy every frame.
Recommended Conditional gate
Only run Virtual Controller when Fuzzy Matcher succeeded:
fuzzy_matcher.match → conditional (equals true) → virtual_controller
fuzzy_matcher.match_string into virtual_controller.command on the true branch.
Dependencies
| Layer | Requirement |
|---|
| Compile server | No ML extras. paho-mqtt and httpx are base Django/SDK deps. |
| Edge worker | Base cyberwave — client.twins.get, controller-policies API, client.mqtt.publish |
| edge-sync | No model_requirements (no ML weights) |
rapidfuzz is only required on the Fuzzy Matcher node upstream, not on Virtual Controller.
Full voice-pipeline matrix: Edge workflow dependencies.
Controller policy examples
Command-mode UGV policies (control_mode: "command") define bindings like:
{
"key": "W",
"label": "Forward",
"actuation": "move_forward",
"continuous": true
}
move_forward on MQTT when the pipeline passes "Forward", "move_forward", or fuzzy-matched variants. See seeded policies in seed_controllers.py (e.g. controller:ugv-beast:v1).
Universal keyboard (controller:keyboard-locomotion:v1) has no per-asset bindings in seed data; the node uses built-in locomotion defaults when supports_locomotion is true.
- Twin — supplies Fuzzy Matcher
candidates from control_actuations / control_labels
- Fuzzy Matcher — maps noisy STT (uncertain string) →
match_string from a source-of-truth string or array
- Call Model — STT text → Fuzzy Matcher
query
- Conditional — gate on
matched
- Send MQTT — generic publish; Virtual Controller is purpose-built for controller actuations
