Overview
Edge workers are Python modules that run inside thecyberwaveos/edge-ml-worker Docker container on the edge device.
They use the Cyberwave SDK hook API (@cw.on_frame, cw.models.load, cw.publish_event)
to subscribe to sensor data and emit events.
There are two kinds of workers:
Worker files live in
{CONFIG_DIR}/workers/ (default ~/.cyberwave/workers/).
Container Image
The worker container image (cyberwaveos/edge-ml-worker) is pulled automatically by edge-core on first startup, in the background. This keeps cyberwave edge install fast and non-blocking — there is no need to wait for the image download during installation.
Edge-core manages the container lifecycle — you don’t need to run it manually. The container mounts worker scripts and model weights from the host as read-only volumes.
CLI: Managing Worker Files
cyberwave worker list shows the origin of each worker:
Tuning Log Verbosity
Worker containers and edge-driver containers have separate log-level knobs so you can raise one toDEBUG without drowning in the other.
Edge-core auto-forwards any
CYBERWAVE_* env var it sees in its own process
environment into each worker container at docker run time, so the same knob
works whether you set it in the systemd unit, the CLI install env, or the
shell. Workers print one INFO line at startup naming the resolved level and
the variable that supplied it (e.g. Worker runtime starting with log level=INFO (source=default)), so you can verify the knob took effect without
inspecting the container.
In steady state, per-node workflow trace lines (Workflow node started: <uuid>,
Workflow node finished: <uuid>) log at DEBUG, not INFO — they
fire 20–30×/s on frame-driven workers and would otherwise dominate the log.
Errors (Workflow node failed) and per-execution boundaries (Workflow execution finished) stay visible at the default level.
The generated workflow logger is hierarchical
(cyberwave.workflows.<workflow-uuid>), so you can also tune workflow
verbosity per-instance from inside a worker without touching the root logger:
CLI: Monitoring Workers
cyberwave worker monitor opens a live-updating terminal dashboard for the running worker container.
Requirements:
- Docker must be running on the host.
- For Zenoh throughput, hooks, and model metrics:
eclipse-zenohmust be installed (pip install eclipse-zenoh). - GPU metrics require NVIDIA drivers and
nvidia-smion Linux. macOS shows “N/A” for GPU.
CLI: Managing Workflows
Thecyberwave workflow command group lets you manage workflows from the terminal:
--base-url / -u to override the API URL (e.g. http://192.168.10.101:8000). When a UUID is omitted, an interactive arrow-key selector is shown.
The sync command reads the workflow’s camera_frame trigger nodes to discover which twin(s) it targets, then sends a sync_workflows MQTT command to each twin’s edge node, which triggers an immediate HTTP pull of the latest worker files.
Workflow-Generated Workers
When a workflow with a Camera Frame trigger and a connected Call Model node is activated in the UI, the backend generates awf_*.py worker module
for that workflow.
The edge node pulls these generated files during boot and on a periodic sync
(default every ~5 minutes). The file is written atomically — content-identical
files are left untouched to avoid spurious worker container restarts.
Scope of the pull: Edge Core only pulls
wf_*.py files for the twins the
operator selected at install time (the twin_uuids list in
~/.cyberwave/environment.json). Workflows targeting twins that are not in
that list stay off this edge even if those twins happen to share its
fingerprint in metadata — re-run cyberwave edge install to change the
selection.- Activate a workflow in the UI (trigger: Camera Frame → Call Model → edge-compatible model).
- Edge core syncs on next boot,
CYBERWAVE_WORKER_SYNC_INTERVAL_LOOPSexpiry, or immediately when async_workflowsMQTT command arrives on the twin’s command topic. That command is sent bycyberwave workflow sync, byPOST /twins/{uuid}/sync-workflows, and — forrun_on_edgeworkflows — automatically by the activate API itself, so flipping the UI toggle takes effect on the edge within seconds instead of up to one sync interval. wf_<uuid8>.pyappears in the workers directory.- The WorkerWatcher detects the new file and restarts the worker container.
- The worker container loads the new module and activates the hook.
run_on_edge off publishes a
surgical remove_workflow_worker MQTT command to each affected twin
command topic, naming the specific worker file(s) to unlink. Edge core
deletes the matching wf_*.py and immediately reconciles the worker
container lifecycle — the workflow stops running on the device within
seconds rather than waiting up to one full periodic interval (~5 minutes)
plus the bulk-sync two-strikes cleanup. The same path is used for the
cascade cases (workspace or environment deletion that removes the
workflow). The periodic reconcile remains the correctness backstop if
the MQTT publish is dropped.
Raising alerts from detections
call_model no longer publishes alerts on its own (the legacy emit_event config was removed). To raise alerts from a camera_frame → call_model chain, wire an explicit send_alert node downstream. Optional gating nodes preserve the previous emit_mode / cooldown semantics:
detection_event_gate— class filtering pluson_enter(new classes only) /on_change(count changes) modes.timed_condition(mode: debounce,cooldown_s: <seconds>) — minimum delay between consecutive fires, replacing the oldcooldown_seconds.
cw.publish_alert(...) only when the gate + timer agree:
send_alert whose immediate upstream is call_model defaults to force=True — the per-frame publish loop bypasses the backend’s content-hash dedupe so two consecutive identical detections still raise distinct alerts. Set parameters.force to false on the send_alert node to opt back into the dedupe; when a gate or timed_condition sits between call_model and send_alert the heuristic backs off and the long-standing force=False default applies (the gate already debounces).
A timed_condition placed downstream of call_model without a trailing send_alert is inert — the legacy implicit-alert path it used to gate is gone. The compiler attaches a warning to the resulting compilation so the silent-skip case is visible in cyberwave workflow sync preflight output, but the worker still ships and runs inference; nothing is published until you wire the send_alert.
See the emit_event migration table for the field-by-field mapping (including the force=True default).
Eject Pattern
A generatedwf_*.py worker can be ejected into a custom worker when
you need to customise the logic beyond what the workflow graph supports.
- The
wf_*.pyfile is deleted on the next edge sync (because the workflow was deactivated). - Your
my_detector.pyis untouched by edge sync — you own it. - Edge sync never writes to files that do not start with
wf_.
wf_*.py files directly — they will be overwritten
on the next sync. Always eject first.
Writing a Custom Worker
Generated Worker Format
For reference, here is what a generated worker looks like (withon_enter emit mode and cooldown):
cw is injected as a builtin, models are loaded at module level, and events
are published with cw.publish_event. The fps parameter on the @cw.on_frame
decorator controls the frame sampling rate.
Troubleshooting: @cw.on_frame hook never fires
Run cyberwave worker doctor first. It runs both static (“paperwork”) and
runtime (“actual traffic”) checks.
Static checks surface the most common config-level failures:
cyberwave-edge-corenot installed. Worker containers are managed by edge-core. Install withcyberwave edge install.- Worker files not world-readable. The container runs as a non-root user
(UID 1001) and cannot read mode
0600files.cyberwave worker addchmods files to0644automatically; files installed manually may needchmod 0644 ~/.cyberwave/workers/*.py. - No co-located driver container. Workers only receive frames from
drivers on the same Zenoh session. Check
docker ps --filter name=cyberwave-driver-on the edge host. - Env drift between driver and worker. Mismatched
CYBERWAVE_ENVIRONMENT,ZENOH_CONNECT,CYBERWAVE_DATA_BACKEND, orZENOH_SHARED_MEMORYproduces a container that looks healthy but never publishes on the key the worker subscribes to. The doctor compares these across the running driver and worker containers, and flags legacy env-var spellings (e.g.ZENOH_SHM_ENABLED) that are silently ignored.
zenoh-liveness— counts keys seen on the bus during the probe. Zero traffic means either no publisher is running or the doctor can’t reach the router.keyexpr-alignment— for every@cw.on_*(<twin>)hook the scanner can resolve statically, asserts at least one matching key is being put on the bus. Diagnoses three common failure modes:- Sensor mismatch — hook listens on
sensor="default"but the driver publishes onsensor="color_camera"(or vice-versa). - Wrong twin — the channel is flowing, but under a different twin UUID than the hook expects.
- No publisher — nothing on the bus looks anything like the hook’s expected key.
- Sensor mismatch — hook listens on
keyexpr-scoping— flags publishers that put to non-canonical keys likeframes/color_camera(nocw/<twin>/...prefix). Twin-scoped hooks silently drop these. The checker accepts only the canonical shapecw/<twin-uuid>/data/<channel>[/<sensor>].
--no-runtime (pure paperwork) or extend
the window with --window 6 when traffic is bursty. The probe requires
eclipse-zenoh on the host (pip install --user eclipse-zenoh); when
it’s missing the runtime section degrades to an info message rather
than failing.
A healthy edge node shows keyexpr-alignment ok and ~25 fps on
cw/<twin-uuid>/data/frames/<sensor> in the zenoh-liveness top-keys
list, alongside a ticking cw/_monitor/worker_stats key.
Manual fallback: inspect the bus with eclipse-zenoh
If the doctor can’t reach the router (remote edge, unusual network) the same sanity check is one script away:cw/<twin-uuid>/data/frames/<sensor>. If the key list is empty or uses a
different twin/sensor than your @cw.on_frame(...) call, fix the publisher
— the worker will never see those messages otherwise.