> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cyberwave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Edge Workers

> Manage, author, and deploy Python worker modules that run on-device for ML inference and event-driven processing.

<Warning>
  **STUB DOCUMENT:** This page is intentionally minimal and will be expanded
  with deeper technical details in a future update.
</Warning>

## Overview

Edge workers are Python modules that run inside the `cyberwaveos/edge-ml-worker` Docker container on the edge device.
They use the Cyberwave SDK hook API (`@cw.on_frame`, `cw.models.load`, `cw.publish_event`)
to subscribe to sensor data and emit events.

There are two kinds of workers:

| Kind          | Filename prefix       | Managed by                                   |
| ------------- | --------------------- | -------------------------------------------- |
| **Custom**    | anything except `wf_` | You — add/edit freely                        |
| **Generated** | `wf_<uuid8>.py`       | Backend workflow sync — do not edit directly |

Worker files live in `{CONFIG_DIR}/workers/` (default `~/.cyberwave/workers/`).

***

## Container Image

The worker container image (`cyberwaveos/edge-ml-worker`) is pulled automatically during `cyberwave edge install`. To skip it:

```bash theme={null}
cyberwave edge install --without-workers
```

Edge-core manages the container lifecycle — you don't need to run it manually. The container mounts worker scripts and model weights from the host as read-only volumes.

| Tag          | Description                  |
| ------------ | ---------------------------- |
| `latest`     | Stable CPU release           |
| `dev`        | Development CPU build        |
| `latest-gpu` | Stable with NVIDIA CUDA      |
| `dev-gpu`    | Development with NVIDIA CUDA |

***

## CLI: Managing Worker Files

```bash theme={null}
cyberwave worker list                   # list installed workers (shows origin)
cyberwave worker add detect_people.py  # copy a file into the workers dir
cyberwave worker remove detect_people  # remove a worker (with or without .py)
cyberwave worker status                # show worker files + container state
cyberwave worker logs                  # stream worker container logs
cyberwave worker monitor               # live resource/throughput dashboard
cyberwave worker doctor                # diagnose silent failure modes
```

`cyberwave worker list` shows the **origin** of each worker:

```
Installed Workers (~/.cyberwave/workers)
┌───────────────────────────┬──────────┬────────────────────────────┬───────┐
│ Name                      │ Origin   │ File                       │ Size  │
├───────────────────────────┼──────────┼────────────────────────────┼───────┤
│ detect_people             │ custom   │ detect_people.py           │ 512 B │
│ wf_a1b2c3d4               │ workflow │ wf_a1b2c3d4.py             │ 1.2 kB│
└───────────────────────────┴──────────┴────────────────────────────┴───────┘
```

After adding or removing a worker file, restart the worker container so the
change takes effect:

```bash theme={null}
cyberwave-edge-core worker restart
```

***

## Tuning Log Verbosity

<Warning>
  **STUB SECTION:** This page will be expanded with a per-component log-routing
  diagram.
</Warning>

Worker containers and edge-driver containers have separate log-level knobs so
you can raise one to `DEBUG` without drowning in the other.

| Variable                     | Applies to                     | Default | Notes                                                                                                                              |
| ---------------------------- | ------------------------------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `CYBERWAVE_WORKER_LOG_LEVEL` | Worker container only          | `INFO`  | Wins over `CYBERWAVE_EDGE_LOG_LEVEL` when set. Raise to `DEBUG` to surface per-frame workflow trace lines (one per node per tick). |
| `CYBERWAVE_EDGE_LOG_LEVEL`   | Edge drivers + worker fallback | `INFO`  | Read by drivers and by the worker when `CYBERWAVE_WORKER_LOG_LEVEL` is unset.                                                      |

Edge-core auto-forwards any `CYBERWAVE_*` env var it sees in its own process
environment into each worker container at `docker run` time, so the same knob
works whether you set it in the systemd unit, the CLI install env, or the
shell. Workers print one INFO line at startup naming the resolved level and
the variable that supplied it (e.g. `Worker runtime starting with log
level=INFO (source=default)`), so you can verify the knob took effect without
inspecting the container.

In steady state, per-node workflow trace lines (`Workflow node started: <uuid>`,
`Workflow node finished: <uuid>`) log at `DEBUG`, not `INFO` — they
fire 20–30×/s on frame-driven workers and would otherwise dominate the log.
Errors (`Workflow node failed`) and per-execution boundaries (`Workflow
execution finished`) stay visible at the default level.

The generated workflow logger is hierarchical
(`cyberwave.workflows.<workflow-uuid>`), so you can also tune workflow
verbosity per-instance from inside a worker without touching the root logger:

```python theme={null}
import logging
logging.getLogger("cyberwave.workflows").setLevel(logging.WARNING)
```

***

## CLI: Monitoring Workers

<Warning>
  **STUB:** This section will be expanded with screenshots and deeper guidance.
</Warning>

`cyberwave worker monitor` opens a live-updating terminal dashboard for the running worker container.

```bash theme={null}
cyberwave worker monitor               # default 2s refresh
cyberwave worker monitor --update 1    # 1s refresh
cyberwave worker monitor -c <name>     # explicit container
```

The dashboard shows:

| Section              | Metrics                          | Source                        |
| -------------------- | -------------------------------- | ----------------------------- |
| **Resource Usage**   | CPU %, memory, network I/O, PIDs | `docker stats`                |
| **GPU**              | Utilization, memory, temperature | `nvidia-smi` (Linux only)     |
| **Zenoh Throughput** | Per-channel msgs/s, totals       | SDK instrumentation via Zenoh |
| **Worker Hooks**     | Per-hook frame count, drop rate  | SDK instrumentation via Zenoh |
| **Model Inference**  | Avg / P95 / P99 latency, count   | SDK instrumentation via Zenoh |

**Requirements:**

* Docker must be running on the host.
* For Zenoh throughput, hooks, and model metrics: `eclipse-zenoh` must be installed (`pip install eclipse-zenoh`).
* GPU metrics require NVIDIA drivers and `nvidia-smi` on Linux. macOS shows "N/A" for GPU.

***

## CLI: Managing Workflows

The `cyberwave workflow` command group lets you manage workflows from the terminal:

```bash theme={null}
cyberwave workflow list                # list workflows with status and target twin(s)
cyberwave workflow list --json         # JSON output
cyberwave workflow show                # interactive selection, shows nodes and twins
cyberwave workflow show <uuid>         # show specific workflow
cyberwave workflow create -n "Name"    # create a workflow
cyberwave workflow create --template motion-detection
cyberwave workflow activate            # activate (interactive selector)
cyberwave workflow deactivate          # deactivate (interactive selector)
cyberwave workflow sync                # sync workflow to edge device(s) via MQTT
cyberwave workflow sync --edge-active  # restrict selector to active edge workflows
cyberwave workflow sync <uuid>         # sync specific workflow
cyberwave workflow delete --yes        # delete (skip confirmation)
```

All subcommands accept `--base-url` / `-u` to override the API URL (e.g. `http://192.168.10.101:8000`). When a UUID is omitted, an interactive arrow-key selector is shown.

The `sync` command reads the workflow's `camera_frame` trigger nodes to discover which twin(s) it targets, then sends a `sync_workflows` MQTT command to each twin's edge node, which triggers an immediate HTTP pull of the latest worker files.

***

## Workflow-Generated Workers

When a workflow with a **Camera Frame** trigger and a connected **Call Model** node
is activated in the UI, the backend generates a `wf_*.py` worker module
for that workflow.

The edge node pulls these generated files during boot and on a periodic sync
(default every \~5 minutes). The file is written atomically — content-identical
files are left untouched to avoid spurious worker container restarts.

<Info>
  **Scope of the pull:** Edge Core only pulls `wf_*.py` files for the twins the
  operator selected at install time (the `twin_uuids` list in
  `~/.cyberwave/environment.json`). Workflows targeting twins that are *not* in
  that list stay off this edge even if those twins happen to share its
  fingerprint in metadata — re-run `cyberwave edge install` to change the
  selection.
</Info>

**Lifecycle:**

1. Activate a workflow in the UI (trigger: Camera Frame → Call Model → edge-compatible model).
2. Edge core syncs on next boot, `CYBERWAVE_WORKER_SYNC_INTERVAL_LOOPS` expiry, or immediately when a `sync_workflows` MQTT command arrives on the twin's command topic. That command is sent by `cyberwave workflow sync`, by `POST /twins/{uuid}/sync-workflows`, and — for `run_on_edge` workflows — automatically by the activate API itself, so flipping the UI toggle takes effect on the edge within seconds instead of up to one sync interval.
3. `wf_<uuid8>.py` appears in the workers directory.
4. The WorkerWatcher detects the new file and restarts the worker container.
5. The worker container loads the new module and activates the hook.

**Deactivating** a workflow removes the `wf_*.py` file on the next sync,
which triggers a container restart without that worker.

***

## Raising alerts from detections

`call_model` no longer publishes alerts on its own (the legacy `emit_event` config was removed). To raise alerts from a `camera_frame → call_model` chain, wire an explicit `send_alert` node downstream. Optional gating nodes preserve the previous `emit_mode` / cooldown semantics:

* **`detection_event_gate`** — class filtering plus `on_enter` (new classes only) / `on_change` (count changes) modes.
* **`timed_condition`** (`mode: debounce`, `cooldown_s: <seconds>`) — minimum delay between consecutive fires, replacing the old `cooldown_seconds`.

A typical edge chain compiles to a worker that calls `cw.publish_alert(...)` only when the gate + timer agree:

```
camera_frame → call_model → detection_event_gate → timed_condition → send_alert
```

`send_alert` whose immediate upstream is `call_model` defaults to `force=True` — the per-frame publish loop bypasses the backend's content-hash dedupe so two consecutive identical detections still raise distinct alerts. Set `parameters.force` to `false` on the `send_alert` node to opt back into the dedupe; when a gate or `timed_condition` sits between `call_model` and `send_alert` the heuristic backs off and the long-standing `force=False` default applies (the gate already debounces).

A `timed_condition` placed downstream of `call_model` *without* a trailing `send_alert` is inert — the legacy implicit-alert path it used to gate is gone. The compiler attaches a warning to the resulting compilation so the silent-skip case is visible in `cyberwave workflow sync` preflight output, but the worker still ships and runs inference; nothing is published until you wire the `send_alert`.

See the [`emit_event` migration table](/use-cyberwave/workflows#migrating-from-emit_event) for the field-by-field mapping (including the `force=True` default).

***

## Eject Pattern

A generated `wf_*.py` worker can be **ejected** into a custom worker when
you need to customise the logic beyond what the workflow graph supports.

```bash theme={null}
# 1. Copy the generated worker to a new custom name
cp ~/.cyberwave/workers/wf_a1b2c3d4.py \
   ~/.cyberwave/workers/my_detector.py

# 2. Edit the new custom worker
nano ~/.cyberwave/workers/my_detector.py

# 3. Deactivate the originating workflow in the UI
#    (this removes wf_a1b2c3d4.py on the next sync)
```

**Ownership after ejection:**

* The `wf_*.py` file is **deleted** on the next edge sync (because the workflow was deactivated).
* Your `my_detector.py` is **untouched** by edge sync — you own it.
* Edge sync never writes to files that do not start with `wf_`.

**Important:** do not edit `wf_*.py` files directly — they will be overwritten
on the next sync. Always eject first.

***

## Writing a Custom Worker

```python theme={null}
# ~/.cyberwave/workers/detect_people.py
# ``cw`` is injected by the worker runtime — no import needed.
# For IDE support, uncomment: from cyberwave import Cyberwave; cw: Cyberwave

model = cw.models.load("yolov8n")        # cached, safe to call at module level
twin_uuid = cw.config.twin_uuid

@cw.on_frame(twin_uuid, sensor="front")
def on_frame(frame, ctx):
    results = model.predict(frame, classes=["person"], confidence=0.5)
    for det in results:
        cw.publish_event(twin_uuid, "person_detected", {
            "confidence": getattr(det, "score", None),
            "frame_ts": ctx.timestamp,
        })
```

See the [Edge Workers SDK reference](/feature-reference/edge/drivers/edge-workers) for the full hook API.

***

## Generated Worker Format

For reference, here is what a generated worker looks like (with `on_enter` emit mode and cooldown):

```python theme={null}
"""
Generated edge worker for workflow: Alert on Person
Workflow UUID: a1b2c3d4-...

Auto-generated by WorkerCodegen — DO NOT EDIT.
To customise this worker, eject it:
  cp wf_a1b2c3d4.py alert_on_person.py
"""

import time

# ``cw`` is injected by the worker runtime — no import needed.
# For IDE support, uncomment: from cyberwave import Cyberwave; cw: Cyberwave

yolov8n = cw.models.load("yolov8n.pt")  # type: ignore[name-defined]  # noqa: F821

_last_emit_ts_0 = 0.0
_prev_labels_0 = set()

@cw.on_frame("twin-uuid", sensor="front", fps=5)  # type: ignore[name-defined]  # noqa: F821
def on_frame_a1b2c3d4_0(frame, ctx):
    """Camera frame handler for workflow 'Alert on Person'."""
    global _last_emit_ts_0, _prev_labels_0
    results = yolov8n.predict(frame, classes=["person"], confidence=0.5)
    current_labels = {
        getattr(d, 'label', getattr(d, 'class_name', ''))
        for d in results
    }
    new_labels = current_labels - _prev_labels_0
    if not new_labels:
        _prev_labels_0 = current_labels
        return
    now = time.monotonic()
    if now - _last_emit_ts_0 < 5.0:
        return
    _prev_labels_0 = current_labels
    _last_emit_ts_0 = now
    for det in results:
        label = getattr(det, 'label', getattr(det, 'class_name', ''))
        if label not in new_labels:
            continue
        cw.publish_event(  # type: ignore[name-defined]  # noqa: F821
            "twin-uuid",
            "person_detected",
            {
                "severity": "WARNING",
                "model": "yolov8n.pt",
                "confidence": getattr(det, 'score', None),
                "frame_ts": ctx.timestamp,
            },
        )
```

Generated workers follow the same contract as handwritten workers:
`cw` is injected as a builtin, models are loaded at module level, and events
are published with `cw.publish_event`. The `fps` parameter on the `@cw.on_frame`
decorator controls the frame sampling rate.

***

## Troubleshooting: `@cw.on_frame` hook never fires

<Warning>
  **STUB SECTION:** The most common reasons a worker container looks healthy but
  hooks report `frames: 0` indefinitely.
</Warning>

Run `cyberwave worker doctor` first. It runs both static ("paperwork") and
runtime ("actual traffic") checks.

**Static checks** surface the most common config-level failures:

1. **`cyberwave-edge-core` not installed.** Worker containers are managed by
   edge-core. Install with `cyberwave edge install`.
2. **Worker files not world-readable.** The container runs as a non-root user
   (UID 1001) and cannot read mode `0600` files. `cyberwave worker add`
   chmods files to `0644` automatically; files installed manually may need
   `chmod 0644 ~/.cyberwave/workers/*.py`.
3. **No co-located driver container.** Workers only receive frames from
   drivers on the same Zenoh session. Check
   `docker ps --filter name=cyberwave-driver-` on the edge host.
4. **Env drift between driver and worker.** Mismatched `CYBERWAVE_ENVIRONMENT`,
   `ZENOH_CONNECT`, `CYBERWAVE_DATA_BACKEND`, or `ZENOH_SHARED_MEMORY`
   produces a container that looks healthy but never publishes on the key
   the worker subscribes to. The doctor compares these across the running
   driver and worker containers, and flags legacy env-var spellings
   (e.g. `ZENOH_SHM_ENABLED`) that are silently ignored.

**Runtime checks** open a short Zenoh subscription (default 3 s) and
compare the actual traffic against the key-expressions your worker hooks
declare:

* **`zenoh-liveness`** — counts keys seen on the bus during the probe.
  Zero traffic means either no publisher is running or the doctor can't
  reach the router.
* **`keyexpr-alignment`** — for every `@cw.on_*(<twin>)` hook the scanner
  can resolve statically, asserts at least one matching key is being put
  on the bus. Diagnoses three common failure modes:
  * **Sensor mismatch** — hook listens on `sensor="default"` but the driver
    publishes on `sensor="color_camera"` (or vice-versa).
  * **Wrong twin** — the channel is flowing, but under a different twin
    UUID than the hook expects.
  * **No publisher** — nothing on the bus looks anything like the hook's
    expected key.
* **`keyexpr-scoping`** — flags publishers that put to non-canonical keys
  like `frames/color_camera` (no `cw/<twin>/...` prefix). Twin-scoped
  hooks silently drop these. The checker accepts only the canonical shape
  `cw/<twin-uuid>/data/<channel>[/<sensor>]`.

Skip the runtime probe with `--no-runtime` (pure paperwork) or extend
the window with `--window 6` when traffic is bursty. The probe requires
`eclipse-zenoh` on the host (`pip install --user eclipse-zenoh`); when
it's missing the runtime section degrades to an info message rather
than failing.

A healthy edge node shows `keyexpr-alignment ok` and \~25 fps on
`cw/<twin-uuid>/data/frames/<sensor>` in the `zenoh-liveness` top-keys
list, alongside a ticking `cw/_monitor/worker_stats` key.

### Manual fallback: inspect the bus with eclipse-zenoh

If the doctor can't reach the router (remote edge, unusual network) the
same sanity check is one script away:

```python theme={null}
import time, zenoh

cfg = zenoh.Config()
cfg.insert_json5("connect/endpoints", '["tcp/<edge-host>:7447"]')
session = zenoh.open(cfg)

seen = {}
def on(sample):
    seen[str(sample.key_expr)] = seen.get(str(sample.key_expr), 0) + 1

sub = session.declare_subscriber("**", on)
time.sleep(3)
for k, n in sorted(seen.items(), key=lambda kv: -kv[1]):
    print(f"{n:>6}  {k}")
```

You should see canonical keys shaped like
`cw/<twin-uuid>/data/frames/<sensor>`. If the key list is empty or uses a
different twin/sensor than your `@cw.on_frame(...)` call, fix the publisher
— the worker will never see those messages otherwise.