RL Tasks

An RL Task is a workspace-scoped, optionally environment-linked entity that packages a runnable reinforcement-learning plugin. Cyberwave generates the SceneCfg from a linked environment — the same MuJoCo scene used everywhere else in the platform — and you bring the rest of the MDP and RL code as an arbitrary tree of source files. Exporting an RL task produces a ZIP that drops directly into cyberwave-rl/tasks/<task_id>/, where the existing registry discovers it and make run_cfg runs it like any other task.

Cyberwave currently supports basic mjlab: exported tasks are mjlab-compatible (scene config, action and observation terms, skrl training). We’re planning to release a richer Cyberwave mjlab extension with useful helpers in the future.

Why a separate entity?

RL tasks are deliberately kept independent of Environments. Deleting an environment detaches any linked RL tasks (sets environment to none) instead of cascading the delete, so a task keeps its source tree even after the world it was originally trained against is gone. This matches how researchers actually iterate: the SceneCfg is a moving target, while the reward, action, and observation specs are the long-lived artefacts.

Lifecycle

Create

Create an RL task from /rl-tasks, or from File → Export → Create RL task from this environment in the environment editor.

Import or author source

Import an existing task ZIP (for example an in-tree task from cyberwave-rl/tasks/) to seed the source tree, or write files directly in the Files tab.

Configure entrypoints

Set the entrypoint paths — scene_cfg_path, env_cfg_path / env_cfg_factory, rl_cfg_path / rl_cfg_factory, and an optional registry_path. Cyberwave never forces a particular from … import … statement; the import wiring is yours.

Regenerate SceneCfg

Regenerate the Cyberwave-generated SceneCfg module from the linked environment. This also rewrites the spec-derived files (scene_bindings.py, actions.py, observations.py, task_export/, and the strict task spec).

Export and run

Download the ZIP and drop the folder into cyberwave-rl/tasks/. Run it with make run_cfg arg_overrides="--task=<id>" exactly like the in-tree examples.

Round-trip

Edit locally, then re-import the same folder back into Cyberwave to keep the editor in sync. The editor stays authoritative for Cyberwave-generated files.

Editor tabs

The RL task editor is organized into tabs that map onto the export contract:

Tab	What it authors
Overview	Task metadata, linked environment, and entrypoint paths.
Scene entities	The articulation entities, their simulation actuators, cameras, and contact sensors. See RL task scene sensors.
Actions	The policy-controlled action terms and their control-mode presets.
Observations	Grouped observation terms for the policy and critic heads.
RL config	skrl backend, algorithm, graph-based network builder, and trainer parameters.
Files	The full source tree, including read-only Cyberwave-generated files.
Runtime	Runtime target, accelerator, bundle version, and “Train locally” snippets.
Policies	Upload trained checkpoints, publish controllers, and run inference.

Author from Python (SDK)

Everything the editor tabs produce can also be authored from the Python SDK — useful for reproducible setup scripts. The RLTaskClient covers every tab via its set_*_spec methods, so a script and the dashboard produce the same specs. The Actions, Observations, and RL config tabs also ship make_* builder helpers; the command tabs are authored as plain dicts passed to set_training_command_spec / set_inference_command_spec (see step 4b below). The example below is the OpenArm pick-up task: a fixed-base arm, a table, a free-floating cube, and a wrist depth camera. It is the same scene used in the full RL task walkthrough, trimmed to the essentials.

from cyberwave import Cyberwave
from cyberwave.rl_tasks import (
    RLTaskClient,
    make_action_spec,
    make_position_delta_action,
    make_observation_spec,
    make_joint_observation,
    make_previous_action_observation,
    make_default_ppo_rlmodule_config,
)

cw = Cyberwave()            # reads CYBERWAVE_BASE_URL + CYBERWAVE_API_KEY
rl = RLTaskClient(cw)
task_uuid = "<rl-task-uuid>"   # created via POST /api/v1/rl-tasks

# Match the arm + finger joints (twin-namespaced at runtime).
ARM_JOINTS = [r".*__openarm_left_joint[1-7]$", r".*__openarm_left_finger_joint[12]$"]

# 1. Scene entities: which twin is which mjlab entity.
rl.assign_articulation_entity(
    task_uuid, twin_uuid="<openarm-twin-uuid>", name="robot", base_type="fixed",
    actuators=[{"type": "xml_position", "target_names_expr": ARM_JOINTS}],
)
rl.assign_rigid_entity(task_uuid, twin_uuid="<table-twin-uuid>", name="support_box", base_type="fixed")
rl.assign_rigid_entity(task_uuid, twin_uuid="<cube-twin-uuid>", name="cube", base_type="free")

# 2. Actions: one position-delta term driving the arm + fingers.
rl.set_action_spec(task_uuid, make_action_spec([
    make_position_delta_action("joint_delta", entity="robot", target_names_expr=ARM_JOINTS, scale=0.04),
]))

# 3. Observations: a single "policy" group the actor and critic share.
rl.set_observation_spec(task_uuid, make_observation_spec({"policy": [
    make_joint_observation("joint_pos", kind="position", entity="robot", target_names_expr=ARM_JOINTS),
    make_joint_observation("joint_vel", kind="velocity", entity="robot", target_names_expr=ARM_JOINTS),
    make_previous_action_observation("actions"),
]}))

# 4. RL config: PPO with the default shared-MLP network.
rl.set_rl_config_spec(task_uuid, make_default_ppo_rlmodule_config())

# 4b. (Optional) Command specs: a target generator your observations read.
#     Opt-in and split by lifecycle — tasks with no command fall back to a
#     null manager. The TRAINING spec drives mjlab's CommandManager (sampled
#     goals, resampled each window); the INFERENCE spec drives the live/
#     inference command surface (externally supplied targets).
#     A code-backed term names its uploaded source module by dotted path
#     (``commands.viewpoint`` → ``commands/viewpoint.py``) and the class it binds.
rl.set_training_command_spec(task_uuid, {
    "schema_version": 1,
    "commands": [{
        "name": "viewpoint", "type": "custom",
        "module": "commands.viewpoint", "symbol": "ViewpointCommandCfg",
        "resampling_time_range": [4.0, 4.0],
    }],
})
rl.set_inference_command_spec(task_uuid, {
    "schema_version": 1,
    "commands": [{
        "name": "viewpoint", "type": "custom",
        "module": "commands.viewpoint", "symbol": "ViewpointCommandCfg",
        "payload_schema": {"target_position": "vec3"},
        "source": {"kind": "mqtt"},
    }],
})

# 5. Regenerate the Cyberwave-generated files from the specs + linked environment.
rl.regenerate_scene_cfg(task_uuid)

rl.get_orchestration_hints(task_uuid) returns the scene-derived dropdown data (articulated entities, actuator groups, actuated vs passive joint names, available sensors) the editor uses to populate its pickers — call it first to discover the exact entity and joint names before authoring specs.

For the complete version — scaffolding the workspace, environment, and twins, wiring a wrist depth camera and contact sensors, training, and deploying the policy — follow the RL task walkthrough.

Actions

Action terms are authored from five control-mode presets:

Preset	Spec value	Formula	Requires actuator
Position delta	`position_delta`	`target = q_now + baseline_delta + action × scale`	`xml_position`
Smoothed position delta	`smoothed_position_delta`	same as `position_delta`, delta ramped linearly over the decimation substeps	`xml_position`
Position target	`position`	`target = action × scale + offset` (anchored at joint home)	`xml_position`
Velocity	`velocity`	`target = action × scale + offset`	`xml_velocity` or `xml_position`
Effort / torque	`effort`	`target = action × scale + offset`	`xml_motor`

position, velocity, and effort mirror the Isaac Lab / mjlab JointPositionActionCfg / JointVelocityActionCfg / JointEffortActionCfg math one-to-one — exactly the target = action × scale + offset that mjlab implements in BaseAction.process_actions. position_delta is Cyberwave’s own preset: baseline_delta is an explicit per-step bias baked into the target, so an action of zero can drift the joints toward a goal pose. smoothed_position_delta is the same delta, but spread evenly across the policy step’s physics substeps for a smoother velocity profile. A single Use default joint offset checkbox mirrors mjlab’s use_default_offset: when on, mjlab replaces offset with each joint’s default_joint_pos (or default_joint_vel) at runtime so action zero returns the joint to its home pose. Every numeric field (scale, offset, baseline_delta) has a Per joint toggle that switches it between a uniform scalar and a {joint_name: value} mapping. The backend rejects per-joint keys that aren’t in the action’s selected joints, so typos surface at save time. Legacy symbolic offsets ("default", "zero", "none") keep round-tripping unchanged. The Scene tab is where you enable simulation actuators for an entity (which Xml*ActuatorCfg wrapper drives each joint); the Actions tab is where you pick the policy-controlled subset of those actuators. Each action row renders a schema-backed joint table (Joint, Control type, Position min/max, Max velocity, Max torque/force, KP, KD, Selected) with Select all / Clear. Passive joints, spring-only actuators, and joints driven by an incompatible wrapper appear greyed out with a hint. Incompatible presets are hidden rather than shown disabled. Concrete joint names are stored in target_names_expr; a collapsed Advanced — custom regex disclosure lets imported or legacy specs keep regex patterns.

One action per joint. Every (entity, joint) pair can be assigned to at most one action term — the policy emits a single value per actuated DOF. The Actions tab enforces this in three places: joints already controlled by another row render disabled with an “owned by …” pill; Select all only picks unclaimed joints; and PUT /api/v1/rl-tasks/{uuid}/actions rejects overlapping payloads with an error naming both action names and a sample of overlapping joints. To split a robot across two terms (for example arm + gripper), give each term a disjoint subset of joints.

Observations

Observation terms are organized into named groups (policy, critic, …) and drawn from a catalog:

Term	Description
`joint_position`	Joint positions.
`joint_velocity`	Joint velocities.
`joint_effort`	Joint efforts / torques.
`link_position`	Body position; pick among `x` / `y` / `z` components.
`link_pose`	Body pose: `x` / `y` / `z` plus a single `quat` token keeping the full `(qw, qx, qy, qz)` tuple grouped.
`link_velocity`	Body twist: `x` / `y` / `z` linear plus `rx` / `ry` / `rz` angular.
`camera`	Depth or color from a sensor configured on a scene entity.
`previous_action`	The previous action tensor.
`custom`	User `module.symbol` plus literal kwargs.

Link terms pick bodies from a scene-derived checkbox list and expose Components toggles so you only export the axes the policy needs. Joint observation terms reuse the shared Joint target picker and — unlike actions — may include both actuated and passive joints, with “All actuated”, “All joints”, “Passive joints”, and “Pick specific joints” exposed directly; “Custom regex” sits behind a small Advanced disclosure.

Shared policy/critic (default)
Separate policy and critic

One group is authored, and the runtime aliases the missing critic group onto policy so both heads see the same tensor.

Observation rows show the term type as a compact chip next to the editable name; the type is fixed after creation (remove and re-add to change it).

RL config

The RL config tab persists the skrl backend, the algorithm (PPO or SAC), a structured graph-based Network builder, the trainer parameters (timesteps, intervals, seed), and the algorithm-config knobs (learning rate, discount factor, ratio clip, batch size, …). Switching algorithm seeds matching defaults. Experiment paths (experiment.name / experiment.directory) default to values derived from the RL task slug and are shown read-only; toggle Override experiment paths to write custom values.

Network builder

The network builder is graph-first: it renders the network as a DAG of typed nodes on an inline canvas, with one canvas per role for the active topology.

shared uses the same architecture for both heads. For PPO this yields a true shared-trunk model (SharedRLModelCfg); for SAC it instantiates two independent modules (policy + Q) with the same block spec, because SAC heads can’t share weights.
separated lets you author one graph per role (policy / value for PPO, policy / q for SAC).

Top-level presets are graph templates (for example PPO – shared MLP, SAC – separated MLP) that materialise the equivalent observation → trunk → head topology directly on the canvas. The palette covers inputs (observation_input), dense layers (mlp, residual_mlp), transforms (activation, layer_norm), recurrent layers (rnn, gru, lstm), vision (cnn_backbone with NatureCNN / IMPALA presets), merge nodes (concat, residual_add), and a role/algorithm-aware Heads section that surfaces actor_head + value_head for PPO and actor_head + q_head for SAC. Drag the canvas background to pan, drag a node to reposition it (positions persist in graphs[role].layout), and click an edge to delete it. Clicking a node opens an inline inspector below the canvas with the outputs picker (features_node_id / head_node_id) and the per-node editor. An Open graph builder button pops the same graph into a fullscreen dialog for serious authoring. observation_input nodes pick a group and a subset of term names from the Observations tab. Each term shows read-only tensor metadata — an image-like / flat badge and a short kind label — so the builder can route image-like terms into a cnn_backbone and reject flat terms feeding one. Gaussian log-std controls (initial_log_std, min_log_std, max_log_std) live in the actor head inspector and apply only to the stochastic Gaussian policy.

Legacy {kind, hidden_units, activation} and architecture: "mlp" / "residual_mlp" / "custom" specs are transparently promoted to a graph on load, so older tasks just render. At runtime, from runners.skrl.network_builder import build_skrl_model_cfg maps the persisted spec onto rlmodule.MlpCfg / ResidualMlpCfg / SharedRLModelCfg / SeparatedRLModelCfg for the skrl runner.

Verify before saving. The Verify button next to Save RL config runs the same validation the save path uses — without persisting — and reports the first problem (for example “mlp needs exactly 1 input; got 2 — merge multiple branches with a Concat node first”). The inline canvas also flags structural issues live as red error badges. A passing Verify guarantees Save will succeed.

Runtime: target, accelerator, and bundle

Open the Runtime tab to choose how the exported plugin runs:

Pick a runtime_target — cyberwave-rl-experimental (plain local Python, no Docker) or cyberwave-rl (the dockerized stack).
Pick a runtime_accelerator — gpu is the default; cpu is the smoke-friendly opt-in.
Pin a single runtime_bundle version (default latest). The bundle ships mjlab + the chosen runtime + skrl together, so callers only ever pin one version.
Configure the entrypoints (scene_cfg_path, env_cfg_path / env_cfg_factory, rl_cfg_path / rl_cfg_factory, optional registry_path) inline.
Copy the matching Train locally snippet for the selected target and accelerator.

Both runtimes consume the same exported plugin contract, and the runtime spec (target + accelerator + versions + policy interface) is persisted on the RLTask and forwarded to checkpoints and inference dispatch with a task → checkpoint → launch merge order.

cyberwave-rl (Docker)
cyberwave-rl-experimental (local Python)

make start
make run_cfg arg_overrides="--task=<task_id>"

Real training expects a CUDA GPU.

# CPU smoke test
python -m train_from_cfg --device=cpu --num_envs=2 --max_steps=10

# Full GPU training (same defaults as cyberwave-rl)
python -m train_from_cfg

Policies and inference

The Policies tab is the post-training counterpart to the Runtime tab’s “Train locally” snippets: upload a trained checkpoint, publish it as a controller, and run inference. Inference deliberately does not run MuJoCo — the worker loops Cyberwave joint state through the policy and publishes commands over MQTT. See the dedicated Uploading Policies page for the full checkpoint upload, publish, and inference flow, and Online Controllers for serving a published policy.

What’s in the exported ZIP?

The export is a minimal Python package that drops into cyberwave-rl/tasks/<task_id>/:

__init__.py — generated header that imports your registry (or the default one).
Your source tree — exactly as you authored it, with arbitrary nesting and free file structure.
scene_cfg.py (at the configured scene_cfg_path) — the Cyberwave-generated SceneCfg, regenerated from the linked environment.
scene_bindings.py — semantic role / twin / sensor aliases plus ACTION_SPEC, OBSERVATION_SPEC, and RL_CONFIG_SPEC.
actions.py (make_actions()) and observations.py (make_observations()) — the named wrappers.
task_export/ — the runtime adapters those wrappers import.
cyberwave_task_spec.py — the strict, canonical Python interchange for scene entities.
cyberwave_export.json — the export manifest (records runtime target, accelerator, and version pins).
MuJoCo assets — one merged models/mujoco_scene.xml, one per-entity models/<name>.xml carved out of it at build time (so each EntityCfg.spec_fn can MjSpec.from_file its own slice), and the shared models/assets/ meshes.
registry.py — a fallback registry wiring env_cfg / rl_cfg factories into register_cyberwave_task, used only when you don’t ship your own registry_path.

Cyberwave-generated vs user-written

scene_bindings.py is the stable interface between Cyberwave-generated scene metadata and user-written training code. It exposes role aliases (ROBOT_ENTITY, OBJECT_ENTITY, SUPPORT_ENTITY), twin namespace prefixes, namespaced joint-position regex keys, actuator target regexes, camera attach paths, contact sensor names, and a body_pattern(entity, suffix) helper. Consume these constants from env_cfg.py so renaming a twin or moving a camera in the composer regenerates the bindings without touching MDP code. The editor owns Scene entities, Actions, Observations, RL config, and all generated artefacts. Task-local Python — reward functions, reset / termination logic, metrics, command shaping, and custom observation functions — stays user-written under task_sources/. Three execution-only modules in cyberwave-rl consume the persisted specs so user-written files only have to provide rewards, resets, terminations, commands, and custom observation functions:

from task_export.action_specs import make_action_cfgs          # ACTION_SPEC → {name: ActionTermCfg}
from task_export.observation_specs import make_observation_groups  # OBSERVATION_SPEC → {group: ObservationGroupCfg}
from task_export.rl_config_specs import make_rl_cfg_from_spec   # RL_CONFIG_SPEC → ReinforcementLearningCfg

ZIP import is intentionally document-only for the generated paths: scene_cfg.py, scene_bindings.py, and cyberwave_task_spec.py are skipped on import and the response carries an explicit warning. Scene entities are not auto-restored from a bundled cyberwave_task_spec.py — re-compose them in the editor. The export ZIP always carries fresh bytes for the Cyberwave-generated files, so re-running the composer (or hitting regenerate-scene-cfg) is enough to refresh the artefact.

API surface

All endpoints are under /api/v1/rl-tasks and require authentication.

Method & path	Purpose
`GET/POST /api/v1/rl-tasks`	List / create RL tasks.
`GET/PUT/DELETE /api/v1/rl-tasks/{uuid}`	Read, update, soft-delete (independent of the linked environment).
`POST /api/v1/rl-tasks/{uuid}/clone`	Duplicate the task definition (metadata, runtime contract, source files, scene entities) into a new row; checkpoints and inference runs are not cloned.
`GET /api/v1/rl-tasks/{uuid}/source-files`	List and read/write/delete the user-written source tree (content endpoints live under the same prefix). Unsafe paths and uncompilable Python are rejected.
`POST /api/v1/rl-tasks/{uuid}/regenerate-scene-cfg`	Rewrite the SceneCfg at the configured path plus every spec-derived file (task spec, `scene_bindings.py`, `actions.py`, `observations.py`, `task_export/`). This is the Files-tab “Regenerate all”.
`POST /api/v1/rl-tasks/{uuid}/regenerate-task-spec`	Regenerate only the spec-derived files (no SceneCfg rebuild); used by the Actions/Observations “Save & regenerate”.
`POST /api/v1/rl-tasks/{uuid}/generate-env-cfg`	Write a runnable starter `env_cfg.py` (user-written). Returns 409 if one already exists unless `overwrite=true`.
`GET /api/v1/rl-tasks/{uuid}/export.zip`	Download the runnable plugin ZIP.
`POST /api/v1/rl-tasks/import.zip`	Import a ZIP as a new task.
`POST /api/v1/rl-tasks/{uuid}/import.zip`	Refresh an existing task from a ZIP (generated files skipped; a bundled task spec surfaces a warning).
`GET/PUT /api/v1/rl-tasks/{uuid}/actions`	Read / write the action spec.
`GET/PUT /api/v1/rl-tasks/{uuid}/observations`	Read / write the observation spec.
`GET/PUT /api/v1/rl-tasks/{uuid}/rl-config`	Read / write the RL config spec.
`GET /api/v1/rl-tasks/{uuid}/orchestration-hints`	Scene-derived dropdown data: articulation entities, actuator groups and types per entity, declared joint/body names, actuated vs passive joint lists, and configured camera / contact sensors.
`GET/POST/PUT /api/v1/rl-tasks/{uuid}/scene-entities`	List, create, and update scene entities.
`PATCH/DELETE /api/v1/rl-tasks/{uuid}/scene-entities/{entity_uuid}`	Partially update or delete a single scene entity.

The RLTask response surfaces the action, observation, and RL-config specs as action_spec, observation_spec, and rl_config_spec JSON fields. It also surfaces the two command specs split by lifecycle — training_command_spec and inference_command_spec — each paired with an explicit lifecycle switch (training_command_setup_enabled / inference_command_setup_enabled). Unlike the actions/observations/rl-config specs (which have dedicated GET/PUT endpoints), the command specs and their switches are read from the RLTask response and written through POST/PUT /api/v1/rl-tasks/{uuid} (mirrored by the SDK’s set_training_command_spec / set_inference_command_spec).

The spec GET, the regenerate endpoints, and the export ZIP all produce byte-identical output for the same scene-entity rows, so re-reading and re-exporting are always consistent. If a body can’t be resolved while splitting the MuJoCo scene, the request fails with HTTP 400 naming the failing entity and the selectors that were tried — check the twin selectors and re-export.

Inference Commands (stub)

RL tasks can declare inference commands that define a live control surface for deployed policies. Each command declares:

A payload schema (typed fields like target_position: vec3, object_position: vec3)
A source binding (transport + topic, e.g. MQTT)

When you deploy an RL task controller, the Online Controllers panel shows schema-driven inputs matching your inference command spec. Submitting values sends them to the running controller over MQTT, updating the policy’s command observations in real time. Common use cases:

Position controllers: Set a target EE pose for reach tasks
Waypoint controllers: Set the detected object center for inspection trajectory generation

Commands are authored as first-class terms, like observation terms: each term references a CommandTermCfg subclass (module.symbol) implemented in the task’s env_cfg.py. Add and edit terms in the editor’s Commands tab (term name, module.symbol, lifecycle fields, kwargs) or from the SDK (set_training_command_spec / set_inference_command_spec); the term’s Python implementation lives in the task’s source files. The Commands tab has two explicit lifecycle switches — Training command setup and Inference command setup. Turning one off clears its spec by design (a disabled lifecycle is “deliberately off”, not a forgotten upload); turning it on requires at least one code-backed term. From the SDK, pass enabled= to set_training_command_spec / set_inference_command_spec to set a switch explicitly, or omit it and the switch is derived from whether the spec has commands.

Where to go next

RL Task Walkthrough

Build the OpenArm pick-up task end-to-end in Python, from empty workspace to deployed policy.

Uploading Policies

Upload trained checkpoints, publish them as controllers, and run inference.

Online Controllers

Serve a published policy as a live controller.

RL Task Scene Sensors

Configure cameras and contact sensors on scene entities.

Simulation

Build the MuJoCo environment that backs your RL task’s SceneCfg.

Concepts

Platform Features

Cyberwave Edge

Technical Reference

Use-Case Recipes

Why a separate entity?

Lifecycle

Editor tabs

Author from Python (SDK)

Actions

Observations

RL config

Network builder

Runtime: target, accelerator, and bundle

Policies and inference

What’s in the exported ZIP?

Cyberwave-generated vs user-written

API surface

Inference Commands (stub)

Where to go next

RL Task Walkthrough

Uploading Policies

Online Controllers

RL Task Scene Sensors

Simulation

​Why a separate entity?

​Lifecycle

​Editor tabs

​Author from Python (SDK)

​Actions

​Observations

​RL config

​Network builder

​Runtime: target, accelerator, and bundle

​Policies and inference

​What’s in the exported ZIP?

​Cyberwave-generated vs user-written

​API surface

​Inference Commands (stub)

​Where to go next

RL Task Walkthrough

Uploading Policies

Online Controllers

RL Task Scene Sensors

Simulation

Why a separate entity?

Lifecycle

Editor tabs

Author from Python (SDK)

Actions

Observations

RL config

Network builder

Runtime: target, accelerator, and bundle

Policies and inference

What’s in the exported ZIP?

Cyberwave-generated vs user-written

API surface

Inference Commands (stub)

Where to go next