> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cyberwave.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Import datasets

> Import robot datasets from HuggingFace Hub or upload zip files.

<Info>
  **Robot datasets first.** Cyberwave currently focuses on robot manipulation and navigation datasets (LeRobot, RLDS, and similar time-series formats). Zip upload supports LeRobot only; HuggingFace Hub import detects any dataset repository, but only robotics datasets are correctly displayed today. Full support for non-robotics formats — playback, conversion, and training — is coming in a future release.
</Info>

## Overview

Cyberwave supports importing datasets from two sources:

* **HuggingFace Hub** — Import directly by repository ID (lazy, no multi-GB copy)
* **Zip Upload** — Upload a pre-packaged dataset archive (max 10 GB)

Format detection happens automatically on the server side and sets the `source_format` field on the dataset record.

## Supported formats

The two import sources have different levels of format support today.

### Zip upload — LeRobot only

Zip upload currently supports **LeRobot v2.1 and v3 only**. Other dataset formats are not yet supported via zip upload.

| Format | `source_format` value | Description |
| - | - | - |
| LeRobot v3 | `lerobot3` | Latest LeRobot format — parquet episodes + `meta/info.json` |
| LeRobot v2.1 | `lerobot21` | Legacy LeRobot format with splits (normalised to v3 on export) |

### HuggingFace Hub — broader import, robotics-only display

HuggingFace Hub import works for **any dataset repository** — Cyberwave queries the repo and detects its format automatically regardless of what kind of dataset it is. However, only **robotics datasets** are correctly displayed and usable in Cyberwave today.

| Category | Examples | Status via HuggingFace import |
| - | - | - |
| Robotics / time-series | LeRobot, RLDS, HDF5, Zarr, GR00T, RoboDM, MCAP, ROS bag | Detected and correctly shown |
| Everything else | COCO, YOLO, ImageNet, video/audio, CVAT, and other image/video/CV formats | Detected and cataloged, but not yet correctly shown or usable |

<Note>
  Playback, conversion, and ML training are fully wired up only for LeRobot today. Other robotics formats imported from HuggingFace are recorded correctly (`source_format`, sample counts) but full episode playback is still being built out.
</Note>

## Import from HuggingFace Hub

1. Navigate to your workspace and click **Import Dataset**
2. Select **HuggingFace Hub** as the source
3. Enter the repository ID (e.g. `lerobot/pusht`)
4. Click **Import**

HuggingFace imports are **lazy by default**: Cyberwave queries the HF API for tags, card data, and the file list, and for LeRobot datasets reads only the small `meta/*.json` files. The dataset card appears immediately with episode counts, FPS, robot type, and cameras — without copying the multi-GB payload. Frames are fetched on demand when a training run or visualisation needs them. The `metadata.import.materialized` flag stays `false` until a follow-up materialisation step runs.

Any repository can be imported this way — Cyberwave will detect and catalog whatever format it finds — but only robotics datasets are correctly displayed and usable today (see [Supported formats](#supported-formats)).

<Info>
  For private HuggingFace repositories, ensure your organisation has configured the HuggingFace token in the deployment settings.
</Info>

## Upload a zip file

1. Navigate to your workspace and click **Import Dataset**
2. Select **Upload Zip** as the source
3. Select your zip file (max 10 GB)
4. Click **Upload**

The system automatically detects the format after the upload completes. Zip upload only supports **LeRobot** (v2.1/v3) today (see [Supported formats](#supported-formats)); other formats should be imported from HuggingFace Hub instead.

## Monitor progress

After starting an import you can track it via:

* The dataset list view (status indicator)
* The dataset detail page (`metadata.upload_progress`)

## API reference

<Card title="POST /datasets/import/init" icon="code">
  Initialise a dataset import. Returns a signed URL for zip uploads, or immediately starts an HF import.
</Card>

<Card title="POST /datasets/import/complete" icon="code">
  Complete a zip upload import after the file has been uploaded to the signed URL.
</Card>

See the [API Reference](/api-reference/rest/DefaultApi) for full details.

## Upload a LeRobot dataset

Cyberwave supports uploading pre-built LeRobot datasets directly to your workspace. This allows you to use existing datasets for VLA model fine-tuning without re-collecting data in Cyberwave.

Supported formats:

* **LeRobot v2.1** - Legacy format with splits-based episode organization
* **LeRobot v3.0** - Current format with chunked data and episodes.jsonl

### Quick Start

1. Open an environment in the Cyberwave editor
2. Navigate to **File → Import → Upload dataset**
3. Fill in the dataset name and select your `.zip` file
4. Wait for processing to complete
5. Your dataset is now available under **Data → Manage Datasets**

### Dataset Requirements

Your LeRobot dataset zip file should contain:

```
your-dataset.zip/
├── meta/
│   ├── info.json          # Required: dataset metadata
│   └── episodes.jsonl     # v3.0 only
├── data/
│   └── chunk-000/
│       └── *.parquet      # Episode data
└── videos/                # Optional: video files
    └── chunk-000/
        └── *.mp4
```

The `meta/info.json` file must include:

* `codebase_version` - Version string (e.g., "v3.0" or "v2.1")
* `fps` - Recording frame rate
* `features` - Feature definitions including action/observation shapes

### Size Limits

* Maximum file size: 10 GB
* Large uploads (>2 GB) may take several minutes

### Processing Stages

After upload, Cyberwave automatically:

1. **Downloads** the zip to processing workers
2. **Extracts** and validates the archive structure
3. **Detects** the format version (v2.1 or v3.0)
4. **Validates** data integrity and required files
5. **Persists** metadata and creates dataset records

You can monitor progress in real-time through the upload dialog.

### Using Uploaded Datasets

Once processed, your dataset can be used for:

* **VLA Fine-tuning** - Train SmolVLA or other models on your data
* **Export** - Download in different formats
* **Review** - Browse episodes and metadata

### Related

* [SmolVLA Training](/feature-reference/ml-models/smolvla-training)
* [Model catalog](/feature-reference/ml-models)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.