EMBODIMENT-AGNOSTIC VLA

COLLECT → TRAIN → DEPLOY

train VLAs

DifoTrain — robotic arm reaching through spaceFIG. 01 — COLLECT. TRAIN. DEPLOY.

from the

(PIP INSTALL)

real world

Embodiment-agnostic framework for collecting demos, training Vision–Language–Action policies, and deploying them — sim or hardware — behind four stable interfaces.

View on GitHub

pip install difotrain

The hard part is not the model — it is the plumbing

Time-aligned data, language conditioning, action normalization, task success, safe deploy. DifoTrain fixes that behind four interfaces — Robot, DataSource, Policy, EpisodeDataset — so the model, robot, and data source stay swappable.

— DifoTrain design principles

THE LOOP

Collect → Train → Deploy → Feedback

01

Collect

Scripted teleop, synthetic experts, or human video (MediaPipe pose → retarget). Every demo becomes a canonical Episode.

02

Standardize

EpisodeDataset on disk — LeRobot/RLDS-style. Shared ActionChunk format and kinematics tokenizer across embodiments.

03

Train

Behavior cloning on native MLP-VLA or Flow-VLA, or wrap OpenVLA / ACT / π0 / Octo. Language-conditioned from day one.

04

Eval & Deploy

Frozen bench tasks, compare harness, safety layer, ONNX export. Deploy rollouts log back into the dataset — the flywheel.

Four Interfaces. Everything else is a plugin.

Robotdifotrain.embodimentPlanarArm · PyBullet · MuJoCo · ROS2Pluggable
DataSourcedifotrain.data.sourcesScripted teleop · Human video · SyntheticPluggable
Policydifotrain.policyMLP-VLA · Flow-VLA · OpenVLA · ACT · π0 · OctoNative + wrap
EpisodeDatasetdifotrain.dataDirectory dataset · .npz episodes · meta.jsonCanonical

A Loop You Can Run on CPU

Reference task: a 2-link arm must reach a target named only in the language instruction — success means real grounding, not memorized motion. No webcam, GPU, or hardware required.

Collect200 demos from the scripted expertdifotrain collect --out data/reach --episodes 200CPU
TrainBehavior cloning → policy checkpointdifotrain train --data data/reach --out runs/policy.pt~5 min
EvaluateLanguage-grounded reach success ratedifotrain eval --policy runs/policy.ptSim
CompareRank models on same seeds & episodesdifotrain compare --models mlp_vla,flow_vlaHTML report
BenchFrozen 10-task Difotrain-Bench suitedifotrain bench --policy runs/policy.ptv1.0
ImproveSelf-improvement flywheel with rollbackdifotrain improve --policy runs/policy.pt --cycles 3Gated

Collect. Train. Deploy.

Collect — teleop and human video demos
COLLECT — teleop & video
Train — native or wrapped VLA policies
TRAIN — native or wrapped
Deploy — safety and self-improvement flywheel
DEPLOY — safety + flywheel

Difotrain-Bench v1.0

Frozen tasks. Honest scores.

Ten tasks covering language grounding, paraphrase robustness, observation noise, precision, OOD starts, and PlanarArm → MuJoCo physics transfer. Submissions are PRs with result JSON.

Leaderboard snapshot

mlp_vla83.3%0.10 ms18.5k
mlp_vla-onnx83.3%0.03 ms—
flow_vla80.4%0.23 ms143k
Full leaderboard

BEYOND THE CORE LOOP

Video, flywheel, fleet

01

Human video → robot actions

MediaPipe pose estimation plus a Retargeter maps webcam demos into any registered robot’s action space. Offline JSON trajectories work too.

02

Self-improvement flywheel

difotrain improve finds failures, synthesizes corrective demos, fine-tunes a candidate, and promotes only if frozen eval improves — else automatic rollback.

03

Enterprise-ready packaging

Signed policy bundles, staged fleet rollouts, and federated fine-tuning for multi-robot deployments — optional layer on the same core interfaces.

Install

Light core. Heavy things optional.

Core is NumPy + PyTorch only. Capture, sim backends, model wrappers, and ONNX export install as extras when you need them.

pip install difotrain
pip install "difotrain[capture]"  # webcam
pip install "difotrain[sim]"      # pybullet / mujoco
pip install "difotrain[export]"   # onnx
PyPIFull docs