DifoTrain

train VLAs from the real world

Embodiment-agnostic framework for collecting demos, training Vision–Language–Action policies, and deploying them, sim or hardware, behind four stable interfaces.

View on GitHub pip install difotrain

DifoTrain, robotic arm reaching through space — COLLECT. TRAIN. DEPLOY.

EMBODIMENT-AGNOSTIC VLACOLLECT → TRAIN → DEPLOYNATIVE + WRAPPED POLICIESMIT · V0.7 · PYTHON 3.10+↓ 5-MINUTE LOOP

The hard part is not the model, it is the plumbing

Time-aligned data, language conditioning, action normalization, task success, safe deploy. DifoTrain fixes that behind four interfaces, Robot, DataSource, Policy, EpisodeDataset, so the model, robot, and data source stay swappable.

DifoTrain design principles

Collect. Train. Deploy.

Collect, teleop and human video demos — COLLECT: teleop & video

Train, native or wrapped VLA policies — TRAIN: native or wrapped

Deploy, safety and self-improvement flywheel — DEPLOY: safety + flywheel

THE LOOP

Collect → Train → Deploy → Feedback

Collect

Scripted teleop, synthetic experts, or human video (MediaPipe pose → retarget). Every demo becomes a canonical Episode.

Standardize

EpisodeDataset on disk, LeRobot/RLDS-style. Shared ActionChunk format and kinematics tokenizer across embodiments.

Train

Behavior cloning on native MLP-VLA or Flow-VLA, or wrap OpenVLA / ACT / π0 / Octo. Language-conditioned from day one.

Eval & Deploy

Frozen bench tasks, compare harness, safety layer, ONNX export. Deploy rollouts log back into the dataset, the flywheel.

Four Interfaces. Everything else is a plugin.

Interface	Module	Shipped	Kind
Robot	difotrain.embodiment	PlanarArm · PyBullet · MuJoCo · ROS2	Pluggable
DataSource	difotrain.data.sources	Scripted teleop · Human video · Synthetic	Pluggable
Policy	difotrain.policy	MLP-VLA · Flow-VLA · OpenVLA · ACT · π0 · Octo	Native + wrap
EpisodeDataset	difotrain.data	Directory dataset · .npz episodes · meta.json	Canonical

A Loop You Can Run on CPU

Reference task: a 2-link arm must reach a target named only in the language instruction, success means real grounding, not memorized motion. No webcam, GPU, or hardware required.

Step	Detail	Command	Note
Collect	200 demos from the scripted expert	`difotrain collect --out data/reach --episodes 200`	CPU
Train	Behavior cloning → policy checkpoint	`difotrain train --data data/reach --out runs/policy.pt`	~5 min
Evaluate	Language-grounded reach success rate	`difotrain eval --policy runs/policy.pt`	Sim
Compare	Rank models on same seeds & episodes	`difotrain compare --models mlp_vla,flow_vla`	HTML report
Bench	Frozen 10-task Difotrain-Bench suite	`difotrain bench --policy runs/policy.pt`	v1.0
Improve	Self-improvement flywheel with rollback	`difotrain improve --policy runs/policy.pt --cycles 3`	Gated

Leaderboard snapshot

Policy	Score	Latency	Params
mlp_vla	83.3%	0.10 ms	18.5k
mlp_vla-onnx	83.3%	0.03 ms	-
flow_vla	80.4%	0.23 ms	143k

Full leaderboard

BEYOND THE CORE LOOP

Video, flywheel, fleet

Human video → robot actions

MediaPipe pose estimation plus a Retargeter maps webcam demos into any registered robot’s action space. Offline JSON trajectories work too.

Self-improvement flywheel

difotrain improve finds failures, synthesizes corrective demos, fine-tunes a candidate, and promotes only if frozen eval improves, else automatic rollback.

Enterprise-ready packaging

Signed policy bundles, staged fleet rollouts, and federated fine-tuning for multi-robot deployments, optional layer on the same core interfaces.

pip install difotrain
pip install "difotrain[capture]"  # webcam
pip install "difotrain[sim]"      # pybullet / mujoco
pip install "difotrain[export]"   # onnx