ROBOT LEARNING / UNIVERSITY SEMINAR · ONGOING · 2026
Teaching the SO-101
to pick and place.
Part of my seminar on AI and extended reality, and my hands-on way into robotics: from recorded demonstrations to a published ACT policy that picks up a block and places it in a box.
- Demonstration episodes
- 61
- Recorded frames
- 17,324
- Camera views
- 2
- Training steps
- 50,000
THE PROJECT
Learning a physical task from examples.
The project is part of my university seminar Recent Advancements in Artificial Intelligence and Extended Reality, where my topic is Teaching Embodied AI through XR: Immersive Robot Training and Teleoperation. It is also how I’m teaching myself robotics after studying reinforcement learning in simulation. I work hands-on with the SO-101 arm and LeRobot to collect demonstrations, prepare a training dataset, and train a policy for a concrete manipulation task: “Pick up the block and place it in the box.”
The published artifacts include the cleaned dataset and an Action Chunking with Transformers (ACT) policy. The policy takes front and wrist camera images plus the robot’s joint state, and predicts chunks of actions.
- 01 / RECORD
Demonstrations
Front and wrist camera streams at 640 × 480 and 30 FPS, synchronized with six joint-state and action values.
- 02 / PREPARE
Cleaned dataset
61 episodes and 17,324 frames published in LeRobot v3 format, with video, actions, and metadata available for inspection.
- 03 / TRAIN
ACT policy
Trained with LeRobot for 50,000 steps. The published checkpoint uses a ResNet-18 vision backbone and 100-step action chunks.
UNDER THE HOOD
The published configuration.
| Hardware | SO-101 follower arm, with front and wrist cameras |
|---|---|
| Policy | Action Chunking with Transformers (ACT), ResNet-18 backbone |
| Inputs | Two RGB images (3 × 480 × 640 each) and six joint-state values |
| Outputs | Six action values per step, predicted in 100-step chunks |
| Dataset | 61 episodes · 17,324 frames at 30 FPS (about 9.5 seconds per episode, 9.6 minutes in total) · one task |
| Training | 50,000 steps · batch size 8 · AdamW · learning rate 0.00001 |
| Release | Policy and cleaned dataset published on Hugging Face under Apache 2.0 |
Where the project stands
The dataset and trained policy are published, and the project is ongoing. The recording above shows demonstration data, not the trained policy. The model card does not yet report real-robot evaluation trials or a success rate; measuring that is the next milestone.
WHERE THIS IS GOING
From imitating demonstrations to learning from experience.
The ACT policy is my entry point into robot learning, not the destination. The direction I’m working toward is a robot that generalizes, recovers from mistakes, and improves through its own experience.
- NOW
Imitation learning
Leader-arm demonstrations and an ACT policy for a single pick-and-place task.
- NEXT
Measure what the policy actually learned
About ten minutes of demonstrations can’t establish broad competence, so I’ll run 20–30 real-robot trials per condition: the training position, new positions, different-looking objects, a disturbance mid-task, and new lighting or backgrounds. Checkpoints from different training steps are compared on the same setup, with failure videos and an analysis of where things go wrong.
- SEMINAR
Immersive teleoperation in XR
Using a VR headset to teleoperate the SO-101 and record demonstrations for training, and exploring how immersive teleoperation compares with the leader arm for collecting data.
- THEN
Learn from corrections
When the robot fails, I take over with the leader arm, record the correction, fine-tune, and repeat: an interactive imitation-learning loop in the spirit of DAgger. The question: do ten minutes of targeted corrections improve the robot more than ten minutes of fresh demonstrations?
- LATER
Improve through interaction
Human-in-the-loop reinforcement learning on the real robot (HIL-SERL) and residual RL on top of an imitation policy, alongside continuous-control experiments in MuJoCo simulation.
- LATER
Reusable skills
Several manipulation skills on the same arm, and a controlled comparison between ACT and a vision-language-action model such as SmolVLA.
A research plan, not a schedule. Each stage depends on what the previous experiments show.
Details checked against the public model card, policy configuration, and dataset metadata on 11 November 2026.