What reinforcement learning means for a small biped
Reinforcement learning (RL) trains a control policy by trial and error. The policy is a small neural network that reads the robot's joint positions, velocities and body orientation and outputs a target for every servo. A reward function scores each step (follow the commanded speed, stay upright, do not stamp the feet) and the training algorithm adjusts the network to collect more reward. For legged robots this happens in a physics simulator, where thousands of copies of the robot can fall over safely at the same time.
Microduck is a good reinforcement learning robot to start with because the whole recipe is open. Pollen Robotics publishes the robot and its RL training environments, and Luwu Dynamics publishes a downstream version for the Arduino UNO Q build. Both train walking, standing up, sitting and tricks with PPO, then export the policy to ONNX so the robot can run it.
RL is not the same as the imitation learning in our LeRobot guide. There, a person demonstrates the task and the network copies the demonstrations. Here nobody demonstrates walking: the reward defines what good walking is, and the simulator supplies the practice.
We wrote this guide from the two projects' published documentation and our own technical notes. We did not train or deploy a policy ourselves while writing it, and both projects change. Treat the commands as the documented starting point and check them against the version of the code you download.
Pick the training project that matches your robot
There are two related training projects, and the robot model matters more than the code. A policy learns the mass, geometry and servo behaviour of the robot it was trained on, so a walking policy trained for one duck can fall at once on the other.
Pollen Robotics' microduck_rl trains the original Microduck, a biped of about 800 g and 25 cm with 14 Dynamixel XL330 servos. Luwu Dynamics' xgoduck_rl follows the same task recipes for the Arduino UNO Q adaptation, whose geometry and inertia differ and whose simulator models 1910-family servos instead of the XL330. The Microduck Robot kits on this site are built around the UNO Q, so start with xgoduck_rl and confirm the servo model against your build.
Scroll sideways to compare all columns.
| microduck_rl (Pollen Robotics) | xgoduck_rl (Luwu Dynamics) | |
|---|---|---|
| Robot | Original Microduck, about 800 g and 25 cm tall | The Arduino UNO Q adaptation |
| Servo model in simulation | Dynamixel XL330 (BAM M6 model) | 1910-family servo (HLS1910 BAM model) |
| Task names end in | -MicroDuck | -XgoDuck |
| Training stack | mjlab on MuJoCo Warp, PPO | mjlab on MuJoCo Warp, PPO |
| Licence | Apache 2.0 for the code; the 3D models are CC BY-NC-SA 4.0 | Apache 2.0 |
How the training loop works
Both projects build on mjlab, which runs MuJoCo physics on the GPU through MuJoCo Warp and trains with PPO (proximal policy optimization) from rsl_rl. The documented commands run 4,096 simulated robots in parallel, so one hour of training collects thousands of hours of simulated practice.
Each skill is a separate task with its own policy. A velocity task learns to walk while following speed commands. A stand-up task learns to get up from lying on its front, its back or sitting. Other tasks learn to sit and stand on command, crouch and touch the ground, kick a ball, roll forward over the head or skate on passive wheels. The original project trains its policies at 50 Hz.
Rewards and randomisation live in each task's configuration. The original project documents its reward-design rules, and keeps the domain-randomisation switches as ENABLE_* flags at the top of each task's configuration file, so you can see what is randomised before you start a run.
What you need before you start
You do not need the robot to train a policy or to check it in simulation. You need it only for the final test.
- A computer with an NVIDIA CUDA GPU. Training runs through MuJoCo Warp on the GPU; the computer inside the robot does not train.
- Python 3.12 and uv, the Python package manager both projects use to install their pinned dependencies.
- The training source code: xgoduck_rl from Luwu Dynamics for the UNO Q build, or microduck_rl from Pollen Robotics for the original Microduck. Our technical docs name both projects.
- Optional: a Weights & Biases or Hugging Face account. The original project can play and export runs logged to Weights & Biases, and documents running training on Hugging Face Jobs when you have no local GPU.
Install the training environment
From the root of the downloaded project, uv sync creates the environment from the project's lock file. Run it inside xgoduck_rl, or inside microduck_rl for the original robot.
cd xgoduck_rl
uv syncOn ARM machines such as DGX Spark (GB10) or Jetson, the first sync downloads a large CUDA wheel (the original project says about 2 GB) and uv's default 30-second HTTP timeout can stop it. Raise the timeout for that first sync.
UV_HTTP_TIMEOUT=600 uv syncThen list the registered tasks. Use a task whose name ends in -XgoDuck for the UNO Q build and -MicroDuck for the original robot.
uv run list-envsTrain a walking policy
Start with the flat-ground velocity task. It is the main task in both projects: the robot learns to walk while following velocity commands.
uv run train Mjlab-Velocity-Flat-XgoDuck --env.scene.num-envs 4096For the original Microduck, the project estimates one to two hours for a usable gait at 4,096 environments. Luwu Dynamics does not publish a time for the UNO Q build, and the real time depends on your GPU.
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096Checkpoints (model_*.pt) and TensorBoard logs are written under logs/rsl_rl/<experiment>/; the walking task uses logs/rsl_rl/xgoduck_velocity/. Train other skills with the same command and another task name, for example the stand-up task. The table below lists the UNO Q build's tasks; the descriptions come from the original project's task list, whose recipes these tasks follow.
uv run train Mjlab-StandUp-Flat-XgoDuck --env.scene.num-envs 4096The original project documents how to continue a run from a saved checkpoint with the agent flags below.
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096 \
--agent.run-name resume --agent.load-checkpoint model_29999.pt --agent.resume TrueScroll sideways to compare all columns.
| Task (UNO Q build) | Terrain | What it learns |
|---|---|---|
Mjlab-Velocity-Flat-XgoDuck | Flat | Walk while following velocity commands |
Mjlab-Velocity-Rough-XgoDuck | Rough | The same walking task on uneven ground |
Mjlab-StandUp-Flat-XgoDuck | Flat | Stand up from lying down or sitting, then hold the stand |
Mjlab-SitStand-Flat-XgoDuck | Flat | Sit down and stand up on command |
Mjlab-GroundPick-Flat-XgoDuck | Flat | Crouch, touch the ground with the beak and stand again |
Mjlab-BallKick-Flat-XgoDuck | Flat | Kick a ball with the right foot; a left-foot task also exists |
Mjlab-Roulade-Flat-XgoDuck | Flat | Roll forward over the head and land on the feet |
Mjlab-RollerStandUp-Flat-XgoDuck | Flat | Stand up from the ground onto passive wheels |
Mjlab-Velocity-Swizzle-XgoDuck | Flat | Symmetric swizzle skating on wheels |
Check the policy in simulation
play opens the viewer for a task. Without a checkpoint flag it loads the newest model_*.pt in that task's experiment directory.
uv run play Mjlab-Velocity-Flat-XgoDuckTo review a specific run, pass its checkpoint file. Replay the exact checkpoint you intend to export, not only the newest one.
uv run play Mjlab-Velocity-Flat-XgoDuck \
--checkpoint-file logs/rsl_rl/xgoduck_velocity/<run>/model_XXXX.ptWatch for what the reward may not catch: feet that scuff, a twitch every few steps, a lean that grows with speed, or a fall when the command changes. A policy that looks unstable in simulation will not get better on the real robot.
Close the sim-to-real gap
With small servos driving a light biped, the original project's authors say actuator fidelity is most of the sim-to-real gap. That is why both projects model the servo with BAM (Better Actuator Models, from Rhoban) down to its voltage control law, back-EMF and load-dependent friction, instead of an ideal PD controller.
Domain randomisation then varies battery voltage, voltage sag under load, command delay and friction across the parallel environments, so the policy cannot depend on one exact servo. In the original project every main task also has a Backlash variant that adds ±1° of gear play in series with each servo joint: insert -Backlash before the robot name, for example Mjlab-Velocity-Flat-Backlash-MicroDuck.
This is also why the servo model decides which project to use. Training the UNO Q build against the XL330 model, or the reverse, trains the policy for servos that are not in your robot.
Export the policy to ONNX
The export script turns a checkpoint into an ONNX file and bakes the observation normalisation into the graph. Always deploy a file produced by scripts/export.py; a hand-converted checkpoint would receive unnormalised observations on the robot.
uv run python scripts/export.py Mjlab-Velocity-Flat-XgoDuck \
--checkpoint-file logs/rsl_rl/xgoduck_velocity/<run>/model_XXXX.pt \
--onnx-file logs/rsl_rl/xgoduck_velocity/<run>/<run>.onnxWithout --onnx-file, the output is written to output.onnx in the current directory. Keep the exported file next to the run that produced it, and record the training project's revision, the checkpoint and the file's checksum.
sha256sum logs/rsl_rl/xgoduck_velocity/<run>/<run>.onnxThe original project can also drive an exported policy from the keyboard in CPU MuJoCo with scripts/infer_policy.py. It simulates the servos with the same BAM model used in training, which makes it a useful last check before the robot.
uv run scripts/infer_policy.py --walking output.onnxRun the policy on the robot
On the original Microduck, Pollen Robotics' onboard runtime loads exported policies and switches between them (walk, recover, trick) behind one shared observation of 61 values: 48 proprioception values plus commands for body velocity, head pose and body pose. Its publishing tool refuses a graph that is not [1,61] → [1,14], one output for each of the 14 servos.
For the UNO Q build, load the policy through the runtime that matches your firmware branch; our technical docs describe the runtimes. Before the first run, check that the runtime's joint order, observation fields and normalisation match the training task. A mismatch usually shows as a robot that kicks out a leg or folds instead of standing.
- Charge the battery and run the calibration from the assembly guide, so the zero pose of the robot matches the zero pose of the model.
- Hold the robot or put it on a stand for the first start, with the power switch within reach.
- Load a standing or stand-up policy first, then the walking policy at a low commanded speed.
- Write down the checkpoint, the ONNX checksum and the runtime revision, so a later repair can return to a combination that worked.
Troubleshooting
If a problem stays, send us your kit version, the training project, the task name and a short video of the robot, and we will check it against the build issues we know.
uv syncstops partway on an ARM machine: setUV_HTTP_TIMEOUT=600for the first sync.list-envsdoes not show your task: check the suffix (-XgoDuckor-MicroDuck) and that you are in the right project.playshows an unexpected run: it picks the newest checkpoint in the experiment directory, so pass--checkpoint-file.- The robot shakes or reacts late although simulation looks good: check the servo model, the battery voltage and the gear play first. These are what the domain randomisation is meant to cover.
- The robot moves the wrong joint or falls at once: compare the runtime's joint order and observation layout with the training task, and confirm the file was exported with
scripts/export.py. - The robot stands crooked: calibrate again before you retrain. A policy cannot correct a servo whose zero is offset from the model.