Microduck Robot●microduck-robot.com

Microduck Robot / RL

Reinforcement learning for robotics: train a biped robot duck to walk

This guide explains reinforcement learning for robotics on a real, open-source biped: the Microduck robot duck. It follows the published training projects from Pollen Robotics (the original Microduck) and Luwu Dynamics (the Arduino UNO Q build), from installing the training environment to exporting a policy and testing it on the robot.

What reinforcement learning means for a small biped

Reinforcement learning (RL) trains a control policy by trial and error. The policy is a small neural network that reads the robot's joint positions, velocities and body orientation and outputs a target for every servo. A reward function scores each step (follow the commanded speed, stay upright, do not stamp the feet) and the training algorithm adjusts the network to collect more reward. For legged robots this happens in a physics simulator, where thousands of copies of the robot can fall over safely at the same time.

Microduck is a good reinforcement learning robot to start with because the whole recipe is open. Pollen Robotics publishes the robot and its RL training environments, and Luwu Dynamics publishes a downstream version for the Arduino UNO Q build. Both train walking, standing up, sitting and tricks with PPO, then export the policy to ONNX so the robot can run it.

RL is not the same as the imitation learning in our LeRobot guide. There, a person demonstrates the task and the network copies the demonstrations. Here nobody demonstrates walking: the reward defines what good walking is, and the simulator supplies the practice.

We wrote this guide from the two projects' published documentation and our own technical notes. We did not train or deploy a policy ourselves while writing it, and both projects change. Treat the commands as the documented starting point and check them against the version of the code you download.

Pick the training project that matches your robot

There are two related training projects, and the robot model matters more than the code. A policy learns the mass, geometry and servo behaviour of the robot it was trained on, so a walking policy trained for one duck can fall at once on the other.

Pollen Robotics' microduck_rl trains the original Microduck, a biped of about 800 g and 25 cm with 14 Dynamixel XL330 servos. Luwu Dynamics' xgoduck_rl follows the same task recipes for the Arduino UNO Q adaptation, whose geometry and inertia differ and whose simulator models 1910-family servos instead of the XL330. The Microduck Robot kits on this site are built around the UNO Q, so start with xgoduck_rl and confirm the servo model against your build.

Scroll sideways to compare all columns.

microduck_rl (Pollen Robotics)xgoduck_rl (Luwu Dynamics)
RobotOriginal Microduck, about 800 g and 25 cm tallThe Arduino UNO Q adaptation
Servo model in simulationDynamixel XL330 (BAM M6 model)1910-family servo (HLS1910 BAM model)
Task names end in-MicroDuck-XgoDuck
Training stackmjlab on MuJoCo Warp, PPOmjlab on MuJoCo Warp, PPO
LicenceApache 2.0 for the code; the 3D models are CC BY-NC-SA 4.0Apache 2.0

How the training loop works

Both projects build on mjlab, which runs MuJoCo physics on the GPU through MuJoCo Warp and trains with PPO (proximal policy optimization) from rsl_rl. The documented commands run 4,096 simulated robots in parallel, so one hour of training collects thousands of hours of simulated practice.

Each skill is a separate task with its own policy. A velocity task learns to walk while following speed commands. A stand-up task learns to get up from lying on its front, its back or sitting. Other tasks learn to sit and stand on command, crouch and touch the ground, kick a ball, roll forward over the head or skate on passive wheels. The original project trains its policies at 50 Hz.

Rewards and randomisation live in each task's configuration. The original project documents its reward-design rules, and keeps the domain-randomisation switches as ENABLE_* flags at the top of each task's configuration file, so you can see what is randomised before you start a run.

What you need before you start

You do not need the robot to train a policy or to check it in simulation. You need it only for the final test.

  1. A computer with an NVIDIA CUDA GPU. Training runs through MuJoCo Warp on the GPU; the computer inside the robot does not train.
  2. Python 3.12 and uv, the Python package manager both projects use to install their pinned dependencies.
  3. The training source code: xgoduck_rl from Luwu Dynamics for the UNO Q build, or microduck_rl from Pollen Robotics for the original Microduck. Our technical docs name both projects.
  4. Optional: a Weights & Biases or Hugging Face account. The original project can play and export runs logged to Weights & Biases, and documents running training on Hugging Face Jobs when you have no local GPU.

Install the training environment

From the root of the downloaded project, uv sync creates the environment from the project's lock file. Run it inside xgoduck_rl, or inside microduck_rl for the original robot.

Install the pinned dependencies
cd xgoduck_rl
uv sync

On ARM machines such as DGX Spark (GB10) or Jetson, the first sync downloads a large CUDA wheel (the original project says about 2 GB) and uv's default 30-second HTTP timeout can stop it. Raise the timeout for that first sync.

First sync on an ARM machine
UV_HTTP_TIMEOUT=600 uv sync

Then list the registered tasks. Use a task whose name ends in -XgoDuck for the UNO Q build and -MicroDuck for the original robot.

List the registered tasks
uv run list-envs

Train a walking policy

Start with the flat-ground velocity task. It is the main task in both projects: the robot learns to walk while following velocity commands.

Train the walking policy (UNO Q build)
uv run train Mjlab-Velocity-Flat-XgoDuck --env.scene.num-envs 4096

For the original Microduck, the project estimates one to two hours for a usable gait at 4,096 environments. Luwu Dynamics does not publish a time for the UNO Q build, and the real time depends on your GPU.

The same task for the original Microduck
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096

Checkpoints (model_*.pt) and TensorBoard logs are written under logs/rsl_rl/<experiment>/; the walking task uses logs/rsl_rl/xgoduck_velocity/. Train other skills with the same command and another task name, for example the stand-up task. The table below lists the UNO Q build's tasks; the descriptions come from the original project's task list, whose recipes these tasks follow.

Train the stand-up policy
uv run train Mjlab-StandUp-Flat-XgoDuck --env.scene.num-envs 4096

The original project documents how to continue a run from a saved checkpoint with the agent flags below.

Resume a run (original Microduck project)
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096 \
    --agent.run-name resume --agent.load-checkpoint model_29999.pt --agent.resume True

Scroll sideways to compare all columns.

Task (UNO Q build)TerrainWhat it learns
Mjlab-Velocity-Flat-XgoDuckFlatWalk while following velocity commands
Mjlab-Velocity-Rough-XgoDuckRoughThe same walking task on uneven ground
Mjlab-StandUp-Flat-XgoDuckFlatStand up from lying down or sitting, then hold the stand
Mjlab-SitStand-Flat-XgoDuckFlatSit down and stand up on command
Mjlab-GroundPick-Flat-XgoDuckFlatCrouch, touch the ground with the beak and stand again
Mjlab-BallKick-Flat-XgoDuckFlatKick a ball with the right foot; a left-foot task also exists
Mjlab-Roulade-Flat-XgoDuckFlatRoll forward over the head and land on the feet
Mjlab-RollerStandUp-Flat-XgoDuckFlatStand up from the ground onto passive wheels
Mjlab-Velocity-Swizzle-XgoDuckFlatSymmetric swizzle skating on wheels

Check the policy in simulation

play opens the viewer for a task. Without a checkpoint flag it loads the newest model_*.pt in that task's experiment directory.

Play the newest checkpoint
uv run play Mjlab-Velocity-Flat-XgoDuck

To review a specific run, pass its checkpoint file. Replay the exact checkpoint you intend to export, not only the newest one.

Play a specific checkpoint
uv run play Mjlab-Velocity-Flat-XgoDuck \
    --checkpoint-file logs/rsl_rl/xgoduck_velocity/<run>/model_XXXX.pt

Watch for what the reward may not catch: feet that scuff, a twitch every few steps, a lean that grows with speed, or a fall when the command changes. A policy that looks unstable in simulation will not get better on the real robot.

Close the sim-to-real gap

With small servos driving a light biped, the original project's authors say actuator fidelity is most of the sim-to-real gap. That is why both projects model the servo with BAM (Better Actuator Models, from Rhoban) down to its voltage control law, back-EMF and load-dependent friction, instead of an ideal PD controller.

Domain randomisation then varies battery voltage, voltage sag under load, command delay and friction across the parallel environments, so the policy cannot depend on one exact servo. In the original project every main task also has a Backlash variant that adds ±1° of gear play in series with each servo joint: insert -Backlash before the robot name, for example Mjlab-Velocity-Flat-Backlash-MicroDuck.

This is also why the servo model decides which project to use. Training the UNO Q build against the XL330 model, or the reverse, trains the policy for servos that are not in your robot.

Export the policy to ONNX

The export script turns a checkpoint into an ONNX file and bakes the observation normalisation into the graph. Always deploy a file produced by scripts/export.py; a hand-converted checkpoint would receive unnormalised observations on the robot.

Export a checkpoint to ONNX
uv run python scripts/export.py Mjlab-Velocity-Flat-XgoDuck \
    --checkpoint-file logs/rsl_rl/xgoduck_velocity/<run>/model_XXXX.pt \
    --onnx-file logs/rsl_rl/xgoduck_velocity/<run>/<run>.onnx

Without --onnx-file, the output is written to output.onnx in the current directory. Keep the exported file next to the run that produced it, and record the training project's revision, the checkpoint and the file's checksum.

Record the file's checksum
sha256sum logs/rsl_rl/xgoduck_velocity/<run>/<run>.onnx

The original project can also drive an exported policy from the keyboard in CPU MuJoCo with scripts/infer_policy.py. It simulates the servos with the same BAM model used in training, which makes it a useful last check before the robot.

Drive the exported policy in CPU MuJoCo (original project)
uv run scripts/infer_policy.py --walking output.onnx

Run the policy on the robot

On the original Microduck, Pollen Robotics' onboard runtime loads exported policies and switches between them (walk, recover, trick) behind one shared observation of 61 values: 48 proprioception values plus commands for body velocity, head pose and body pose. Its publishing tool refuses a graph that is not [1,61] → [1,14], one output for each of the 14 servos.

For the UNO Q build, load the policy through the runtime that matches your firmware branch; our technical docs describe the runtimes. Before the first run, check that the runtime's joint order, observation fields and normalisation match the training task. A mismatch usually shows as a robot that kicks out a leg or folds instead of standing.

  1. Charge the battery and run the calibration from the assembly guide, so the zero pose of the robot matches the zero pose of the model.
  2. Hold the robot or put it on a stand for the first start, with the power switch within reach.
  3. Load a standing or stand-up policy first, then the walking policy at a low commanded speed.
  4. Write down the checkpoint, the ONNX checksum and the runtime revision, so a later repair can return to a combination that worked.

Troubleshooting

If a problem stays, send us your kit version, the training project, the task name and a short video of the robot, and we will check it against the build issues we know.

  1. uv sync stops partway on an ARM machine: set UV_HTTP_TIMEOUT=600 for the first sync.
  2. list-envs does not show your task: check the suffix (-XgoDuck or -MicroDuck) and that you are in the right project.
  3. play shows an unexpected run: it picks the newest checkpoint in the experiment directory, so pass --checkpoint-file.
  4. The robot shakes or reacts late although simulation looks good: check the servo model, the battery voltage and the gear play first. These are what the domain randomisation is meant to cover.
  5. The robot moves the wrong joint or falls at once: compare the runtime's joint order and observation layout with the training task, and confirm the file was exported with scripts/export.py.
  6. The robot stands crooked: calibrate again before you retrain. A policy cannot correct a servo whose zero is offset from the model.

Project sources

Contact us