Module 6: Reinforcement Learning for Locomotion
Author: Umema Sultan
Learning Objectives
By the end of this module, you will be able to:
- Explain why reinforcement learning is well-suited for locomotion control
- Describe the components of the RL loop: state, action, reward, and policy
- Design reward functions that produce stable, efficient walking gaits
- Apply domain randomization to enable sim-to-real transfer
- Outline the deployment pipeline from trained policy to real robot execution
Walking seems effortless for humans, yet it remains one of the most challenging problems in robotics. A bipedal robot must continuously balance on a small support area while coordinating dozens of joints in precise sequence. Traditional control methods struggle with this complexity. Reinforcement learning offers an alternative: instead of programming every motion by hand, we let the robot discover how to walk through trial and error in simulation.
The Challenge of Bipedal Locomotion
Walking is fundamentally an act of controlled falling. With each step, the body tips forward, catches itself with the swing leg, then repeats. This elegant process conceals remarkable complexity.
Why Walking is Hard for Robots
Unstable dynamics. A humanoid standing on two feet—or worse, one foot during mid-stride—has a tiny base of support. Any small perturbation can lead to a fall. The robot must constantly adjust to maintain balance.
High dimensionality. A humanoid may have 20 to 40 actuated joints. Coordinating all of them in real-time to produce smooth, stable motion requires solving a complex optimization problem at every timestep.
Contact dynamics. The interaction between feet and ground involves impacts, friction, and intermittent contact. These discontinuities are difficult to model accurately and even harder to control.
Environmental variation. Real floors are uneven. Surfaces vary in friction. The robot encounters stairs, slopes, and obstacles. A walking controller must generalize across all these conditions.
Traditional approaches—hand-tuned trajectory generators, zero-moment-point controllers, model predictive control—can produce walking, but they require extensive engineering and often lack robustness. Reinforcement learning offers a path to more natural and adaptive locomotion.
Reinforcement Learning Fundamentals
Reinforcement learning frames locomotion as a decision-making problem. The robot observes its state, takes an action, receives feedback, and gradually learns which actions lead to good outcomes.
The Learning Loop
The RL process cycles continuously:
- Observe — The robot senses its current state: joint positions, velocities, body orientation, foot contacts
- Decide — A policy (typically a neural network) maps the observation to an action
- Act — The action is applied to the robot's motors
- Experience — The environment responds; the robot moves, potentially falls, or makes progress
- Learn — Based on the outcome, the policy updates to improve future decisions
This loop repeats millions of times during training. Early policies produce chaotic, falling motion. Over time, they converge toward stable, efficient gaits.
The Policy
The policy is the brain of the walking controller. It takes in sensory observations and outputs motor commands. For locomotion, policies are typically neural networks with a few hidden layers—small enough to run at high frequency on embedded hardware.
The policy does not encode explicit knowledge about walking. It learns everything from experience: how to shift weight before lifting a foot, how to swing the leg forward, how to absorb impact on landing.
The Reward Signal
The reward function defines what "good" locomotion means. Designing this function is one of the most important decisions in the training process.
A typical locomotion reward combines several terms:
Forward progress. The robot should move in the commanded direction. Reward is proportional to velocity toward the goal.
Energy efficiency. Wild, jerky motions waste energy and stress hardware. Penalizing large motor torques encourages smooth, efficient movement.
Stability. The torso should remain upright. Penalizing tilting or excessive oscillation promotes balanced gaits.
Foot clearance. Dragging feet causes trips. Rewarding adequate ground clearance during swing prevents stumbles.
Symmetry. Natural gaits are symmetric. Encouraging left-right balance produces more natural-looking motion.
Survival. Above all, the robot should not fall. A large penalty for termination (falling over) provides strong pressure toward stable behavior.
The art of reward design lies in balancing these terms. Too much emphasis on speed produces aggressive, unstable gaits. Too much emphasis on energy efficiency produces slow, shuffling motion. Finding the right balance requires experimentation.
Training in Simulation
Real-world training is impractical—a humanoid robot falling thousands of times would destroy itself. Instead, we train in physics simulation, where falling costs nothing but computation.
Why Simulation Works
Modern physics engines can simulate robot dynamics with reasonable accuracy. More importantly, simulation offers:
Speed. Thousands of simulated robots can train in parallel, compressing years of experience into hours.
Safety. Falls, collisions, and failures have no real-world consequences.
Control. We can systematically vary conditions—terrain, friction, disturbances—that would be difficult to reproduce physically.
Reset. After each fall, the robot instantly resets to try again.
Massively Parallel Training
The breakthrough enabling practical locomotion learning is massive parallelization. Modern frameworks like NVIDIA Isaac Gym simulate thousands of robots simultaneously on a single GPU.
Each robot runs independently, experiencing different random conditions. A robot on flat ground might be learning basic balance while another on rough terrain learns recovery. All their experiences feed into the same policy update.
This parallelism transforms training time from weeks to minutes. A walking policy that might require a billion timesteps of experience can be trained in under an hour on modern hardware.
The Training Environment
The simulated environment includes:
The robot model. A physically accurate representation including masses, inertias, joint limits, and motor characteristics. This typically comes from CAD models or system identification.
The terrain. Starting with flat ground, training progressively introduces slopes, stairs, rough surfaces, and obstacles.
Disturbances. Random pushes, varying friction, and payload changes force the policy to develop robustness.
Observation noise. Real sensors are imperfect. Adding noise to simulated observations prepares the policy for real-world perception.
The Training Pipeline
Training a locomotion policy follows a systematic process.
Stage 1: Environment Setup
First, we configure the simulation. This includes loading the robot model, defining the terrain, setting physics parameters, and specifying the observation and action spaces.
The observation space typically includes:
- Joint positions and velocities for all actuated joints
- Orientation of the torso (roll, pitch, yaw)
- Angular velocity of the torso
- Linear velocity of the base
- Binary contact indicators for each foot
- Commanded velocity (the target direction and speed)
The action space defines what the policy controls. Most commonly, actions are target joint positions that a low-level controller tracks. Some systems use direct torque control for more dynamic motion.
Stage 2: Reward Engineering
The reward function is implemented and tuned. This often requires iteration—initial guesses rarely produce good behavior.
Common strategies include:
Start simple. Begin with just forward velocity and survival. Add refinements once basic walking emerges.
Observe failure modes. If the robot shuffles without lifting its feet, add foot clearance reward. If it oscillates wildly, add smoothness penalties.
Use curriculum. Early training might reward any forward progress. Later stages demand efficiency and naturalness.
Stage 3: Domain Randomization
To ensure the policy transfers to real hardware, we randomize simulation parameters:
Physical properties. Mass, friction coefficients, motor strength, joint damping—all varied within plausible ranges.
Sensor characteristics. Noise, bias, and latency added to observations.
Environmental conditions. Terrain roughness, slope, and obstacle placement varied across episodes.
The goal is that real-world conditions fall somewhere within the distribution of training conditions. A policy robust to simulated variation will likely handle reality.
Stage 4: Training Execution
With everything configured, training begins. The standard algorithm is Proximal Policy Optimization (PPO), valued for its stability and sample efficiency.
Training proceeds in iterations:
- Rollout. All parallel robots execute the current policy, collecting experience.
- Evaluation. Rewards are computed for each timestep.
- Advantage estimation. The algorithm determines which actions were better or worse than expected.
- Policy update. The neural network weights are adjusted to increase probability of good actions.
- Repeat. The cycle continues until performance plateaus.
Monitoring during training reveals progress. Early episodes show chaotic motion and quick falls. Gradually, robots learn to stand, then shuffle, then walk. Advanced training produces running, turning, and recovery from pushes.
Stage 5: Policy Refinement
After basic locomotion emerges, additional training stages add capabilities:
Velocity tracking. The robot learns to match commanded speeds and directions, not just move forward.
Terrain adaptation. Curriculum introduces increasingly challenging surfaces.
Disturbance rejection. Random pushes train recovery reflexes.
Natural motion. Reference motion data from human walking can shape the gait toward more natural appearance.
From Simulation to Reality
A policy trained purely in simulation will not work on a real robot without additional steps. The gap between simulated and real physics—the "sim-to-real gap"—must be bridged.
Sources of the Gap
Modeling errors. Simulated dynamics never perfectly match reality. Friction, contact, and motor behavior differ in subtle ways.
Sensor differences. Real sensors have noise characteristics, biases, and latencies not captured in simulation.
Actuator limitations. Real motors have bandwidth limits, backlash, and thermal constraints.
Environmental unknowns. The real world contains conditions never encountered in training.
Bridging Strategies
Domain randomization. By training across wide parameter ranges, the policy learns to handle uncertainty. If real-world friction falls within the randomized range, the policy can adapt.
System identification. Careful measurement of real robot parameters allows simulation to more closely match reality.
Observation filtering. Real sensor data is processed to match the statistics expected by the policy—filtering noise, compensating for bias.
Action smoothing. Limiting how quickly commanded positions can change prevents exciting unmodeled dynamics.
Online adaptation. Some systems fine-tune the policy on the real robot using limited real-world data.
Deployment Architecture
On the real robot, the trained policy runs in a control loop:
- Sensor processing. Raw sensor data is converted to the observation format expected by the policy.
- Policy inference. The neural network computes the action given current observations.
- Safety filtering. Actions are checked against joint limits and rate constraints.
- Command execution. Filtered actions are sent to motor controllers.
- Repeat. The loop runs at high frequency, typically 50 to 500 Hz.
The policy itself is computationally lightweight—a few matrix multiplications—enabling real-time execution on embedded processors.
Real-World Performance
Sim-to-real locomotion has achieved remarkable results in recent years.
Demonstrated Capabilities
Modern RL-trained humanoids can:
Walk on varied terrain. Flat floors, grass, gravel, and uneven surfaces pose no problem for well-trained policies.
Climb stairs. Ascending and descending stairs requires significant adaptation in stride length and foot placement.
Recover from pushes. Policies trained with disturbances develop reflexive recovery, maintaining balance under perturbation.
Carry loads. Domain randomization over mass allows carrying objects without retraining.
Move at variable speeds. Velocity commands produce smooth transitions between slow walking and jogging.
Remaining Challenges
Despite progress, challenges remain:
Manipulation while walking. Coordinating arm movements with locomotion adds complexity not fully solved.
Highly dynamic motion. Running, jumping, and acrobatic movements require more advanced training techniques.
Long-horizon planning. Navigating complex environments requires integration with higher-level planning systems.
Energy efficiency. RL gaits, while stable, often consume more energy than optimized analytical controllers.
Generalization. Truly general locomotion across all possible environments remains elusive.
Practical Considerations
Deploying RL locomotion involves engineering beyond just training.
Hardware Requirements
Compute platform. Policy inference requires minimal computation, but sensor processing and communication must meet real-time deadlines.
Motor controllers. The low-level controllers must accurately track position or torque commands at the policy's output rate.
Sensing. Joint encoders, IMU, and optionally force sensors provide the observations needed by the policy.
Safety Systems
Joint limits. Hard stops prevent exceeding mechanical limits regardless of policy output.
Emergency stop. Human-triggered immediate shutdown for unexpected behavior.
Fall detection. Automatic motor shutdown when falls are detected to prevent damage.
Current limiting. Prevents motor overheating from sustained high-torque commands.
Testing Protocol
Before deployment:
- Simulation validation. Verify the policy performs correctly across randomized conditions.
- Hardware-in-the-loop. Run the policy with real sensors but simulated dynamics.
- Suspended testing. Execute on the real robot while suspended, unable to fall.
- Supported testing. Walk with safety harness or rails preventing hard falls.
- Free walking. Gradual progression to unsupported operation.
Summary
Reinforcement learning has transformed bipedal locomotion from a painstaking engineering challenge into a learnable skill. The key insights:
Learn from experience. Rather than programming every aspect of walking, let the robot discover effective strategies through millions of simulated trials.
Train in parallel. Massive simulation parallelism compresses training time from years to hours, making iteration practical.
Randomize for robustness. Domain randomization ensures policies handle the inevitable differences between simulation and reality.
Bridge the gap carefully. Sim-to-real transfer requires attention to observation processing, action filtering, and safety systems.
The result is humanoid robots that walk with unprecedented robustness—handling terrain variation, external disturbances, and environmental uncertainty with learned reflexes rather than hand-coded rules.
Key Takeaways
- RL discovers locomotion through experience rather than requiring hand-coded gaits
- Reward design is critical—it shapes what the robot learns to optimize
- Massive parallelism in simulation compresses years of experience into hours of training
- Domain randomization builds policies robust to the inevitable sim-to-real gap
- Deployment requires care—observation processing, action filtering, and safety systems are essential
- The gap is closing—modern RL-trained humanoids walk on varied terrain, climb stairs, and recover from pushes