Tuning Best Practices
SUMMARY
A reward curve tells you a run improved. It does not tell you whether the next thousand iterations will improve or collapse — Diagnostics/kl and Policy/action_std do.
This page is the short list of things that actually decide whether a run learns, each with the measurement behind it.
Read these curves, in this order
tensorboard --logdir logs| Metric | What it tells you |
|---|---|
Episodes/mean_length | On a locomotion task, watch this before the reward — staying upright is what the return is built on |
Episodes/mean_reward | The headline number. Also what model_best.pt is scored on |
Diagnostics/kl | How far each update moved the policy. The single best predictor of a collapse |
Diagnostics/clip_fraction | Fraction of samples the surrogate clipped. Rises with KL |
Diagnostics/explained_variance | How well the critic predicts the returns. Near zero means the value function is not learning |
Policy/action_std | Exploration. A monotone decay toward zero means the policy is going deterministic and will stop improving |
Loss/value, Loss/surrogate, Loss/entropy | The loss terms. Loss/symmetry appears when the extension is on |
Learning/learning_rate | Under the adaptive schedule this moves; it is the schedule reacting to KL |
Performance/collect_time, Performance/learn_time | Where the wall clock goes. Collection dominating means the environment is the bottleneck |
A reward that climbs and then collapses is almost always KL far above target.
Keep the adaptive schedule
schedule="adaptive" with desired_kl=0.01 adapts the learning rate to hold each update inside a KL trust region. This is the default, and it is the difference between a run that holds and one that falls over.
Measured on Humanoid-v5 over 250–400 iterations, at the same reward:
| Schedule | Diagnostics/kl | Diagnostics/clip_fraction |
|---|---|---|
Fixed 1e-4 | 0.137 — about 14× the target | 0.52 |
| Adaptive | 0.016 | 0.18 |
If you switch to schedule="fixed", you own the stability problem that the schedule was solving.
Give the standard deviation a floor
The entropy bonus has no target. It pushes the standard deviation up until the policy gradient balances it, which means it can lose. Without a floor the standard deviation decays until the policy is near-deterministic and simply stops improving.
Measured on Humanoid-v5 over 6000 iterations without a floor: action standard deviation went 1.00 → 0.29 while the reward peaked at 1788 and fell back to 611.
actor=MLPConfig(
hidden_dims=(256, 256, 128),
obs_normalization=True,
distribution_cfg=GaussianDistributionConfig(init_std=1.0, std_range=(0.2, 1.5)),
)(0.2, 2.0) is the usual range for mjlab and Isaac Lab locomotion tasks, where actions are joint targets.
Use a plain Gaussian, not the squashed one
SquashedGaussianDistribution is numerically fragile at high action dimension. At 17 action dimensions, once the mean drifts, tanh saturates, its log probability explodes and the ratio overflows. Standard PPO on MuJoCo uses a plain Gaussian with the action scaling applied by the environment wrapper, and that is what the examples do.
Match the hyperparameters to the task size
A hard, many-actuator task wants a small learning rate and few epochs; smaller ones want a larger rate, more epochs over each batch and more entropy. These values are measured, and each already ships as configs/gymnasium/<task>.yaml — so the recipe is the file, not a set of flags:
| Config | Envs | Steps | Iterations | Learning rate | Epochs | Entropy | Hidden dims |
|---|---|---|---|---|---|---|---|
Pendulum-v1.yaml | 8 | 128 | 120 | 3e-4 | 10 | 0.01 | (64, 64) |
MountainCarContinuous-v0.yaml | 8 | 128 | 300 | 3e-4 | 10 | 0.01 | (64, 64) |
Reacher-v5.yaml | 8 | 128 | 500 | 3e-4 | 10 | 0.001 | (64, 64) |
Hopper-v5.yaml | 32 | 64 | 1500 | 1e-4 | 5 | 0.001 | (256, 256, 128) |
Humanoid-v5.yaml | 32 | 32 | 3000 | 1e-4 | 2 | 0.0011 | (256, 256, 128) |
Read across the rows rather than down them: the easy tasks share a shape (few environments, long rollouts, a high rate and many epochs) and the hard ones share the opposite.
When changing entropy_coef, watch Policy/action_std — that is the thing it moves.
Size the networks to the observation
Capacity follows the observation dimension. The Gymnasium example picks (256, 256, 128) above 100 observation dimensions and (64, 64) below, so Ant-v5 (105) and Humanoid-v5 (348) get what they need while Pendulum (3) is not over-parameterized.
Always turn on obs_normalization for both networks. Observation entries span angles, velocities and contact forces with scales differing by orders of magnitude, and the normalization is exported with the policy, so deployment sees the same inputs training did.
Batch size is num_envs × num_steps_per_env
That product is what one iteration optimizes over, and the two factors are not interchangeable:
- CPU Gymnasium tasks: tens of environments with more steps each, e.g.
-n 32 -s 64. - GPU mjlab / Isaac Lab tasks: thousands of environments with few steps each, e.g.
-n 4096 -s 24. This is the point of a GPU simulator.
num_mini_batches=4 splits that batch per epoch. num_learning_epochs then decides how many passes it takes — more epochs squeeze more out of each batch but push KL up, which the adaptive schedule then counteracts by lowering the rate.
Turn off check_for_nan for real runs
Each check forces a GPU-to-CPU sync, so on a GPU-resident environment it caps utilization. Enable it while bringing up a new environment or reward function, then turn it off.
Symmetry, if your robot is symmetric
A legged robot is left-right symmetric, so a policy that has learned to trot leading with one leg has in principle learned the mirrored gait too. Telling PPO that is worth real sample efficiency and is what stops a policy settling into a limp: mirrored samples are appended to every mini-batch, and an optional term penalizes the policy for disagreeing with itself on them. This follows Mittal et al., ICRA 2024.
It is off by default and configured in Python rather than from the command line, because the piece it needs cannot be shipped: a mirror function saying which observation entry mirrors which, which only somebody who knows the robot's joint order can write.
from telekinesis.rlbotics.config import PPOConfig, SymmetryConfig
PPOConfig(
symmetry_cfg=SymmetryConfig(
# The function, or an import path to it so the config survives config.json
data_augmentation_func="my_robot.symmetry:mirror",
use_data_augmentation=True, # mirrored samples in every mini-batch
use_mirror_loss=True, # the auxiliary term
mirror_loss_coeff=1.0,
),
)The function is called as func(env=env, obs=obs, actions=actions) — the environment is passed in, so it can read the observation layout — and returns each argument as the originals stacked with their mirrored copies along the batch dimension. Either argument may be None.
Setting both flags to False still computes and reports the loss, detached, which is a cheap way to watch how symmetric a policy is without changing what it optimizes. Symmetry is not supported for recurrent policies.
Writing a mirror function
For a manager-based environment (mjlab, Isaac Lab) the layout is metadata you can read, not something to guess:
env.venv.observation_manager.active_terms # term names per group
env.venv.observation_manager.group_obs_term_dim # their dimensions, so you get each term's offsets
env.venv.scene["robot"].joint_names # carries the side and the axisFor the Unitree G1's 99-dimensional actor group that yields base_lin_vel[0:3], base_ang_vel[3:6], projected_gravity[6:9], joint_pos[9:38], joint_vel[38:67], actions[67:96], command[96:99]. Each term then follows from what kind of quantity it is:
| Term | Under a left-right mirror |
|---|---|
base_lin_vel, projected_gravity | Polar vectors: (+, −, +) — the lateral component flips |
base_ang_vel | Pseudovector: (−, +, −) — the in-plane components flip, the normal one does not |
command (vx, vy, yaw rate) | (+, −, −) |
joint_pos, joint_vel, actions | Swap left↔right joints, negate those whose axis flips |
| Per-foot terms (height, air time, contact) | Swap the two feet |
| Contact forces | Swap the feet, negate the lateral component |
The joint part is derivable rather than typed: names like left_hip_roll_joint give both the partner (left_↔right_) and the sign — negate when the axis is roll or yaw. For the G1 that produces 26 swapped and 16 negated entries across 29 joints.
Verifying a mirror function
Do not trust a mirror you have not measured
A wrong one trains happily and teaches an invariance the robot does not have. Nothing about the run looks wrong.
Three checks, cheapest first:
- Involution —
mirror(mirror(x)) == x. Catches a bad permutation or a stray sign at once. - Commutes with the dynamics — mirror the simulation state as well, step both with mirrored actions, and confirm the next observation mirrors and the reward is unchanged. This is the real test. Observation noise and randomizing reset events have to be off for it to mean anything, and rough-terrain tasks are not mirror-symmetric in their dynamics even when the observation map is right.
- During training — the reported
Loss/symmetryshould start O(1) with a fresh policy and fall. Pinned at zero from the first iteration means your mirror is the identity somewhere.
Measured this way over 200 random states, three Gymnasium tasks have exact-enough maps — a sign flip per entry, no permutation:
| Task | Observation error | Reward error |
|---|---|---|
Pendulum-v1, obs (+, −, −), action (−) | exact | exact |
InvertedDoublePendulum-v5 | 1e-06 | 1e-10 |
InvertedPendulum-v5 | 4e-03 — solver noise, it grows with derivative order | exact |
Swimmer-v5 was measured too and is not symmetric under a sign flip (0.4 on observations, 1.0 on the reward), which illustrates why the measurement matters: it looks like it should be. Humanoid and the walkers are left-right symmetric, but their mirror is a joint permutation over body blocks (cinert, cvel, cfrc_ext), not a sign flip.