Anomalies: debugging a whole-body controller at millisecond scale
The open problem, stated plainly. The controller intermittently demands torques the hardware cannot deliver — up to 199 N·m from a 120 N·m joint — and we do not yet know why. The firmware saturates, the joint pins against a limit it cannot reach, and what follows is a cascade the robot does not recover from on its own.
These pages are the evidence trail. Both events were caught on a 50 Hz ring buffer that only dumps when something trips, so what you are reading is reconstructed from 20 ms samples of the run-up.
How any of this was recoverable: the black box
A humanoid that falls tells you almost nothing after the fact. The useful information is in the run-up — the few hundred milliseconds before anything visible happens — and by the time a human reacts it is long gone.
So the deploy carries a pre-trigger ring buffer: the last 30 seconds of full state at 50 Hz, continuously overwritten and never written to disk. It dumps only when something trips — an operator e-stop, a tilt watchdog, a stale-state fault. What lands on disk is therefore the approach to the fault, not the damping that follows it.
Crucially it records the whole command chain, not just the outcome, so a divergence can be attributed to a stage rather than guessed at:
| stage | channel |
|---|---|
| what the planner asked for | ref_pos_mj |
| raw policy output | action_pre_clip_il |
| after the action clip | action_post_clip_il |
| target before the safety clamp | target_raw_mj |
| target actually published | target_final_mj |
| what the control law asked of the motor | pd_torque_demand_mj |
| what the joint actually did | joint_pos_mj, joint_vel_mj |
| what the body did | base_quat_wxyz, base_gyro_xyz |
| motor thermals, per-source staleness | motor_temp_c, src_age_s |
Both findings on this page come from that: the commanded target left the reference while the body was still upright is only sayable because the reference, the target and the measured position are all in the same dump on the same clock, 20 ms apart. Every number below is read from it — nothing is inferred from video.
Anomaly 1 — hip roll burst (2026-08-25)
The same event in MuJoCo, against the planner's reference
What the recorder shows
| value | meaning | |
|---|---|---|
peak demand, left_hip_roll | 84.5 N·m | against 42° of tracking error |
peak demand, right_knee | 199.4 N·m | on a 120 N·m joint — 1.7× rating |
| body tilt when the hips first passed 40 N·m | 1.9° | upright: the command led the fall |
| first threshold crossing → release | 160 ms | the whole burst |
| final tilt | 31.5° | operator e-stop |
No automatic guard fired. The tilt watchdog sits at 72.5° and the velocity trip at 35 rad/s; neither was close. A human pressed the button.
Anomaly 2 — torque reversal under softened hips (2026-08-26)
It does not
| kp ×1.00 | kp ×0.80 | |
|---|---|---|
| peak hip demand | 84.5 N·m | 84.1 N·m |
| tracking error at that peak | 42.0° | 44.3° |
| outcome | cascade → e-stop | arrested by the gantry |
Cutting stiffness by 20% produced the same torque, because the
tracking error grew to compensate: τ = kp·err, and the plant
simply moves along that curve. Three hardware rungs were tried — at ×0.60 the
robot could not hold a stand and was worse by torque; at ×0.70 with
softened knees the peaks fell but sustained load and audible knee clicking
both rose.
Why gain scaling could never have worked. The action scale is
computed from the unscaled plant and baked into the deployed
constants, so kp_scale does not touch it. No gain rung could
reduce the commanded target excursion — only the torque produced per unit of
error, which the error then grew to cancel.
It does not reproduce in simulation
This is the part that keeps the cause open. The event has never been reproduced in sim — not for want of trying, and not on a mismatched rig.
The full loop was rebuilt end to end — planner → pose watchdog → deploy → MuJoCo — and driven by the operator's own recorded command tape from the 13:39 event, replayed at its recorded inter-event timings. Not an approximation of the gesture: the same intents, the same 20 ms stick step, the same 4 s pause after a full-speed walk.
It was run twice: once on the robot's own aarch64 binary, on the robot's own compute, which controls for build, ONNX runtime and system libraries as well as configuration; and once on a workstation where physics keeps real time. Byte-identical across both: planner ONNX, policy ONNX, tuning preset, all three actuator-plant files, and the environment profile.
| hardware | simulation | |
|---|---|---|
| hip tracking error during the turn | 42° | 0.9–1.8° |
| peak hip demand | 84.5 N·m | — |
| body tilt | 31.5° | < 3.4° |
| outcome | cascade, e-stop | nothing |
The turn phase — the exact gesture and the exact moment — is the calmest part of the simulated trace. Neither platform produced anything resembling the event.
One thing sim did reveal, by accident: the safety clamp that was supposed to prevent this saturates during ordinary walking, harmlessly, for the same 200 ms duration seen in the event. It is not a bound the failure exceeded; it is a bound the policy rides routinely — which is why it never served the purpose it was added for.
So the gap is not configuration, not the build, and not the operator's input. It is somewhere in contact reality, actuator behaviour, or something the simulator does not model at all — and until it can be reproduced, every proposed guard is being sized against a single recording.
Where this leaves the transfer
The frozen-G1 approach is proving substantially harder against the recently released fine-grained manipulation checkpoint than against the earlier core. That checkpoint targets slow-precision manipulation and wrist accuracy, and the whole-body controllers derived from it are the ones exhibiting this behaviour.
The central unknown is the one at the top of this page: the model demands torques the hardware cannot produce, for reasons we have not yet isolated. It may be something the cross-embodiment transfer simply cannot achieve — the frozen prior asking for dynamics that this embodiment's actuators cannot realise, with nothing in training to penalise the ask because in simulation the limit is absorbed for free.
Rolling back to the natively-trained incumbent removes the behaviour entirely: peak knee demand 69 N·m against 160, six times less time above the break-away threshold, and smooth walks and turns reported by the operator.
kp·(target−pos) − kd·vel, what the control law asked for. No
motor exceeds its rating: the firmware saturates, so demand above the limit
means the joint was pinned there. Nothing on the deploy side measures
delivered torque.
· Back to the project page