How Robot Policies Learn From Demonstration and Simulation

Demonstrations provide examples, not instructions

Human demonstrations can be collected through video, teleoperation or motion capture. They show a system how observations and actions relate across many examples. The data must still be filtered, aligned to the robot's body and labelled or structured for training.

Figure's Helix work describes vision-language-action models that connect language and images to robot control. Its Project Go-Big describes collecting human video for broader physical reasoning. Google DeepMind positions Gemini Robotics around vision, language and physical action. These programmes share an interest in generalization, but their training data, action spaces and evaluations are not interchangeable.

Simulation supplies controlled variation

Simulation lets developers repeat falls, contacts and object arrangements without damaging hardware. Parameters such as mass, friction, delay and sensor noise can be varied to prevent a policy from fitting one perfect virtual environment.

NVIDIA's Isaac and GR00T tools are built around this development loop. Simulation is valuable because it produces volume and controlled tests. It also contains a model of reality, and errors in that model can appear when a policy reaches the physical robot.

Retargeting connects behaviour to one body

Human motion cannot be copied joint for joint into a humanoid with different proportions, strength and range of motion. Retargeting maps an example to the available body. A lower-level controller then has to maintain contacts and balance while following that reference.

The same high-level task can therefore behave differently across platforms. Hardware limits, actuator response, sensors and control frequency all affect whether a learned action is repeatable.

Physical evaluation closes the loop

A useful evaluation states the starting conditions, number of attempts, success criterion and intervention policy. Failed attempts belong in the record beside successful runs. For sport, testing should include disturbances, communication loss, repeated impacts and performance after repair.

The reporting question is reliability

Terms such as general-purpose or autonomous are too broad without a task boundary. Better reporting asks which observations the policy receives, which actions it controls, what data produced the behaviour and how often it succeeds under stated conditions.

Learned policies can expand the range of behaviours a machine attempts. Performance still depends on the mechanical body, safety system and human operating procedure, especially when the robot leaves its training distribution.