What is ‘Sim2Real’, and why does it matter for physical AI?

When we say physical AI, we mean AI capable of navigating and interacting with the real world through intelligent robotics. This ambition is no small feat: humanoid and other intelligent robots are complex, fragile and highly dynamic systems where a single mistake in balance, contact or timing can mean hours of recovery or damaged hardware. Training and validating every behaviour to stop these mistakes directly on a physical robot is slow, expensive and often unsafe.

This is where Sim2Real comes in. Short for simulation-to-reality, Sim2Real is a powerful approach to training AI policies in simulation before deploying them directly in the real world with minimal additional fine-tuning. Working in simulation allows us to iterate at speeds that reality can’t match, meaning we can safely test control algorithms, refine learning policies and probe failure modes overnight. In short, it allows teams to move faster without breaking hardware.

But this speed comes at a cost. Simulations are, by nature, approximations. No matter how refined the models or equations, they can never fully capture the complexity of reality. A policy performing flawlessly in simulation is meaningless if it instantly fails in the real world. This is exactly why Sim2Real must be more than an afterthought – instead, it must be the critical step between promising research and a robot that can actually walk, grasp, interact and succeed.

This discrepancy is known as the ‘Sim2Real gap.’ Subtle differences in physics, sensors, lighting, timing and even network behaviour can separate the clean predictability of simulation from the messy richness of the real world. Bridging this gap is one of the central challenges in the successful deployment of AI and robotics, and the core challenge Sim2Real seeks to solve.

For our team’s exploration of physical AI, Sim2Real goes beyond making motion policies work in specific simulated and real environments. It’s about building a complete integration pipeline that can scale to the real world: one that connects machine learning inference to real-world whole body control, continuously incorporates sensor feedback and robustly handles the unpredictability of real deployments. The same principles that allow a humanoid robot to balance on a lab floor can power safer, more adaptive robotic systems across almost every domain. In healthcare and assistive robotics, it enables extensive testing before systems ever interact with patients. In industrial and hazardous environments, virtual training supports safer deployment. In other words, it removes a huge amount of risk before the robotics enter the actual playing field.

Key challenges of bridging the Sim2Real gap

The Sim2Real gap isn’t caused by a single flaw. It emerges from the accumulation of hundreds of small mismatches that only become visible when theory meets hardware. That said, a few consistent themes have emerged that we must tackle to close the gap:

1. Latency

In simulation, everything is perfectly aligned. Control signals, sensor updates and physics advance together in neat, predictable steps. The real world doesn’t play by those rules. Network delays, hardware buffers and operating system scheduling introduce tiny inconsistencies that drift over time. Even a few milliseconds of delay can destabilise a balance controller or cause high-frequency loops to fight against themselves.

2. Control frequencies and update rates

Simulators will happily run at whatever frequency you choose. Real robots will not. Most humanoid and other intelligent systems rely on embedded controllers running at fixed rates between 500 and 1000 Hz, while higher-level balance or learning policies typically operate at 30 to 60 Hz. Bridging these layers is delicate work. If updates arrive too slowly, the robot feels sluggish. Too fast, and the system becomes overwhelmed by noise and timing drift. In simulation, you can make the world wait for your loop, but in the real world, the clock keeps ticking.

There’s also a practical trade-off between the desired control frequency and available compute. Large AI models, particularly visual language action models (VLA), simply cannot run at high update rates without expensive, power-hungry GPUs. Designing control systems that meet real-time requirements within tight compute budgets is a core part of making Sim2Real work.

3. Imperfect and missing data

Simulation gives you perfect, complete data. Every joint angle, velocity and contact force is available instantly and exactly. In reality, sensors are noisy, vision is partial, and orientation must be estimated from imperfect IMU and kinematic data.

In early tests, we discovered our control code had been trained assuming the robot always started facing a specific direction. The first time we launched it with the robot rotated 180 degrees, it promptly fell over. This kind of failure is a powerful reminder that deployment often reveals assumptions developers didn’t realise they were making and feeds directly back into better training design.

4. Multiple simulators, multiple truths

No simulator captures reality perfectly, and each makes different compromises. To avoid overfitting to a single physics model, we trained policies in NVIDIA Isaac Gym and validated them in MuJoCo, which handles contact and stability more realistically. If something works in both environments, chances are it will also work in the real world. Cross-simulator testing is one of our most reliable indicators of generalisation long before any hardware gets powered on.

5. Sensors and visual realism

Vision is one of the hardest gaps to bridge. Lighting, textures, lens distortion and motion blur never match perfectly between simulation and reality. Even small discrepancies matter – a few pixels of distortion or a slight delay in a video stream can throw off a vision-based controller that otherwise worked flawlessly in simulation. Domain randomisation helps by injecting variation into synthetic data, but it can’t reproduce every nuance of a real camera pipeline.

6. Hardware and compute constraints

Training happens on powerful desktop GPUs, while deployment happens on compact embedded systems with a fraction of the compute budget. Models that run comfortably on a workstation often need to be compressed, quantised or pruned before they can run onboard.

Then there are the physical limits: actuator power, thermal thresholds, battery capacity and mechanical tolerances. A policy that performs an impressive leap in simulation may simply trip a current limit or drain the battery in seconds on real hardware. In other words, what works in the lab often doesn’t work on the real machine. The solution to this is that ML models and control policies must be designed not just to work, but to work within tight computational and physical constraints or they will fail the moment they leave the simulator. This isn’t something you can fix down the line – we must be designing for reality, not just simulation, from day one.

Progress so far in bridging the gap

To manage these challenges, our system is built around the principle of Sim2Real equivalence. The same ROS 2 interfaces, message types and control nodes run in both simulation and hardware. Switching domains means changing the world, not the code.

The bridge between inference and joint control is deliberately minimal, translating policy outputs directly into the robot’s command format. Networking and synchronisation receive equal attention. Robots operate on fixed IPs and domains, with a parent network bridge keeping simulators, hardware and motion-capture systems aligned. It may not be the most elegant, but it’s robust and, vitally, it works.

Precise time synchronisation using network time protocol (NTP) and precision time protocol (PTP) ensures consistent timestamps, while a Vicon tracking system provides ground-truth pose data for debugging. Together, these form a controlled ‘reality sandbox’ where simulation-trained behaviours can be observed, measured and refined under real-world conditions.

We can now train motion policies entirely in simulation, validate them across multiple physics engines and deploy them directly to hardware without altering the control stack. The result is stable balance, repeatable joint-space motion and early demonstrations of onboard, perception-driven control.

These successes represent foundational building blocks toward a unified framework in which behaviours move seamlessly from virtual to physical with minimal human tuning. In other words: faster training, faster real-world deployment.

The road ahead for physical AI deployment

Reality is always more complicated than simulation – but with the right Sim2Real design choices, it can also be more forgiving. Small assumptions, such as a fixed world origin or an ignored delay, can undermine an entire controller. Shared interfaces, consistent timing and cross-sim validation make transfer stable and predictable.

We’ve learned that robustness isn’t a final step; it’s something designed in and deeply integrated from the beginning. Interestingly, many of the changes required to make a controller work in reality also improve its performance back in simulation.

Our goal is a humanoid platform that learns safely and efficiently in simulation, then walks into the real world without needing to relearn the basics. Sim2Real is more than a technical hurdle – it’s a way of designing systems that accept uncertainty, adapt to imperfections and still perform.

Each improvement brings us closer to robots that can function reliably not just in structured labs, but in the unpredictable world we live in. Follow our journey into physical AI and reach out to continue the conversation.