First steps: from a remote-controlled to an AI-controlled robot

Since the start of the year we have been working on the foundations. Before a robot can work on its own, you first need to be able to move it safely, observe it and measure it.

Teleoperation in simulation

We have built an environment in NVIDIA Isaac Sim where we guide robots through virtual rooms by remote control. Every movement is recorded. These are the first datasets robots will later learn from.

A robot controlled by an AI agent

In parallel, we have built a command-line interface through which an AI agent controls a real robot. Instead of entering joint angles, you state the goal, and the agent translates it into movements.

Navigation and object detection

In a simulated kitchen, our robot now drives on its own to named areas such as "kitchen table" or "shelf". There it detects objects with a camera model, uses the depth camera to determine their position in the room and approaches them.

What we have learned

Simulation saves an enormous amount of time, but only if it is realistic enough. Reflective and transparent objects, for example, are hard for depth cameras. Knowing such limits early is at least as valuable as any successful demo.