WHO
Agility, Amazon, Covariant, Robust, Toyota Research Institute
WHEN
Now
Generative AI is causing a paradigm shift in how robots are trained. It’s now clear how we might finally build the sort of truly capable robots that have for decades remained the stuff of science fiction.
Robotics researchers are no strangers to artificial intelligence—it has for years helped robots detect objects in their path, for example. But a few years ago, roboticists began marveling at the progress being made in large language models. Makers of those models could feed them massive amounts of text—books, poems, manuals—and then fine-tune them to generate text based on prompts.
The idea of doing the same for robotics was tantalizing—but incredibly complicated. It’s one thing to use AI to create sentences on a screen, but another thing entirely to use it to coach a physical robot in how to move about and do useful things.
Now, roboticists have made major breakthroughs in that pursuit. One was figuring out how to combine different sorts of data and then make it all useful and legible to a robot. Take washing dishes as an example. You can collect data from someone washing dishes while wearing sensors. Then you can combine that with teleoperation data from a human doing the same task with robotic arms. On top of all that, you can also scrape the internet for images and videos of people doing dishes.
By merging these data sources properly into a new AI model, it’s possible to train a robot that, though not perfect, has a massive head start over those trained with more manual methods. Seeing so many ways that a single task can be done makes it easier for AI models to improvise, and to surmise what a robot’s next move should be in the real world.
It’s a breakthrough that’s set to redefine how robots learn. Robots that work in commercial spaces like warehouses are already using such advanced training methods, and the lessons we learn from those experiments could lay the groundwork for smart robots that help out at home.