OpenAI is evidently ramping up its own robotics efforts, too. Last week, Caitlin Kalinowski, who previously led the development of virtual and augmented reality headsets at Meta, announced on LinkedIn that she was joining OpenAI to work on hardware, including robotics.
Lachy Groom, a friend of OpenAI CEO Sam Altman and an investor and cofounder of Physical Intelligence, joins the team at the conference room to discuss the business side of the plan. Groom wears an expensive-looking hoodie and seems remarkably young. He stresses that Physical Intelligence has plenty of runway to pursue a breakthrough in robot learning. “I just had a call with Kushner,” he says in reference to Joshua Kushner, founder and managing partner of Thrive Capital, which led the startup’s seed investment round. He’s also, of course, the brother of Donald Trump’s son-in-law Jared Kushner.
A few other companies are now chasing the same kind of breakthrough. One called Skild, founded by roboticists from Carnegie Mellon University, raised $300 million in July. “Just as OpenAI built ChatGPT for language, we are building a general purpose brain for robots,” says Deepak Pathak, Skild’s CEO and an assistant professor at CMU.
Not everyone is sure that this can be achieved in the same way that OpenAI cracked AI’s language code.
There is simply no internet-scale repository of robot actions similar to the text and image data available for training LLMs. Achieving a breakthrough in physical intelligence might require exponentially more data anyway.
“Words in sequence are, dimensionally speaking, a tiny little toy compared to all the motion and activity of objects in the physical world,” says Illah Nourbakhsh, a roboticist at CMU who is not involved with Skild. “The degrees of freedom we have in the physical world are so much more than just the letters in the alphabet.”
Ken Goldberg, an academic at UC Berkeley who works on applying AI to robots, cautions that the excitement building around the idea of a data-powered robot revolution as well as humanoids is reaching hype-like proportions. “To reach expected performance levels, we’ll need ‘good old-fashioned engineering,’ modularity, algorithms, and metrics,” he says.
Russ Tedrake, a computer scientist at the Massachusetts Institute of Technology and vice president of robotics research at Toyota Research Institute says the success of LLMs has caused many roboticists, himself included, to rethink his research priorities and focus on finding ways to pursue robotic learning on a more ambitious scale. But he admits that formidable challenges remain.