This $1.5 Billion AI Company Is Building A ‘General Purpose Brain’ For Robots

Robots, whether they are bipedal humanoids handling basic factory tasks or four-legged military “robot dogs” intended for urban combat, need brains. Historically, these have been highly specialized and purpose-built. But a Pittsburgh-based robotics startup claims it’s created a single off-the-shelf intelligence that can be plugged into different robots to enable basic functions.

Founded in May 2023 by Abhinav Gupta and Deepak Pathak, two former Carnegie Mellon University professors, Skild AI has created a foundational model for what it describes as a “general purpose brain” that can be slotted into a variety of robots, enabling them to do things like climbing steep slopes, walking over objects obstructing its path and identifying and picking up items.

The company announced Tuesday it has raised $300 million at a $1.5 billion valuation in a Series A funding round led by Lightspeed Ventures, Softbank, Coatue and Amazon founder Jeff Bezos with participation from CRV, Felicis Ventures, Menlo Ventures, Amazon and General Catalyst, among others.

Raviraj Jain, the Lightspeed partner who also led the company’s seed round in July 2023 told Forbes he was wildly impressed with Skild AI’s models when he first saw them being pressure tested last April. Robots using them were able to perform tasks in environments that they’d never seen before and hadn’t been designed for demos. “The robots at that time were able to climb stairs, and I think it’s really crazy how well they were able to do it because it’s a very complex stability problem,” he said.

More impressive, still: The robots using Skild’s AI models also demonstrated “emergent capabilities”— entirely new abilities they weren’t taught. These are often simple, like recovering an object that slips out of hand or rotating an object. But they demonstrate the model’s ability to perform unanticipated tasks, a tendency that occurs in advanced artificial systems like large language models.

Skild has pulled this off by training its model on a massive database of text, images and video — one it claims is 1000 times larger than those used by its rivals. To create this massive database, the cofounders, both former AI researchers at Meta, blended a mix of data collection techniques, which they have developed and tested over years of research.

One way was to hire human contractors to operate robots remotely and collect data about those actions. Another was to have the robot carry out random tasks, record the results and learn by trial and error. The AI model was also trained on millions of public videos.

As a PhD student at UC Berkeley, Pathak developed a way of instilling “artificial curiosity” into robots by rewarding the system for producing outcomes that come about when it can’t predict the results of its actions. “The more uncertain the agent is about the prediction of the effect of its actions, the more curious it gets to explore,” he explained. The technique incentivized the AI to navigate more scenarios and collect more data.

His research on curiosity-driven learning was published in 2017 and has been cited more than 4000 times, he said. Pathak also devised a way for robots to use written information from large language models like GPT (of how to open a can of milk, for example) and convert that into actions.

“In 2022 we figured out a way to put these things together in a single coherent system,” Pathak said. “The notion of learning from videos, learning from curiosity, learning from real data but combined with the knowledge from simulation.”

Skild AI faces steep competition from a string of robotics companies that have emerged with billions of dollars in venture funding thanks to the AI boom. Industry behemoth OpenAI recently revived its robotics team to supply models to robotics companies, Forbes first reported. Then there’s outfits like humanoid robotics company Figure AI, helmed by billionaire CEO Brett Adcock, and Covariant, an OpenAI spin off that is building ChatGPT for robots and has raised over $200 million to do it.

Cofounder Gupta claims that Skild AI’s access to large amounts of data separates it from others in the space but declined to disclose exactly how much data its model is trained on.

Ken Goldberg, a professor of robotics and automation at UC Berkeley agrees that data is the key to scaling robotics, but robots require a specific type of data that isn’t widely available on the internet. Plus, using data gathered from simulation doesn’t always translate to the real world.

“The whole idea that robotics is excited about right now is that we can do something analogous to large language models and large vision language models where they both have internet scale data accessible where you have billions of examples,” he said. It’s not a straightforward task for robotics, but Skild AI aims to address the issue by combining all of its data collection techniques with more information drawn from simulations.

Pathak and Gupta envision a future for their company that is similar to OpenAI, where different use cases and products can be built on top of Skild’s foundational model by fine tuning it. “This is exactly how we aim to disrupt the robotics industry,” Gupta said, adding that eventually they want to achieve artificial general intelligence (a hypothetical AI system that can rival or surpass human capabilities) for robots but one that people can interact with in the physical world.

“A GPT-3 moment is coming to the world of robotics,” said Stephanie Zhan, a partner at Sequoia Capital and an existing investor in Skild AI. “It will spark a monumental shift that brings advancements similar to what we’ve seen in the world of digital intelligence, to the physical world.”

MORE FROM FORBES

ForbesOpenAI Is Rebooting Its Robotics TeamBy Kenrick CaiForbesMeet The New AI-Robot BillionaireBy Kerry A. DolanForbesThis $110 Million Startup Is Building A Nervous System For AIBy Rashi ShrivastavaForbesBillionaire Steve Cohen’s Point72 Ventures Lays Off Fintech Team In Pivot Towards AIBy Emily Mason