Physical AI Is Redefining 'Smart' Beyond LLMs

It’s almost laughable to look back at what we used to call smart. Early Alexa devices that played music when you asked for it, Google’s Nest thermostat, TVs that enable apps and interactivity. They’re all still smart, to a degree, and they all still have room to grow. But the bar has skyrocketed upwards over the last two years as physical AI is changing what we call smart. Now we’re seeing robots do household chores, build airplanes, and lift heavy things in factories. And LLMs are not enough. LLMs excel at language, but they were never built to “understand” a room, a hallway, a factory, a construction job site, or the objects inside them. That’s where world models and spatial intelligence come in. Perhaps surprisingly, they have impact both in the physical world, as you’d expect, and the virtual world. And that’s changing where AI researchers are focusing. “World models are more focused on physical space: ingesting and simulating the real-world environment,” Kirin Sinha, founder and CEO of Illumix, told me recently on the TechFirst podcast. “You can see obvious applications for this. For example, with gaming, how can we create these highly expansive, complex worlds in a whole different way with a single continuous model? I think that’s really interesting. Robotics is the other big one you tend to hear about. How can we simulate all of these different environments so that robots can learn more effectively how to operate in the real world?” All of which means, Sinha says, that AI is starting to focus more on physics and real-world functionally over language, where LLMs have traditionally lived. Interestingly, that’s made AR, which her company invested in to build experiences for Disney and Six Flags, relevant again after a long break from the hype train. MORE FOR YOU The question in robotics is how to make AI operate in physical environments reliably and efficiently. That means understanding everything from geometry and depth to contextual meaning and relevant action, all of which require spatial perception, scene understanding and contextual intelligence. Perception is the baseline: what’s around us? How far away is that object? What is the shape of the space? Scene understanding goes further, looking for meaning in the arrangement of objects: is this a library? Are books strewn about because someone is researching? Finally, contextual intelligence asks: what should we do with that information? How does it relate to the user and their intent? All of this is tough enough for robots in static environments. It gets even more difficult in dynamic, quickly changing environments where all of a sudden there’s Lego on the floor or a pallet falls and spills its contents. This focus on physical space contrasts sharply with much of the AI conversation over the past two years. Meta and others poured billions into virtual reality and “metaverse” platforms that emphasize fully digital worlds. Meta specifically has just taken a giant step back from VR and metaverse investments. But Sinha says the real opportunity lies in augmented reality: systems that merge digital intelligence with the real world, rather than replace it. “Historically AR has focused on understanding space and building foundational blocks to add digital elements into our world," she says. In a happy coincidence, understanding space and also how it could be different is actually a pretty useful skill for an autonomous digital being that needs to plan out a path, a series of physical actions, or a task. The hard party here is translating continuous video into intelligence. “Humans are incredibly good at nuance," Sinha says. "Lighting changes, a kid throws a blanket on a chair, a new chair appears. Humans instantly know it’s the same space. For computers, that’s actually really challenging.” All of which means edge intelligence – real-time physical AI running on-device on limited hardware – is increasing important. Other things like long-term intelligence or perhaps higher-order planning can happen in the cloud, but a moving heavy metal object in human space needs to have a pretty good sense of direction on its own part as well prediction about where people might go and what they might do. “You don’t want a device to say, ‘Hold on, I’m querying ChatGPT,’ and then ten seconds later you get an answer,” she says. Instead, physical AI needs to orchestrate a blend of on-device processing for speed and cloud computation for long-term memory and ambient intelligence. This hybrid approach mirrors human cognition: only the most relevant information is actively processed at any moment. Looking forward, Sinha expects hardware to evolve in tandem with software. Custom chips optimized for physical AI with differentiated power profiles and compute pathways will be essential as robots and wearables proliferate. But even with specialized silicon, the key to real-world AI isn’t raw compute: it’s efficiency, context and spatial understanding. The idea is that physical AI’s last mile — the transition from digital simulation to real-world action — won’t be solved by language models or raw compute alone. Instead, it will need architectures that see, understand, and act in the physical world with the same seamless intuition that humans take for granted. And that’s a tall order.