Robots Achieve Human-Like Lip Movements for Natural Communication

Robots struggle with something humans barely think about: lip movement. Nearly half of our attention in a face-to-face conversation lands on the way someone’s mouth moves. That’s a lot of brainpower spent on a small patch of skin. It explains why a video call with bad audio feels off, or why a person who can’t move their face well can seem hard to read, even when the words are right. Robots run straight into that problem. A robot can walk, pick things up, and even answer questions – but if its mouth doesn’t match its voice, people tense up. We cut robots some slack for clunky steps or stiff arms. We don’t cut them slack for a face that looks almost human but not quite. That standard is known as the “uncanny valley.” Lips are challenging for robots A realistic mouth isn’t just a jaw flapping open and shut. Human speech is built from tiny sound units, and each one has a matching pattern of lip shapes and timing. The lips stretch, press together, curl, and relax in fast sequences. They also sync with cheeks, jaw, and the area around the mouth. Our brains notice when the timing is off by a fraction of a second. That’s part of why many humanoid robots look lifeless or even creepy. Even the expensive ones often do what the researchers call “muppet mouth gestures.” If a robot has a face at all, it’s usually rigid. The motion is often pre-planned, like a puppet routine, instead of reacting smoothly to the sounds it’s making. A robot that studied its own face A Columbia Engineering team has built a robot that can learn facial lip motions for tasks such as speech and singing. The team showed the robot speaking in multiple languages and singing from its AI-generated debut album “hello world.” The key is how it learned. Instead of feeding it a big set of rules about where each motor should go for each sound, the robot learned by watching. First, it learned its own face. The researchers put a robotic face with 26 motors in front of a mirror and let it experiment. The robot made thousands of random expressions and lip gestures, then connected what it saw to what it did. This is called a “vision-to-action” language model (VLA). After that, the robot studied people. It watched hours of YouTube videos of humans talking and singing. That gave the AI driving the face a chance to pick up how real mouths move when specific sounds happen. With those two pieces together, the system could translate audio into motor actions that move the lips in sync. What the robot can do now In the tests, the robot didn’t need to understand the meaning of the audio clips. It just had to match motion to sound, across different voices, languages, and even songs. That’s important because it separates lip timing from language understanding. A robot can, in theory, learn the physical rhythm of speech before it learns the message. The researchers noted that the results aren’t perfect yet. Professor Hod Lipson is the director of Columbia’s Creative Machines Lab, where the work was conducted. “We had particular difficulties with hard sounds like ‘B’ and with sounds involving lip puckering, such as ‘W’. But these abilities will likely improve with time and practice,” said Lipson. That fits with what we know about learning systems in general. When an AI model trains on more examples, it often gets better at the rare, tricky cases. In speech, those tricky cases include fast changes and sounds that depend on precise lip contact, like when the lips must fully close and pop open. Missing link in human-robot life Getting the mouth right isn’t only about looking less awkward. It’s about communication. People lean heavily on facial cues to decide whether someone is friendly, bored, joking, or serious. A face also tells you when it’s your turn to talk. Those cues matter in classrooms, hospitals, customer service desks, and anywhere a robot might need to work around people without making them uneasy. “Much of humanoid robotics today is focused on leg and hand motion, for activities like walking and grasping,” said Lipson. “But facial affection is equally important for any robotic application involving human interaction.” Yuhang Hu, who led the study for his PhD, ties the face to modern chat systems that already handle conversation well. “When the lip sync ability is combined with conversational AI such as ChatGPT or Gemini, the effect adds a whole new depth to the connection the robot forms with the human,” explained Hu. “The more the robot watches humans conversing, the better it will get at imitating the nuanced facial gestures we can emotionally connect with.” The future of humanoid robots According to Hu, the longer the context window of the conversation, the more context-sensitive these gestures will become. If humanoid robots become common, that pressure to feel natural will only grow. The team noted that some economists predict over a billion humanoids will be manufactured in the next decade. That’s not a niche future. That’s a world where faces on machines show up in everyday places, and people will judge them in seconds. “There is no future where all these humanoid robots don’t have a face. And when they finally have a face, they will need to move their eyes and lips properly, or they will forever remain uncanny,” Lipson estimates. “We humans are just wired that way, and we can’t help it,” said Hu. “We are close to crossing the uncanny valley.” The full study was published in the journal Science Robotics. —– Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates. Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com. —–