Physical AI uses both sight and touch to manipulate objects like a human

In everyday life, it’s a no-brainer to be able to grab a cup of coffee from the table. Multiple sensory inputs such as sight (seeing how far away the cup is) and touch are combined in real-time. However, recreating this in artificial intelligence (AI) is not quite as easy.

An international group of researchers created a new approach that integrates visual and tactile information to manipulate robotic arms, while adaptively responding to the environment. Compared to conventional vision-based methods, this approach achieved higher task success rates. These promising results represent a significant advancement in the field of multimodal physical AI.

Details of their breakthrough were published in the journal IEEE Robotics and Automation Letters.

Machine learning can be used to support artificial intelligence (AI) to learn human movement patterns, enabling robots to autonomously perform daily tasks such as cooking and cleaning. For example, ALOHA (A Low-cost Open-source Hardware System for Bimanual Teleoperation) is a system developed by Stanford University that enables the low-cost and versatile remote operation and learning of dual-arm robots. Both hardware and software are open source, so the research team was able to build upon this base.

However, these systems mainly rely on visual information only. Therefore, they lack the same tactile judgments a human could make, such as distinguishing the texture of materials or the front and back sides of objects. For example, it can be easier to tell which is the front or back side of Velcro by simply touching it instead of discerning how it looks. Relying solely on vision without other input is an unfortunate weakness.

“To overcome these limitations, we developed a system that also enables operational decisions based on the texture of target objects—which are difficult to judge from visual information alone,” explains Mitsuhiro Hayashibe, a professor at Tohoku University’s Graduate School of Engineering.

“This achievement represents an important step toward realizing a multimodal physical AI that integrates and processes multiple senses such as vision, hearing, and touch—just like we do.”

The new system was dubbed “TactileAloha.” They found that the robot could perform appropriate bimanual operations even in tasks where front-back differences and adhesiveness are crucial, such as with Velcro and zip ties. They found that by applying vision-tactile transformer technology, their Physical AI robot exhibited more flexible and adaptive control.

The improved physical AI method was able to accurately manipulate objects, by combining multiple sensory inputs to form adaptive, responsive movements. There are nearly endless possible practical applications of these types of robots to lend a helping hand. Research contributions such as TactileAloha bring us one step closer to these robotic helpers becoming a seamless part of our everyday lives.

The research group was comprised of members from Tohoku University’s Graduate School of Engineering and the Center for Transformative Garment Production, Hong Kong Science Park, and the University of Hong Kong.