Neura Develops Multimodal Robot Foundation Model Integrating Touch and Audio

Robotics company Neura has developed a new foundation model that combines three types of sensory information: visual, tactile, and audio. The model aims to enhance robots' perceptual capabilities and represents a shift away from the traditional design philosophy of robot AI that has relied heavily on visual information.

Conventional robot AI has primarily been designed around "vision" from camera footage and similar visual inputs. Just as humans use touch to feel an object's shape and firmness, or listen to sounds to assess situations, robots performing actual tasks require sensory information beyond vision alone. Against this backdrop, there is growing interest across the industry in multimodal approaches that incorporate tactile and audio information—methods that can handle multiple input formats.

The foundation model developed by Neura is designed to process not only camera footage but also tactile data from physical contact with objects and surrounding audio information simultaneously. This is expected to enable robots to make judgments difficult with vision alone, such as adjusting the precise force needed to grasp a soft object or detecting anomalies from sound. However, the company acknowledges that learning this model with sufficient quantities of real-world data presents a significant challenge.

This challenge is common across robot AI development. While visual data can be collected in large quantities through photographs and videos, tactile sensor data and audio must be painstakingly gathered through actual robot operations, with substantially higher collection costs and effort. Since model performance is directly tied to data quality and quantity, securing training data has become a universal difficulty in research and development within this field.

Nevertheless, Neura's decision to pursue this approach appears to stem from a judgment that addressing the limitations of vision-only models can enhance robot competitiveness. In settings such as manufacturing lines and logistics warehouses that demand delicate manipulation, tactile and audio information significantly influence work precision. A model that integrates multiple senses can be viewed as a step toward practical deployment in such environments.

The key focus going forward lies in how to efficiently collect and utilize real-world data. Research is also proceeding in parallel on methods to supplement data scarcity, such as advances in sensor technology and the use of simulation environments. How Neura's model overcomes this challenge, and whether other robotics companies adopt similar approaches, will be important indicators of future industry trends.