Audio World Model
Sensing the World through Sound
Listen as you read. Each world fades in while it is on screen.
Our ears have evolved over millions of years to help us make sense of our surroundings. We can hear someone coming around a corner, sense the size of a room through its echoes, or follow a passing car with our eyes closed. Hearing lets us pick up cues from our surroundings that are hidden from our other senses, including vision. This makes it central to our ability to make sense of the world around us.
Sound sources on a rainy street
In this world
Giving machines the ability to understand the world around them requires them to make sense of these cues too. At Attractor Labs, we build audio world models so that machines can learn about the physical world through sound.
What is an audio world model?
An audio world model learns a representation of the world and how that world evolves by listening. Modeling a world from audio involves both perception and generation: perception forms this representation, enabling understanding, prediction, and planning; generation brings that world to life.
Perception
Let’s consider a person walking through a room with a boiling kettle and a fireplace. A model needs to identify the footsteps, estimate where they come from, and distinguish them from the other sounds in the room. As the person moves, it needs to update their position while preserving their identity. If they stop walking or leave the room, the model should retain what it has learned and update its confidence as time passes.
The listener’s movement matters too. When the listener turns, a stationary sound source changes direction relative to them. A model following the world through sound needs to account for both the listener’s movement and the movement of surrounding objects.
Tracking sound sources
In this world
This information needs to be encoded in the model’s representation. We can use it to estimate a sound source’s direction, distance, or speed. We can connect it to a language model to ask questions about what is happening, to a robot’s policy to help it choose how to act, or to a vision model to give it information beyond what it can see.
We can also use the representation to predict how the world might evolve. Given the current representation and an action, a model can predict the next representation: what the world might be like after that action. This lets us explore what could happen if we move or interact with our surroundings.
We are developing perception models to form this understanding of the world from sound and enable applications in robotics, wearables, and agentic AI.
Generation
Perception infers properties of a world from sound. Generation conditions sound on those properties: which sources are present, what they are doing, the surrounding geometry and materials, and the listener’s position and orientation. A world state does not specify every acoustic detail, so a generative model should represent a distribution of possible sounds consistent with that state.
The model needs to capture dependencies between sources. Ecological validity concerns which sounds plausibly coexist in a setting, their timing and relative levels, and the relationships between events. These dependencies should constrain both source selection and event timing. When using recordings, their background noise and reverberation also need to suit the intended environment. The constellation below illustrates source selection using different groups of recordings from a shared library.
Soundscape compositions
Out in the open wood, just after first light.
In this world
Generation also requires temporal state. A source should retain its identity and ongoing activity as its position and surroundings change. New events should depend on that history and on interactions in the scene: a foot contact initiates a step sound, while a continuously burning fire continues through a change in viewpoint. The model should allow variation in individual sounds while preserving these relationships over time.
The sound emitted by a source and the sound received by a listener impose different constraints. Contact with a surface affects the emitted footsteps; distance, geometry, and materials determine how that sound propagates. Sources in the same space should share a consistent acoustic environment, with reflections and occlusion determined by their positions. The model should update these propagation paths as the listener moves or a door closes, while maintaining the histories of the sources. The world below illustrates these changes at the listener.
Sound changes with the environment
In this world
Outside · birds, insects and a gravel path
At Attractor Labs, we're beyond excited to be building machines that perceive the world like we do and enable creators to create worlds of their own. Reach out if you're building worlds, robots, wearables, or agents.