Augmented Reality in Your Headphones

When we speak of Augmented Reality, we often refer to a visually enhanced experience: when, for example, the printed images of a book animate with the help of a tablet, or when a planned building can be seen virtually at the site of its future location (even before construction begins), with the help of VR glasses. But what role can audio play in this enhanced reality? And what potential does augmented reality (AR) audio have for the visitor experience in museums and visitor centres? Our experiments show how AR audio creates new spaces of perception.

With AR Audio, the sounds existing in the room are supplemented or replaced by virtual sounds. The audio, with its apparent spatial quality is not actually in the room, rather, everything emanates from the headphones. Today it is possible to track user movement in real time and calculate respective distances to a virtual sound source. Coupled with sensors which measure the direction in which the user is looking, an astonishing spatial listening experience is enabled. One seems to be surrounded by fictitous aural events. This is the acoustic dimension of augmented reality.

In comparison to visual augmented reality, the differences between the “normal” experience and the audio augmentation is more subtle. We are accustomed to not being able to visually locate a sound source, and thus “capture” it, for example when a car honks behind us, or someone knocks on the door. Also, artificially generated sounds are not clearly distinguishable from sounds organic to the environment, so with headphones it is not always possible to pinpoint the “real”. It is this propensity for “deceptive realism” that makes AR Audio an exciting proposition.

Today, more than ever, museums and visitor centres are faced with the challenge of innovative storytelling and wayfinding in guiding visitors skillfully around spaces and objects. In our own test rooms, we have probed the role of AR Audio in these contexts, and found AR Audio to open up to three perspectives for museums and visitor centres:

• With AR Audio, inherently silent exhibition objects can now acquire a new voice. For example, as one approaches an object, an aural experience such as the history or context of the object increases in volume the closer one gets. In this way, user behaviour influences individual experience.

• Entire exhibitions or even outdoor spaces cand be expanded with acoustic zones, invoking virtual landscapes. Since the sound is experienced individually, from headphones, there is no aural chaos—even in a complex soundscape accomodating many people.

• Visitor tours with musem guides take on new levels with virtual moderators located in the space. One can experience the tour as part of a group setting, or gravitate to another stationary narrator in a different corner of the room.

AR Audio is based on a virtual model in which the physical space is represented in 3D. Sound sources are placed at specific locations within the model. Furthermore, a high-precision location tracking system, based on ultra-wideband technology, is installed in the physical space. The movements of the visitors are recorded and transferred into the virtual model. Now the system sends those exact same sounds to the headphones placed at the respective location. The site-specific listening experience is completed by spatial information about the direction in which the users are looking. This is determined by a sensor attached to the headphones.

AR Audio extends the acoustic dimension of exhibitions by combining content mediation with an amazing sensory experience of space. It is this interlocking of different sensory perceptions that is exciting: What we hear changes our perception of what we see. Conversely, what we see changes our perception of what we hear. In this way, exhibitions and visitor centres have the potential to become extraordinary experiences.