In the exhibition "Camille Pissarro. The Studio of Modernism", a distinctive sound art experience called "Pissarro Sounds" offers a sensory exploration of the artist's life and work. With the help of binaural spatial sonics and precise location tracking, visitors are immersed in three-dimensional sound spaces generated based on their position in the space. This installation, commissioned by the museum, was created by the artist Moritz Fehr. It is the first public application of the Innosuisse research project "Immersive Audio Guiding System" (IAGS), a collaboration of the FHNW School of Music, iart and Idee und Klang Audio Design.
- Highly precise location tracking
- Binaural sonics in your headphones
- Mixed Reality Plattform: Digital Twin is the basis of interaction in the room
- Virtual 3D model of the exhibition spaces with sounds and trigger zones
- Three-dimensional soundscapes
Generative Sound Artwork
"Pissarro Sounds" spans spans five rooms of the exhibition. With each space, visitors explore a dynamic soundscape embedded with Pissarro's original texts and ambient sonic textures. A special feature of the project: many of the recordings are taken from the actual site of Pissarro’s daily life. For example, visitors experience the sonic environment of his studio, a converted barn that still stands in Eragny, surrounded by the family garden.
Mixed Reality Platform
"Pissarro Sounds" expands the perception of the real exhibition space with live-generated soundscapes. Based on an interactive 3D model, an exact replica of the exhibition space, it is comparable to a computer game consisting of virtual rooms, various objects, sounds, interactions and trigger zones. This "digital twin" concept is the core of the Mixed Reality Platform (MxRP) developed by iart especially for museum application.
MxRP Headphones with UWB Tracking
High-precision location tracking records the movement of visitors in real time and transmits their position to the 3D model. The tracking is based on ultra-wideband technology (UWB), which is particularly suited to real-time localisation indoors. It is carried out via special headphones, which in terms of appearance and handling, are indistinguishable from conventional headphones, but in which additional technology is integrated. Each headphone receives signals from UWB transmitters distributed in the exhibition space, and can determine its own position with an accuracy of twenty centimetres. An IMU sensor (Inertial Measurement Unit) also determines the head orientation of the visitors. In this way, the system not only senses the users’ position, but also the direction they are looking in.
Algorithm for Binaural Synthesis
All this information is processed in real time. In the so-called binaural synthesis, both the properties of the human ear and spatial acoustics are taken into account: How big is the room? Are we indoors or outdoors? What material are the walls made of? Where is the sound source located in relation to the user? The answers to these questions give the experience of Pissarro's studio in Eragny a deceptively real three-dimensionality, in which visitors can hear the voice of a virtual Pissarro wandering through the space.
Potential of the Mixed Reality Platform
iart's Mixed Reality Platform enables future-oriented applications for museums and exhibitions on the basis of a digital twin. Apart from the current application, it also offers, for example, an ideal groundwork for the implementation of visual augmented realities, the provision of context-driven additional information in the language of the respective user or automatic light, sound, and video triggered by the position of the visitor.
Idee und Klang Audio DesignAcademy of Musik FHNWMoritz Fehr
Technical PlanningSystem DevelopmentSoftware DevelopmentSystem IntegrationProcess ManagementCoordinationElectronics DevelopmentInteraction DevelopmentControllingMedia technical planning