The euphoria.
There is currently enthusiasm around spatial audio, also known as 3D audio, terms that frequently appear at technological events. Today's technology leaders such as Apple, Meta, Microsoft and Dolby announce immersive experiences, of which spatial audio is an integral element. The trend is confirmed in the Gaming, Cinema and Music industries, which announce productions providing a growing sense of immersion [1][2]. The spatial audio industry is experiencing rapid growth. In a 2023 report, the market represented a value of US$5.7 billion in 2022 with estimated growth to US$11 billion in 2028 [3], and US$31.9 billion in 2032 [4].
What is Spatial Audio?
Spatial audio is a sound generation and reproduction technology that aims to match the experience of how we naturally hear through our ears. At its essence is signal processing that translates the characteristics naturally imposed by human anatomy and the surrounding environment into the propagation of sound waves, from the emitter to the ear canal of each ear.
The basic idea of this technology is to faithfully reproduce the perception of natural human hearing.
Spatial audio differs from traditional technologies, such as stereo, in the richness of information contained in the signal. With more information available, the brain has a more complete perception of the sound environment, particularly its spatial representation.
Spatial audio formats can be reproduced through loudspeakers or, more conveniently, through headphones, in which case the resulting signal is binaural audio.
Figura 1 - Representação do uso do theatrophone (adaptado de [5])
Immersive experiences.
Our natural experience as humans builds our perception of the environment by integrating information from our senses.
A technological goal has (historically) been to be able to reproduce the natural experience in an increasingly convincing way – an immersive experience. The contribution of spatial audio to immersion is the capacity it provides for an auditory experience that is perceptually indistinguishable from the natural one, providing the sensation of being at the centre of a virtual soundscape.
A pioneer in the search for a more faithful spatial representation, the theatrophone (1881 to 1932) is considered the first device for public reproduction using binaural audio. But it was through cinema that spatial audio reached a wider audience when a surround system was implemented for the screening of the popular Walt Disney film “Fantasia” (1940).
Figure 2. “Oscar”, the mannequin with microphones embedded in his face (1931). His pioneering binaural captures surprised by the authenticity of the spatial representation (adapted from [6]).
Continuing to explore more realistic systems, the development of the 1973 Dolby Laboratories surround system, aimed at the cinematography industry, would be the starting point for the popularization of this format of stereo, both in cinemas and in-home use. At the same time, with advances in transducer technology, binaural audio also gained prominence. In 1978, Lou Reed released Street Hassle, the first commercial pop album recorded in binaural audio.
The modern era has brought renewed interest in spatial and binaural audio, in part due to easy access to equipment, as well as, in a never-ending quest for richer experiences, a general increase in commercial interest in the technology. In 2021, the Apple-Dolby partnership made its binaural audio format available on the streaming platform, placing both companies at the media centre of today's spatial audio revolution.
Modern immersive systems are usually supported by three sources of information, vision, sound and touch, and aim to reproduce the perception of real events. Current VR technology has established itself as the platform of choice for implementing immersive environments. With its increasing adoption, it is possible to see a greater prevalence of spatial audio in the generation of increasingly realistic immersive environments, granting the sensation of presence in a virtual world.
Figure 3. Tools like VR Glasses [7] (left) and CAVE systems [8] (right) are paired with spatial audio for immersive experiences
Spatial audio in scientific research.
Far from the media centre is the news about the contribution of spatial audio to scientific research. The authenticity conferred by spatial audio opens a window to new opportunities. Its usefulness when combined with VR is notable, providing a multimodal platform for carrying out ecologically valid and replicable experimental protocols. In other words, it enables audiovisual experiences with participants in the laboratory with a high degree of realism, allowing precise manipulation of complex use cases.
An obvious case is the contribution of spatial audio to progress in understanding the hearing aid. The realistic simulation of the effects of different types of hearing loss and hearing aids allows us to investigate the consequent deterioration in auditory perception [9]. Such simulations also have practical advantages, such as in the process of installing hearing aids, which are traditionally based on simplistic stimuli and acoustic conditions. Using VAEs, complex acoustic conditions that resemble real cases can be simulated in the evaluation of different algorithms and hearing aid configurations [10].
Another example, in this case, applied to fundamental research in the study of the perception of the movement of a sound source. Previous studies in this area were often disturbed by interference and limitations of the mechanical apparatus for generating the movement of the sound source (loudspeaker). In VAEs, these limitations are easily overcome, allowing complex trajectories and precise control of external noise [11].
Research on urban soundscapes in the transition to electric vehicles is another topic where several studies use this technology to generate realistic and easily manipulated soundscapes, including studies on the perceptual effect of the spatial arrangement of sound sources [13] [14]. In a study investigating the decision process when crossing a road crossing, the effect of noise emitted by an approaching vehicle was evaluated. To carry out experiments with participants, a VR crossing scenario was implemented incorporating dynamic audiovisual simulations in which the soundscape included the noise emitted by the vehicle, characterized by intensity, directionality, trajectory and orientation [15].
Figure 4. Experiences in a crossing scenario carried out in the CAVE immersive environment [15].
The application of immersive environments has been seen as an increasingly important tool in healthcare, where stress reduction is an active topic of research. As a complement to music therapy, environments employing spatial audio are effective in reducing stress in individuals diagnosed with depression and anxiety disorders [16].
Concluding:
Spatial audio technology has become crucial for the implementation of highly immersive environments, increasingly driven by progress in VR [17]. As these technologies gain popularity, exciting new spatial audio applications are expected, not forgetting that the implications of spatial audio will have an impact beyond the world of entertainment. Although there are obstacles to overcome, such as the lack of standardization of communication formats and protocols, the future of spatial audio appears promising. As history seems to show, technological advances fuel the desire to replicate realistic experiences with increasing fidelity.
Per Frederico Pereira - HTIR Senior Development Technician
[1] https://appleinsider.com/articles/23/01/29/why-spatial-audio-is-the-future-of-the-music-industry-even-if-you-hate-it
[2] https://www.rollingstone.com/culture-council/articles/spatial-audio-next-revolution-in-audio-1234609914/
[3] https://www.marketwatch.com/press-release/3d-audio-market-size-report-estimated-at-usd-575003-million-in-2023-with-growing-at-a-cagr-of-1152-forecast-period-2023-2028126-pages-report-2023-02-15
[4] https://www.futuremarketinsights.com/reports/global-3d-audio-market
[5] https://medium.com/museum-of-portable-sound/the-théâtrophone-worlds-first-stereo-audio-broadcast-paris-1881-a4bf302cde35
[6] S. Paul Binaural recording technology: A historical review and possible future developments
[7] https://varjo.com/products/vr-3/
[8] https://steantycip.com/projects/virtual-reality-tore/
[9]. Cuevas-Rodriguez, M., Gonzalez-Toledo, D., Rubia-Cuestas, E., Garre, C., Molina-Tanco, L., Reyes-Lecuona, A., ... & Picinali, L. (2017, May). An open-source audio renderer for 3D audio with hearing loss and hearing aid simulations. In AES Convention 142.
[10]. An Extended Binaural Real-Time Auralization System With an Interface to Research Hearing Aids for Experiments on Subjects With Hearing Loss, Pausch et al 2018
[11]. Carlile, S., & Leung, J. (2016). The perception of auditory motion. Trends in hearing, 20, 2331216516644254.
[13] Rajguru, C., Obrist, M., & Memoli, G. (2020). Spatial soundscapes and virtual worlds: Challenges and opportunities. Frontiers in Psychology, 11, 569056
[14] Sanchez, G. M. E., Van Renterghem, T., Sun, K., De Coensel, B., & Botteldooren, D. (2017). Using Virtual Reality for assessing the role of noise in the audio-visual design of an urban public space. Landscape and Urban Planning, 167, 98-107
[15] Soares, F., Silva, E., Pereira, F., Silva, C., Sousa, E., & Freitas, E. (2020). The influence of noise emitted by vehicles on pedestrian crossing decision-making: A study in a virtual environment. Applied Sciences, 10(8), 2913.
[16] Greenberg, D. M., Bodner, E., Shrira, A., & Fricke, K. R. (2021). Decreasing Stress Through a Spatial Audio and Immersive 3D Environment: A Pilot Study With Implications for Clinical and Medical Settings. Music & Science, 4, 2059204321993992
[17] Kern, A. C., & Ellermeier, W. (2020). Audio in VR: Effects of a soundscape and movement-triggered step sounds on presence. Frontiers in Robotics and AI, 7, 20.], [ [Li, H., & Lau, S. K. (2020). A review of audio-visual interaction on soundscape assessment in urban built environments. Applied acoustics, 166, 107372.
