Creating an Ecosystem to Provide an Immersive Sound Field to Even More Customers
360 Reality Audio
November 4, 2020
Sony’s 360 Reality Audio makes listeners feel as if they are immersed in sound from all directions, as if a band is playing right in front of them. Since its launch last fall, this new music experience, which is the result of combined knowledge across the Sony Group, has been gaining attention through successful collaborations with famous artists. Sony is currently building an ecosystem to bring this experience to people around the world.
Tokyo Laboratory 20
Product Technology Dept.2
V&S Business Group
Sony Home Entertainment &
Sound Products Inc.
Business Development Dept.
V&S Business Group
Sony Home Entertainment &
Sound Products Inc.
Sony Music Entertainment
Sony Music Entertainment
The sense of immersion produced by a spherical sound field
──What exactly is 360 Reality Audio?
Maki Tomioka: 360 Reality Audio is a new music experience that uses Sony’s object-based audio technology to create a realistic sound field. We are creating an open ecosystem by providing the technology needed for the production, distribution, and playback of compatible contents. Specifically, during content creation, location information is attached to each sound source, such as the vocals, chorus, and instruments, and placed within a spherical space. Sony is proposing a wide range of technologies to creators and artists in order to smoothly introduce this new form of entertainment to a wide scope of users. Since the service’s launch in fall 2019, compatible music in a variety of genres has been distributed by multiple streaming services in Europe, the US and Asia.
──How is 360 Reality Audio different from other three-dimensional sound field experiences?
Yukiko Unno: It has three main characteristics. Firstly, because it is object-based from production to distribution, it can be adjusted to a variety of listening styles on different playback devices, allowing for high quality sound field reproduction. In this context, “object-based” refers to a method that delivers sound material and metadata to generate audio signals according to preferences and sound systems on the receiving side. Secondly, during the content creation process, creators and artists can freely position the vocals, choruses, instruments, and other sound sources in a spherical space.Thirdly—and this is the most significant feature—is that by analyzing an individual’s hearing characteristics based on photos of their ears taken with a smartphone, acoustics can be optimized for that individual, allowing them to enjoy the 360 Reality Audio’s sound experience even with headphone playback. Apart from that, it is based on the MPEG-H 3D Audio international standard, so it can be used widely throughout the music industry, and its format allows for streaming through smartphones and other mobile devices, making it ideal for music distribution services.
──What technologies are used to analyze individual hearing characteristics?
Kazumi Fukuda: To reproduce an immersive spatial sound field, it is very important to optimize the playback for each listener’s unique hearing characteristics. By the time a sound reaches your eardrum, it has already been reflected and diffracted on several different surfaces, including the floor, walls, and even your own body. The sound’s characteristics change depending on where one’s ear is relative to the sound source, and people subconsciously determine where a sound is coming from based on these differences. 360 Reality Audio uses these sound characteristics to create the perception that sounds are being heard from outside of the headphones, thus reproducing a spatial sound field. However, people perceive sounds differently depending on the shape of their ears and head. Therefore, individual optimization is necessary for deeper immersion. We have developed a personal optimization algorithm that uses machine learning to predict hearing characteristics from photos of ears based on our proprietary database that catalogs the relationships between head/ear shapes and hearing characteristics. Thus, by taking pictures of their ears using the "Sony | Headphones Connect" application, users can use the algorithm on the cloud to analyze their own hearing characteristics and create an optimized sound experience.
──What was the most difficult part of developing and launching this service?
Unno: Improving the accuracy of individually optimized sound experiences was a challenge. Sony’s proprietary technology for extrapolating individual hearing characteristics from photos of users’ ears uses a combination of image processing technology, AI technology, spatial acoustic theory and a database of the human ear, all of which were developed in-house. We were very proud of the idea, but to put it into practice, we needed to review our database and establish a method for users to take the photographs. A lot of work went into getting the model ready for launch. In the beginning, we struggled to improve the sound quality and auditory localization, making it difficult to justify individual optimization. So, we reviewed the processing flow from the beginning and identified the processing blocks that were affecting accuracy. By concentrating on improving the algorithms for these blocks, we were able to improve the sound quality and auditory localization. We are very grateful to those who helped us with the numerous evaluations. Ultimately, we were able to achieve a more immersive sound for the launch. The product was evaluated both in Japan and overseas. We set up listening rooms where people could experience 360 Reality Audio and conducted sound quality evaluations for general listeners. We were nervous about how this new music experience would be received, but we got very positive feedback, such as "It was like being surrounded by sound" and "It was just like being at a live concert." This gave us confidence leading up to the launch.
Tomioka: Leading up to the launch, we faced all the challenges of a new business. There were tough negotiations with labels, artists, mixing engineers, and music distribution companies. We also traveled overseas to demonstrate the value of 360 Reality Audio to as many people as possible. This new music experience has the potential to stimulate the industry, and we thought that if people could understand the value of what we were offering, it would expand our network of partners. As it happens, we received very positive feedback after the demos, even from people who were initially skeptical about spatial audio, and the demos often led to further discussions as well.
Involving the whole music industry in content production
──You are also focusing on creating compatible content. Why are you focusing on content creation in addition to technology development?
Tomioka: The production of a wide range of high-quality, attractive content is key to creating a new ecosystem for people to enjoy spatial sound such as 360 Reality Audio. Sony’s R&D Center, Sony Home Entertainment & Sound Products and studio engineers closely collaborated to promote content production leading up to the service launch.
──How did you collaborate to create content for 360 Reality Audio? Can you tell us about your specific workflow?
Fukuda: The R&D Center has developed 360 Reality Audio production tools and provided them to each label’s production studio. These production tools allow engineers to freely place instruments and vocals in a spherical space, and the on-board spatial sound technology allows them to confirm the location using both headphones and speakers.
Tomioka: The Business Development Dept. has introduced production infrastructure at Sony Music Entertainment (SME) and other affiliated studios, and trained their engineers on how to produce content. After discussing potential content with streaming services, we approach the labels and ask them to produce it. We have also developed a 360 Reality Audio app to proceed with artist approval for remixed songs.
Unno: Product Technology Dept.2 collaborated with the Business Development Dept. and studio engineers to create a collection of tips for content creation. Through many trials and experiments, we have put together a method of effectively producing 360 Reality Audio’s key characteristics of auditory localization, spaciousness, and movement.
Mark Wilder: Battery studios has worked hand-in-hand with each team offering ideas for software and technology advancements as well as helping with the preparation of content from SME and other record labels. Since Battery Studios is committed to the idea of One Sony, this has been a very fulfilling process.
Delivering moving music experiences to
customers around the world
──Please tell us about future plans and possibilities.
Tomioka: Since the 360 Reality Audio launch event last October in New York City, we have been receiving a lot of positive feedback from industry professionals and customers, and 360 Reality Audio’s market presence has been steadily growing. Recently, we have been working to distribute songs from popular SME artists such as The Chainsmokers and Kane Brown. Going forward, Sony intends to build partnerships in each country, with the aim of further expanding their user base.
Unno: I think that we were able to create this new musical experience in the emerging field of spatial audio thanks to the diverse knowledge of our employees. We hope you will enjoy this new, immersive sound. We will continue taking on the challenge of developing new technologies in the audio field.
Mike Piacentini: It’s been a great honor and pleasure over the past few years to see the collaborative efforts of all Sony group employees come together to launch a product, across many different business groups in different countries.
Fukuda: With entertainment continuously evolving in our rapidly changing social environment, I would like to provide new musical experiences to our customers and search for new ways to deliver kando to them.