In the late 1880s, the Edison Manufacturing company was producing wax cylinders for their Ediphone. These wax cylinders were fragile, wore out quickly, and the audio quality was much to be desired. However, for the first time, we were able to record live music, store that recording, and play it back at our leisure. In 1982, the compact disc popularized digital audio, and we’ve since continued to push the limits of quality and immersion when it comes to digital audio recordings. When it comes to recording classical music, there are many questions we must ask ourselves when deciding on how to record a piece – likewise, we also have to decide on how we want to experience those recordings. The variables are endless: is the performance in a concert hall or a studio? How large is the orchestra? How long is the piece? What are the acoustics of the venue? Is video being recorded alongside audio? Where/how is the recording being presented? One answer to all of these questions is to simply record everything; that is, to try as accurately as possible to represent the live experience of the concert in an immersive audio / video experience.
Part of the experience of listening to a classical composition live is the acoustics of a venue. The interaction between the sounds of the music and the physicality of the space create unique listening experiences that can’t be replicated, only captured. One such technique to do so is binaural audio recording. Binaural audio has some similarities to stereo, in that there are two channels (left and right) that allow you to perceive directionality in sound. However, binaural audio is special in that the microphones used are designed to mimic the input of sound through human ears. Binaural microphones rely on the shape of the human pinnae (the outer part of your ear on the side of your head) to mimic the spatial audio cues that we normally get when listening to sounds in person. Binaural microphones are either placed within one’s own ears, or mounted on a dummy head to further capture the interactions between our head, our ears and the space around us. Take a listen to the Eastman Saxophone project performing Peer Gynt Suite No.1, Op, 46 in binaural audio.
Taking things one step further, we can combine 360° video and high fidelity spatial audio to produce immersive, point of view videos. For these kinds of recordings, we need to use ambisonic microphones that record an entire sphere of sound around a single point. For example, this em32 Eigenmike contains an array of 32 individual microphones, all designed to record sound from different directions. Combining all these inputs allows us to create a 3D soundscape that, when combined with a 360° video, invites you inside the soundscape for a truly immersive experience. Take a listen to this live recording of Leonard Bernstein’s Symphony No, 1 “Jeremiah”.
Meet the man behind the media.
My name is Ming-Lun Lee, and I’m from Taiwan. I got my bachelor’s and master’s degree in ECE from National Taiwan University. After I served in the military, I entered the graduate institute of musicology also at National Taiwan University. After that, I got admission from the University of Buffalo and I got my PhD in musicology and music theory. And my dissertation is on stereo opera recording, the recordings by DECA producer John Culshaw, and I focused on the composer Benjamin Britten – he’s one of the most famous British composers in the 20th century. That also started my interest in stereo recording. I started my interest in classical music when I was a college student. I joined a choir, and my friend that liked classical music a lot, gave away a lot of classical music cassettes because at the time, people were moving to CDs as the cassette quality wasn’t that good. So I started listening to his cassettes and I found that I liked classical music a lot. So after that, I listened to a lot of music and a lot of radio programs and I decided to studied musicology.
Is there a specific time period, genre, composer that is your favourite?
No, I basically listen to anything from the middle ages to the 21st century!
What were some of your early experiences doing live concert recordings?
I started live concert recording by just recording my wife’s concerts and also my concerts. At first, I didn’t have a lot of experience doing that, I just read a lot of books and listened to a lot of recordings and watched some videos and found that at least it’s not difficult to record classical concerts. If you want me to record pop concerts then I’d need to set up a lot more microphones – it’s not as easy.
Three years ago, I had an idea to do some research because I listened to some binaural recordings. I was amazed at the sound quality, as it’s very different then stereo recordings. For stereo recording, you have 2 channels – left and right – and you can do some things like stereo panning, and get things closer to your eyes for a sense of virtual source – but that’s always between 2 speakers. After I listened to binaural recording, I was amazed how the sound simulated our actual hearing. Because our ears are so sensitive, we can hear sound coming from behind, above, below – I thought, why not? If I can use a microphone to record 3D audio then I only need to wear headphones to listen to 3D sound. Then, if I could develop a project to record concerts at Eastman, to find the best spots to put microphones, then everyone can listen to the recordings and it will sound like their always sitting at the best seats in the concert hall.
There’s a difference between just trying to record a song and capture the experience of a live performance.
I think many recording engineers like to do studio recordings so they can manipulate the sound to get the best quality. But, I like the sound in the concert hall, I like the acoustics. That’s why I was always trying to work around the concert hall to compare different acoustics to find the best spots to set up microphones. Even for live recordings, we aren’t just trying to capture the live sound, but you’re still trying to find the best techniques for recording to get the best sound quality possible. That’s why I tell all my students to always go to the dress rehearsals, so they can listen to the performance beforehand. Also for different pieces – a concerto or symphony, or for a choir or large scale work, or church organs – we need to go to the space to listen to the rehearsals to determine the best spots.
Is there a lot of documentation on techniques for classical concert recording?
There are only a few resources we can check. For recording, most of the time it requires your experience. So I was building the best workflows to record live concerts to record with 3D microphones.
What drove the decision to add the video component into these 3D recordings?
That was also three years ago when the university had an AR/VR pilot project, and I and a couple other professors – here and at Eastman – started a project together and applied for the funding to purchase microphones and 360 degree VR cameras. With audio, we can close our eyes but with video, to assist you, you can localize sound sources better using your eyes – it’s more of a complete listening experience.
How has technology changed the way we preserve and think about capturing musical experiences?
I think my musicology background is very important, because I know music very well. I know about different styles and genres, and so that’s why I have many friends at Eastman and I can talk to them about music. Even after the concert we can discuss our recordings. We cannot fix some errors in live recordings. But we can still try to talk about our microphone setups and figure out what gives us the best sound.
When you say errors, do you mean errors in the recording process?
Like wrong notes in the performances. We try to block an area so that we can record cleaner sounds without too much noise from the audience. But sometimes you can’t avoid that – even if you close your eyes, you still hear noises – people reading the programs, coughing.
It seems like you’ve taken up a bit of a challenge, as a recording engineer. In the studio, you have a lot more control over your environment and the kind of sounds that you’re are picking and producing. At a concert hall, you’re at the mercy of your environment. What are some specific challenges that you’ve run into?
The challenge of doing this is that there are still a lot of unexpected things that happen, and the amount of preparation required. For example, before any concert, we have to contact all the musicians to get their permission and to get permission from Eastman and the concert office, the department of multimedia, production and recording – thankfully they are very supportive, I’m very fortunate to be able to record a lot of concerts. But, even for live recordings, if you miss something – if you miss the button press to record, or you forget to format the memory card and you run out of space – if you miss something, you can’t do it again.
What do you think the musicians get out of having their performances captured in such a complete way?
That’s a very interesting question. I know for the directors that listen to the recordings, some of them don’t like it. Some of them do – it depends. It’s still very different from the sound they hear on stage. Think about if you are a conductor – I’m also a conductor – and you’re on stage; the musicians are very close to you. When you’re in the audience, you get a very different sound, more reverb. If you are on the stage, you get more direct sound, so they need to adjust to it – it’s not easy for them.
Also, another thing is that there are still some technical difficulties. For example, when we upload our videos to Youtube it compresses our video and audio, and the quality is not as good as our original audio and video recordings. Maybe 10 years later we can re-render all the recordings and have better sound, audio and video.
That’s interesting – no matter how high a fidelity you record your media on, if people are just watching things on their phone through their phone speakers, then they’d lose some of the quality that was there.
Yeah, we record in 8k and record lossless audio, but you still need to be able to play 8k video with a fast computer – also high speed internet too! So we’re just trying to prepare for the future. At least for the audio part, we’ve made some really good audio recordings. At least we can play the high resolution ambisonic recordings in our lab.
Is there any specific genre of classical you’re most (or least) fond of recording?
No, I think I’m interested in recording a variety of genres. Since two and a half years ago, we’ve recorded almost every concert of the Eastman Saxophone Project. I do like this ensemble very much, and I enjoy going to their concerts. They memorize the music – that’s amazing, usually they have movements on the stage. They’re very professional so we usually get very good sound for our recordings as well. We also record organ recitals at churches, operas, orchestras, wind ensembles, jazz ensembles, choirs, and chamber music – so basically we try to record as many genres as possible. Like tomorrow (December 13th, 2019) it will be our first time recording a Chinese music ensemble – they’re on tour and in Rochester, and it’s our first time to record Chinese instruments.
Are there any recording opportunities that you haven’t had yet, that you’re still looking for?
Yes! There are still a lot of things, different types of music – even just at Eastman. They also have period instrument ensembles, or more contemporary works. I’d like also to record some popular music concerts. I’d also want to record different ensembles - professional ensembles - if I had the chance. For example, the RPO, if I could record some of their concerts, that’d be great. I think for Eastman, they have such high quality concerts, and most of the concerts are free and they let me put lots of microphones and cameras in the audience. It’s hard if you want to record a concert where the audience has to buy tickets.
Keep in mind recording a live performance is so different from recording in a studio. Performers have a different kind of energy that they get from having an audience, so the sound is sometimes different, and it’s important to capture that.
Ming-Lun Lee is an Assistant professor of Electerical and Computer Engineering at the University of Rochester. With master’s degrees in electrical engineering and musical from national Taiwan University and a PhD in historical musicology and music theory from the University at Buffalo, Lee is well versed in the analysis and aesthetics of classical music recordings. Alongside academics and teaching, Lee is a talented choral conductor and baritone. He is the director of the Chinese Choral society of Rochester, and the vocal coach of the Taiwanese Choral Society of Rochester.