The author of this special report is chief engineer of WBUR Boston and the former tech editor of Radio World Engineering Extra.
The subject of Audio over Internet Protocol is large, wide-ranging and frequently discussed. But for this article we’ve spoken to a number of radio industry experts to get a fresh point of view.
These are people who have built, maintained and used systems in large and small installations. The kinds of projects profiled here range from the basics of building out studios that are simply reliable, to wildly imaginative ways to use this technology.
First let’s reflect on where we started and how we got to this point.
A short history
While today there are a range of IP-based audio products to cover nearly every application in radio, including fully virtual solutions, it has taken several decades to get to this point.
I well recall the early efforts of streaming with TCP/IP over the public internet back in the days of 33 kbps telephone modems, just 25 years ago. While streaming audio in those days was mostly a “proof of concept” rather than listenable, those early demonstrations showed the potential of using the rapidly expanding public internet as a way to “broadcast” audio.
Using IP, packets of audio data could be addressed for delivery to specific destinations across local- and wide-area networks spanning continents.
A second enabling technology was the use of Ethernet protocol to build scalable computer networks at ever-increasing speeds via low-cost, short-distance, copper connections. It too played a role in early AoIP development. Due to the lack of copyright limitations, manufacturers rapidly jumped into this technology. Improvements in Ethernet performance have since followed a logarithmic curve, driven by the vast marketplace for personal computer networking. The audio industry now has adopted Ethernet wholesale as a transmission medium, allowing us to take advantage of the incredible cost and performance advantages of this technology.
Even further back, the most important enabling technical developments were the digital signal processing advances in the 1950s, ’60s and ’70s, in which the theories of Shannon, Nyquist and others were proven to work in practice. The ability to convert analog signals to digital and back again without loss of quality — rapidly, reliably and cheaply — represents the crucial first step toward digital transmission, freeing audio from the inherent limitations of analog.
The combination of these three enabling technologies has led to the world of AoIP that we enjoy and which becomes more powerful and reliable every day.
One of the first problems to solve in the progress toward AoIP was a method of using Ethernet protocol to transmit signals in real time. While the use of network switches is taken for granted now, the original Collision Detection algorithm and network hubs used for data were fundamentally unsuited for uninterrupted (real time) signals.
Early systems such as Cobranet and Ethersound solved the problem of packet collisions by creating a master timing clock that controlled permission to transmit on the shared receive and transmit wiring. It was during this era that the importance of timing in the transmission of networked audio was discovered. High-speed variations, known as jitter, would cause audio to distort. Long-term variations due to delayed packets and network propagation times would cause the original timing to drift or the feed would go periodically silent.
By using network speeds that far exceeded the audio demand, it was possible to compensate for jitter and long-term delay by creating a buffer for data that could be filled in high-speed bursts. The audio output could then be spooled out of this buffer at a slow, but maintainable, speed with a local clocking system restoring the original audio timing.
Early systems using 10 Mbps Ethernet transmission — 10 Mbps representing the maximum speed at which individual bits could be transmitted in the absence of overhead packets and collisions — were more than adequate to handle the roughly 1.5 Mbps data rate of PCM digitized stereo audio. This rate had been standardized by the massively popular audio compact disk (CD).
As Ethernet hubs increased speed to widely available and inexpensive 100 Mbps models (later improved to switches which by design are not limited by packet collisions on the network wiring), it became feasible to multiplex a number of PCM streams on a single network branch without them interfering with one another.
Although limited to in-house systems where the timing could be successfully synchronized to one master clock system, LAN-based audio transmission was born. Unlike analog systems, the transmission distance did not affect the phase and frequency response of the audio signal after digitization. The promise of inexpensive and virtually perfect fidelity, digital transmission came into view, a huge step toward today’s AoIP systems. Another bonus was the ability to combine multiple signals on a single wiring system.
It’s amusing to recall the primary competitors for audio transmission in that era. While copper audio pairs within the public telephone systems were already on a path to obsolescence, many radio stations still relied on them to transmit analog audio over a distance of many miles. Powerful equalizing and phase compensating amplifiers were used to repair the distortions caused by the capacitance of lengthy wiring segments.
Still, there was a limit beyond which this compensation could not be used. Digital audio was the solution to this distance limitation.
In the late 1980s, to replace those analog copper circuits, the various Regional Bell Operating Companies introduced a means of digital transmission of audio signals in the form of Switched 56, or 56 kbps clock synchronized data circuits. Shortly after, Integrated Services Digital Network, or ISDN, service was widely made available and rapidly became the most popular form of digital audio transmission. It was limited initially to 64 kbps, and then later used two-channel bonding to achieve 128 kbps.
Radio, the recording industry and even film producers relied on ISDN for more than two decades. What enabled this technology to succeed was the development of coder-decoder techniques that pre-treated the digitized audio stream, reduced it to a much lower rate of transmission and then restored it after digital transmission.
The devices that incorporated these data reduction techniques were called codecs, the combination of the words “coder” and “decoder.” They were the first cost-effective means of digital audio transmission over long distances. ISDN “calls” could made between any two capable telephone centers all over the United States and overseas to Europe.
The widely used Fraunhofer MPEG II audio codec for a time became nearly universal on bonded ISDN links at 128 kpbs. This type of codec divided the audio into frames, analyzed the frequency content in each frame using a Fast Fourier Transform, and then discarded 85% of the audio bits that were effectively inaudible to human hearing due to acoustic masking. At the far end it would reconstitute the original audio signal. Note the key use of rapid DSP techniques in these operations.
Codecs using acoustic masking techniques are still widely utilized, now combined with the power of IP networking in place of serial data transmission systems like ISDN.
AoIP remote links
A widespread use of ISDN technology was for remote broadcasting. In the early 2000s standalone IP codecs were produced that could take advantage of IP encapsulation. As AoIP moved beyond the LAN, it added large buffers and user-settable dynamic buffers to allow for a wide variation in timing and jitter as experienced on links over networks not controllable by a local master clock.
This development coincided with the deployment of internet access into public and private locations. Telephone and cable companies that had copper wiring, coaxial cable and even fiber optic connections into businesses and homes shifted their marketing to high-performance network connections to meet escalating demand.
At the same time, they let it be known that ISDN was a sunset technology that would not be supported for much longer. Indeed, after Hurricane Sandy in 2012 literally flooded out a large central office technical center in Manhattan, Verizon New York announced it would soon eliminate any new ISDN services in the Northeast.
The slow withdrawal of ISDN allowed IP codecs to be deployed with advantages that would soon outweigh the value of the older technology.
The introduction of high-speed wireless data in its “3G” or third-generation model in 2001 made mobile internet access possible, creating a whole new type of product that allowed remote broadcasting anywhere that wireless data was available.
By 2010 ever-faster speeds were achievable over wireless, in some cases up to 100 Mbps download speeds, when 4G and 4Ge systems were deployed.
Claims for 5G systems now being built are to offer 1 Gbps downloads to mobile phones. Portable codecs with wireless modems can connect with seemingly endless bandwidth for remote broadcasts from virtually anywhere within the site of a cell tower.
STL and remote systems
AoIP made early inroads into radio as a studio-to-transmitter link, or STL. Audio codecs form the core of AoIP audio transport over interconnecting networks that don’t necessarily maintain the same timing or path length. By encapsulating an audio codec within IP packets, the modern, routable codec combines the power of LAN-based audio transmission with the widely available public internet, capable of providing high-quality audio links to almost anywhere.
The use of AoIP codecs as modern STLs is demonstrated today by a variety of models that largely use the public internet as an inexpensive means of distribution. At both ends a device with an Ethernet interface (also known as NIC, or Network Interface Card) is associated with a unique address on the public internet. These connections are installed at the customer location by internet service providers.
Using a pair of codecs to create a one-way audio path from studio to transmitter is as simple as pointing the local codec to the remote address. Since the connection provided by the ISP is bidirectional, the corresponding codec on the far end is typically, but not always, pointed back to the local (a widely-used application is to have the output of an air monitor feed a confidence stream back to the studio).
Since the 1996 Telecommunications Act, the radio industry has seen a continuing trend toward consolidation of radio license ownership and shared transmission facilities. AoIP STLs are a necessary support for that consolidation as they allow inexpensive and readily available internet access contracts virtually anywhere.
One example is The Joy FM, a group based out of Sarasota, Fla., that has more than 20 transmitters in Florida, Georgia and Alabama. Acting Chief Engineer Avery Ditmars spoke with me about their deployment of AoIP codecs to link all of these sites.
“We have a total of roughly 50 Telos Z/IP One codecs that we use to connect to all our transmitters, which includes 12 sites in Florida and a sister station with nine transmitter sites in Georgia,” said Ditmars. “A Telos Z/IP per station is used to encode 12 individual streams of audio to different locations. I chose the Telos line because it supports the Livewire interface from our studios, making it a completely AoIP path. All the way from the mic processor to the transmitter.”
The Joy FM first began with a satellite delivery system over C-Band. The transition to AoIP codecs for STL was accomplished over the course of several years.
“We decided to convert to using the internet because it gave us the flexibility to do what we needed to do, without costing an obscene amount. We discontinued C-Band in 2014 and as we’ve added stations we’ve built around the Telos Z/IP as our core STL,” he continued.
Ditmars’ experiences have taught him some lessons about the best way to build STLs using AoIP over the public internet.
“In this business you have to eliminate single point of failures as much as possible,” he said. “So we also maintain a backup network of links using Barix codecs to every site, and we’re able to switch these into service automatically if we detect silence on the main links. We also have two, if not more, ISPs for every site in case of problems with one of our Internet providers.”
He noted the flexibility of the Barix codec as a fully compatible RTP device. ”We use Barix as our backup audio device, streaming from a server on a completely second path from our main VPNs. The compatibility of Barix with whatever encoder we put on the other side is why we went the direction we did.”
One other recommendation from Ditmars is about the importance of security on these critical links.
“It goes without saying that you have to change ALL the default passwords on every device that you connect to the internet,” he said. “You have to build these links with full security to prevent someone from hacking into your system. All of our codecs are located so they connect within a VPN tunnel. Nothing is ever connected directly to the internet at any site.”
Work from home
AoIP codecs of course aren’t just used as links between the studio and transmitter. For instance they have been used for years to build out remote or home studios.
This capability turned out to be transformative for SiriusXM when it decided to move most of their hosts to home during the lockdowns put in place last spring when the pandemic began to spread throughout the United States.
Senior Director, SiriusXM Enterprise Operations Daniel Berdiel explained the scope of the problem.
“We needed to provide program hosts, guests and studio teams a way to work from home safely,” he said.
“On the hardware side, we needed a way to scale up our remote sessions capacity in a small hardware format. Because we already had substantial experience with Comrex portable codecs we chose the ACCESS Multirack. The AES67 interface and its ability to make five simultaneous connections made it easy to integrate in our facility. In most cases, each show uses two or three instances for hosts, and one instance for the remote producer/board op.”
For the remote side, SiriusXM used a standard setup. “The primary remote kit uses an ACCESS or NX Portable, Beyer DT297 headset and sometimes a cough switch. An IFB was already in place that the studio can interrupt to speak to a host,” said Berdiel.
The challenges facing SiriusXM were more than what one would find in the average large radio cluster. “We have about 400 hosts who are on remote operation,” said Berdiel.
While AoIP was finding early application in remote broadcasting and STLs, the developments were not limited to transmission to external locations. Studio systems began to develop at the same time.
In many cases AoIP mixers were added as studio expansion, creating islands of operation. However, it was clear from the beginning there were advantages to AoIP being a full facility standard, interconnecting every device in every room.
At Corus Radio Inc., Greg Landgraf, manager, radio technology for the western region, describes how they came to build out an entire studio facility with AoIP after a fire badly damaged their complex in 2012.
“The main driving factor in choosing AoIP for the temporary and then the permanent installations in Calgary was speed of installation. The systems can be racked, wired and configured in a fraction of the time required for an analog build,” he said.
“Obviously, the incredible power, redundancy and scalability were huge factors as well. In the past a desired change in a studio involved significant effort to accomplish. Pulling wire and connecting equipment has been replaced, for the most part, by a few mouse clicks in configuration, in an AoIP world. Also, the vast options available for configuration of audio and control in AoIP simply are not available in traditional analog without significant time and available dollars.”
The new home for Corus Radio Calgary occupied about 24,000 square feet, of which around 11,000 were technical spaces such as studios and the TOC. There are five main control rooms plus three general control rooms, four production rooms and talk studio space.
The core systems are based on Wheatstone’s Wheatnet AoIP. The project took about nine months. “The AoIP portion of that was in the two-month range. This is very impressive considering an analog build could take two or three times that.”
In spite of the need to move quickly, the project exceeded its goals. “All expectations were met. Those were to replace and mimic our previous analog setup, which it did without breaking a sweat,” he said.
“The virtually forced transition to AoIP has opened new doors in studio configurations and uses. A control room AoIP system can be loaded with a number of configurations for immediate changes of that room to function in different modes.
“Take an FM MCR; once the morning show is finished using the room in a traditional fashion, it can be reconfigured, by a simple button push, to become a voice/talk studio or visual radio space, all the while sending automation to air uninterrupted.
“The only thing I can share as a warning or concern revolves around system stability,” said Landgraf. “In a hardwired analog system, there is not much that can go wrong as it relates to audio wires … they are hardwired and carry one signal. In AoIP those are defined by software, crosspoints, AoIP packets travelling the network, network switches, etc.
“As in any computer/network-based system, there is potential for glitches to disrupt that flow and even, on occasion, mysteriously change audio sources, settings and configurations. Having said that, I am willing to deal with the occasional glitch for the incredible power of AoIP.”
Landgraf also deploys a wide selection of Tieline AoIP codecs. Like Joy FM, Corus has converted its affiliate distribution to AoIP codecs.
“We have a national Genie Distribution network set up in Toronto to distribute syndicated programming to all Corus radio sites. This system was installed to replace our previous satellite distribution,” he said.
Corus uses the Report-IT application for live news and special programming. “A number of our commercial remotes are also done on the Report-IT app using the FTP functionality to record and upload, which is then ingested into our Burli system for playout — we call this ‘As Good as Live.’
“This is how we receive traffic reports from our traffic helicopter auto ingested into Burli for playout, similar to our commercial remote setup. We have also used this method to do a number of remote live shows, on our music stations, from faraway lands, as sometimes their connects are suspect, and this increases the reliability and integrity of each break. We have a national Tie-Server system set up for live remote contributions from radio and TV news staff via Report-IT and their phone device (iPhone or Android). Corus has one of the largest Tie-Server systems on record.”
Landgraf, like Bardiel, also mentioned demands for “Work From Home” solutions. “A number of ViA codecs are in use, as show hosts are working from home. With the ViA’s capabilities we can feed local computer audio and as well as record off-line audio for later playout all on the ViA. These are connecting through our Peplink/Pepwave VPN system to ensure reliability and security.”
Mic to transmitter
Dennis Eversoll, Midwest regional director of engineering at Cumulus Radio, discussed with me via email the importance of AoIP in a 2019 consolidation project of 17 studios in Kansas City.
He and Assistant Chief Engineer Kevin Cox, who plays a lead role in the new digital systems, installed Axia for their infrastructure and an air chain from the studio all the way to the transmitter using AoIP STLs.
“Our facility was originally built out in 1997, and it was a mix of analog-only control rooms, some hybrid digital/analog rooms from 2006, and even one AoIP studio from 2016,” said Eversoll.
“We decided on a new space of 17,000 square feet and determined from the start that it would be entirely AoIP, since we had successfully done similar builds in San Francisco and Chicago already, based on the Axia Fusion and IQ Series consoles. The studio phones would use an all-new VX [phone system]. ”
The use of analog and AES3 digital formats was minimized as much as possible.
“The only analog audio in the facility originates in specific source equipment like CD players and flash recorders. Those devices are immediately interfaced to the Axia network via nodes. Literally every source anywhere in the building can be routed anywhere, and this is completely controlled by the routing software, so we no longer had the need for extensive audio routing system we had previously employed,” said Eversoll.
“The biggest change for me personally was the fact that we would no longer be using any conventional method of program delivery to our eight different transmitter sites around the city. Our primary and backup STL system would now also be IP-based using the GatesAir IP Link units.
“We established an MPLS circuit for the primary program delivery and, by using the second network connection on the IP Link 200 units, we have the failover set up to move to an open internet feed that is available at all our sites.”
The new STL is an improvement over legacy techniques. “This setup allows us to perform all our audio processing at the studio and then send the exact same processing to multiple sites including several auxiliary sites where previously we had to operate separate processing at each location,” said Eversoll.
He noted the extensive change in design for an AoIP Technical Center.
“Our technical operating center has no audio flowing within it, it is simply multiple networks including business systems, Axia audio, phone systems (both business and studio phones), as well as other networks designed for both security and accessibility,” said Eversoll.
Plan for advantages
Their decision to use AoIP was motivated by a desire to make a better studio complex, taking advantage of technology improvements but also to improve usability for staff by selecting a common platform.
“Like any large cluster build, it was our objective to make it easy for talent to move from studio to studio and find identical setups from one room to the other,” Eversoll continued.
“We had enjoyed this at a time in our history but had strayed away from that through years of ownership changes and new technologies. With the proper programming a single button can reroute everything involved in an intricate studio setup and make the move to another studio seamless with no loss of functionality. This was possible before, but not without a variety of patching and multiple routing.”
Recent events have also driven home the unexpected benefits of AoIP technology for Cumulus.
“Less than a month after completion, along came the pandemic, and we effectively closed the facility. Fortunately in the move we had dropped all the older technologies such as ISDN and had installed about 10 new codecs [a mix of Comrex, Tieline and Telos models]. These, along with the outstanding remote voicetracking capabilities of our on-air systems, allow us to do both live and automated shows with virtually no personnel in the building.
“Using our VoIP telephone capabilities, we are even sending studio calls to talent operating from home studios. If necessary, our consoles are capable of being remotely operated from off site,” said Eversoll.
“Those of us in engineering also have the capability of analyzing and changing anything in the facility from off-site since almost the entire infrastructure is IP-based.”
Eversoll had some final words for anyone preparing for an AoIP buildout: “In my opinion AoIP is the only way to go in these times; but the importance of intense planning is paramount. Just be prepared for intense research and finding people with experience to lean on early in these projects.”
Another notable example of AoIP in action is at Onda Madrid, a public station in Spain that has a major commitment to news and sports. It recently installed an AEQ BC-2000 D Router audio matrix, updating its digital consoles to support AoIP connectivity and adding an Arena digital mixing console in its Recording Studio 1.
The modular router central control room matrix includes 16 analog, eight AES/EBU digital and 128 AoIP Dante inputs/outputs; it’s complemented by three NetBox 32 AD Dante AoIP interfaces, one for additional routing of the central control and link rooms, a second to provide booth connectivity and the last as a backup. Each interface connects 32 AoIP inputs and outputs to 16 analog and eight AES/EBU digital stereo I/O.
The communications system is a notable part of the installation. An AEQ VoIP phone system and fleet of Phoenix Stratos codecs are integrated into software that unifies control of broadcast telephony and audio routing for the ISDN/IP codecs.
“On one hand, in the Central Control Room, there is an AEQ Systel IP 16 system providing 16 voice over IP lines and supporting four AEQ SystelSet+ communications terminals, which include an IP phone and a touchscreen running the communications control software,” according to a project description.
“These have been installed in the Continuity Studio and the three recording ones. The 16 available phone lines can be dynamically assigned among these studios as required by each of them at any particular moment.”
Meanwhile the system also includes 10 AEQ Phoenix Stratos codecs supporting both ISDN and IP connectivity; these can be assigned to create permanent links to transmitter centers, external studios and remote vans.
“Eight workplaces have been licensed for this unified communications software. It allows for the management of the audiocodecs and the IP broadcast telephony system from a single screen.” Each user can operate the external links using codecs and prepare phone calls for a studio — acting as a producer, supervisor or custom function depending on their access rights.
The deployment also included 10 AEQ ALIO external reporting systems with IP connectivity in portable cases, with router/modems for connectivity over 3G/4G mobile IP networks.
Another AEQ user, Tom Walker, technical director at Black Country Radio in the United Kingdom, was pleased to find an easy integration path to the Dante audio transport system, often used by live sound systems for its high quality and efficient transport. He said Dante “was our chosen audio environment to future-proof our new studios.
“We have no audio loss anywhere in our broadcast chain from studio to our transmitters. We really are at the forefront of a fully Dante digital station.”
Walker was enthusiastic about the capabilities that came with an AoIP installation. “We have had so much fun installing the new AEQ consoles using Salvos and virtual I/O signals; we were finding new features daily, especially when it came to Talkback. We had a lot of fun humoring the install team.”
The WAN network
With AoIP addressability there is theoretically no limit to the size of the network that can be connected, although it presents challenges in getting sufficiently fast network connections.
I spoke with Bill Jackson, principal studio engineer, Educational Media Foundation, about their goal of building an AoIP network in which every device in every room is controllable across a network of 32 studios physically located in three cities. In effect, an AoIP router is being built across large geographical distances. He is currently working on the buildout of its Franklin, Tenn., facility, with a total of four control rooms.
“We’ll link AoIP facilities via wan/fiber to provide any available multicast stream at a given facility available at either location. As this is implemented, it will allow us to begin consolidation of playout and delivery systems and simplify the processes to deliver the content to various distribution avenues,” said Jackson.
Jackson noted in a particular advantage of AoIP systems is the ability to prewire and test systems in a lab environment while construction of the walls and finishes proceeds. This leads to much faster and more reliable installations because it gives an opportunity to work out many of the bugs that occur during a deadline driven installation.
Like Eversoll, Jackson emphasized that the initial planning process is essential.
“Plan, plan, plan, review, plan some more,” said Jackson. “Have peers review your plans. Make appropriate changes and then proof your final plan. While you’re planning your current build, make sure a five-year plan for expansion is included in your current design. Don’t approach is from the perspective of, ‘Get it on the air now.’ Focus on, ‘Get it on the air RIGHT.’ This will save you time in the long run.
Perhaps none pushes the limits for a studio quite as extensively as a recent project in Switzerland for Radio Zürisee. Markus Stocker, systems integrator with Media Engineering, shared details of its innovative design. Many of the systems described would have been impossible without AoIP infrastructure.
“The system consists of two identical on-air studios, each equipped with a 15-channel Wheatstone LXE surface with motorized faders, redundant power supplies and a huge number of custom scripts for Talkback functions, studio switchover functions, monitoring selects, paperless news reader monitors and customized video wall operation,” said Stocker.
“It was time to look for something new and more modern. Because as good as all radio stations sound, they are all more or less the same as others. Something had to be different in order to distinguish this radio station from all the others.”
This led to the idea of building an open performance space called the Loft, where the talent are encouraged to roam freely while announcing.
The Loft is an old ballroom that is original to the hotel building that now contains Radio Zürisee. It consists of an open performance space with a small stage area but also an open kitchen, a large dining table, some comfortable seating and a full complement of video cameras to capture it all. There’s a working espresso bar and even a pinball machine — all with mics to capture their sounds.
The talent use wireless mics and have wireless tablets with custom screens using the ScreenBuilder application for WheatNet to control the audio systems. Sources can be selected, faded up remotely in the active studio and monitored with in-ear monitors while walking around. The cameras use automation to select views based on the active microphone. Or they can be manually controlled from the tablets.
Even the consoles look different from typical designs. In addition to having motorized faders that move themselves, the consoles use a frameless design with fader panels installed into the tabletops in different orientations but integrated into one mixer system. This physical separation allows different controls to be segregated for different show formats at the touch of a button to recall a new configuration.
For the live performance area, the WheatNet AoIP is integrated with a Dante audio system for the PA system. The whole station has a special feel to it due to the creative combination of technologies used in radio and live performance.
Stocker confirmed the project was completed on time and under budget, but noted that it takes extra time and expertise to build such an extensive number of custom scripts for the tablet controls. He also warned about system complexity when so many systems are integrated in one space.
“As expected, there were few problems with the AoIP technology within the Wheatstone system, but it becomes more complicated as soon as there are connections to other subsystems and providers. Such integration never works on the first try. But it’s worth it to look at it carefully and to make it happen. The number or severity of problems grows exponentially with the number of technical partners,” said Stocker.
AoIP has demonstrated it can provide both cost-effective and high-performance systems to handle just about any technical need for a radio station plant. It has a place anywhere in the transmission chain, and with IP addressing, the physical size of the network is no longer a limit to available connections.
New standards for equipment that address interoperability improvements, such as signal discovery and controls, are in development for AoIP in standards such as AES70. Once fully implemented AoIP will overcome any final barriers to interconnecting equipment from different manufacturers.
Starting from humble beginnings, AoIP has grown to be a core technology for broadcast, now capable of doing things that were impossible in the old analog world. Imagine what will be possible in another 10 years.
Comment on this or any story. Email [email protected] with “Letter to the Editor” in the subject field.