The authors are director of technology and product manager, respectively, for StreamGuys.
Every engineer and station manager knows that actively monitoring their broadcast operations is critical. Techniques for monitoring over-the-air broadcast signal chains have been honed over many decades and may be well understood.
But streaming distribution brings with it a new set of potential points of failure that must be monitored in different ways. Understanding each step of how your streamed content goes from point A to point B, and how you can monitor each of those steps, is essential to fixing problems if they occur.
Stream monitoring for radio broadcasters starts with source production in the studio and extends all the way out to the audience consuming the content. At StreamGuys, we talk in terms of four primary layers of the signal chain: content production; the facility network that connects the production to a streaming service provider or CDN; the service layer that hosts and streams the content to the world; and the application layer where listeners access the content.
The content layer
Streaming radio content essentially takes one of two forms: on-demand content such as podcasts, or live streams as a corollary of traditional linear broadcasts.
Verifying on-demand content is fairly straightforward, as you’re only really worried about whether the media file is “good” — for example, whether it is encoded in the correct codec and bitrate, with proper metadata, and normalized correctly for its purpose (such as the TD1008 spec for speech-centric content).
Validating live content is more complicated as it requires multiple facets to be verified in near real time, starting with playout from the automation system. Normalization and digital signal processing may occur prior to the encoder, and you need to ensure the encoder receives a valid audio signal that is true to the producer’s intent. The output of the encoder itself — MP3, AAC, or even HLS — must also be validated.
The network layer
The network is how your signal is transported to your streaming media provider or CDN, which in turn reflects that signal out to the world. The network comprises the switches, routers and gateways that make up your local area network (LAN) at the origination source, and also interacts with a dedicated internet service provider (ISP) connection for uploading streams from the encoder.
Tools like ping, traceroute and PingPlotter should be used constantly to monitor the consistency of the network and ensure connectivity to the outside world. You should be monitoring against the points of presence where your CDN is ingesting your streams. Generic speed tests don’t offer useful perspective, as they don’t accurately represent the path between your network and the service layer to which you’re broadcasting.
Persistent tracing provides the operational visibility necessary to identify intermittent issues. This long-term approach to monitoring the health of your network enables you to correlate issues reported by listeners — such as stream buffering at specific times of day — against network events such as packet loss or jitter on the link between the studio and CDN. Such data can help when working with your ISP to resolve such issues.
The service layer
After the live feed reaches your CDN or you have uploaded file-based media file to your CMS, the service layer is the delivery mechanism that distributes your content to your audience. It consists of multiple purpose-built streaming servers and usually is operated by a CDN or streaming media company, spanning multiple points of presence globally.
For monitoring purposes, look at the service layer as if you are part of your audience. First, ensure you can connect to the stream from your location, like a listener in your own DMA. From there, cast a wider net to ensure the stream is available from other parts of the country, or even globally. Automated monitoring agents can be set up in different geographic areas and even on specific consumer networks to help narrow down issues affecting only specific groups of listeners.
In addition to verifying that streams are accessible, you also want to ensure listeners are correctly receiving your audio. Silence monitoring is an important aspect of that, and can help detect issues with the automation system or encoder. The silence detector “listens” to the stream, and if it falls below a particular audio level for longer than a specified time threshold, triggers an alert. Audience measurement can also help identify issues, as sudden dips in traffic can correlate to technical issues with your streams.
The application layer
The application layer is essentially any mechanism the audience can use to interact with your content. This could be via your website using various browsers; it could be a dedicated mobile app; or it could be through increasingly popular smart speakers.
Monitoring the application layer comes down to verifying proper playback of your content. Are there any audio issues that weren’t present in the source, but are audible through the player? Such problems may be platform specific. For example, the audio specifications you’re delivering might be fine in one browser but not in another.
Analyzing the user experience is also important, as how people interact with your content — such as voice commands — can impact their engagement.
There isn’t yet a great, automated way of testing the human experience. Manual testing is still the best way to verify that your player is outputting the right content properly in various locations, and that commands are fulfilling the interactions you want your consumers to have.
While you should obviously perform your own testing, you can supplement it with feedback from people who are essentially constantly monitoring the application layer for you — your audience! Pay close attention to their feedback, even unique problems you haven’t heard of elsewhere. They may be the clues you need to track down problems elsewhere in the chain.
