I once had the good fortune to work with a skilled chief engineer of the analog era, who would rant and rave anytime a new piece of hardware needed a firmware update. I enjoyed his rants immensely but always thought this particular one was silly; software is the future, let’s just accept that and move on. I was fully on board with the rapid digitization that was overtaking both audio and broadcast tech at that time, but he was full of paranoia and dire warnings.
Then last year, the station had a severe off-air incident that changed my whole outlook. My old chief engineer’s anti-firmware rants echoed in my head.
First, some background
When I became GM of WFMU, I inherited a lightning-damaged 30-year-old transmitter. Dead air happened frequently and, until we saved enough for repairs, the preferred solution was to drive up the mountain to the transmitter and broadcast directly from a RadioShack mini-mixer. DJs fought over the privilege of being able to spin from the parking lot outside the shack – it was guerilla-like and cool.
Back then, you were on-the-air or you were off-the-air. There was no in-between.
Eventually we purchased a backup transmitter. Then we got our first repeater station on the air. Then came internet streaming, additional web channels and two more translators. The days of being off the air and dead in the water were over! For several decades, WFMU was never completely off the air. If a stream was dead, the FM stations were on. If one station was off, the other three were still transmitting and the streams were still rolling.
Until that fateful day a few months back. First the streams for all four channels died and then all four FM stations went off the air. It was too much to be a coincidence.
It wasn’t. It was a paperwork error, updated for the internet age.
Our security certificate (the magic payment that adds the “s” to “http” in your URL) had expired and our ISP had bungled updating their records. When the old certificate timed out, everything that connected us to the web died at once, including the fancy new EVPL T1s that had just replaced our old copper-based STL circuits.
We fixed the problem soon enough, but the incident was a red flag, especially as we contemplated rewiring all our studios, scrapping old fashioned copper wires for AOIP.
The lessons here for me were many. First and foremost, watch your security certificate like a hawk. But the larger issue that hit me over the head is that broadcast studios and transmitter facilities require backups, aka redundancy. If you lose one fader, you don’t want to lose all your faders. If you lose one studio or transmitter, you don’t want to lose ALL your studios and transmitters.
Redundancy was easy and common sensical in the analog era. But digital redundancy is way harder and more expensive. Smaller stations like mine struggle to afford even a single full-time IT specialist, and there are thousands of independent commercial, religious and public stations in the same boat.
For understaffed stations that try to employ best digital practices like local network QOS, or failovers with multiple ISPs and multiple streams, it doesn’t take long before nobody has any idea what’s really going on under the digital hood.
It’s not the first time I’ve realized the superiority of analog, even as I race to upgrade to the latest digital approach. Reel to reel and cassettes are a much easier, cheaper and more reliable long-term storage format than hard drive backups which need to be constantly refreshed and re-copied. Analog audio fidelity is richer and clearer than digital audio fidelity. We shouldn’t be shy about acknowledging specific analog advantages.
Just like it’s easier to be on the air with paper notes, as opposed to a stack of bookmarked URLs.