An impending storm is coming, and I am driving to the radio station to fix my own stupid mistake.
Often, particularly in radio, failure is not an acceptable solution; but maybe it should be. Like all radio stations, we strive for 100% uptime. This mindset is so common in today’s business world; but as I drive, I am wondering if that is holding us back.
I have helped my in-laws with their radio stations over the past 20 years, and much more extensively over the past few years. As I look back at all the changes we have made, I’ve realized that the vast majority of them started with some sort of failure.
GOOD THINGS HAPPEN
In September 2016, the station manager had recently departed, and my in-laws had a planned trip. I told them to continue with the trip and I would watch over the station while they were away. In some ways, that too was my own stupidity, but that is a conversation for another day (radio has a way of sucking you in).
That week, the station’s main computer that runs the automation software went bad. It was multiple failures from hard drives to the motherboard. I would guess this was caused by some sort of power surge.
That computer was probably the oldest functioning Windows computer I have ever seen. I would estimate based on the processor that it was at least 12 years old. Why can’t I ever get a Windows computer to last more than a couple of years? (Again, a conversation for another day).
Anyway, this failure led to a series of improvements starting with a modern computer with solid-state hard drives and updated automation software.
Computer failures do happen, but good things can happen because of them. After spending endless hours working to recover the lost data, today we have backups on backups. We automatically save all files to a local backup. In addition, we also set up a cloud backup service. We will never lose a file again.
The key with any failure is looking for ways to prevent disaster in the future. This has traditionally been a key aspect to radio engineering, but in some ways, today’s leaders struggle with the idea of embracing failure.
We would have many other failures over the next few years at Holy Spirit Radio, but through each of them, we found success. All sorts of problems created by mice eating wires led us to moving to a Wheatstone system. It was more than the mice, but also thinking through how the station would survive once our engineer decided to retire. Unfortunately there are very few engineers like him left in our business (I wrote about this transition in Radio World in 2018 (“A Digital Guy Tackles Broadcast Radio Challenges,” visit radioworld.com, keywords Digital Guy).
Like many stations, we utilize the Broadcast Tools WVRC Plus to alert us to problems at the stations. As I moved to Wheatstone, I changed our WVRC unit to call my phone numbers when the station was off the air. We affectionately refer to the voice calling as the “little man.”
Well, I started to hate the little man! There is nothing like getting woken up by an automated call when the station was down. Every call led to changes.
Today at our primary station, we have a multitude of backups to avoid ever going down. We use some of the technology within Wheatstone as well as other sources to ensure a signal is always going over the air. We even use a software clock called vClock as well as a companion software called vMeters and an online technology called IFTTT to control many aspects of this (See my story from RWEE earlier this year). I will write an update soon with more ways we use this software.)
When we noticed our satellite would sometimes have garbled sound, I set the receiver up with a SmartThings plug and programmed it to reboot during a break once a week. To avoid internet outages, we installed a Cisco router (RV-325 — there is now a newer model RV-345) with wireless 4G backup via USB. The only outages we have seen at this location have been electrical failures. Hopefully, someday we will be able to purchase the right automatic backup generator, but for a small non-profit, that is not in the cards yet.
Even with all these changes, I was still receiving calls from the “little man.” It seemed to always be in the middle of the night. The calls were not from our primary station, but from our secondary station. I would often drive an hour-plus to figure out what the problems were, but often, before I would arrive, things would magically be fixed.
Over the next few months, I would perform similar upgrades at our other station. As we did the Wheatstone upgrade at our primary station, I replaced that computer, which was relatively new, so I brought it to our secondary station. I also added one Wheatstone blade and a Broadcast Tools device to pick up various streaming services. We changed the primary connections between the locations from Barix to Comrex BRIC Link II.
The setup at both locations was now virtually identical except the secondary location also has the capability of shuffling through various streaming options if the BRIC connection goes down. It also has backup recordings that change every week to ensure a good listener experience if we are down (I will write more about this in a future article).
I changed the router to the Cisco RV-325 with a wireless backup, but things do not always work out as we planned. It turns out the newer USB sticks from Verizon do not work with the RV-325.
Over the next few weeks, I would realize our biggest challenge was probably our internet connection. Even though it was fiber, it would go down once a week or so. The outages were usually only for a few moments, but that was enough to create alarms.
To fix this, instead of going out and purchasing a used Verizon USB stick, I bought another router by Cradlepoint (MBR1200). I bought it used on Amazon for about $140.
Instead of switching all the settings to the Cradlepoint router, I continued to use the Cisco and added the Cradlepoint as a secondary connection. It all worked, knock on wood! In the six months since making the change, the station has only been down once, and that was due to an issue inside the transmitter. (It’s always good to make sure connectors are clean.)
IN PRACTICE
This brings me back to my ride to our primary station this morning. Every once in a while, usually in the middle of the night, our router would go down and lose connection for some devices within our network. It was usually not a big deal, but, accidentally, I made it a big deal last night.
In my quest to solve this problem, I plugged the router into a SmartThings power jack and copied the programming from our satellite plug so it would reboot weekly. I was not thinking straight when I set this up because SmartThings will not work properly if the network is not working. I knew this; but we all make mistakes.
As I was driving along and thinking about this, I came up with the perfect solution. Most things within our stations have a primary and a backup. The one area that does not is the router. I am going to purchase the Cradlepoint router to serve as the backup connection, move the USB stick to that and run the SmartThings plug on it (as well as a secondary connection to the Cisco router). This will allow me to schedule the reboot as well as make it always possible to reboot the main router from anywhere in the world with the use of my phone. I can also build rules that can do this automatically if anything goes down.
We are in a competitive world, with radio stations striving to beat not only other radio stations but also online content sources, satellite and so much more. If we think back through the history of radio, it has always been about failures and the way we work around them. This has not just been about engineering, but even formats and so many others.
As I was writing this article, I came across a blog post by National Radio Hall of Famer Fred Jacobs, titled: “Why ‘The Edge’ Radio Station Was Our Biggest Failure and Success.” If we want to bring innovation back, the first step is embracing failures!
Frank Eliason is a consultant helping Fortune 500 brands with customer experience and digital disruption. He is an author and director of operations for Holy Spirit Radio in the Philadelphia area.
