The author is chief engineer at Educational Media Foundation.
What do you do when your organization operates 900 radio stations and FM translators across 50 states, but only employs 34 engineers to maintain them all?
While running for the hills screaming seems like a reasonable response, the healthier, saner thing to do is to build a NOC.
A standard equipment rack for a station is being assembled in the EMF Tech Shop.
At Educational Media Foundation, our Network Operations Center is absolutely vital to keeping our signals on the air and operating legally. Having a NOC allows fewer engineers to support more signals than would otherwise be practical, and to do that task remarkably well.
The NOC demonstrates just how effectively a large radio network can operate when comprehensive, real-time data are captured in a unified setting and acted upon by skilled professionals. The result is increased uptime, better operational efficiencies, healthier staff and ultimately a better experience for our listeners.
Today, each of the 34 staff engineers at EMF manages between 25 and 45 signals, and yet our networks maintain an overall reliability nearing 99 percent, and that number continues to improve each year. This would simply not be possible without our NOC and the eight dedicated technicians who keep it running 24/7/365.
Fifteen years ago, there was a small handful of engineers at EMF who collectively maintained and managed the 57 stations and roughly 150 FM translators between the K-LOVE and Air1 Radio networks. Each week, the engineer who drew the short straw took home a Palm Pilot and the on-call phone, and was solely responsible for making sure all the stations stayed on the air. While it wasn’t always a “good time,” it was manageable.
However, as the networks grew, the on-call engineer got less and less sleep, to the point where it became physically impossible to handle the volume of issues that arose each day. The need for a centralized solution was soon recognized.
Inside the EMF NOC with David Pelz, NOC technician, at the helm of one of the three main workstations.
We began looking at different options, even visiting the NOC for a large telephone service provider to get ideas, and toiled over various iterations of how we could implement a solution that was practical, functional, scalable and cost-effective. With the commitment of our team and senior leadership, and the vision and brilliance of the late Alan Guthrie, the EMF NOC was born in May 2005.
Initially, our NOC was staffed with folks who could answer incoming phone calls from dial-up remote controls and dispatch engineers to solve problems. Limited telemetry was available, but we had a handle on the issues that affected our listeners.
As the station count continued to explode over the years, so did the need to monitor more equipment and data points to keep the engineering staff ahead of the curve. We could no longer rely on bits and pieces of information coming in via phone calls from synthesized voices.
Changing requirements for incoming data also meant the NOC staff had to become considerably more technically proficient, therefore additional training and documentation was developed to support it.
In our quest to improve signal reliability and listener experience, EMF’s transmitter facility equipment was upgraded and modernized so we could remotely monitor every aspect of a signal’s operation. We instituted a “cookie-cutter” approach to building equipment racks for stations, standardizing the layouts and wiring of each site as much as possible.
This allowed any engineer or NOC technician to be able to work through issues at any site in the networks. These standards were updated as equipment inevitably changed over the years, but the idea of creating carbon copy facilities (within reason) has been foundational in our engineering and NOC teams’ abilities to do their jobs.
The standardization of equipment installations allowed for more accurate troubleshooting and a growing number of repairs to be done remotely, significantly reducing the workload on the engineers in the field.
REAL-TIME MONITORING
The heart of our NOC is real-time data. Without it, efficiency and overall effectiveness is stifled. Beginning in 2008, every station at EMF was equipped with an IP connection, the standard being a VSAT (Very Small Aperture Terminal) link, which sends two-way data over our existing satellite carrier.
NOC display shows the Ku-band uplink health, with a live spectrum analyzer view of the downlink monitor.
Click To Enlarge
FM translators in our network use either VSAT or a custom-designed remote control system that transmits and receives basic telemetry and commands over low-earth orbit (LEO) Iridium satellites. New systems are currently being developed in-house to add IP connectivity to FM translator sites using cellular, wireless terrestrial or satellite technologies.
One downside to the VSAT system is available bandwidth, and because of this, a great deal of thought and planning was put into how to best utilize it. When there are roughly 500 stations and thousands of individual devices sharing a symmetrical 1 Mbps satellite IP connection, elaborate GUIs and Flash-based web pages are excruciatingly slow to load and basically unusable.
The VSAT system monitor shows bandwidth utilizations and status per site.
Click To Enlarge
Methodical testing is done on each piece of equipment before it can be placed into service to assess its ability to communicate quickly and effectively over the VSAT system. We determined Simple Network Management Protocol was the clear choice as our standard protocol for communication between the transmitter plant and the EMF NOC. SNMP is incredibly efficient, and allows for hundreds of sites to be monitored simultaneously on a relatively tiny Internet pipeline. What’s even more surprising is that, even with so many stations and devices connected, the bandwidth utilization on our VSAT network generally hovers around 10 percent.
By being proactively efficient with bandwidth, a few creature comforts could be implemented at our remote sites, including VoIP telephones (as cellular service can often be sketchy or nonexistent on isolated mountaintops), audio streaming from off-air confidence monitors, and remote desktop connections to HD Radio Importers. Of course, not all of these services operate simultaneously, but their presence on the network and occasional use is easily supported by the VSAT system.
Additionally, delivering telemetry over VSAT allowed us to cancel telco lines at our sites, saving hundreds of thousands of dollars per year.
CUSTOM REMOTE CONTROL
In 2011, we began replacing the dial-up remote controls at each station with the Worldcast Systems Audemat Relio, which ingests SNMP data from on-site equipment supporting the protocol, and analog/digital metering and closures from other systems, and sends it to the NOC over VSAT.
Satellite systems NMS, and K-LOVE / Air1 network RDS monitors.
Click To Enlarge
For the most part, every piece of equipment that supports remote control (either via IP or GPI/GPO) is monitored, no matter how unassuming its function. Being able to monitor everything allows for a level of remote troubleshooting that could previously only be achieved with an engineer standing in front of the equipment rack.
Broadcast Manager, a scalable network monitoring solution built by Worldcast, captures all this data for the NOC. Any alarm or out-of-tolerance condition captured by a Relio remote control is instantly displayed on a screen in the NOC, allowing the NOC technicians to intervene as events occur, minimizing downtime.
For those issues that can’t be corrected remotely, the NOC will dispatch the local staff or contract engineer to investigate. Broadcast Manager also captures readings from each transmitter, which are formatted and placed in the weekly station logs reviewed by the chief operators.
The NOC features an array of monitors for viewing the health of every facet of network operation in real time. The satellite uplinks are the bread and butter of our networks, for both program and content delivery and maintaining positive control of each station. The uplinks, downlinks, NMS (Network Management System), and VSAT are all monitored on dedicated screens.
Other displays host maps showing National Weather Service radar and storm tracks, the stations and FM translators that are currently experiencing issues, a virtual white board for sharing information, and the text being sent from studio automation to the RBDS encoders. There is even a screen showing the forecasts for E-layer ducting, which often affects the propagation of FM stations.
Remote telemetry is not the only method the NOC uses to determine a station’s operational health. As sophisticated as equipment is today, at times stations develop issues not captured by remote sensors. The NOC places special priority on listener reports. Therefore, when a report of a signal problem comes in directly from a listener, it’s immediately sent to the NOC so it can be investigated and corrected as quickly as possible.
As one might imagine, with the large number of signals in the EMF networks, equipment commonly fails. Part of a NOC technician’s duties is to routinely go through the networks and check for systems that are not functioning properly. If equipment is not replying to pings or seems to be offline, the technician will attempt a remote reboot or notify the local engineer.
Each station is equipped with a remotely-switchable backup audio path, which not only helps keep program audio flowing if something fails, but also enables the NOC to troubleshoot a faulty device while the station is on the air.
As equipment is found to be faulty, the NOC will often assist an engineer in the field by creating a trouble ticket to get a replacement unit shipped. When the replacement arrives and is being installed by the engineer, the NOC will remotely test it and complete any final configuration needed before the engineer leaves the site.
KEEPING IT LEGAL
At EMF, the legal operation of our signals is absolutely paramount. If a problem occurs which places a signal out of FCC compliance, and it cannot be immediately corrected, the station is turned off until repairs are made. The NOC works hand in hand with the engineers in ensuring FCC compliance.
To confirm RF systems are operating within limits, the transmitter’s remote control is tested and calibrated during every routine or emergency visit to a site by an engineer. The NOC also checks for positive control and performs routine log reviews to alert engineers to potential issues before they become more serious problems.
Additionally, if for some reason a station loses its ability to communicate with the NOC, sophisticated ATS (automatic transmission systems) scripts, written into each Relio, kick in automatically and allow the remote control to self-manage its systems, with the authority to turn the transmitter off if needed, until two-way communication is restored.
The NOC also works closely with our many dedicated and fantastic contract engineers, who perform a great deal of routine maintenance and emergency repairs on the signals, and follow up with a notification to the EMF staff engineer about the work completed. This enables the staff engineer in charge of the station to keep tabs on the condition of all the sites in his region with fewer phone calls and emails.
The NOC also acts as an important safety net for the engineers. Working at remote tower sites is often a lonesome matter, and the roads to these sites can be treacherous, especially in mountainous regions. As a matter of protocol, EMF staff and contract engineers are required to check-in with the NOC when they are at any tower site. This not only allows the NOC to do a full systems check of the on-site equipment, but also provides a last-known location for an engineer in the event he doesn’t make it back to town on schedule.
The servant’s heart of each technician is a key component to the NOC’s success, and allows the engineer in the field, who spends a lot time driving and on desolate mountaintops, to be much more effective at keeping the stations well-maintained and operating properly and legally, and less time on their laptop connected to VPN through a mobile hotspot while sitting at a rest stop along the highway.
Because the NOC is in operation all day, every day, an engineer in any time zone can pick up the phone at any hour and reach a NOC technician for assistance with remotely troubleshooting an issue, filing a NOTAM for a tower light failure, checking the status of a power outage at a site, or even getting the phone number of a site manager.
By God’s great mercy, EMF has been able to invest in new and advancing technologies for our engineering department, constantly seeking to improve communications, bolster training and expertise in the radio broadcast field, increase uptime at our sites, and operate our networks to the highest standards.
The NOC has proven itself to be fundamental in this process, allowing EMF to expand its K-LOVE and Air1 music formats into new markets while significantly reducing workload and maintenance costs. By being the first line of defense for our networks, the NOC allows the engineers to have more restful nights and productive days, drive fewer miles, and spend more time with their families, all while sustaining a great number of signals across widespread areas.
Ultimately, the EMF NOC helps keep the tools God uses (i.e. K-LOVE and Air1) on the air, playing the music and sharing His love to millions of listeners and supporters every day.
