Flashback to 1975 in Cedar Rapids, Iowa: I had just accepted my first job as a chief engineer for an AM/FM combo operation. Times were simpler then, with turntables, cart machines and reel-to-reel decks. Most importantly, my AM and FM transmitters were at the same site, with no alternate transmitters.
Despite the simplicity of those times, I was always busy with remotes (a lot of local sports games with a temperamental old tube-type RPU), transmitter failures, monitor points, DA problems, tower lights not coming on at sundown — and yes, I did an air shift.
I frequently drove to the transmitter to fill out the maintenance log, calibrate the remote readings, inspect the grounds, and check the AM parameters prior to taking monitor points. Iowa thunderstorms and monstrous winter weather triggered problems with equalized telephone lines, power outages, blown fuses and an iced-up FM antenna.
Today, many of us are responsible for four or more call signs (I have 10). We have more work, more stations and decreasing staff. Most engineers make site trips to test generators, A/C systems and, when time allows, backup systems. Often our many responsibilities leave little time for these routine weekly inspections.
To me, there is nothing more frustrating than learning of a problem with a backup transmitter, generator, audio processor or cooling system because it failed to work when needed; and, it is equally frustrating to know that the problem could have been discovered early and the outage avoided had there been more time for routine inspections.
Looking forward, demands on our time are likely to increase, so problems like this aren’t going away.
Regardless of your market ranking, an off-air situation is gut-wrenching. The purpose of this article is to teach you how to avoid those moments by minimizing station downtime and maximizing station reliability and efficiency by the use of automated testing. When you lack the time to make frequent and regular site inspections, automated equipment testing can provide the solution by alerting you to a system problem before that problem escalates into an actual off-air emergency.
I understand that many broadcasters are conservative about adopting new techniques and feel reluctant about going “off the reservation” for different (and often better) solutions. However, experience has shown that a remote equipment monitoring system with automated testing can help you by effectively allowing you to be in more than one place at a time, thus increasing your productivity and simply allowing you to get more things done during the week.
VIGILANCE IN TESTING
In order to minimize the chances that an undiscovered equipment failure will catch us off-guard, my facility now employs a remote monitoring solution, which consists of two parts:
1. Automatic, scheduled testing of backup systems
2. Continuous testing and monitoring of critical/important systems (we’ll explore this in greater detail in future articles).
At our stations, we use sensors and out-of-range email alerts to monitor many parameters: transmitter room temperature, radio STL signal level, generator running and online status, transmission line pressurization, three-phase line voltage, nitrogen tank pressure (if used), de-icer current (if used) and audio chain silence alarms. I program the software to send us daily equipment “I’m here and online” emails, as well as “I have been monitoring the generator block heater for three hours and it has cycled on at least once” notifications. If I don’t receive these daily emails, it’s time to investigate.
I use this continuous automatic testing approach to alert my staff and me to various out-of-tolerance conditions. My application of continuous automatic testing to broadcasting equipment has evolved over many years of trial and error.
AUTOMATIC SCHEDULED TESTING
Let’s examine a sample automatic scheduled testing routine for an auxiliary (backup) transmitter. There is no single procedure that will address all needs, so for discussion purposes, I’ve created a generalized flow chart for a weekly auxiliary transmitter “qualification” to serve as an example. My hope is that you will use it as a guide to spark some ingenious monitoring ideas of your own!
Please refer to the flowchart while reading the discussion.
We begin with the initial software setup: Select a day and time for the program to run the automatic test. I recommend Tuesday/Wednesday at 10 a.m. for these tests. Mid-morning tests provide sufficient time for you to travel to the site if a problem is uncovered, without interfering with your early morning routine duties. Further, it’s early enough in the week to address any issues without being sidetracked with the usual Monday deluge. In any case, choose what makes the most sense for your requirements.
Now let’s follow the steps as the automated test routine begins at the selected date and time.
The first detail the program checks (using status or metering signals) is that the aux transmitter is not the active transmitter. If it is, the routine exits without further action.
If the aux transmitter is not on-air, the program then turns on the dummy load blower (if applicable). It follows this by sending an alert email to the technical staff informing them that the weekly test has begun.
Next, the program turns on the transmitter. This process varies depending upon if the transmitter is a tube-type or solid-state transmitter. For example, if the transmitter is tube-type, the program can turn on the filaments for 30 seconds, and then turn the HV on.
After the transmitter is turned on, the program waits 30 seconds and then checks if the power output is normal. “Normal” is defined by what the engineer considers to be the acceptable operating range for the particular transmitter being tested. If the transmitter power output is excessively low, the program gives the command for an instant shutdown and sends an alert email to the technical staff informing them of a test failure.
If the transmitter power output is within acceptable limits, it is time to check the rest of the parameters. The program continues by setting a loop counter to zero (the reason for this is addressed below).
The program enters the main test loop. Here, it can inspect as many parameters as an engineer chooses, and he/she can add, alter or delete a parameter by simply editing the software. In this example, the loop will examine power output, audio level modulation and PPM decode OK/fail. I monitor these particular parameters because I believe their combined results provide a good qualification of the system as a whole.
The program now cycles through the loop. During each iteration, the program checks one parameter and then increments the loop counter by one. The first loop iteration checks the power output level; the second iteration checks the audio level; the third checks PPM status again; and so on. If a parameter failure is detected at any point during the loop, the program exits the test routine, sends an alert email and performs a shutdown procedure (transmitter off, delay, dummy load blower off, etc.).
The settings for the delay timer and the loop counter are important. My system’s delay timer is set for a relatively short delay time (10–15 seconds), while the loop counter maximum is set so that the product of the loop counter maximum setting and the delay timer equals 15 minutes. I chose this time limit because I believe that 15 minutes is long enough to run a stable test without stressing air conditioning and significantly raising utility bills. Keeping delay timer settings of short duration allows the routine to more or less continuously monitor parameters during the test and more quickly react to failures during the test process.
Once you initiate a remote monitoring process at one site, it can become quite addictive. Your first testing routine will likely inspire other ideas and improvements. You can tune your routine by editing the software. Find what works best for your operation — that’s half the fun of automated testing!
I should mention at this point that at minimum, three remote control vendors (Burk, Davicom and WorldCast Systems) offer scripting software that will permit you to implement a flow chart such as the one presented here as well as even more intricate routines to fit your specific requirements.
REAL WORLD IMPLEMENTATION
Let me provide two recent personal experiences to demonstrate how remote monitoring saved the day at my facility.
Recently, we received an email alert that the standby transmitter at our site in Beverly Hills failed the weekly automatic test. After driving through L.A. traffic to the site, we discovered the transmitter had multiple issues, including an open filament variable voltage control transformer, an inoperative SSR (which controlled the bias on/off), a failed bias circuit breaker and a control relay that had fused contacts. Obviously, something nasty had happened up there, but what’s really important is this: Without the alert from our automatic monitoring system, the failure might have gone undetected until the standby transmitter was needed.
In another instance, the weekly test for an auxiliary transmitter at one of our AM stations (the flagship for the Los Angeles Dodgers) alerted us that, although the transmitter was operating normally and at full power, its audio chain was silent! The radio STL link selected as the audio source for this transmitter had no audio output (we have three audio sources and select a different source for each of the three transmitters). This is a four-hop link (yes that’s right— four-hops) and in this case, the source of the trouble was at one of the intermediate points in the path. We selected an alternate audio source until the root cause with the radio link was resolved. The automatic test alert had again saved us from a surprise outage. We have a saying here in LA: “Got to keep Vin Scully on the air!”
CLOSING COMMENTS
I spend a good deal of time in my job working on spreadsheets and budgets, reading long emails and attending meetings and conference calls. While busy with these office duties, it’s always reassuring when an email pops up with the message, “KXXX auxiliary transmitter test results normal.”
I hope the approach I’ve outlined in this article can benefit you and your facility. Let’s put this technology to work for us!
Sloatman is the director of engineering and IT for iHeartMedia Los Angeles.
