Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×

Cris Alexander on How to Avoid a Cluster of Failures

And what to do when everything breaks at once

When it snows, it pours. This FM antenna coincidentally failed at the same time as an audio processor and a whole-site UPS unit.
In my company we have a first-rate bunch of market chief engineers.

Each is a veteran and has been in the business for many years. Each has tremendous experience with studio equipment, transmitters, antennas, towers, STL systems and the other components that make up the broadcast technical infrastructure. Some have a crew of engineers working under them. One has a large engineering staff. I am proud of our people and their technical abilities.

But even with all their experience, qualifications and staffing, each of these folks faces something that every broadcast engineer will confront on occasion: multiple fires to put out at the same time. We may wish things would always happen at a metered pace, but the reality is that when it rains, it pours. It doesn’t much matter why this happens. The problem becomes one of priorities: What do we fix first, and in what order do we address the rest?

There is no easy answer. Each situation is different and dynamic. And sadly, whatever you do, it often will be the wrong thing to do, from at least someone’s perspective.

THE NIGHTMARE

Consider this situation: Severe thunderstorms march across the city, bumping the power at the studio and scrambling the brain on the automation, taking out Internet service and locking up the AM’s on-air console.

As the storms roll on, lightning hits the power lines at one of the FM sites, killing the power there and doing some other unknown damage — the generator is running but neither transmitter will come up. And the digital STL at the other FM site dies when the storm passes over.

To each station’s PD, his or her station should have priority. They don’t need PPM data to tell them that their listeners are all tuned to the competition now and that many just might not come back.

In recent years, I have observed competent, capable and committed engineers terminated because of situations very much like this one.

“But that’s not fair!” you say; and you’re right! Yet everyone needs someone to blame. In many cases, blame for poor performance in ratings and sales will be (unfairly) shifted to the chief engineer for any number of stated reasons. When you boil it all down, the problem often is poor communications. We can truthfully say, “Commu¬nication is our business — it’s not our policy.”

So how do we avoid this? In what order does our overloaded chief engineer address the myriad of problems and issues in this or a similar situation?

The process involves regular communication with a management-level decision maker, usually the GM or owner, the one person who has a dog in every fight in the cluster.

It really is up to him or her to help you set priorities, sometimes on the fly, and deal with the issues in the order that makes the best sense for the operation. This will be good for some of the people involved and bad for others, but that is unavoidable. But with the GM making or helping you make the calls, the blame is shifted to where the buck should always stop: at the top.

It is an excellent idea to sit down with the GM during a quiet period and talk about scenarios like this one. You know your market and your facilities. You know where the vulnerabilities are. The GM knows where the money is and what facilities need protecting, perhaps at the expense of another. You and he can flesh out a “what-if” list that you can use to prioritize things without a lot of additional consideration.

For example, most clusters have a billing leader, the one station in the cluster responsible for the lion’s share of the revenue. It is a simple fact that the air time of that station is worth more than the air time of the other stations in the cluster. Accordingly, priority should always be afforded the lead station. The GM can identify the pecking order and help you set the respective stations’ priorities.

DEVELOP A PRIORITY LIST

In other cases, the situation may be more dynamic.

Consider a cluster that has a billing leader that would normally get top priority, but during baseball season a sister station that would otherwise be at the bottom of the list might actually be assigned the top slot.

There can be all sorts of factors that figure into this, things to which the chief engineer would not normally be privy. For example, there may be contract negotiations in process for next season that could be jeopardized by an outage during a game. And because the situation is dynamic, regular communication with the GM is required to stay on top of it.

Some GMs are all over this. They’re always thinking, always managing, always communicating. Others — and these are more typical — don’t give engineering a thought until something goes seriously wrong. It’s up to you to figure out which type you’re dealing with and adjust the way you interact with them accordingly.

With the first type, you likely have a good and regular flow of communication already. With the second, the burden will be on you to initiate and maintain this flow.

Now we’ve considered the worst-case nightmare scenario, what about the everyday stuff? Like most engineers, you have your project list, your regular maintenance duties and a bunch of sleeve tugs in the hallway. Projects and maintenance are fairly easy to prioritize. Sleeve tugs easily can derail your priority train if you let them.

I would suggest the following as a starting point for the priority list for just about any station or cluster:

  1. Compliance (FCC, FAA, etc.)
  2. Transmitter Operations

    a. Transmitters (main and auxiliary)
    b. Antennas and Lines
    c. STLs
    d. Audio Processing
    e. Remote Controls
    f. Generators

  3. Studio Operations

    a. Mixers
    b. Automation Systems
    c. Routers
    d. Source Equipment

  4. Production
  5. Remote Equipment
  6. Facilities (building, etc.)

Looking at this list, you probably can see some conflicts already, but that’s unavoidable. Your priority list should be developed with the help of your GM or director of engineering, and it will in most cases follow the money.

Starting at the top, your licenses are critical to your operations and they must be protected at all costs. Stations simply can’t afford trouble with the FCC or other regulatory agencies, especially trouble that can affect license renewal.

Your signals are the delivery mechanism for your product (programming) to the consumer (the listener). If you can’t deliver, you’re done. It won’t matter a whit how great the programming is or how good the audio from the studio sounds. The transmitters main and aux, the antennas, transmission lines, STL links, processors and remote control systems have got to work all the time. And don’t even think about deferring the maintenance on that aux transmitter. That’s like neglecting to get the flat fixed on your spare tire — sooner or later you will be walking.

Identify vulnerabilities and don’t even think about deferring maintenance on that aux transmitter! Much the same argument can be made for studio facilities; without programming, all you’ll have is a dead carrier. But you typically have more options at the studio. Most automation systems, for example, can operate in some sort of emergency mode for a day or more. On-air consoles, mixers and routers can be bypassed altogether if necessary to get back on the air. But otherwise, studio facilities are high on the priority list.

Production is important, but you can be on the air without production for awhile, and again there are often other options (multiple production rooms, control rooms not in use, etc.).

You get the idea.

Once the priority list is developed, the trick is to stick with it. That’s where the sleeve tugs come in. You’re walking down the hall and one of the producers calls your name, telling you that Mic 2 in Production B isn’t working. The temptation is to run in there and see about the mic problem, but you can’t do that if there are other things in play that are higher on the priority list. You have to be disciplined here. You have to explain to all sleeve-tuggers that they must use the established discrepancy reporting system and write up the problem; you’ll get to it as soon as you can. You can’t allow the urgent to get in the way of the important.

DON’T FAIL TO COMMUNICATE

I hope that if you’ve gotten anything out of this discussion, it’s that you must communicate. Good communication with your GM or owner, PDs, producers and others is key to a smooth-running, properly prioritized technical operation.

Let me add one more thing. Once a week, write a detailed report of your activities to the GM. Copy everyone else that might be interested — PDs, producers, etc. Describe in easy to understand terms what you have done this week, provide the status of each item (complete, parts on order, etc.), and give a brief overview of the plans for the next week.

This report will do more for you than anything else you say or do. Each person in the operation sees only his or her own part of the operation. It’s easy for them to think that you give all your attention to station “B,” or that “He spends all his time at the transmitter site goofing off.” Your professional, well-written, detailed weekly report will put an end to all that, document your activities for the record and — here’s the best part — help you evaluate your own performance and priorities. If they don’t sound right when you write them up, they probably aren’t.

Cris Alexander is director of engineering at Crawford Broadcasting Company, a longtime RW contributor and a past recipient of the SBE’s Broadcast Engineer of the Year Award.

Close