The author is tech support manager for Comrex Corp. This story is one in a series that appear in the Radio World ebook “Getting Data Up the Hill” exploring aspects of studio/transmitter links.
In my job at Comrex, I help customers maintain and support their video and audio broadcast equipment. Much of this support relates to IP audio codecs used in studio-to-transmitter links.
Support cases arrive in waves, so we are constantly riding the tide — sometimes things are relaxed, sometimes we are extraordinarily active. It’s not uncommon to experience a lot of activity prior to a heavy sports season or before a big election, but our team supports users all over the globe, so an attempt to predict how busy things will get is easier said than done.
For each case, the associated details and context could be similar or different compared to the previous encounter. Our team has to play the role of detective and acquire as many details as possible. Additionally, we need to refresh the data collection process without assumptions every time to be sure we’re gathering details in an unbiased manner.
As you can imagine, if the details are not accurate or are filtered, someone could head down the wrong troubleshooting path, wasting a lot of time and creating confusion — a lose-lose situation. So precision is key.
[Related: “MPX Codecs Reduce Operational Costs”]
As mentioned much of this work involves STLs as well as station-to-station links. Of course, no one wants anything to go wrong. But in reality, an enormous number problems can occur even without conscious intervention — power surges, ISP changes, firewall policy updates, etc.
It is clear from our experience in recent years that even more stations now operate with fewer full-time engineers. We interact more frequently now with part-time contract engineers who manage the engineering of multiple stations, which can reduce the amount of time an engineer is available to help solve a given on-site problem.
We also more frequently now encounter engineers who know few details about a station because they are new to the outfit and are just learning about it for the first time, as are we.
In either situation, time is precious.
A common example might involve audio dropouts: “Paul, my tower codec is indicating a lot of packet loss, what happened to it?”
Typically, this will boil down to a network issue rather than the codec, but we need to put on a detective hat first and start asking pertinent questions, as well as administer some tests to gather accurate data points.
We certainly don’t want to assume we know what the problem is and start throwing spaghetti at the wall. A better approach is first to establish where the problem is, then what it is, because when you are pushing or pulling IP data, you don’t really know if it is an issue with the source location pushing data or with the receiving location pulling. If you can test each location against a third, you can establish a much clearer picture to help isolate the problem.
This is an ideal scenario, but let’s say we have a studio codec sending data to a tower codec. If the studio transmits to the tower, disconnect them and connect the studio to an alternate codec/location. Does the alternate codec exhibit the same issue? If not, connect the alternate codec to the tower. Does the tower still exhibit the same issue?
If the problem is reproducible regardless of which codec/location connects to the tower, and assuming everything else looks stable, you likely are dealing with an issue associated with the tower.
Essentially, we want to see if the problem stays put or moves. Though we don’t yet know what the problem is we know where to focus additional efforts. This saves an incredible amount of time.
Following up, one can start testing the simple stuff — swapping cables, assessing bandwidth or even contacting an ISP to have them confirm if the site is having an issue.
We manufacturers each have our own secret sauce and baked-in codec features to try to workaround adverse network conditions. But if you’ve identified a bad link, Comrex advocates isolating and fixing that link before it gets worse. For example, a faulty transmission line on the exterior of the building or down the street could spiral further out of control if inclement weather impacts the situation.
People will ask me what working in tech support is like. I describe it as a 50/50 ratio, where 50% is technical and the other 50% is relational. We must keep in mind that we might be communicating with a user who is a talent, or a general manager, or a broadcast engineer who has 40 years of experience.
Then of course we can ask questions that are appropriate to their role and background. But how we ask is equally important because the person’s state of mind makes a difference too. If a user is frustrated because of an off-air situation, our team must remain calm so that we don’t excite the situation and delay a resolution, especially when a system is down and we don’t have the luxury of time.
After we gather essential details and apply effective communication that benefits both sides, we can make progress. Many engineers want to pull themselves up by their own bootstraps. We appreciate and sympathize with that mindset. But when time is a factor, when the mission is critical, we encourage our users to lean on us for support so their gear can operate as expected in a timely manner, as well as understand the best practices involved for their workflow.
That’s a win-win!
