A few weeks ago, I woke up to around 50 email alarms. Not exactly the way you want to start your day. All the alarms came from the Nielsen PPM Multi-Channel Encoding Monitor, or MCEM, that monitors our internet streams.
The first thing I did was to confirm whether the streams were actually down. Eventually, after figuring out that the streams actually were on, I diagnosed that the MCEM was not getting audio.
The MCEM is fed audio for each station from Inovonics 610 Internet Radio Monitors. The issue seemed to lie with these units. But it was strange that all four of these went down at the same time. It had to be something external to them.
Since I could bring up their Web GUI, I knew the network was not an issue. I began to wonder if the firewall we had installed at the end of the year might have stopped the data traffic from getting to the units. It had already been in place for a couple of months, so it didn’t seem likely. Still, it was worth a check.
Reconnected
When I logged into the PfSense firewall, I did see that there was traffic being stopped to the Inovonics 610 units. I tried using the “Easy Rule” feature in PfSense, which essentially tells the firewall to stop blocking the traffic it is blocking to that IP address. This didn’t fix the issue.
I then turned to Todd Dixon, the guru of PfSense here at Crawford Broadcasting. He saw the same things I was seeing, but we couldn’t really see why it was stopping the URLs from getting through. Earlier, I had tried the URLs on my desktop computer on the exact same subnet and the link would bring up the streams on my computer just fine.
While Todd and I were looking at the issue, I went into the network configuration of one of the Inovonics 610s. I noted that the Internet Radio Monitor only allowed one entry for a DNS server.
[Related: “Malaysia’s RTM Uses Inovonics for Its Revamp”]
In this instance, we were using the main Comcast DNS static IP address. We have been using this address for DNS for more than 10 years and on lots of computers and equipment where we used static IP addresses, and we’d really never had any issues. We often used a combination of the Comcast and Google DNS servers when it was possible to enter two or more servers.
On was a bit of a lark, I decided to change the DNS server from Comcast to Google. Within 30 seconds, the stream was once again connected to the Inovonics Internet Radio Monitor. I felt bad that I’d wasted Todd’s time, but I was glad to see it working again.
As you might imagine, we might have other issues show up. One of the first things we noticed was that our NexGen timekeeping computer was off on time. The time sync software was displaying a message that it was unable to connect to the time servers. I checked the DNS entry in the network setting and sure enough, it was only the Comcast DNS server. Once changed to Google’s DNS, the time sync started working again.
We also had a Burk ARC Plus Touch not sending us a notification of a generator exercising as expected, and it was also the same problem. One DNS entry was allowed and it was pointing to Comcast.
So we are going to have to check a lot of equipment to see what is still pointing to the Comcast DNS server. It would be nice if Comcast had let us know that they were making a change or having issues.
Rick Sewell, CSRE, CBNT, AMD, is Chicago engineering manager for Crawford Broadcasting. This story originally appeared in the company’s Local Oscillator newsletter.
