Not that long ago, having a “network” of transmitter sites to maintain meant having a long list of telephone numbers you would use to reach specific remote controls. Today, you likely have a list of IP addresses.
What’s behind this evolution in technology? For starters, more and more remote sites have access to IP connectivity (either via the public internet or private IP links). Furthermore, transmitter manufacturers have added IP-based user interfaces, along with SMTP, user-configurable alarm generation and SNMP support for control and alarming to their product lines.
Solid-state transmitters have become more reliable over the last couple of decades; and now when they do experience failures, it’s often of a “soft” nature, involving one power supply, or one power amp, out of many. Power output is reduced, but the transmitter can keep on running. (If you are still using vacuum-tube transmitters, you should consider that.) This means that the engineer responsible for the site can essentially be farther away.
In reality it’s now more practical than ever to build a network operations center or NOC that can be used to gather information, and to provide control, to transmitter sites widely spread out geographically. That’s the topic of this article: use of remote monitor and/or control systems in a NOC environment.
CORTEX 360
Davicom recently introduced the Cortex 360. In addition to its normal, locally-based remote control functionality, Cortex 360 can be used in the construction of a widespread network of station control systems. I asked John Ahern, president of Comlab, the parent company of Davicom, about the new product.
“The Cortex 360 is actually Davicom’s third generation of remote site management system. We started building these in the 1990s. We’ve incorporated many of the suggestions that have come from customers over the years, and we think we’ve added lots of capability that allow it to monitor and control stations with a great variety of equipment,” he said.
As one example, Cortex offers analog inputs that can measure up to 80 volts with excellent common mode rejection. Status inputs are also all provided with individual ground contacts — allowing the installer to keep all the various pieces of gear isolated from one another.
I asked him about the ultimate number of stations that could exist in a network made up of the Cortex 360 units.
“Well, there’s not really any limit. The unit itself is usually installed at a transmitter site, although lots of customers are using them at the studios as well,” he said. “If you’ve got a network, you can use a distributed architecture, where you’ve got one unit at each transmitter site, and one back at your studios, along with NOC software that sits at a central site, collecting information, building a database, and connecting to all the sites as required.”
So as a user, I could have a computer and a piece of software running that would go out and poll all the various sites during the day, however many times a minute I wanted to — in order to keep track of what’s going on — but that’s not the only option.
“That’s right — if you have at a central site, but some customers don’t really have that,” said Ahern. “They’ve got the network, but they’ve got local technicians or local engineers maintaining a certain cluster of sites in a town, and they want them to be able to access the units directly and locally, and that is one way of setting it up. You can have your NOC software that’s polling everything, or have it just waiting for information coming in from all the sites, and then it’s advising or notifying personnel that something’s going on.”
Regarding a dashboard display, Cortex 360 provides various options. “The Cortex 360 now operates on HTML5, so you don’t need any special software. You connect to the unit, after which you have access to the GUI that is user-programmed. You can see what is going on locally, but you can also access the NOC software and get an overall view; a map view of all your sites so that you can see what is going on in different regions.”
I asked how easy it would be to have engineers from the next closest company cluster provide vacation relief.
“Up to 16 different user accounts can access any particular unit, and you can fine-tune permissions for various levels of access. In other words, the ‘vacation relief’ would be able to access the device, but wouldn’t have any rights to make modifications,” he said.
For those new to the concept of SNMP, the letters stand for Simple Network Management Protocol. It was developed early, as part of the internet protocol suite, but has only come into regular use for broadcasting of the last dozen years or so. It is a means by which data can be read, and control provided, between an agent and a manager. SNMP “get” is a way to retrieve a logic state, or other data, from the far end (ideal for reading status and telemetry); “set” is a way to provide a remote command at the far end; and you can think of “traps” as alarms that are configured on the agent, and sent to the manager should they come into being.
One of the most important features of any remote control today is its SNMP capability. The Cortex 360 can act as both and agent and manager, with the capacity of 1,024 SNMP Gets, 1,024 Sets, and 1,024 traps.
WORLDCAST NMS MANAGER
Another option for those that need to monitor a network of remote transmitter sites is WorldCast’s NMS Manager. WorldCast’s U.S.-based engineering rep Tony Peterle discussed the basic hardware requirements with me.
“With WorldCast Manager it depends on the scale of the operation. The operating system is Ubuntu server — I think it’s version 10.8 right now. We find that the OS is very light and efficient, and if you’re connecting 50 units or 100 units, you could probably get away with some fairly insubstantial hardware. If you need to go into the hundreds or thousands of units for monitoring then you might want to invest in something that’s more typical server hardware, with robust dual power supplies.”
For those who have used Audemat Control, or Relio before that, note that WorldCast Manager is designed to work with many different devices, not just remote controls also made by WorldCast.
“One of the significant things about WorldCast Manager is we determined from the onset of the development of this new platform that we were going to be vendor-agnostic,” Peterle said. “In our previous network management software we paid attention to both the Audemat side and the APT side for the codecs. We were very brand-specific.”
He continued, “This time we wanted to do something that was much more open and easier to configure and now we have already integrated into the software 350 to 400 devices from all different manufacturers all over the world. It’s very easy to integrate anything that speaks the SNMP protocol.”
Before getting systems such as the Cortex 360 or WorldCast NMS manager to “read” far-end devices via SNMP it’s necessary to pick out “objects” using what is known as their Object ID. These OIDs are identified by the user from a text file known as a MIB (short for management information base). Once the appropriate OIDs are identified, the system addresses them via “gets” and controls them via “sets.” Once a user becomes more familiar with this process, it isn’t hard to identify the OIDs needed.
If that sounds a bit complex to a beginner, WorldCast can help out. “That’s something that we can do for the customers free of charge — or if the customers have the tools to do that themselves so they can specifically adjust the software to give them the information deemed most important,” said Peterle.
“The other significant thing is that SNMP is not the be all and end all of the WorldCast Manager. We can speak other protocols as well like Modbus and CANBUS. We also have, both from our own catalog products and from the world market in I/O, a large set of edge devices that can convert older units that don’t support SNMP to systems that do,” said Peterle. “We have hardware that can convert data points like door sensors and smoke alarms into something that we can more easily network with and integrate into the overall monitoring scheme of the WorldCast Manager.”
I asked Peterle about connectivity requirements for use of WorldCast Managers. “It’s fairly limited. SNMP is a pretty lightweight protocol and the software gives the customer the opportunity or the ability to adjust the polling rate,” he said. “The WorldCast Manager can receive alarm messages from equipment in the field using traps or it can operate as an active polling device and just query parameters at certain intervals. The customer can adjust how often polling occurs, on a site-by-site basis, to compensate weak spots in the network.”
Finally, WorldCast Manager provides user-configured customized displays so that the user can see the status of your network, and to drill down to the critical issues when you do have problems.
It’s important to note that control capability (i.e., “set” commands) will be available in WorldCast Manager in Q3, delivery in September.
BURK AUTOPILOT WITH WARP ENGINE
The Burk Technology ARC Plus Touch, and its companion software AutoPilot, form a well-known remote control system used by many radio and TV stations in North America.
Chuck Alexander of Burk talked about a new standard feature of AutoPilot called Warp Engine. It enables fast, bandwidth-efficient, real-time monitoring of hundreds of ARC Plus or ARC Solo sites at a rate of 100 sites per second, with each site reporting as many as 32 status and/or meter values. Warp Engine requires less than 400 bytes per second of IP capacity for each monitored site. If communications bandwidth is at a premium cost, it can be further reduced by selection of a slower polling rate.
“Warp Engine communicates with one or multiple ARC Plus and ARC Solo systems,” he said. “A station can use AutoPilot to monitor and control individual sites, and not strictly transmitter controls, because we can interface with virtually everything at the site through a combination of distributed IO units, which communicate back to the ARC Plus over the IP network, and SNMP.”
Increasingly SNMP is used for direct, IP-based communication with any device in a system that supports the protocol. A centralized location running AutoPilot can monitor and control multiple ARC Plus or ARC Solo sites. “Our customers use it for network operation centers of all sizes at the market or regional or national level, so you could have 20, 30, or hundreds of stations monitored using AutoPilot, again communicating out to the ARC Plus systems at each site,” said Alexander.
Burk’s system architecture is based on the distributed intelligence of the ARC Plus. “We like to think of ARC Plus as an island of reliability. With an ARC Plus unit at the remote site, even if communications is lost between your central point and that remote site, normal operation can continue because of the automated capabilities of the ARC Plus for responding to off-air events and timed events like AM pattern changes.”
Alexander said that as a NOC grows and handles dozens if not hundreds of sites, efficiency of network communications protocols becomes more important.
“As you move toward the larger numbers of sites, going from 20 or 30 sites to 100, or 200, or 300, two factors become important. One is the efficiency of the communication both at the network operations center and the remote site. There’s a need for many of our users to be very efficient in bandwidth utilization and also efficient in computing capacity at the network operations center. We’ve designed a series of features, one of which is Warp Engine, to facilitate these very large NOCs.”
When the NOC needs to handle hundreds of sites, enabling Warp Engine switches from a TCP connection to connectionless UDP and uses very small UDP packets, polling at a fast rate — typically up to 100 sites per second, bringing back as many as 32 values per site. The user can configure those parameters if there is a need to use less bandwidth overall.
He also described the navigation pane as seen in the user interface for AutoPilot.
“If you do have a large network of stations then likely there’s some hierarchy that you will use to manage those stations. It might represent the hierarchy of engineering personnel with the responsibility perhaps — regional responsibility down to market responsibility. The user can create a network view that represents your management structure. Basically you’re just dragging and dropping folders to organize the way you look at your network; you might have a nation view and then three or four regional views, and inside each regional view there can be several market groupings, and these are all represented by folders in the left navigation pane.”
According to Alexander, that becomes particularly useful with another feature called summary status indicators, both at site and group levels.
“A summary indicator can be assigned for a particular site or folder of sites and it gives you a numeric and a color-based indication of everything at that group of sites,” he said. “Is everything good and green? Or do you have some number of red alarms or yellow alerts all summarized in the little block for that site or group? Warp Engine monitors hundreds of sites very fast, and if one of them comes up indicating alarm conditions then you can click on that site and establish a full TCP connection. Now while you’re monitoring all the other sites fast with UDP you can be in full communication with sites of particular interest using TCP connections, so you always have access to that control.”
As engineers get more and more spread out in their daily routines, the ability to “be in more than one place at the same time” becomes more and more important. Fortunately the tools needed to help you do just that are available now, and not expensive or hard to learn.
Comment on this article or suggest other topics for this series. Email radioworld@nbmedia.com with “Trends in Tech” in the subject line.
Doug Irwin, CPBE AMD DRB, is vice president of engineering at iHeartMedia in Los Angeles and a technical advisor to Radio World. His Trends in Technology columns will appear here regularly.
