System redundancy is an important aspect of automation systems so I asked various developers about their approaches they take.
“With Simian, most users set up their production machine so that all of their audio files are mirrored between the air and production computers using the Simian File Sync utility, said Alex Roy of Simian developer BSI.
“If the air system fails, they swap the validation code and hardware keys between their two systems so that the production machine is now using the on-air license. If you want satellite source switching as well, you would route your satellite audio to the production system as well.”
At Arrakis, said Ben Palmer of the Digilink system, “We keep all of our software files in a single folder on the C drive. This folder contains all your schedules, audio, settings, etc. … We recommend to customers to back up this folder, either with an external drive, a server, or on another networked PC or NAS. If you have the unfortunate experience of having your on-air PC die on you, it would then be as simple as placing these files onto a new PC and you’re back on air. Customers will often have a PC on their LAN that is mirroring that particular folder. This PC would be the standby, and would minimize any downtime.”
Ron Paley of DJB described redundancy in its system. “The DJB Radio automation system uses the audio server principle whereby the on-air software resides on top of the shared audio database, audio assets and archived files. In this case it’s a RAID mirror for the Windows Server 2016 operating system. Redundancy is provided by the second file server that runs as a file server in Win 2016. The main on-air machine seeks the primary database or it can connect to File Server 2 for operations, while its local database runs as a backup.
“There are also many cases of off-site backups at transmitter sites whereby the assets are copied over a VPN to a desktop at the transmitter site(s),” he said. “We have found a software product called Syncback that does a great job mirroring the assets and with little overhead (which is user adjustable).”
Jeff Wilson of Broadcast Electronics provided specifics about AudioVault Flex: “With fast networks and large hard drives, the simplest solution is to copy everything everywhere with sound cards in the studio machines. If someone spills their coffee in the air console, hop into the prod room and fire-up the air-screen and as long as you can route the audio you’re back in business with minimal downtime
“For larger setups we do redundant servers that have multiple engines for automatic fail-over in case of server failure. In those sites, IP audio makes the backup possibilities endless,” Wilson continued. “Typically, we work with the engineer and design the system with the level of redundancy their budget allows.”
Jeff Zigler of RCS responded to my question about redundancy for Zetta. “One of the big benefits of Zetta is the hardware-agnostic approach, which allows it to be deployed on many different platforms (real or virtual) and hence be able to accommodate the varied redundancy schemes that are suitable for a given platform. We have many sites that have anxiously transitioned to the world of AoIP, but want to hold on to a bootstrap type of backup where audio is played out synchronously in both the AoIP platform as well as on a real hardware audio card,” he said.
“The design philosophy for Zetta was to natively impose no functionality limits within the software, thus leaving the limiting factors to be within hardware domain. Since hardware platforms are always changing and typically improving with technological advances, Zetta’s configuration possibilities and redundancy designs will vary and evolve over time.”
While RCS strongly believes in backup and redundancy, he added, “We also believe there really isn’t one path that would be applicable to all customers and situations. We provide the customers with the capabilities, tools and assistance to develop and deploy the ‘right’ redundancy plans for them. We have had customers use as little as an external USB drive to backup and transport the data via a cloud storage solution to a get to a virtual machine, and all the way to customers that built a redundant system and then built an off-site second system that is in and of itself fully redundant and operating in concert with the initial ‘primary’ site.”
OMT’s iMediaTouch has several approaches to system redundancy, Gary Kristiansen told me about its solution for mission-critical applications.
A central server is used for audio, logs and the database and through a feature called Store Forward. Both a “main” and “secondary” server pull copies and keep them locally. Further, the two workstations run in “teaming mode” whereby the secondary can take over for the main should the main fail. In the event that the central server fails, the main on-air workstation keeps on going since it has the necessary files, previously pulled from the server. Likewise, the secondary can take over for the main since it too has pulled all the needed files from the central server. There is no single point of failure, he said.
