“Always have a backup.”
That phrase has been a cornerstone of good engineering practice since the earliest days of broadcasting. In the digital era, where information such as client lists, commercials and music libraries can be worth more than the hardware it runs on, backing up data should be an important part of every station’s routine.
Basic concepts
Current storage technology for the music, commercial and log data in automation systems is based on hard drives.
As mechanical devices with moving parts, they are the most likely failure point in the system. Manufacturers quote a Mean Time Between Failures of over 1 million hours, or greater than 100 years. More frequent failures occur due to infant mortality and environmental conditions such as heat and vibration being exceeded during shipping or in the field.
In addition to bad sectors on a hard drive disk, the data on drives may also be corrupted by software glitches or computer viruses. The challenge for broadcasters is to make sound decisions regarding the tradeoff between performance vs. fault tolerance.
Tape backups have long been used to preserve data. Backup software is used to stream data from the hard disk onto some sort of tape cartridge, which is removable. The transfer rate of data is very slow when compared to hard drive speeds, and backing up takes considerable time. These backups represent a snapshot of data at a given time, and are always somewhat out of date.
Since tapes store the data in a linear fashion, recovering a particular file which has been corrupted or lost can be time-consuming. Despite some of the drawbacks, tape systems have been a convenient way to recover from catastrophic failure.
A more elegant system of backing up files involves disc mirroring or Redundant Array Inexpensive Drive systems. RAID systems have a multi-channel controller and three or more drives. Other systems provide full or partial backup of files on different machines throughout the station.
Storage, small, medium, large
Storage systems need not be complex or expensive, especially at the small-market level.
Jim Trapani, owner of JT Communications, notes that stations on a tight budget may wish to prioritize and backup only commercials and other locally-produced materials. “Music libraries are more generic, and can be easily replaced. Not so with local spots.”
John Fulbright, director of studio applications development for RCS, adds that another popular scheme for low-budget operations is rotating portable USB hard drives. Backups can be done weekly, with one drive taken home while the other stays at the station.
Trapani feels there is a certain virtue in simplicity.
“Some vendors use proprietary schemes to write data to the hard drive, or use a special database to store it. If corrupted, either by a bad hard drive or a software glitch, it may need to be rebuilt.”
He adds that proprietary software may be more difficult to upgrade, as it is harder to write the software. “This is really an area where keeping it simple can pay off. Systems that rely on off-the-shelf software such as Norton Ghost or Windows utilities are usually easier to maintain.”
Criss Onan, AudioVault national sales manager for Broadcast Electronics, notes that when archiving small files such as seasonal music and commercials, the medium of choice still seems to be CD-R or DVD-R. He adds, “USB sticks are a great way for transferring small files to an external device, but won’t hold enough WAV files for large archiving.”
Automation systems with less than half a terabyte of capacity have several solutions. Onan notes a trend away from conventional backup/recovery systems. In their place, drive arrays with the associated software seem to be gaining favor.
“Multiple online drives are generally cheap, require no labor, are resistant to data corruption, are automatic and provide instant recovery.” He said that most AudioVault configurations include distributed redundancy, so that all audio files are stored in more than one computer in the system.
Don Backus, vice president of sales and marketing at ENCO Systems, notes that in truly large systems with storage capacity in the 12 to 15 TB range, the use of removable media can be challenging. “Options are generally limited to redundant servers, or dual servers and tape carousels.”
Best practices for backup also include preventive measures. Fulbright said some basic steps can keep viruses out of your system.
“Keep the audio and automation system on its own subnet, do not connect it to the business network. Don’t enable Internet Explorer or other Web browsers, and use firewalls.”
Off-site backup
At some point, backup systems should intersect with a disaster recovery plan. Off-site storage can protect the station from data loss when the studio building goes away due to fire, floods or other disasters.
For smaller operations, this can be as simple as backing up information to CDs or DVDs at the end of the week and taking the discs home. Larger stations with more resources may want to devise an automated system for off-site backup.
Onan said that as more stations implement HD, they have an IP connection at the transmitter site, so a backup computer with automatic file mirroring is becoming increasingly common.
Using an Internet connection is certainly possible, although Trapani and Backus caution that issues of access, reliability and security need to be considered. Best practice calls for a system that is completely isolated from other communications channels.
Fulbright notes another option for large users is Network Attached Storage. NAS is a form of hard-disk storage that has its own network address, rather than being connected to the same computer that is delivering audio or performing other administrative tasks. This system can be faster because it is not competing with the server for computing resources.
Sometimes it takes a close call to remind stations about the importance of a backup plan. Fulbright notes that events like lightning strikes or the loss of a server can make believers out of casual users quickly.
While the future can be difficult to predict, storage and backup operations may one day be the province of Software as a Service (SaaS). This is a delivery model where software is hosted by a vendor over the Internet. Customers pay for using the software rather than owning it. Google Earth is one well-known example of SaaS.
Richard Heitmann, vice president of product management for EVault, notes advantages for both small and large media operations.
“For smaller organizations with limited resources, the SaaS model is attractive because they don’t have to invest in infrastructure or time to do these things themselves.”
“Larger operations with many remote sites may have the expertise for data storage and backup at the corporate headquarters, but it may not be available in the rest of the organization. For all users, SaaS provides an easy-to-use Web interface for managing storage and backup operations.” EVault maintains seven data centers in North America.
Other advantages of an SaaS model are that sensitive data can by encrypted before being sent to the data center, and is stored that way, and that only new or changed blocks of data are sent to the data center, so the service is fast, and data is available for immediate restore in the event of a catastrophic failure at the studio site.
Heitmann said that in the case of EVault, the service is available as SaaS, a managed service or as traditional software. Customers frequently migrate between the three as they grow and needs change.
“We have a customer in the media industry that operates more than 50 radio stations in North America. They started with EVault’s SaaS, and as their needs grew and changed, they switched to a licensed software model. Over 4 TB is protected and managed anytime, anywhere via the central management console.”
Conclusion
The hardware and software for backing up and recovering files has undergone a revolution in the past five years. Once a series of manual procedures that had to be performed by someone with an understanding of IT, the technology has become almost invisible.
This can be a mixed blessing, and there is a real danger that it can be forgotten or taken for granted. As Backus reflects, “There are only two types of systems: those that have failed, and those that have yet to fail.”
For more about radio IT management, click on that tab at radioworld.com.
