They say the difference between paradise and hell can be one mosquito.
When 9/11 hit, Radio Free Europe/Radio Liberty’s beautiful location in the center of historic Prague turned into a logistics and security nightmare, both for the city and the organization. Eight long years later, RFE/RL has finally finished its move into its new facilities in Hagibor, not far from the city center, built from the ground up with technical capacity and security in mind, while its recent home, the former Czechoslovak communist parliament building, is being turned over to the National Museum of the Czech Republic.
RFE/RL President Jeffrey Gedmin greets Secretary of State Hillary Clinton, who serves on the Broadcasting Board of Governors and visited the new headquarters in April. ‘You and the president frequently talk about soft power and smart power,’ Gedmin said, ‘so I think the first thing to say is: Welcome to smart radio, Radio Free Europe/Radio Liberty.’ Photo courtesy RFE/RLRadio World asked me to offer a summary of the project for readers. I’m a long-term resident of Prague, now providing broadcast radio, TV and mobile phone solutions, upstreaming for mobile video bloggers and converged TV/new media systems. But I worked previously for Radio Free Europe and Radio Free Asia in RFE/RL’s old building and I keep in touch with the organization. I’ve had the opportunity to watch this recent move unfold. I also work with some of the vendors mentioned in this article, though I was not involved in the move.
The story of the project begins earlier than Sept. 11, 2001.
In 1993, Václav Havel offered RFE/RL the use of the old Parliament building at the top of Wenceslas Square for a token three pennies a year, which allowed RFE/RL a chance to move out of its expensive Munich headquarters and refocus its mission for the changing east.
But free is not always inexpensive. Old phone systems, ungrounded wiring, outdated networks, unstable power grids, leaky generators and in general the job of retrofitting radio facilities into a space built for Soviet-era politics proved challenging. Its location on an island in a tourist zone between two highways made security tricky.
IT-based
Given its past, RFE/RL wanted to get things right this time, and after an unsuccessful search for existing facilities that met security, space and technical needs, management finally settled on building them from scratch.
Photo courtesy RFE/RL Just as RFE/RL had used its move to Prague to upgrade to new VSAT and digital audio systems, this new move allowed it to convert to Axia Ethernet-based audio systems for its 49 studios, as well as networked TV and radio in-house distribution.
Instead of the reams of audio cabling that filled the closets and ceilings of the old building, sparse fiber runs and CAT-6 Ethernet provide the bulk of the distribution infrastructure, much closer to an all-IT shop than a traditional radio facility.
While most of RFE/RL is broken up into individual language services, there’s a central atrium that holds the consolidated news and research staff.
With a staff of more than 500 and a six-story building of 220,000 square feet, there’s a lot of room to cover for services and maintenance.
One big function for large news facilities is monitoring, both internally produced shows and external TV and radio programs. The old facility had an extensive building-wide coax system to carry the house feeds, and in the spirit of network consolidation, this has also been moved to a LAN-based solution.
In this case Amino video servers and playout boxes and dedicated displays work in tandem with the open source VideoLAN VLC program on the average desktop. (Since users can be editing material at their desktop, simultaneous monitoring on the same machine is often frowned on.)
For audio distribution, the Telos iPort is used to uplink multiple streams to an IceStream server, and special Power-Over-Ethernet networked speakers have been designed to play out the MP3 streams, while PCs can access MP3 through normal players.
Technology Director Luke Springer in the studio. Photo by Jonathan MarksOnce you hit the studios, the IT feel starts to change. The legacy Wenger booths have been moved from the old building, in a now-streamlined process that takes four hours to dismantle, four hours to reassemble. New Czech hand-crafted furniture was designed both for aesthetic and functional purposes, keeping audio and Ethernet cabling well out of sight while easing production workflow with its horseshoe shape.
A mic boom, speakers, Axia console and Telos multistudio call-in system share equal time with the keyboards and LCD displays for the David digital recording system and other PC devices. The fanless PCs themselves are tucked into the furniture, while under the desk, a half-filled double rack holds the processing and network gear, the Axia distribution nodes and the mic I/O units. Since soundcards are replaced by Axia’s network audio driver, the number of physical audio connects is minimal.
The overall studio feel is spacious, uncluttered and convenient.
Not all of the studios are full production spaces with separate broadcast rooms; 11 are combo production-broadcast, 15 are minis and two extra tiny 4-by-4-foot booths are used for basic news reads. Console size, number of mic inputs and other equipment choices are adjusted for the expected purpose of each type of studio, such as Zephyr hybrids for phone-in rather than full call-in systems.
Consolidation
With 27 language services and hundreds of broadcasters, training on the new systems requires serious planning to work around people who are on continuous deadline.
The Central News Room provides background news data for individual language services. With 27 such services and hundreds of broadcasters, training on new systems requires serious planning to work around people who are on continuous deadline.Language services are moved one at a time, and while most of the training took place last year, refresher courses are given just before the move. The broadcasters are well familiar with their David Systems Digas editing stations, having used them for several years, but the new Axia consoles and call-in systems take some getting acquainted, even with configuration presets on the board that handle most of the dirty work.
When I visited recently, one Iraqi broadcaster was running back and forth between adjacent studios doing two tasks that could have been handled on a single console. Still, the amount of broadcaster support required has decreased with the new facilities, while the number of on-air errors have dropped.
Part of that can be attributed to per-broadcaster presets that ease sharing studios without leftover setup mistakes, more intuitive interfaces, better integration of console, editor and call-in systems and good training.
The Master Control display wall allows easy monitoring of multiple satellite feeds (aside from its own countries, RFE/RL provides backup for Voice of America), as well as signal status, channel meters, network conditions, weather maps and other useful indicators.
The control room originally was designed to house Help Desk and other support functions as well. But in practice a stationary standby team is a thing of the past; with mobile phones, response personnel can handle other tasks while still “on call,” and most networked systems are accessible remotely.
The move allowed the help line to be consolidated to one phone number for all systems, saving users much confusion, while computer, telco, network and broadcast audio racks have been consolidated in one space (aside from the satellite uplink room and per-floor network closets).
UPS on steroids: RFE/RL’s basement holds enough backup charge and emergency cooling to keep the most important systems operational during even lengthy outages.By using audio over gigabit Ethernet, an audio cable mess seen previously has largely disappeared, so that maintenance and debugging are simplified and cable tracing is much less chaotic.
Unlike the old days of “home run” audio connects back to the control room, the network audio breakout point is pushed as close to the end equipment as possible, typically in-studio, and overstuffed cable trays give way to much smaller floor-to-floor fiber runs and in-furniture wiring.
PRINCE2
When Radio Free Europe started working with digital audio 14 years ago, it was with leftover networking from the Prague building’s Communist days, along with a then-state-of-the-art 12-node FDDI network.
These days, communications means a team familiar with its sophisticated 10-gigabit Cisco core network providing high-definition video conferencing over Internet, VoIP to 23 bureaus, video, Wi-Fi phones and data in-house, redundant Internet links and a flexible SkyWAN FTDMA satellite network system for bureaus and distribution. Media assets alone takes up 8 Terabytes on the EMC2 DMX 1500 storage array.
Using several hundred VLANs in a dual fault tolerant architecture in its Cisco Access switches, critical audio system traffic can be separated from normal office use and other networked systems.
Bill Cline, director of broadcast engineering, monitors satellite signals and weather conditions from RFE/RL’s operations center, making sure levels don’t drop and 24 channels of audio get to the right places.Recent bouts of hacker attacks around the Internet continue to up demands for network security. RFE/RL uses nSense and nVision for intrusion detection, Cisco firewalls and Cisco Meeting Place to integrate desktop application sharing, Webcams and conferencing. A combination of CA Spectrum and CiscoWorks supplies overall network management, and Websense handles content filtering.
Raised floor cable runs in the rack room are now dominated by power lines to all the equipment. Having learned from the previous building’s HVAC and power problems, management paid much attention to designing the new.
The building requires 2 Megawatts power, and the basement holds two 1 MW UPS systems — an impressive store of car batteries — with failover and power shedding capabilities to keep critical systems on air. Important areas of the building such as the rack room and master control are dual-supplied to run on either UPS. For HVAC, there are four chillers — two redundant — with a cross-matrix for guaranteed power supply, and four PACUs (precision air conditioning units) for the main equipment room with dual water supply.
Some of the staff had been involved in the earlier Munich-Prague move, which in some ways resembled a mad cross-border dash. There was more effort this time in formalized planning, using Microsoft Project, PRINCE2 project management methodology and other techniques.
This led to a more evolved business continuity plan to keep languages on the air and systems running. A halfway staging area was outfitted so that a system out of the old facility would come to staging, its operational capability would be confirmed, and then it would be moved to the new building as another system replaced it.
Tech Team Key members of the tech team were RFE/RL’s Director of the Technology Division Luke Springer, Deputy Director of Technology Chris Carzoli, Director of Broadcast Engineering Bill Cline, Director of Technical Operations & Facility Management Don May, Director of Networks and Telecommunications Manfred Hanspeter and Director of Information Technology Ron Crozier.
Technical Division project management was by Michal Simice, while overall RFE/RL project management was by Nick Kavelakis, working under EC Harris, a consultant specializing in “built assets.”
Members of the tech team and colleagues gather after grand opening ceremonies. Lower row, from left: Luke Springer, Manfred Hanspeter and Chris Carzoli. Standing: Neil Kinnock, Arup HVAC consultant; Don May; John Meyers, EYP Mission Critical Facilities security consultant; Bill Key, director of corporate security; Bill Cline; Beth Portale, chief of staff; Terry Dix, Arup consultant director; Mark Bartlett, Arup M&E consultant; Nick Kavelakis; and John Lindberg, RFE/RL corporate counsel. (Other titles in sidebar text.) “Bill and his engineers worked together with Don and his tech ops guys for control room design,” said Carzoli. “Engineers did the studio design with input from the sound techs and broadcasters, and their own knowledge and experience.” The work involved on the studio, control and satellite portions of the project, Carzoli said, was a Broadcast Engineering team effort led by Bill Cline.
Don May was the “overall go-to guy for the entire project, with his CAD and facilities management experience, on top of his TV/video engineering expertise, and ability to be the pit bull when needed,” Carzoli said.
The technical budget for the project was approximately $12 million.Focus on user-acceptance of the new systems and facilities was much greater under PRINCE2 than it would have been without. System testing and acceptance was much more methodical and rigorous than prior efforts, identifying some major contractor flaws that might have slipped through otherwise.
One weak point in RFE/RL’s management structure is that it has a split front-office between the United States and Prague, so that response and diligence from above can be difficult to harness for schedule-impacting conflicts and contract issues.
In the end, large delays did pop up due to a contractor’s organizational difficulties and unforeseen circumstances such as huge swings in currency exchange rates (the dollar depreciated 30 percent or more compared to the Czech crown since the project started, an important consideration for international projects).
Some tasks were reprioritized to lessen the schedule delay, such as moving the uplink satellite dish. Final transmission output still went back to the old building via a fiber connect, running 24 channels of audio across Axia nodes, until the move could be rescheduled. (With a 20 foot dish mounted to the opening inside of the building, traffic on the highway has to stop while a crane maneuvers over the outside walls).
This cross-town fiber was thrown in as part of a rather large telecommunications contract, heavy international calling being an essential part of daily reporting from dozens of countries.
While risk management can deflect or avoid some of these effects, with the specialized needs of large international radio broadcast from a small country, it’s not easy to pick and choose perfect solutions.
Much has changed in the last eight years. The Czechs are now full EU members; RFE/RL has expanded broadcasts for Afghanistan and Iraq with most of its East European services dropped; and IP-based digital audio has come of age.
The new building not only had to meet operational requirements, but post-9/11 it needed to pass State Department security as well. With Secretary of State Hillary Clinton’s visit to the new facilities in April, a ribbon cutting in May and the move of the Russian language service completed, RFE/RL could look forward to a summer coming-out party.
While the surroundings may not be quite the aesthetic paradise they were in RFE/RL’s last location, at least the organization finally has found its safe European home (cue up The Clash).
Bill Eldridge can be reached at[email protected].
