Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×

Mendes Report Shines Light on BBG Technical Operations

It discusses shortwave ‘sunset,’ engineer morale and other important matters

Its attention-grabber is a recommendation to lay out a plan to “sunset” U.S. shortwave operations, but there’s lots of interesting reading for any engineer or technical manager in the “2010–2012 BBG Technology Strategic Plan.”

A Freedom of Information Act request has led to publication of the report, apparently written in early 2010 shortly after the top engineer at the International Broadcasting Bureau came on the job. It appears on nonprofit website governmentattic.org.

André V. Mendes set out a two-year plan of how to “transform” technical and IT operations for U.S. government international broadcasting. If his recommendations have been made public in other forms before, I’ve not seen them; regardless, they are fascinating reading to any technically minded person interested in U.S. civil international broadcasting.

André V. Mendes

Notably, he recommended that the organization create a strategy to “sunset” shortwave operations, then implement that strategy starting about now, in the second year of his plan — though the term “sunset” appears to be a bit misleading in this context. He also pointed to morale issues among the engineering staff and highlighted poor IT practices, though generally praised the organization’s technical team.

Let’s take a closer look.

Shortwave questions

Mendes in late 2009 was named director of engineering and technical services for the International Broadcasting Bureau, so he’s a member of senior management there. The IBB provides engineering support for the Broadcasting Board of Governors, which encompasses the Voice of America, Radio Free Europe/Radio Liberty, Radio Free Asia, the Middle East Broadcasting Networks and the Office of Cuba Broadcasting. Mendes is a former senior VP of Special Olympics International and former CTO for the Public Broadcasting Service. In his plan he aimed to set out strategic technological guidance for the BBG.

The future of U.S. shortwave has been a contentious one, of course; but it is not immediately clear whether his shortwave discussion represents a further change in thinking about SW at the organization, or is merely a discussion by Mendes about how to carry out its existing policy. It also was not immediately clear whether all of his recommendations were adopted or whether this report is the most current version; I e-mailed the BBG to ask, and a spokesman promised to get back to me. I will share what I learn. (See update here.)

But we know the BBG has in fact been scaling back on its shortwave infrastructure, as Radio World readers are aware. The question likely to hover now is whether Mendes’ use of the term “sunsetting” indicates a move to abandon that platform entirely.

“Sunsetting” generally is taken to mean “the end, turn out the lights, it’s over.” But in other settings, the BBG has indicated that’s not the plan; and beyond using the word “sunsetting,” the Mendes report does not in fact recommend an utter end to the use of shortwave; his language is more about continued reductions. So I suggest it be interpreted with caution.

Clearly, however, it’s noteworthy that a top technical executive at the organization is using this language, and it shows the direction BBG media management continues to go. (And as we’re reported before, we know BBG had proposed closing the last U.S.-based shortwave broadcasting center, in North Carolina, in its 2011 budget request.)

Mendes wrote that despite “substantial” closings of transmitting stations and other steps in the past, high-powered shortwave remains costly in relation to the “rapidly diminishing effectiveness of shortwave within a growing number of countries.” The cost effectiveness continues to wane, he said, a reality acknowledged by other international broadcasters. “Despite likely political pressures to continue widespread use of shortwave, its thoughtful and targeted reduction presents the best potential for cost savings and substantial increases in price performance ratio for the entire agency.”

Mendes also noted that the organization continues to employ shortwave as the most important transmission mechanism to many target areas, and that so-called “surge activities” actually add transmissions, which end up as a permanent part of the schedule without reductions elsewhere.

“In essence, the decision process for station closing does not appear to follow an overt decision and stated plan to reduce shortwave usage,” he wrote.

Mendes lays out several possible paths for shortwave. In the second year of his plan (which would be starting now, June 2011), Mendes wrote that the organization should initiate deployment of the sunsetting strategy by reducing BBG-owned and operated global SW assets, or outsourcing more transmission operations; or exploring resource sharing agreements with other broadcasters, or leasing additional time and frequencies with third parties, or a combination of the above. (Those underlines are mine.)

So whether a “sunsetting” strategy means the BBG has a real “end date” to shortwave in mind would be a dubious conclusion. But again, the direction is clear.

Much more than SW

Shortwave is just one part of his report about technical challenges at the agency, though.

Interesting to any engineer or operations manager will be how the report paints problems within each of its technology worlds — traditional engineering and IT — and also how those worlds overlap, not always smoothly. “Today’s effective separation and isolation between engineering and IT constitutes a serious impediment to progress,” Mendes wrote.

First he laid out the problems facing engineering staff. Mendes noted the proliferation of platforms — AM, FM, satellite radio, satellite TV, Internet and telephone-based content distribution — without cuts elsewhere. He describes a “silent but ever-growing burden of a burgeoning distribution methodologies portfolio.”

While BBG’s engineering operation generally functions well and its personnel show commendable willingness to “go the extra mile on behalf of the agency,” he said the engineering organization is “highly reactive” and ends up ignoring important operational and risk mitigation procedures.

He talks about the complexity of the systems BBG engineers must now administer; declining financial resources; a disorganized and incoherent Network Control Center; and lack of a solid business continuity/disaster recovery plan.

He described a shift in job skill relevance among engineers as dependence on shortwave shifts to third-party operations, satellite and other “direct to consumer” platforms. “This issue is further compounded by the relatively difficult transition from a traditional RF, antenna, transmitter design and maintenance knowledge base to the technologies involved in digital satellite and IP-based networking systems,” an issue that will be familiar to many Radio World readers.

Mendes also identified a problem of low morale among engineers, calling the mood “palpable and often present in conversations that address historical perspectives on a particular station closing, transfer of technologies around the network and any other such topics. Precipitated by the long periods of employment that are relatively standard in the engineering area and perfectly understandable, this grieving process is a natural consequence of the pride involved in creating a state-of-the-art technical facility only to see it being dissected piece by piece as technology continues its relentless creative destruction.” The lack of clear, BBG-approved plans for future direction doesn’t help, he said.

Meanwhile, on the IT side of BBG’s technology operation, he also saw problems. Even though IT “manages to deliver a relatively high level of service,” he identified multiple single points of failure in critical infrastructure; poor infrastructure layout and maintenance; “nonexistent” business continuity and disaster recovery plans and procedures; no overall platform standards; and other issues. He said that at the time of the report, the organization’s network was dependent on a single enterprise-class Cisco core router whose failure would “severely cripple the entire agency” for an extended period.

“Although some of its systems are quite sophisticated and perform admirably, they often do so as a by-product of employee dedication, substantial expense and ultimately luck,” he continued. “Overall [IT] systems design and coherence between project deployments is sorely lacking and is reflected in a highly complex environment with poor overall integration.”

He provided photos as evidence that “the cabling infrastructure that supports the BBG network is in serious disarray,” criticized past management of the agency’s e-mail platform and spoke of ��blurry lines of responsibility, finger pointing, morale issues and lower overall performance” in the BBG’s IT operation. A brief discussion of business continuity and disaster recovery was redacted from the report, apparently for security reasons.

Without accusing individuals, he wrote in the summary that various circumstances “conspire to create an organizational insularity that, despite everybody’s best efforts and intentions, requires a reset of expectations, priorities and operational imperatives to rejuvenate itself, catch up with the industry at large, and return to a perception of excellence. That time has come.”

Playing catchup

Mendes then laid out a plan to help BBG’s technology team address operational issues and “catch up to the broadcast industry at large.”

On a broad level, he recommended that the organization consolidate its platforms. “In the engineering arena, this approach will be mostly focused on the usage of shortwave and the pragmatic analysis of its expected lifespan and overall scope of global operations but also will include continued migration to MPEG-4 compression technology and steady introduction of IP transmission protocols. In the IT arena, this process will endeavor to quickly reduce the number of computing and storage platforms.”

He recommended use of server, storage and networking virtualization, emulating other agencies in the government. He said BBG should colocate systems into high-availability data centers with multiple redundancy layers, and eventually should migrate much of its application portfolio into a cloud computing scenario; he pointed out that BBG is already using the cloud in some applications.

He also laid out immediate operational recommendations, starting in the first year with changing the name of the Office of Engineering and Technical Services (to Technology, Services and Innovation); setting up a new organizational and tech management structure; creating an application and database to manage all of the agency’s transmission and content distribution assets; and create a Security, Business Continuity and Disaster Recovery Office.

Also in the first year, he called for a continued migration to MPEG-4 content encoding; an upgrade of the Network Control Center; outsourcing of station operations in Bangkok and Udorn in Thailand; and applying what they learn from that outsourcing to other shortwave operations. (This is the part of the report where Mendes describes “a multi-year strategy for physical asset outsourcing, consolidation within BBG’s network, schedule consolidation with other broadcasters and third-party leasing” to optimize use of SW.)

He recommended a big IT consolidation effort; fixing the IT wiring infrastructure; eliminating single points of failure; implementing a hosted e-mail, IM, conferencing and Blackberry server system; and expanding the use of internal intranet sites.

It’s not clear how many of his goals, if adopted, were accomplished in the last 12 months. He also laid out broader goals for the second year.

Mendes also referenced the BBG’s implementation of a Dalet digital media system; he said that after “substantial initial difficulties associated with poor requirements definition, insufficient budgetary allocations, lack of end-user buy-in and lack of appropriate staff skills, the project has made significant advances and is now progressing at a relatively steady pace,” but said the Dalet project would continue to receive “special attention and scrutiny” because it’s so important to the organization’s mission.

I never met André V. Mendes; indeed I learned only after writing this article that he is a former contributor to Radio World’s sister publication TV Technology. But based on my own limited exposure to the technical people and facilities that support U.S. international broadcasting, his report seems an intelligent discussion of real problems. His comments about engineer morale are poignant.

I can imagine he has upset some apple carts though. Who wants photos of their bad wiring spread around the Internet? And whether Mendes and the staff have the chops to change an entrenched technology culture embedded within a federal bureaucracy is hard to predict. But if you are interested either in U.S. international broadcasting strategy or its technical implementation, it’s fascinating reading.

You can read the report here in PDF form. Note that the report is followed by separate, apparently earlier information about the organization’s technology and new media efforts.

André Mendes replied to this article with comments about his report’s timing and the board’s use of shortwave. Read that here.

Paul McLane is U.S. editor in chief.

Close