Can the Broadcast IT Facility Rise to the Management Level of a Well-Run Circus?
In the March 1 issue, the author presented the third in a multi-part series on IT Service Management. The series continues here. Readers can review the series at www.rwonline.com Click on the tab IT Service Management.
Over the years I’ve heard (and occasionally referred to) the workplace referred to as a “three-ring circus.”
Having seen a few circuses lately with my kids, enjoying the well-coordinated changes between acts, the fluid choreography between performers and support crew alike, the attention to detail and the perfection that comes with practice, I’ve come to look at three-ring circuses with much more respect.
At a minimum, the tightrope is kept in good repair, the lions are always fed and the trapeze artists discuss changes before they go on – a sense of priorities that may not be matched in a broadcast facility.
In this installment on IT Service Management (ITSM), we look at three rings of Service Support -Configuration, Change and Release Management – to see if the broadcast facility can rise to the level of a well-run circus.
The introduction of IT equipment and services into broadcasting is often a haphazard process – old-style tape decks exist side by side with new digital consoles, analog processes painfully replaced by digital ones.
Even where new facilities are created from scratch, inventories may soon be out of date, or tracking systems may not be geared towards what is useful.
In keeping with the business focus of ITSM, Configuration Management – tracking the infrastructure, processes and capabilities in the facility – should first address the business case and planning of an organization.
This takes several angles – long-term capabilities and maintenance, financial planning, emergency recovery, and daily production – but the information concerning these needs to be available. But if configuration chaos rules and there are no obvious shared qualities across systems, it’s hard to plan.
Unfortunately, IT systems contain an illusory bit of cross-compatibility. Whereas hardware-based broadcast systems contained relatively true-to-self model numbers, the differences between “similar” x86 PCs and Windows versions are substantial.
Not only might replacement parts be different between models, changes in software drivers for network cards or USB devices might break audio software in the process. Changing from Windows 98 or 2000 to XP is even more drastic, and with normal office desktop functions mixed in with broadcast systems, it’s easy to reach the point where any change at all is a scary step into the unknown. Even black box equipment can contain different chip versions on the same model, while digital disk or tape formats go obsolete much faster than analog tape.
The first step to handling this chaos is to get a grip on what matters – detailing and managing the facility’s Configuration.
Equipment needs to be catalogued not only with respect to model, parts and purchase date, but also in regards to performance and function. IT and network systems are typically not standalone – they interact as subsystems, such that the ways and formats in which they communicate need to be noted. Do not ignore human factors. Appropriate noise levels, ease of usage and availability at peak times in the day are just some of the elements.
As usage of IT and the Web grows in broadcast, they frequently have their own departments – with an increased lack of communication with broadcast and production. Improved documentation and process understanding become crucial to effective cooperation.
If problems or observations cannot be addressed right away, they should still be noted so that later facility enhancements take them into account. It is unlikely that anyone sane would catalog equipment to the chip level, but when you note problems, make sure to jot down the affected parts while you have the box open.
A shared maintenance record can do wonders for non-obvious relations between problems – a computer that has first recording difficulties and later network dropouts might turn out to have bad memory as the root cause. It’s hard to see performance changes if there aren’t initial measurements, and some problems creep up slowly enough to be confused with unrelated changes in the meantime.
At the same time, when a computer vendor frequently updates drivers, keeping your own local copy is a waste – maintain lists of direct links to the vendor’s update sites instead. (Some sites such as www.oldversion.com keep earlier versions if you need to downgrade.)
While getting the Finance Department to plan three to five years down the road might be nigh on impossible, a technical plan that addresses obsolescence and upkeep makes it easier. Track staff time and maintenance costs; the practice may help to highlight the fallacy of stocking too many brands of equipment or the illusory cost savings of going to the lowest bidder. A bad lot of hard drives or memory in an order of 100 machines can cost weeks of maintenance and catastrophic failure.
Inventory emergency backup equipment as well. Flag any cannibalization before it affects the facility’s business plan and survival itself. Maintain emergency and standard processes across staff changes, noting not only the appropriate training levels but any associated equipment and resources.
Tracking software licensing is one of the more obvious new IT functions. This can be a source of liability or an unacceptable risk if discontinued or used improperly. Poor upkeep of virus software endangers office and broadcast systems.
In a broadcast facility, “configuration” might extend to program and news material, video or audio libraries, storage, licensing, tracking of on-air programs and commercial spots, rights management, theft protection and other less tangible but critical functions. The rules for Web delivery of radio and TV differ significantly from traditional broadcast, while mobile phone streaming provides yet another playing field to track.
It is not usually the responsibility of Configuration Management to conduct training, but the database system used should make it obvious to the affected parties what areas are deficient. Similarly, Configuration may not fix performance problems, but baselining and assessing changes in systems performance are two of its hats.
The good news is you can choose from a number of affordable, off-the-shelf packages to track computer and network performance. Scripting languages such as Python and Perl, and Windows-specific WSH, ADSI, WMI and VBScript ease automation of broadcasting tasks without heavy programming.
Other ITSM areas impacting Configuration include Availability, Capacity and Continuity Management, as well as Service Level Agreements. In reality, all functions of ITSM have a strong need for effective Configuration Management, and unfortunately its typical implementation is usually little more than asset tracking.
While putting a facility’s configuration in a single database may be impractical, try to consolidate systems into reasonably complete areas.
The facility’s configuration relies heavily on Change and Release Management, two sides of a similar coin.
Obviously daily operations produce changes, whether the date on a log or the occasional replacement of defective equipment. More important are substantial changes involving types of equipment used, procedures and processes altered or job tasks.
Changes that seriously alter the workplace, business plan or risk assessment need to be evaluated before adoption. Specify them in a Request for Chance (RFC), whether via a small note or a full-blown plan. The configuration record should then be updated while you inform affected staff, management and other interested parties.
More than one ITSM function can be handled by the same person, but specifics of each role should be carried out with some independence – one hat at a time.
For broad scale compatibility and functionality, Release Management typically is employed. Here, hardware or software of a particular type might be replaced throughout a facility or org at one time, such as a new digital console or audio software version.
The multiple roles and inter-relatedness of PCs, software, network functions and other IT factors may require simultaneous release changes of several subsystems, more often than that in old-style broadcast facilities. While pre-planning usually goes into major Release upgrades – with lab systems, real-life prototypes and test run-throughs – sadly Change Management is often handled haphazardly (or to be precise, not managed at all).
Plan, prototype and practice are three important factors in predictive change, yet time, equipment and budget constraints often preclude giving them more than lip service.
Isolated changes should be evaluated – even if only quickly – as to priority and impact, with possible effects on financial, technical and business concerns. The combined effects of incremental Change in components or subsystems can strain the resources of an organization where a larger, more structured Release would serve better. It is easier to get the full attention of management, users and other stakeholders with infrequent broad changes than with continuous tinkering and alteration.
Change is a political process as well, and requires time to initiate buy-in and understanding. There is also the danger that with frequent unplanned Change, important issues won’t be noted in the Configuration record.
Facility Security is one multi-faceted concern that’s easy to compromise. Between increasing risk, accidentally allowing unauthorized access, increasing system instability and lowering availability, forgetting to tune performance, and ignoring tie-ins with emergency procedures, a “simple” change has multiple ways to go wrong.
Create a small Change Advisory Board to decrease the temptation to skip steps. The Board provides some formality to the system, decreases the temptation to skip steps, as well as lowers the chance that one pair of eyes will miss something. The Board can make recommendations on the timing of the Change or Release, set standards for evaluating success or failure, and provide a backup plan where implementation fails.
Since these changes are typically carried out as a Project, and Projects tend to be temporary in scope, side effects of a change may not be felt until the system is online for some time past the Project’s life. Schedule follow-up evaluations under Problem Management or as a special task force.
Don’t forget to coordinate and evaluate changes with external parties such as service providers. As you add more data to Internet connections or leased lines, or introduce more call-ins for a radio show, you may introduce unforeseen problems. Similarly, changing vendors always requires care.
One area of frequent neglect is training, thanks to cost, time, difficulty in scheduling and lack of expertise. Setting up effective training systems is tough, and talented staff members are not necessarily talented trainers or available for training. Combine this with an extensive Release that needs different types of training, and it’s understandable but unfortunate why this step is often skipped.
It is unreasonable to expect every broadcast organization to implement a full ITSM structure. The reality is that there are plenty of tiny low-budget stations just as there are one-ring travelling circuses that still manage to entertain.
ITSM is an iterative process; early simple savings build to later more difficult improvement. As in a circus, planning and coordination are essential, and some tasks are critical.
The end goal of broadcast ITSM is not just a robust facility, it’s the successful delivery of a flawless show. Too often, the backstage crew forgets to have fun too. But even in the best of circuses, things can go wrong. Next time we take up Incident and Problem Management.