It is an IT turnaround story. In late 2013 David Bray joins the FCC as its chief information officer. He inherits a creaky IT infrastructure with 207 legacy systems and maintenance costs eating up 85 percent of his budget. He finds a staff that had been led by nine permanent or acting CIOs over the previous eight years, its team members relatively unenthusiastic about the prospects of modernization or moving to cloud services.
The FCC IT team involved with part of “Operation Server Lift” poses at the commercial service provider that assumed operation of the commission’s systems at its remote facility. Nevertheless a bold plan is developed to move as much of it as possible into the cloud. It’s a bumpy ride at times, one that culminates in a 55-hour marathon over Labor Day weekend 2015.
The hard work paid off and was recognized when the FCC IT team won the 2015 AFFIRM Cloud Computing Leadership award. The commission says its new cloud-based IT achieves results “in half the time at one-sixth the cost.”
RW talked to Bray about the technical aspects of the turnaround as well as a pivot in staff attitudes that made it possible. This is a story not just about software and servers but also about rallying the “troops,” creating posters on IT, signing coffee mugs in recognition of thanks and decorating a “Thank-o-saurus Rex” dinosaur.
AS A SERVICE
Bray says he knew that updating the 207 IT systems one by one was not an option. It was time to think differently and at the same time bring the infrastructure into the 21st century.
“Cloud computing and ‘as a service’ technology seemed to be the way to go. Freeing data from legacy IT systems and moving toward thin user interfaces with reusable code made sense,” said Bray. “That way we don’t have to do the same things 207 different times. Rather, we can reuse that code as part of a service catalog, and be more efficient and effective in what we are doing.”
At the same time, Bray assessed the mindset of the commission’s IT staff. “Three months in, I assembled the entire team to candidly discuss how they were feeling about the challenges and changes facing them. I adopted the role of listener and paid attention to the different views, concerns and perspectives. Only about 15 percent were excited. Another 45 percent were cautiously optimistic. The remainder wanted to return to the status quo or the way things were 15 years earlier.”
Bray felt that success hinged as much on transforming the culture as the technology.
For any leader, prioritizing resources is only part of the challenge. The human factor too is critical. “Transforming an organization’s culture involves explaining the vision, engaging team members and the public, incorporating their feedback and giving them a license to be creative co-laborers when it comes to problem solving,” Bray said.
FCC CIO David Bray, center, talks with Managing Director Jon Wilkins and Deputy CIO Christine Calvosa in the commission meeting room. “A lot of history preceded my arrival. I knew it would be important to listen and learn, while also encouraging a ‘think different’ mindset about the agency’s technology. I focused on letting everyone know I was open to thinking outside the box, and encouraged them to do the same. I reward creative problem solvers who get stuff done to move the mission forward.”
An example of input from creative change agents cited by Bray is staffer Dusty Laun. He made a pitch for developing a Software-as-a-Service solution for getting something done in six months rather than one to two years.
“Dusty brought data that demonstrated with labor costs and the SaaS solution annual license, it would cost around $450,000 to do the venture, rather than $3.2 million — an 85 percent reduction in cost. The cost to maintain the system would be cheaper too: only $100,000 per year versus an estimated $640,000 to maintain a system on premise.”
For the first four months of 2014, Bray and his team implemented a Virtual Desktop for all 1,750 FCC employees. “That eliminated user’s desktops,” said Bray, “which meant we could direct resources previously spent maintaining each PC towards implementing more cloud-based solutions.”
The second move was replacing the commission’s 15-year-old Consumer Help Desk. By utilizing Software-as-a-Service, the move was complete in six months at a fraction of the cost estimated by private sector contractors.
By April 2015, he said, the percentage of IT staff enthusiastic for the modernization effort had grown to 85 percent. “One team member … said it was like Thanksgiving because everyone is hopeful, you have your family gathered together, and they may still be a dysfunctional family, however everyone is hopeful, on their best behavior, and glad to be together for the event. That’s when I realized we had turned a corner together.”
MAN OF THE WORLD
Dr. David A. Bray, who tweets as @fcc_cio, has become something of a celeb in the CIO world and was named “Most Social CIO” by Forbes Magazine and the Huffington-Post last year.
Bray began working for Uncle Sam at age 15 on computer simulations at a Department of Energy facility. Later he designed telemedicine interfaces and space-based forest fire forecasting prototypes for the Department of Defense.
He has volunteered for Habitat for Humanity International and worked as a project manager with Yahoo and a Microsoft partner firm, according to an FCC bio. He was IT chief for the Bioterrorism Preparedness and Response Program at the U.S. Centers for Disease Control and Prevention, leading the program’s technology response during 9/11 and other public emergencies. He earned a Ph.D. in information systems from Emory University and two post-doctoral associateships at MIT and Harvard.
He became a Senior National Intelligence Service Executive, advocating for “increased information interoperability, cybersecurity and protection of civil liberties.” In 2012, he became executive director for the bipartisan National Commission for Review of Research and Development Programs of the United States Intelligence Community. He has received a number of leadership and achievement awards, and as an Eisenhower Fellow he met with leaders in Taiwan and Australia on cyber strategies for the “Internet of Everything” in 2015.
POWER-DOWN
With these successes, Bray said, he built momentum in the commission for the idea that moving legacy systems to newer cloud platforms could be done for significantly less money than creating new systems on-premise. It could also be done in a lot less time.
The first step to total cloud migration was moving the commission’s e-mail and documents. In this way, they wouldn’t be disrupted by what was to come. This was necessary because the commission didn’t have the funds to replicate and mirror all of its servers at the commercial server provider. Instead, the only option was to power down the servers, retire those too old to keep, do a physical move of remaining services to the server farm, and power back up. Soon the big move had a name: Operation Server Lift.
Migrating e-mail to the cloud was estimated to take about five months but the team did it in two. Helping move things along was the possibility of a government shutdown in October. Once email and documents were in the cloud, planning for the physical migration could begin in earnest.
An RFP was issued in early 2015 for the movement of the FCC’s servers, including data center consolidation and optimization. A contract was awarded shortly thereafter. In the meantime, a beta version of a “stakeholder-driven” refresh of website FCC.gov was unveiled. It had been six months in the making. Much of that time was spent talking to users to make sure the FCC got it right, he said.
Operation Server Lift was beginning to look more and more possible. Twenty-eight days out, the team started a digital countdown clock marking the days, hours, minutes and seconds till the big move.
Are props such as these really important, or are they gimmicks? Bray feels that they can have a big impact.
“Visual aids are tremendously important in helping to inspire and encourage teams to collaborate, work together and ensure we are all on the same page regarding priorities. We encourage team members to create posters of projects they’re working on, in order to visually communicate deliverables and timelines.”
Twice a week, the IT team has 20-minute “boardwalk” meetings, where the highest priority projects are rated. There is also a large chart on the wall that anyone can write on regarding status, any issues and resolutions for these projects. These visual images are then translated into digital form so they are accessible across the team.
An example of the thousands of cables that need to be managed when moving servers to a commercial service provider. Bray adds that these boardwalk meetings are an opportunity to build a sense of community.
“We circulate thank-you gifts to recognize above-and-beyond performers from the previous week. There’s a mug from Harvard, which is doing a case study on the FCC, and a jacket from the Defense Advanced Research Projects Agency (DARPA) robotics competition. Every Monday, whoever has the jacket or mug signs it and passes it on to somebody else they wanted to give thanks to for their work. In addition, on Thursdays, we have a dinosaur that we dress up and decorate — we call it our Thank-o-saurus Rex. It’s these little things, the shared rituals that we do as a community, that bring the team together.”
As the big day neared, Bray and his team tried to prepare for every eventuality. At stake: about 400 terabytes of data. Bray wondered, “What if the truck transporting the storage area network is in an accident? We could restore from backup, but that’s a painfully slow process.” Instead, the IT team made a complete copy of the FCC’s storage area network. The plan called for the two versions to travel in two separate trucks taking two different routes. “One would not leave the FCC until we had confirmed that the other had arrived safely at the commercial facility,” he said.
OPERATION SERVER LIFT
When planning for big events such as Operation Server Lift, you have to factor for Murphy’s Law. Bray and his team had pretty well dodged the bullet thus far, but their luck was about to run out.
On the Friday before Labor Day weekend, the team began to reconnect servers at the commercial site. Then they discovered that the thousands of inter-server cables needed to reconnect the 200 servers didn’t match up with the network topology documentation that was provided to the commercial vendor.
Steps had been taken, as Bray notes. “We had hired two vendors to do the cabling work, a sub to do the work and a prime to verify the work had been done. Even with that check of two different contractors in place, the problem happened for some of the necessary cables.” The bottom line was around half of the thousands of cables didn’t match what was needed, and that meant all of them had to be replaced from scratch.
Deputy CIO Christine Calvosa takes a brief break after not sleeping for more than 30 hours on Labor Day Weekend. Associate CIO Erik Scheibert used a sleeping bag on the floor. Several people volunteered to work extended shifts. “It could have been the moment that people just got frustrated or started pointing fingers,” he said. “We were facing 55 hours of work over a holiday weekend in order to fix the issue. People could have gotten angry or sad or paralyzed by the overwhelming task ahead, and I would have understood completely. What I celebrate is how they didn’t. Instead it was more like, ‘What can we do to help? What can we do to get through this? Let’s start running replacement cables. Let’s get this done.’ The team rallied.”
Despite sleep deprivation and the complexity of the task, servers were being powered up with the correct cables on Labor Day morning. The deadline for having mission critical systems up and running by Tuesday morning was still in place. And sure enough, the two most important systems, the electronic commenting filing system and public electronic documents were online. Additional applications soon followed. By 8 a.m. on Sept. 10, all of the commission’s external applications were online. Operation Server Lift was over.
Looking to the future, Bray sees further gains in efficiency and cost savings.
“Our rack count went from 90 to less than 72, and the goal is to be down to 60 in the next six months. Each rack eliminated is an instant cost reduction. We’re going to keep going over the next two years until the rack count is zero. Then, everything will be in the cloud.” He adds that an additional benefit is the reduced square footage needed for storing servers in the expensive D.C. real estate market.
Bray said that from spending 85 percent of the IT budget to maintain existing systems, the FCC has reduced that to under 50 percent. At a Forbes CIO Summit in March, he said, federal CIO Tony Scott praised the commission IT team as a model of “positive change” for other parts of the federal government.
Bray feels Operation Server Lift was the moment when the group really came of age.
“The biggest joy as a leader, even though it’s frustrating when you encounter surprises and delays, is when your entire team pulls together and you don’t actually have to do anything about it because your team is rallying on their own.”
Comment on this or any story. Email [email protected].
