I don’t know about you; but when I think of a server move, I think “Headache.”
So many things can go wrong, especially if there is no real backup somewhere. If you have ever replaced a file server for a broadcast digital media system and only had one server to begin with, you might know my predicament.
The new RCS NexGen server sits safe in its rack home.
At our Denver studio facility, a hub for four local stations, we operate with the RCS NexGen automation system, which in our case uses a single RAID-equipped file server.
To begin the process of preparing for the server move, I installed a second hard drive in one of our NexGen workstations. I then mapped the drive to all the NexGen computers so the board ops and producers could begin transferring files they needed to keep, many of them sound files that are not necessarily part of the NexGen database or library but are stored on the server nonetheless. In our setup, they save file pieces to the R drive, a mapped folder on the NexGen server. Over the years, files were forgotten and the amount of material on our NexGen server grew. We started off at just over a terabyte on the server.
New drive
After a week, the board ops and producers had deleted everything they did not need. I then copied over all of their files to the new drive. This brought the amount of material stored on our server down to 500 GB.
Next, I sent an email out and told our board ops and producers I’d begin deleting production files that had not been played or modified in a year. It took me nearly a full work week to get through all of our audio files, but in the end, I cleared out another 300 GB of space.
Then I started on the scheduling of the move. This involved several calls to RCS, mainly to check and double- and triple-check I was prepared.We discussed what to expect. They informed me that becausewe use a single server, in order to change out that server you have to put any control rooms that run live shows in emergency control room mode (“ECR,” in NexGen parlance), and everything else in local database mode. What this meant for us was that we had to have two of our stations in ECR and the others in local database mode.
Because RCS informed us to expect two full days for this move, we had decided that in order to minimize the effect on paid programming, we would do the move over the weekend. We let everyone know when the move would take place so they could help us plan. One of our stations tends to air satellite programming as well as live programming on the weekends, and some of our weekend shows are church-oriented programs provided to us shortly after the services on Sundays. Accommodating the satellite programming and church programs would take some planning.
Saturday came and Director of Engineering for Crawford Broadcasting Cris Alexander and I headed to the office to begin the process. While he was making sure everything on the file server was ready to go, I went around and began switching the control room workstations to either ECR or local database mode. I also switched the audio servers to local database mode. Then began the copy process.
The last moments for the old RCS NexGen server as its files are copied over to a portable hard drive.
We turned off the database on the old server and changed the IP address so we could avoid any mishap. We then moved the old server over to our office network so I could monitor the progress from home. We have an external hard drive that I use periodically to do a utility backup of NexGen. I cleaned off all the data and we used that to copy our database over. We were expecting a 6–8 hour transfer. As I monitored the progress from home, I was surprised. It only took five hours to transfer everything to the external hard drive.
Back to the office and we turned off the old server, took it out of the rack and put the new server in. We plugged it in, made sure it was on the office network and then began to copy the database over to it. We were planning five hours based on how long the transfer took from the old server to the external hard drive, and that’s about how long it took.
Following the script
With that done, we headed back yet again and began the fun process.
We had to make sure the proper folders were shared on the network. Once we did this, we decided to bring the server up by itself on a switch connected only to a NexGen utility workstation. The idea was to make sure that everything was working before we put the server on the NexGen network. We didn’t want to risk any type of corruption of the new server, which could happen if something was not configured properly.
Not surprisingly, it did not work. In Denver, we have some VB scripts that run on each NexGen computer. These scripts were around before my time at Crawford Broadcasting. They unmap the shared NexGen folders and then remap them, also checking for any updates to the NexGen system on boot-up. It was that script which didn’t work, although it appeared to be running.
We had been working with the help of Stephen Poole from our Birmingham, Ala., cluster, who had done a server swap himself recently and created a cheat sheet of sorts for us. He was at a loss and was digging for answers.
Lessons Learned
Contemplating a server replacement in your digital media system? Here are thoughts that may help:
Clean House — Board ops and producers are really good at forgetting about files and not cleaning them out, even with significant warning. Set aside enough time to get through all of your files. I recommend one full week, which may include working remotely from home or just longer hours, because as we all know, other problems tend to come up when we are trying to work on something else that is important. Remember, the less material on the server, the faster the swap will go.
Pick Your Time — Plan the server swap during a time when few people are at the controls. The fewer people around, the less you have to worry about.
Don’t Stress — Do what you have to do to get the swap done. People will be unhappy about it; but they will get over it, especially when the move is over. And in our case, we doubled the amount of available storage and provided for full redundancy in the RAID array (RAID 10). That’s bound to make everyone a little happier.
Just Do It — Server swaps are never fun. Someone always seems to scream about it because, let’s face it, servers are used for a reason: storage. When one goes down, it affects someone. Don’t let this keep you from replacing a server.
— Amanda Alexander
I am an impatient person, so I decided to go to the source, RCS. I called up tech support and after a little while, we found that a VB script must run on the server for everything to work. We also found that the VB script caused a login. What this means is we needed to create another user account on the server (in addition to the admin user). So we did this based on the script and then proceeded to run the script on the server and the workstation.
BAM! There it was, all except for the Update folder. RCS tech support didn’t really change anything. He edited the script to just change the name a little and it worked. He changed it back and it continued working. This is good enough for me. I am all for things working the way we want.
Now for the fun, putting the server back on the NexGen network and switching things back over. We were mainly worried about KLZ(AM), which is the flagship station for Colorado State University football and basketball, and there was a game being aired on this particular Saturday night.
In the past, when switching a station back to normal from ECR, we have always seemed to have trouble with things starting playing back again, usually from an hour earlier. So we decided to do KLZ last. We began with KLTT(AM) as it was in normal programming, with no live shows. That switch went seamlessly.
Then we did KLDC(AM) and it too was flawless. Finally, KLZ. We had to wait for when we knew there wouldn’t be a break. We switched everything back and what happened? NexGen reloaded every spot from the beginning of the game! And we were near the end of the game when this happened! Thankfully, our board op had time to delete everything before the last spot block aired.
So the entire server swap took place in just 11 hours.No programming or air time was lost as a result. If it had not been for the help of my colleague Stephen Poole or the support folks at RCS this move would have been a bust. I definitely recommend never doing a server move for an automation system without first contacting the provider and discussing in detail what to expect and to make sure someone will be around to help if need arises.
It is better to replace a server before something happens such as a power supply going out. If you can control when it goes down, you can minimize the interruption to other people using that server.
Amanda Alexander, CBRE, is the chief engineer for Crawford Broadcasting Co.’s Denver cluster.
