I don’t know about you; but when I
think of a server move, I think “Headache.”
So many things can go wrong, especially
if there is no real backup somewhere. If you have ever replaced a file server for
a broadcast digital media system and only had one server to begin with, you
might know my predicament.
The new RCS NexGen
server sits safe in its rack home.
At our Denver studio facility, a hub
for four local stations, we operate with the RCS NexGen automation system,
which in our case uses a single RAID-equipped file server.
To begin the process of preparing for
the server move, I installed a second hard drive in one of our NexGen workstations.
I then mapped the drive to all the NexGen computers so the board ops and
producers could begin transferring files they needed to keep, many of them sound
files that are not necessarily part of the NexGen database or library but are
stored on the server nonetheless. In our setup, they save file pieces to the R
drive, a mapped folder on the NexGen server. Over the years, files were
forgotten and the amount of material on our NexGen server grew. We started off
at just over a terabyte on the server.
After a week, the board ops and
producers had deleted everything they did not need. I then copied over all of their
files to the new drive. This brought the amount of material stored on our server
down to 500 GB.
Next, I sent an email out and told our
board ops and producers I’d begin deleting production files that had not been
played or modified in a year. It took me nearly a full work week to get through
all of our audio files, but in the end, I cleared out another 300 GB of space.
Then I started on the scheduling of the
move. This involved several calls to RCS, mainly to
check and double- and triple-check I was prepared.We discussed what to expect. They informed me that becausewe use a single server, in order
to change out that server you have to put any control rooms that run live shows
in emergency control room mode (“ECR,” in NexGen parlance), and everything else
in local database mode. What this meant for us was that we had to have two of
our stations in ECR and the others in local database mode.
informed us to expect two full days for this move, we had decided that
in order to minimize the effect on paid programming, we would do the move over
the weekend. We let everyone know when the move would take place so they could
help us plan. One of our stations tends to air satellite programming as well as
live programming on the weekends, and some of our weekend shows are church-oriented
programs provided to us shortly after the services on Sundays. Accommodating
the satellite programming and church programs would take some planning.
Saturday came and Director of
Engineering for Crawford Broadcasting Cris Alexander and I headed to the office
to begin the process. While he was making sure everything on the file server
was ready to go, I went around and began switching the control room
workstations to either ECR or local database mode. I also switched the audio
servers to local database mode. Then began the copy process.
The last moments for the
old RCS NexGen server as its files are copied over to a portable hard drive.
We turned off the database on the old server and changed the
IP address so we could avoid any mishap. We then moved the old server over to
our office network so I could monitor the progress from home. We have an
external hard drive that I use periodically to do a utility backup of NexGen. I
cleaned off all the data and we used that to copy our database over. We were
expecting a 6–8 hour transfer. As I monitored the progress from home, I was surprised.
It only took five hours to transfer everything to the external hard drive.
Back to the office and we turned off
the old server, took it out of the rack and put the new server in. We plugged
it in, made sure it was on the office network and then began to copy the
database over to it. We were planning five hours based on how long the transfer
took from the old server to the external hard drive, and that’s about how long
Following the script
With that done, we headed back yet
again and began the fun process.
We had to make sure the proper folders
were shared on the network. Once we did this, we decided to bring the server up
by itself on a switch connected only to a NexGen utility workstation. The idea
was to make sure that everything was working before we put the server on the
NexGen network. We didn’t want to risk any type of corruption of the new server,
which could happen if something was not configured properly.
Not surprisingly, it did not work. In
Denver, we have some VB scripts that run on each NexGen computer. These scripts
were around before my time at Crawford Broadcasting. They unmap the shared
NexGen folders and then remap them, also checking for any updates to the NexGen
system on boot-up. It was that script which didn’t work, although it appeared
to be running.
We had been working with the help of
Stephen Poole from our Birmingham, Ala., cluster, who had done a server swap himself
recently and created a cheat sheet of sorts for us. He was at a loss and was
digging for answers.
Contemplating a server replacement in your digital media system? Here
are thoughts that may help:
— Board ops and producers are really good at forgetting about files and not
cleaning them out, even with significant warning. Set aside enough time to get
through all of your files. I recommend one full week, which may include working
remotely from home or just longer hours, because as we all know, other problems
tend to come up when we are trying to work on something else that is important.
Remember, the less material on the server, the faster the swap will go.
— Plan the server swap during a time when few people are at the controls. The
fewer people around, the less you have to worry about.
— Do what you have to do to get the swap done. People will be unhappy about it;
but they will get over it, especially when the move is over. And in our case,
we doubled the amount of available storage and provided for full redundancy in
the RAID array (RAID 10). That’s bound to make everyone a little happier.
— Server swaps are never fun. Someone always seems to scream about it because,
let’s face it, servers are used for a reason: storage. When one goes down, it
affects someone. Don’t let this keep you from replacing a server.
I am an impatient person, so I decided
to go to the source, RCS. I called up tech support and after a little while, we
found that a VB script must run on the server for everything to work. We also
found that the VB script caused a login. What this means is we needed to create
another user account on the server (in addition to the admin user). So we did
this based on the script and then proceeded to run the script on the server and
BAM! There it was, all except for the
Update folder. RCS tech support didn’t really change anything. He edited the
script to just change the name a little and it worked. He changed it back and
it continued working. This is good enough for me. I am all for things working
the way we want.
Now for the fun, putting the server
back on the NexGen network and switching things back over. We were mainly
worried about KLZ(AM), which is the flagship station for Colorado State
University football and basketball, and there was a game being aired on this
particular Saturday night.
In the past, when switching a station
back to normal from ECR, we have always seemed to have trouble with things
starting playing back again, usually from an hour earlier. So we decided to do
KLZ last. We began with KLTT(AM) as it was in normal programming, with no live
shows. That switch went seamlessly.
Then we did KLDC(AM) and it too was
flawless. Finally, KLZ. We had to wait for when we knew there wouldn’t be a
break. We switched everything back and what happened? NexGen reloaded every
spot from the beginning of the game! And we were near the end of the game when
this happened! Thankfully, our board op had time to delete everything before
the last spot block aired.
So the entire
server swap took place in just 11 hours.No programming or air time was lost as a result. If it had not been for the help of my colleague Stephen Poole or the
support folks at RCS this move would have been a bust. I definitely recommend
never doing a server move for an automation system without first contacting the
provider and discussing in detail what to expect and to make sure someone will
be around to help if need arises.
It is better to replace a server before
something happens such as a power supply going out. If you can control when it
goes down, you can minimize the interruption to other people using that server.
Amanda Alexander, CBRE, is the chief
engineer for Crawford Broadcasting Co.’s Denver cluster.