How Great is Your IT Department?

For a tax preparer, it is an audit.  For an athlete it’s the big game.  For a chef, it’s a food critic.  Many jobs have an ultimate test of strength and skill.  A test to prove you have what it takes to continue doing what you do.  For the IT department it is a server upgrade.  What does it take for an IT department to change and move services to ensure that customers have adequent and reliable services when they need them?    To the IT department it means they’ve got work ahead of them to do.  Work that cannot fail, that has to be implemented in the best timeline, with a perfect failover in case it doesn’t work and the time set aside to make sure that the mission critical server upgrades are seemly transparent to implementation.  This is when the super heroes really shine, where all the IT skills and learning pays off.

As recently as 2 months ago our customers faced a tremendous challenge in accessing our online training solutions.  Simply logging in could take anywhere from 15-40 seconds, and large enterprise reports were occasionally getting timed out error message if they were run during “peak” hours.

What all could cause that?  Just about every piece of hardware and software in the system.  So it becomes an elimination game.  What happens if I do this, what about that.  It almost resembles an eye test.  Does option 1 work better, or option 2?  Option 2 or option 3?  etc.

Here come the super heroes.  They  work day and night, they never get discouraged, they look at and consider all the issues, and  discard those that are not effective.  They work, monitor, analyze and finally come to the operations side and say “This is going to cost us (x-amount);  the hardware/software has been ordered;  I’ll call my family and let them know I won’t be home till I’m done.”

The result?

Things are much better now.  Today a log in can take up to 8 seconds, but it’s usually closer to 5.  Large enterprise reports are done in a fourth of the time it used to take, and the ones that used to time out or fail seem to work great in just a few seconds.

So what did they do, these unsung super heroes?

It seems that over the labor day weekend, our IT department labored.  They quadrupled the processing power and memory capacity of our servers by a factor of four. (Yes I know that is redundant, and no I don’t mean the increased it by sixteen.  I just thought it was important enough to say it twice).

So how do they do what they do?  They study, they learn, they know their systems, they absorb new information everyday in gobs.  When necessary they work for 72 hours straight and sleep on Tuesday…And they do what ever it takes to know their stuff.

So remember, our systems all run because two guys sacrificed a national holiday to do an upgrade, apply a patch or replace some hardware and software.  Maybe we should have an “IT Department Day”