Skip to main content
No. of Recommendations: 71

Allow me to offer you my tale of woe, that you might find knowledge in the catastrophic series of events that took place in my life earlier today.

And remember kids: Always back up your files. The following will help illustrate why.

I run a medium-sized network of about 1,200 workstations and 50 servers. Many of the servers are "popcorn" servers (i.e.: they pop-up all over the place, seemingly out of nowhere, running on commodity hardware), but I have about 12 rack-mounted systems as well. The rackmounts are Dell PowerEdges, some five years old, most brand new 2550, 2650, and 6650 machines with all the trimmings.

The majority of the popcorn servers are scattered across a seven-building network that spans a few miles of territory over our private fiber backbone. Only six of the popcorn servers live in the "server room," a converted supply closet, and only one of them materially participates in the doomsday scenerio I'm working up to describing.

For awhile now---going on five years---I've been asking for air-conditioning in the server room. I'm willing to forgive the carpeted floor, but tropical temperatures are a little over the top.

My winning gambit was this:

The server room is right next to the Accounting department, basically in their office area. I simply left the server room door open due to "over-heating." The thunderous roar of the cooling fans finally won them over... It's amazing how they'll drop 300K on servers but pinch pennies on protecting them.

Granted, it really was quite hot, but my servers are extremely hardy. (Any marginal equipment has long since failed. You wouldn't believe how many drives and cooling fans I've gone through... :)

Rough timeline of events follows:

(All times 7/20/2005, CDT)

08:40 - I'm home, looking out the window at the downpour. It's the first rainy day in over a month, and it isn't fooling around. I decide I'm not going out in this mess, and get ready to call in sick.

08:41 - While I'm contemplating playing hookie, my cell phone rings. It's Candy from the office, and she tells me it is, and I quote, "raining in the server room."

Me: "Turn the servers off! Turn everything off! Hit the breakers!"

Her: "It's really wet in there. Nobody knows where the breakers are!"

I develop a nervous twitch.

09:00 - My ABS brakes fire as I skid to a halt in the parking lot.

09:01 - First sight of the disaster area.

Water is pouring in from the ceiling, cascading onto my favorite 1U rackmounted KVM. The keyboard is soaked, but some kind soul has placed a plastic bag over the flat-panel display. The KVM drawer is blocking the water's natural trajectory towards the primary file-server, so it didn't give its life in vain.

Even the UPS units are soaked. I hurredly run around pushing power buttons, sloshing through the quarter-inch of standing water, careful to only touch things that are dry.


I guess you really should be careful what you wish for.

The air-conditioner was being installed today. The contractors had cut a 4-foot by 6-foot hole in the roof above the server room immediately prior to a two-inch drenching. (I know it was two inches because that's how much standing water there was in my wall-mounted plexiglass document holder.)

After we stopped most of the water and put up tarps (my server room now looks like a tent city...), I began firing the machines back up to avoid condensation issues while a custodian ran the shop vac. Bilge pumps would have come in handy.

The master replica for my NetWare eDirectory network was unhappy about the unexpected shutdown. It was a simple problem, but I proceeded cautiously and had it fixed in a couple hours.

My firewall system, a popcorn box, had its hard drive crash. Perfect timing... and I almost tripped carrying my spare upstairs.

The water also found its way into an adjoining copy-room, there ruining a few (dozen) reams of paper.

Amazingly, aside from that traitorous firewall, there have been no other ill effects from the water (yet... knock on wood).

The only thing that really kept me going today was that I knew where the backups were.


Print the post  


What was Your Dumbest Investment?
Share it with us -- and learn from others' stories of flubs.
When Life Gives You Lemons
We all have had hardships and made poor decisions. The important thing is how we respond and grow. Read the story of a Fool who started from nothing, and looks to gain everything.
Contact Us
Contact Customer Service and other Fool departments here.
Work for Fools?
Winner of the Washingtonian great places to work, and Glassdoor #1 Company to Work For 2015! Have access to all of TMF's online and email products for FREE, and be paid for your contributions to TMF! Click the link and start your Fool career.