2009

A couple of years into The Great Recession nobody was too sure about their future, and I was no exception.  I was employed by a subsidiary of one of the largest property title companies in the country.  Working inside of the property industry while a housing bubble converted into a housing crisis was not much fun at all.  Every pay period some of my colleagues would get a pink slip with their paycheck and the next workday the rest of us took our sigh of relief and did what we could to ignore the survivor's guilt. That period of my career is what I think about whenever I'm stressing out about a deadline or how to address boarding a large client.  Problems caused by fast growth and aggressive schedules are fine problems to have compared to the ones caused by a lack of revenue.

Ironically my particular division was the only division in the entire company that was in the black during this time.  I would love to say we were just that well organized and managed that we somehow weathered a storm so bad it bankrupted entire countries.  The truth is that while we did have a fantastic team, old fashioned dumb luck had much more to do with our success.  For every bank that collapsed and cost us a customer, a foreclosure attorney showed up to take on all their assets.  The nature of our business meant we got paid no matter which side of the equation needed to get data on potential assets.  Unfortunately, our division being profitable didn't spare us from most of the problems felt by the organization at large.  Costs and headcounts were cut just as aggressively for us as anybody else; that's just how being a small part of a large organization has to work sometimes.

A rather incredible opportunity for me came out of that climate.  Amongst all the layoffs of all the divisions in all the subsidiaries of an organization that employed around 10,000 people at the start of the crisis, a cut or two were simply too deep.  One such cut was to technicians and developers for a little known international division of the company.  That division wrote and supported custom software sold to governments all over the world for their official records of how they defined real estate.  The software was designed to take filing cabinets filled with plat maps and land surveys and convert them into fully digital systems.

An open call was put out to anybody in the company who felt they could get a handle on the software enough to accompany two veterans of the division to a customer in the Caribbean to perform an upgrade of all hardware and software.  Internet connectivity would be spotty, support from home would be limited, and Best Buy would be nonexistent.  Additionally, the trip had to be taken over Easter weekend because that was the only time in the near future that even the municipal services to the island were more or less closed for everything long enough to come up with whatever needed to happen to have them up and running the next business day.  Accepting the chance it all went disastrously wrong and I only ever saw the insides of airports, data centers, and office buildings; I went ahead and accepted the all-expenses-paid business trip to the Carribean.

There was a short time to prepare for the trip and I tried to make the most of it.  I prepared an arsenal of spare sticks of RAM, common sizes and speeds of SCSI drives, bootable CDs, bootable USB sticks, operating system CDs, a fresh passport, and my swimming trunks. I got to know the two veteran coworkers that would be accompanying me and was grateful to get a crash course from them on what the software did and depended on.  I bought my girlfriend a round trip ticket to match my own and explained to her that she might not see much of me depending on how it went, but we were both sure she could enjoy a tropical island without me just fine if it came to that.  I was glad to have her company, and I was glad to have her carry-on capacity as I refused to check my bags full of critical electronics.  The trip was probably over before it really started if we lost a bag on the flight.  I also traded cell phones with her because I simply didn't trust the battery life of my Cingular 8252 "smartphone" and decided to carry her pink Motorola RAZR around the island instead.

The first working day there was Good Friday, and we got to know the representatives from our client that would be our escorts for the duration of our time on site.  My colleagues were accompanied to the various workstations to start upgrading the software while I was accompanied to the data center to have a look at the servers.  I asked about a call center en route that was fully staffed and learned that it was essentially their 911 operation.  My hosts offered me a jacket from a coat rack kept just outside the data center, and I politely declined.   Datacenter temperatures should range from 68 to 82 degrees Fahrenheit, and while that seems chilly to anybody native to a Caribbean island, it is t-shirt weather to anybody coming hot off the heels of a Chicago winter.

The data center itself was unique in that it was designed to be easily portable in the event of hurricanes.  Rather than the standard giant A/C units on the walls with rows and rows of hanging ethernet cables, everything was contained in the racks themselves.  It was a custom design by APC that had every single rack self-contained with its own cooling unit, UPS unit, switch and patch panel, and finally the servers themselves.   Every other row was dedicated to full-size racks packed top to bottom with air conditioning units.  Good cable management is not uncommon in data centers, but I've never seen anything quite as good as one that was designed to be picked up and moved if it had to be.

I was shown my rack and when I opened the door I found a few 1U servers attached to a SAN.  I had known from connecting remotely that the SQL backups were running OK, but they were ultimately stored on the same set of disks as everything else, thus making SAN a single point of failure.  My plan was to hook up a simple USB external hard drive I brought with me to at least get one other bit of hardware in the mix, then upgrade SQL and our application's server components while the previous night's backups copied over.  As if to say hello to me, when I first laid eyes on the SAN  (which was entirely healthy up to that moment), two drives flickered red error lights simultaneously and my servers rebooted as they lost their OS disks.

I had not even managed to login yet, and all drive health monitors were green when I last reviewed from Chicago.  It was a RAID-5 array that cannot support two disks failing at the same time and simply looking at the SAN was enough to trigger that exact scenario.  I got that sinking feeling, took some deep breaths, shut down the servers, and rebooted the SAN to force it to redetect its drives.  An eternal 10 minutes later both drives flashed back to green and everything appeared healthy again.  I booted the servers back up and everything came up perfectly normal.  New plan: get the data off that SAN before doing absolutely anything else.  Do nothing at the same time.  Just get that data off there and then proceed as originally planned.

I hooked up my USB drive and started copying everything that I was agonizing about being unable to recover during the boot process.  I had about 200 gigabytes of stuff I wanted on that drive before I could rest easy again.  Copy time for that was about an hour and if I stayed there staring at that screen for an hour I was bound to start working on something aside from just letting that backup complete.  I asked my client's representative if we could get some early lunch.  We had at least an hour before we could do anything but let this process finish.

The subject of lunch completely transformed my previously sour escort.  He stood up from his folding chair that he had placed close to the exhaust side of the server aisle, removed his shivering hands from his armpits, and told me he knew a place that would be open.  He explained that the island was very devout and that Easter weekend would include Good Friday and Easter Monday.  The only people working on the whole island that weekend would be us, a skeleton crew at the resorts, and foreign-owned businesses.  He showed me that even the radio stations had static before he switched to his CD player.   He drove us into the capital city to eat at the only location he was sure would be open and serving food.  A Subway that had an identical menu to every franchise location they have in the states.  I still find it so amusing that the first touristy stop on my "exotic island trip" was a Subway.

Over lunch, I had learned that my client's representative was the most recent hire and that's why he was saddled with the job of babysitting the American in the data center over a major holiday weekend.  The guy that was escorting my colleagues was their Director of IT and he was overseeing their work from the comfort of his usual office.  From that perspective, my new friend's initial unwelcoming demeanor was entirely understandable.  I was glad for the opportunity to get to know him sooner rather than later and relax the whole relationship.  When we got back to the data center my new friend's sour expression returned as he put on a jacket off the coat rack.  He asked how long we might have to be sitting in the cold data center.  Fearing the worst about the state of my copy process and that shaky disk array I told him I honestly no idea.  I suggested he hope for the best but plan for the worst and expect to be in there all weekend.

I opened the rack door and was pleased to see all green lights on the SAN.  I pulled out the rack-mount KVM to find the copy process had finished without error.  I disconnected my USB drive and had to resist the urge to bubble wrap it before finding a safe place for it.  There was nowhere to buy bubble wrap and I was doubtful of it being in stock anyway.  I asked if there was any sort of file share we might be able to setup some kind of copy job to just in case, and my new friend explained that sort of thing would need to be asked to the director.  I told him that would not be an issue to setup remotely so no worries about getting the necessary information later.

The rest of the afternoon went just fine.  SQL upgraded without issue.  My application setups ran as planned.  I filled the downtime waiting on copy and installation dialogs writing up a document that would have to eventually become a purchase order for replacement drives for the two troublemakers.  I pulled out my pink cell phone and called a coworker to ask if they could connect from the office they were in.  They confirmed success and I was thrilled to inform my new friend that we could leave the data center early, and that it was very unlikely we would need to return to it this weekend.

That night at the hotel I learned a few things.  I learned that my new friend was correct about only skeleton crews only working that weekend, and the skeleton crew did not extend to the gift shop in the hotel.  The gift shop had bottle after bottle of sunblock on display in the window you walked by on the way to your room.  I also learned the shop owner was independent from the hotel, and that meant hotel staff did not have keys to the shop.  The next thing I learned was that UV rays from the sun that close to the equator have no problem reflecting off sandy beaches.  My very fair-skinned girlfriend, who had spent the day reading a book sitting under a beach umbrella and towels instead of swimming because she was unable to acquire sunblock... had been replaced by a tomato.  My girlfriend also learned about these things and that sun poisoning is a thing.  Something I only knew about from spending a full day in the sun playing baseball some summer in my youth.  To this very day, I occasionally get grief for insisting that we not check any bags on this trip which meant that we couldn't pack our own sunblock due to TSA restrictions on traveling with liquids at the time.

We joined my coworkers for dinner that evening.  They had been to the island dozens of times over the years and handled the rental car, all the driving, and had packed lots of extra sunblock to gift to me and my girlfriend.  They also knew of restaurants that were a few steps up from Subway and were open that weekend.  They were excellent friends and guides to have for the remainder of the trip.

With my sysadmin work done I was free to join them in completing workstation setups, reviewing code, troubleshooting any errors, and just generally helping them get ahead of their own schedule for the trip.  That left us with a spare day to just be tourists and explore the whole island.  I still think about how different everything could have gone had that disk array not come back up, and I still count that trip as the single greatest perk of any job I have ever had.