On Call

When you’re on-call your phone magically transforms into a grenade with the pin pulled out and you spend the duration waiting for it to go off. That’s how I felt whenever it was my rotation on-call for whatever production issues may crop up.

For the week’s rotation I was on-call I may have gotten 3 hours sleep tops. Between the anxiety of not really knowing what I was doing and praying I would not get a call while I was on to waiting for that 3am call how could I sleep. The days were no better than the nights where your life is essentially put on hold. Your choices on-call were either to be tethered to your home for an entire week never too far away from any Wi-Fi connection or lugging your laptop everywhere you went in case you got “the call”. Holidays were even worse. At any moment your family time, cooking, gift giving, football game or especially drinking could be interrupted by your mobile going off like a little cheer destroying timebomb. When you have pardon yourself from whatever fun you were having (or sober up) in order to TRY and fix something which wasn’t your fault in the first place!

I got off pretty easy when I was on-call. It was 11pm on a Friday and I happened to still be up when the “bat phone” went off. The developer who was also on-call described the issue we were having. Apparently, one of automated jobs was failing because of duplicates of the entry in the table where it was sucking data from. We ran a quick query and found the duplicates but there was no indication which was the correct entry (shocker, right?) To be fair the dev I was working with didn’t know anymore than I did about the job that was supposed to run but decided to ask me, “Which one do you think it is?” Well, how the hell should I know?? At that point I made an educated guess. When I say “educated guess” I really meant do I cut the red wire or the blue wire. I told the developer to delete the bottom entry and rerun the job. My logic was the bottom entry was older assuming the top entry was newer and causing the job to fail. I had a 50/50 shot at being right or bringing the whole system down. Luckily, we cut the right wire, and the job ran successfully.

Needless to say, a few short weeks later I was looking for a new job. I was not cut out to be part of the “digital bomb squad”. I’ll leave the wire cutting to the professionals.