Monday, November 28, 2011

Hell


Reunion Island is great but life is different when nothing is going right at work.



My work is fairly simple, I have to setup different types of machines and have them talk to each other. The reality is of course quite complex and you really see it when those nice machines decide not to do what they are supposed to. I came back to Reunion Island to increase the capacity and the features of the machines, to do so I need to upgrade the software they use. This type of operation usually occur at night when most customers are sleeping and are not using their phones. I often start working around 11 PM, end up in bed around 3 or 4 AM if I am lucky and much later if I am not.



Things usually go like this:
we get to the workplace
we check that everything is working well (this way if something is not working afterwards we know that it is our doing)
we do whatever needs to be done
we check that everything is still working well
if there is no issue we go home
if there is something wrong we try to find a solution
if we haven't found anything around 5 PM we undo whatever we did hoping the issue will disappear
if the issue is still there once we are back to the way things were before then we are screwed
Issues usually occur a few seconds after modifications have been done but they can sometime be sneaky.



One day a problem occurred 21 hours after our modifications. In this case I have the pleasure of spending my evening on the phone and in front of my laptop to try and bring everything back to normal.
A customer once complained but was unable to tell us when his issue started (knowing when an issue first occurred helps us compare with the modifications we did on that day). We tried to reproduce what was described by the customer but weren't successful, in this case this means for us that the customer doesn't know how to use his phone. The customer kept on complaining but we would still not be able to reproduce his problem. In the end the customer was right to insist, he was the only one with the problem and we would have never seen anything if not for him.
The thing I like best is getting emails after working during the night where I would find diagrams showing how everything was great before and how it was all bad since I changed some things. In some cases it becomes so bad that I can't even wait till the night to change everything back but mostly I am good for another night to erase the previous nights work.
The crown goes to the change that will create 3 or 4 different problems and give the impression of originating from different machines. With this one you get long nights in front of the laptop coupled with lengthy emails and phone calls to explain those who are supposed to help you what you did before trouble started.



It is great once things calm down. It was to a point where everybody was coming to us when there was an issue even if it was not related and we were starting to doubt everything we were doing. After 2 hard months things are back to normal and I have more energy to enjoy the beach.

No comments:

Post a Comment