As the fresh\u00adly brewed cof\u00adfee enters my mouth, I expe\u00adri\u00adence my first glimpse of con\u00adscious\u00adness for the day. \u201cWhere am I?\u201d I mut\u00adter, in bro\u00adken English. The gray walls around me slow\u00adly come into focus, lit by the flick\u00ader\u00ading of a long-in-the-tooth flu\u00ado\u00adres\u00adcent bulb. The top half of a man\u2019s face appears over the top of my cubi\u00adcle wall.<\/p>\n
\u201c<\/span>How\u2019s the won\u00adder\u00adful world of iNetConjoinApp?\u201d<\/p>\n The caf\u00adfeine must have made it past my blood-brain bar\u00adri\u00ader, as I rec\u00adog\u00adnize at once that I\u2019m at EnergyModCo, where I am one of a hand\u00adful of employ\u00adees. The half-head belongs to Freyr, EnergyModCo\u2019s COO<\/span>, lead cus\u00adtomer ser\u00advice rep, and deploy\u00adment technician.<\/p>\n \u201c<\/span>Umm, it\u2019s, well, I just start\u00aded work\u00ading on the \u2013\u201d<\/p>\n \u201c<\/span>Great, that sounds good. You remem\u00adber BigBoxoCo?\u201d<\/p>\n \u201c<\/span>You mean, as in our biggest cust\u2013\u201d<\/p>\n \u201c<\/span>There\u2019s a prob\u00adlem at one of their ware\u00adhous\u00ades. Something to do with our light\u00ading controller.\u201d<\/p>\n \u201c<\/span>Oh?\u201d<\/p>\n \u201c<\/span>I just got off the phone with the ware\u00adhouse man\u00adag\u00ader. All the lights went out for a few min\u00adutes, but they\u2019re back on now.\u201d<\/p>\n \u201c<\/span>Uh, that\u2019s bad. Thank good\u00adness they have skylights.\u201d<\/p>\n \u201c<\/span>Nope. This is their first two-sto\u00adry ware\u00adhouse. The only light the first-floor cus\u00adtomers had was from the emer\u00adgency floodlights.\u201d<\/p>\n My throat tight\u00adens. \u201cWell, I\u2019m on it. We can\u2019t let that hap\u00adpen again.\u201d<\/p>\n The weight of the sit\u00adu\u00ada\u00adtion slams into me like an over-packed palette of giant may\u00adon\u00adnaise jars. After being awake for only 25 sec\u00adonds, I\u2019m not ready to douse this kind of blaze. I don\u2019t have a choice though, so I lean back in my chair and gaze at the craque\u00adlure on the ceil\u00ading tiles. How could this have hap\u00adpened? I recall that at one time we did have prob\u00adlems with the smart-break\u00aders that switched the lights. They would some\u00adtimes mys\u00adte\u00adri\u00adous\u00adly ignore the com\u00admands sent to them by EnergyModCo\u2019s soft\u00adware. But I fixed that by adding a watch\u00addog that would retry the switch com\u00admands if they did not take effect. After a brief pal\u00adpi\u00adta\u00adtion sub\u00adsides, I admit to myself that the light\u00ading con\u00adtrol watch\u00addog must con\u00adtain a nasty bug.<\/p>\n After some brief email dig\u00adging, face palm\u00ading, and silent curs\u00ading, I man\u00adage to get a VPN<\/span> con\u00adnec\u00adtion set up so that I can SSH<\/span> into EnergyModCo\u2019s on-site light\u00ading con\u00adtroller. I bump up the log\u00adging ver\u00adbosi\u00adty, which requires that I restart the sys\u00adtem, and start look\u00ading for clues. After a few min\u00adutes, I see the periscope that is Freyr\u2019s fore\u00adhead rise above my cubi\u00adcle wall.<\/p>\n \u201c<\/span>The lights are off again! I\u2019ve got the BigBoxoCo man\u00adag\u00ader on hold, and he\u2019s about to lose it!\u201d<\/p>\n My lip quiv\u00aders as I strug\u00adgle to sup\u00adpress my fight-or-flight instinct. It can\u2019t be a coin\u00adci\u00addence that the lights went off right when I restart\u00aded the soft\u00adware. What have I done? In a pan\u00adic, I force our soft\u00adware to turn the lights back on, and thank\u00adful\u00adly it works. At this point, I am par\u00ada\u00adlyzed with fear. I want to dis\u00adable our soft\u00adware entire\u00adly, but what if stop\u00adping it is what made the lights go off just now? I pull my hands away from the key\u00adboard, fear\u00ading that any\u00adthing I do might cause the BigBoxoCo man\u00adag\u00ader to enter sud\u00adden car\u00addiac arrest, or worse.<\/p>\n Without being able to touch the on-site soft\u00adware, I dive into the source code, hop\u00ading to track down the bug ana\u00adlyt\u00adi\u00adcal\u00adly. I pore through the entire stack, fol\u00adlow\u00ading the data flow and log\u00adic for the rel\u00ada\u00adtive\u00adly sim\u00adple light\u00ading con\u00adtrol sub\u00adsys\u00adtem. The sched\u00adul\u00ading code makes sense. So does the the timer code. The trick\u00adi\u00adest code, for the watch\u00addog sys\u00adtem, looks entire\u00adly cor\u00adrect. I rack my brain; what am I miss\u00ading? I am star\u00adtled by the crack of thun\u00adder, but there does\u00adn\u2019t seem to be a storm out\u00adside. As my nerves res\u00adonate with the imag\u00adined sound, Freyr\u2019s fore\u00adhead crests my cubi\u00adcle wall.<\/p>\n \u201c<\/span>The BigBoxoCo man\u00adag\u00ader is flip\u00adping out. He says, and I quote, that \u2018There\u2019s a God damned dis\u00adco par\u00adty going on\u2019 in his ware\u00adhouse. They are going to have to stop accept\u00ading customers.\u201d<\/span><\/p>\n Content with hav\u00ading sur\u00adpassed even my worst expec\u00adta\u00adtions, Freyr jogs back to his office. I fol\u00adlow him briskly.<\/p>\n \u201c<\/span>Freyr, can\u2019t the man\u00adag\u00ader hit the man\u00adu\u00adal light\u00ading over\u00adride? I think it might take me a while to fig\u00adure out the problem.\u201d<\/p>\n \u201c<\/span>What are you doing away from your desk? No! Only BigBoxoCo\u2019s main\u00adte\u00adnance engi\u00adneer has a key to the enclo\u00adsure, and he\u2019s AWOL<\/span>. Go!\u201d<\/p>\n I save three sec\u00adonds by run\u00adning back to my desk. At this point, I\u2019m bounc\u00ading ideas off our oth\u00ader pro\u00adgram\u00admer, Nate. No good; he has nev\u00ader worked on this sys\u00adtem, and can only offer lim\u00adit\u00aded advice. I go back to star\u00ading at the code. I may have been uncon\u00adscious an hour ago, but now the fire of my mind is burn\u00ading with the focused inten\u00adsi\u00adty of a TIG<\/span> welder. The cof\u00adfee is gone. I begin ques\u00adtion\u00ading all of my assump\u00adtions. Compiler bug? Memory cor\u00adrup\u00adtion? Cosmic rays? Nate com\u00adplains about the sound of my fore\u00adhead slam\u00adming against the desk. In my height\u00adened state of aware\u00adness, I per\u00adceive the ghost\u00adly sound of foot\u00adsteps come to a stop out\u00adside my cube. Moments pass before the shrunk\u00aden form of Freyr emerges from the hall\u00adway. I am calmed by his lack of speed as well as the fact that he is not using his periscope.<\/p>\n \u201c<\/span>I\u2019m sor\u00adry,\u201d he says.<\/p>\n \u201c<\/span>Uh, hey Freyr, what\u2019s up\u2026?\u201d<\/p>\n \u201c<\/span>I fixed the lights. It was my fault.\u201d<\/p>\n Until this point, it had not crossed my mind that the prob\u00adlems may have been caused by the light\u00ading con\u00adtroller being con\u00adfig\u00adured incor\u00adrect\u00adly. \u201cWha \u2014 what the hell happened?\u201d<\/p>\n \u201c<\/span>I had con\u00adfig\u00adured the light\u00ading con\u00adtroller at a dif\u00adfer\u00adent site with the IP<\/span> address of the smart-break\u00ader at the dis\u00adco ware\u00adhouse. The oth\u00ader site was in a dif\u00adfer\u00adent time\u00adzone, and its sched\u00adule said that the lights should be off.\u201d<\/p>\n The prob\u00adlem was too sim\u00adple. Why did\u00adn\u2019t I think of this? One con\u00adtroller thought the lights should be on, and the oth\u00ader thought they should be off. Thus, the light\u00ading watch\u00addogs at each site were fight\u00ading over con\u00adtrol of the lights. Neither con\u00adtroller knew about the oth\u00ader one; they just thought that the smart-break\u00aders were dis\u00adobey\u00ading them and retried their com\u00admands. Over and over. I sub\u00addue my first instinct to tack\u00adle Freyr on the spot, and mur\u00admur, \u201cOkay. Thanks for let\u00adting me know.\u201d<\/p>\n I sit still for a few min\u00adutes, allow\u00ading the tur\u00adbu\u00adlence of my rage to sub\u00adside. My ini\u00adtial response is to be angry at Freyr for wast\u00ading my time and ter\u00adri\u00adfy\u00ading me. However, as I calm down and regain clar\u00adi\u00adty, I real\u00adize that he did noth\u00ading wrong. Who has\u00adn\u2019t mistyped an IP<\/span> address before? I know I cer\u00adtain\u00adly have, many times. Freyr made a sim\u00adple and under\u00adstand\u00adable mis\u00adtake. The prob\u00adlem was that the BigBoxoCo net\u00adwork was set up in such a way as to allow a sim\u00adple mis\u00adtake to wreak utter chaos.<\/p>\n As it turned out, BigBoxoCo had all of their hun\u00addreds of ware\u00adhous\u00ades on the same vir\u00adtu\u00adal net\u00adwork. Not only could BigBoxoCo\u2019s cor\u00adpo\u00adrate head\u00adquar\u00adters reach machines at every sin\u00adgle ware\u00adhouse, but so could any indi\u00advid\u00adual ware\u00adhouse. A PC<\/span> at a BigBoxoCo in New York could ping a PC<\/span> at a BigBoxoCo in Oregon with no prob\u00adlem. Even ignor\u00ading the secu\u00adri\u00adty reper\u00adcus\u00adsions of such a set\u00adup, there are good rea\u00adsons to avoid it. If the net\u00adwork was set up with a star topol\u00ado\u00adgy, with only the cor\u00adpo\u00adrate head\u00adquar\u00adters hav\u00ading access to every sin\u00adgle ware\u00adhouse, the dis\u00adco par\u00adty fias\u00adco could have been eas\u00adi\u00adly avoid\u00aded. In oth\u00ader words, seg\u00admen\u00adta\u00adtion is good.<\/p>\n","protected":false},"excerpt":{"rendered":" As the fresh\u00adly brewed cof\u00adfee enters my mouth, I expe\u00adri\u00adence my first glimpse of con\u00adscious\u00adness for the day. \u201cWhere am I?\u201d I mut\u00adter, in bro\u00adken English. The gray walls around me slow\u00adly come into focus, lit by the flick\u00ader\u00ading of a long-in-the-tooth flu\u00ado\u00adres\u00adcent bulb. The top half of a man\u2019s face appears over the top [\u2026]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false},"categories":[3,16],"tags":[20,19,18,17],"_links":{"self":[{"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=\/wp\/v2\/posts\/283"}],"collection":[{"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=283"}],"version-history":[{"count":66,"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=\/wp\/v2\/posts\/283\/revisions"}],"predecessor-version":[{"id":646,"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=\/wp\/v2\/posts\/283\/revisions\/646"}],"wp:attachment":[{"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=283"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/blog.mezeske.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n