Part 1
(This all happened way back, in the mid 2010s. It is still one of the weirder missions I’ve been on as a consultant.)
My collegue got a surprise call late on a friday evening. The caller was obviously in a state of desperation, since the call started with deep sigh, then silence. Eventually, the caller explained who he was, and why he called:
…Unmanned transportation… …Big european City… …Original networking team can’t make it for unknown reasons… …Air-gapped network… …High Security… ..Hardened equipment… …Many protocols to follow…
After asking about our hourly rate, the caller just said: “You need to be here on monday morning at 08:00 AM. Can you make it?” My collegue had already started a chat with me on his computer during the call. We both agreed. Nothing to lose, all to gain.
Let’s rock!
We arrived to the site in good time on monday morning, after spending most of the night in the air. It turned out no one had a clue who we were, who ordered us there, or what work we were supposed to do, but that didn’t matter, because we weren’t allowed to enter the site without passing the security and compliance test anyway. We were lucky though, we were told a security training class was being held at 09:00 in the central office a few blocks away. Perfect! Well, perfect might not be the right word. The educator couldn’t really speak english. At all. And the training class was supposed to be in english for obvious reasons. We were 15 or so in the class, and and several continents was represented: Australia, USA, India, Spain etc. People from all over the world. Regardless of the incomprehensable english spoken, we did understand from the slides shown that we had to wear a hard-hat at all times. And a reflective vest. And protective footwear. Same slides over and over, depicting men in hard-hats. With reflective vests. And protective footwear. That went on until lunch.
After lunch, we started off with some repetition. In case we had forgotten about the importance of hard-hats, reflective vests, and protective footwear. The educator then moved on into chapter two: The things that can happen if you DON’T wear a hard-hat, a reflective vest, and protective footwear at all times… Dear lord. You might fall and hurt yourself! Anyway, it seemed they had implemented this three strike system when breaking the rules on site. After the first strike, they would call your manager. After the second, they would call your boss. After the third strike, you would be escorted off the premise, and never be working there again. Since we run our company together, we’d figured out quickly that we would be fine as long as we didn’t get three strikes… We agreed on calling up eachother if we had to, and to yell insults loudly if needed for the audience.
So, at 3.30 pm, the security training was over, and we did pass with flying colours. We now were members of this secret society that was allowed on-site. Or so we thought. Still no manager to guide us. It turned out it was his day off. Or at least we think so (we actually never met him, ever). We decided to call it a day, check into the hotel, and to explore the local food and drink instead. That went pretty well.
DAY TWO
Our hotel was a mere ten minute walk from the site, so we had an early breakfast and arrived at the site at 08:00. We didn’t get in, because the guard didn’t have any info about us, and noone he called knew we existed nor that any work on the network was supposed to be done. Once again, we headed for the central office in order to try to find someone that could give us some kind of clue to why we were called there. When we arrived, we were shown into a conference room. After a few minutes a clerk showed up and told us that we need to attend a security training class, and pass the test. We explained that we did that yesterday, showed our certificates of the completed training, and told him that we were networking engineers, called in for emergency work. The clerk glanced on our certificates, and told us that they weren’t needed for the job we were there to do. My collegue and I looked at each other, then looked at the clerk. We had to ask:
“What job are we here for, EXACTLY?”
“You are going to work on the core network, aren’t you?”
“Well, maybe, we didn’t get much info at all before.”
“I will call a guy that can follow you on-site, the security training class you attended is for the construction workers only. You are going to be in the data center, it is not a hard-hat area any more.”
A bit annoyed with all this confusion, we waited for the on-site guy outside the central office. At least the sun was shining, it was a nice day, early summer, warm. The guy arrived, looking like your typical carpenter or builder. He just laughed when we told him about the confusion, and how we spent the last day.
“Well then gentlemen, there is no confusion here. I will take you to the data center”.
We walked to the site, got past the guards, walked over the railway yard to a big building out in nowhere - Just railway tracks all around it.
“So, what are you guys supposed to do here?” he asked.
“We don’t know yet, exactly. I guess we’ll have to call someone when we have checked the DC out. We are supposed to do some work on the network core” I replied.
“Network?” he said and gave us an odd look.
We continued to walk up to the DC in silence. He unlocked the door and let us in.
Inside there was complete darkness. He lit his flashlight.
“Well, you see…” he started.
“The building is done, all equipment is racked and connected, but the building was setup with temporary electrical feeds during the construction. When all building work was done, these feeds were removed. Since the DC will draw a lot of power, it has its own discrete feeds from outside the track area, and they have not been connected yet.”
He moved his flashlight, pointing at the wall. In the wall, there was a niche where four huge 64 amp cables, each around eight inches thick protruded out. The mains weren’t connected…
He gave us a flashlight tour around the building, and showed us the different DC rooms. Everything was there, core chassis were racked, connected, there was cooling units, raised floor, fire extinguisher in the ceiling. Racks upon racks filled with servers, SANs, interconnects, etc. It looked like a brand new data center. Except without any power.
That concluded our first visit to this site. Two days spent, no work done. More to come, since we had to go there a second time.