Few things are scarier than 4Chan. But our readers told a few stories that spooked us.
Paul van der Werf
Earlier this week, we asked readers to share their most frightening tales of technology terror and support horror. And via both comments and Twitter (using the hashtag #ITTalesofTerror), in poured stories that raised goosebumps from those of us who have worked in IT at one point or another.
After reading through them, we’ve picked out some reader favorites and a few of our own. Some of us at Ars were inspired to recount further tales of horror from our own IT careers—including one of mine that I’ve saved for last; it should cause a shudder of recognition from our more veteran readers and a bit of schadenfreude from those too young to remember five-and-a-quarter-inch floppy disks.
The chamber of horrors
Danger: Intel inside. CohesionMany readers had short tales of terror about mishaps in the closed spaces where we hide our network infrastructure. Eli Jacobowitz (@creepdr on Twitter) shared a short, shocking scenario by tweet: “Raccoons in the network closet (not kidding).” David Mohundro shared another story of a somewhat more smelly infrastructure invasion that brings new meaning to “data scrubbing”: “I saw our IT guys lugging shop vacs through the lower parking deck one day. There was a sewage backup into the server room.”
Another scary server room story via Twitter comes from Peter S. Kastner: “Alone Halloween night 1967 night shift at Cornell's datacenter. Water deluge thru roof on mainframe in storm. Pull plug!”
A somewhat less traumatic tale of almost-disaster in the server room comes from roadflare16:
It was my first week at my first IT job. A few coworkers got up all of a sudden and quickly started heading for the server room. I asked what was going on, and their response was "there's a fire in the server room, come help!" When we got there, the halon fire suppression system hadn't gone off, and we couldn't smell smoke, so we carefully opened the door to look inside. Nothing. Everything was working fine. In fact, almost the exact opposite happened. After checking the situation out, the main A/C unit had failed, which kicked on the backup A/C unit, which hadn't been tested in months. It had frozen over in the short time it was alive, and when the ice buildup began to melt it flooded the false floor where a smoke detector was, shorted it out, prompting the system to think there was a fire. Interesting first week.
But what has to be the most rattling tale of data center disasters comes from tiltedj—a story of a disaster recovery test that turned into a disaster itself:
Our first air-cooled Cray system pulled air from under the floor with a single giant squirrel-cage fan in the bottom of each rack and pushed it up through all the boards and out the top. This of course required installing a couple dozen new Liebert air handlers to push enough cold air under the floor. The raised floor was self-supporting, essentially just a thin metal framework that required most of the tiles to be in place to remain structurally intact. One day we had a COOP exercise and tested the "big red button" on the wall to EPO the facility. The Cray shut off immediately, including all of its fans. The Lieberts were never wired in and did not. We heard the screeching as the fans spun down, mixed with a rumbling as without anywhere for all that air to go it popped out all the floor tiles instead. Then there was the horrible metal crunching as the entire floor collapsed and dropped all the racks.
If you prefer your server room stories with a supernatural twist, however, Arsian bushrat0011899 has a tale that wins hands-down—an unexpected visitor to the server room claiming demonic forces were at work:
Not anywhere enough moons ago, a child came into our office with a broken netbook. Simple enough problem, he just needed to connect to the guest Wi-Fi, done in seconds. Our guest network is a filtered tunnel straight to the WAN, no local peer connections allowed. He came back the next day, he failed trying to roll his own netbook back from Windows 8 to 7, he had this half Windows 8 half 7 abomination that could only run half of the system tools, it was a miracle it didn't blue-screen-of-death the instant it was turned on. I refreshed Windows 8 and he was on his way again.
The next day we notice a sudden, near complete drop in network performance, LAN and WAN, ping times in the seconds, downlink in the Kb range, completely down. The server was fine, the APs were fine, the switches too, nothing was faulty but we gave a restart to everything we could just in case.
After some investigation I found the source of our lost performance, our main backbone link to our core router was unplugged and we were relying on a tiny backup link for emergencies.
I walk down the hall to the core, unlock the door and see this child. I still have no idea how he got in there. He had disconnected our gigabit link and plugged it into his netbook. On this $100 netbook were 8 individual bitcoin miners, all hashing away, he looked up at me crying, saying the devil had possessed his netbook and that if he didn't get a faster internet connection he'd take this kid's soul.
After performing an exorcism on his netbook to disable remote access and purge it of malware it started acting normally. He left and I never saw him again.
I started telling this story to the other support guys, none of them believed that this kid could get in our server room, it can't be locked from the inside either. So I went to pull the security camera footage.
All the cameras were down for the 3 days he was there.
The mystery machine
Enlarge / Best not look too closely at that hardware, citizen. For those of you who need more evidence to prove that someone really is out to get us all, there’s this story from Pehr:
I once was involved in the decommissioning of a few server halls in what had been a NOC. It was mostly mundane stuff; disconnecting, removing and throwing away equipment nobody cared about. There was just one odd thing in the server hall. In a corner stood a truly ancient server. It was home built, with a wooden chassis of all things, and connected directly to a core router using an Ethernet-to-Token-Ring bridge. Unlike the rest of the equipment in the room it had no serial number, and therefore no known owner.
So, one late Friday evening I simply unplugged that machine, turned it off and put it into the pile of trash. After all, we were supposed to clear everything out of the server hall a few days later, and the core router was going down soon anyway. After that I drove home.
At 3 am in the morning somebody started banging and ringing on the door to my apartment. As I lived in a "lively" neighborhood I didn't think too much about it. I did think more about it about 30 minutes later when the police had fetched a locksmith and greeted me in my own bedroom, hands on guns.
The policemen told me, in no uncertain terms, that they KNEW I had disconnected some equipment, and they wanted that equipment back online right now. I was informed that answering "No" to the request would result in a trip to jail for an indefinite time.
So, I ended up going back to work at 4 AM, digging through a container to find that ancient computer and plugging it in again. The police thanked me, told me that I could remove it in a week and gave me a lift back home. They claimed not to know anything about the server, only that they had been told by somebody "higher up" to get it up and running again as soon as possible.
To this day I do not know what the server did, or how they figured out I had disconnected it within a few hours. Perhaps it was part of an important operation. Perhaps it contained the porn collection of the police chief. Nobody else in the office knew anything of the server. A few days later I found it had been turned off remotely, and when I decommissioned the server I took the hard disk out. In youthful curiosity I tried to see what it contained. I found that the hard disk had been wiped.
Another mystery—although one not so sinister—was uncovered at what seemed to be a haunted bank branch. But the case was solved by Arsel, as he explains below:
I worked for a company that provided security for banks and government agencies. In the banks, our alarm system piggybacked its communication onto the house ATM modem: an industrial Racal-Milgo unit that was about 18x12x4".
Every weekday between 1 and 2pm the modem would go offline, taking down the ATM and our alarm system. Somewhere around 3:30 it would magically come back up. The bank's IT people checked into it, even swapped out the modem, but the problem kept occurring.
[I] got the telco involved to confirm the lines were okay, and everything checked out - there was nothing we could put a finger on. So one of our people went in just to see he might find something the alarm could be doing that was being overlooked.
Not being familiar with the branch, he asked the manager if she knew where the ATM modem was. She said she wasn't sure what he was asking for, so he described it as a beige box about so big (holding his hands to roughly show its size). "Oh!" she replied, "You mean the doughnut warmer! It's over here in the break room."
Thoroughly confused, our tech went to check it out and, sure enough, there was the modem... with a large box of doughnuts sitting on top blocking all its ventilation holes. Seems that in the morning when they came in, they'd plop the box down on top of it to keep the doughnuts warm. Made for nice late morning and lunchtime snacks, but by early afternoon the poor little thing went into thermal shutdown from lack of air.
After the branch closed at 2:00 they'd do some house cleaning, throw away the empty box and the little puppy would start to breathe again and come back up about an hour or so later. After a rather protracted "You've got to be kidding me" moment, the cause of their daily comm failures was politely explained and the problem never recurred.
Those demon electrons
Enlarge / Please pay attention to polarity. John Electricity—both its presence and absence—were a common theme in many reader horror stories. Some involved stupid tech tricks that ended in hardware-scorching disaster, while others were more like ghosts in the machine.
Longtime Arsian azazel1024 recounted three stories of electronic mayhem:
Back in the day when RAM came on chips, instead of on DIMMs, my brother was installing an upgrade to his; I want to say 386 16MHz (it may have been an early 486, I don't recall exactly). I think it was from 256 to 512KB of RAM. The conversation...
"Hmmm, I wonder if this have a "right way" to go in. I mean, they fit both directions..." 10 seconds later and the power button is flipped. The cloud of smoke that immediately started emanating out of the case told us that, yes, they had an insertion polarity that needed to be followed (after the fact, we noticed the tiny white dot on each, helpfully on the UNDERSIDE of the RAM chips). Amazingly, after much coughing, choking and yanking out of power cords...it still worked once we swapped the chips around.
Next one, more smoke. Around about 2000 helping a friend upgrade an old machine (Pentium 166, or actually might have been a K6-166) with a new SCSI HDD in it. Again, polarity is queen. No keying on the cable, inserted it wrong, powered it up...all was fine for a few seconds, and then cloud of smoke coming out of the case. Yanked the power, one of the wires melted clean through the sheathing on the SCSI cable. Reversed it, continued to work (despite the wire being bare. We just wrapped the whole thing in electrical tape so that the bare wire couldn't contact any metal in the case).
Final hijinks. A friend was working on an old CRT monitor that had ceased working. It had just been plugged in. I warned him that before monkeying around in the thing, he might want to wait a half an hour or so for any capacitors to discharge. He thought I was being silly. His screwdriver crossed a capacitor. The resulting current discharged caused his muscles to contract so hard he leapt backwards (involuntarily) 5ft in to a concrete wall and then lay on the ground for about a minute before I could get anything coherent out of him.
I'll admit about 45 of those 60 seconds I spent doubled up with laughter. Surprisingly we are still friends.
Another hair-raising tale of fun with capacitors comes from blackbearnh:
Way back in my youth, when dinosaurs ruled the IT industry, I worked at LISP Machine Incorporated. One month we got a batch of power filtering capacitors delivered, in which some of them had been incorrectly labeled with positive and negative grounds. Those suckers could store quite a dollop of electrons in them before they decided to explosively destruct, and for a while it was a fun trick to play on the techs smoke-testing boards to walk up behind them quietly and clap your hands together hard. My, how they would jump! One day I walked into the testing area just in time to hear a loud crack and see a flaming capacitor land at my feet from a machine a good 100’ away.
But the most supercharged story of electrical terror comes from flyingq:
Years ago I was 'watching' whilst a techie was commissioning the cabinets for a large Air Traffic search radar and their associated computer controllers in a military air station in the UK. It was a tense day as 'things were not behaving as they should' so we started opening cabinets and testing contacts and a whole heap of voltage readings.
Bear in mind that these cabinets work at some fearsomely high voltages, we never the less took appropriate precautions.
Comes the point. Techie X was seen flying backwards in a graceful arc to land in a snotty heap on the other side of the concrete block room, all the while there were a few 'Large Blue Sparks' coming from his body and a distinct burnt ozone/flesh/component smell. He was alive and very dazed when he came round in hospital later that day and has a livid scar running in and out of his skin from his right hand down to his foot. The screwdriver he was holding ended up embedded in the wall on the other side of the room where he THREW it.
On testing afterwards we discovered that the voltage between the 3 phases of the supply was absolutely spot on and the voltage to neutral was also good but the cabinet chassis was for some reason hovering around 115,000 volts to ground. Not good.
We traced all the phase lines back and all connections were good, as were all the neutral lines and so was the earth strapping. Puzzling.
Eventually we traced all lines back to the substation and transformer that had been installed specifically for this new equipment. Curiouser and curiouser - all was correct but still neutral was floating at 115,000 volts.
For simplicity you can think of neutral as just another earth but grounded at the transformer not at the building. All the neutral lines come back to a 'star point' and then connected via a metal webbing strap to a big stake in the ground. Someone had forgotten to link said 'star point' to said stake 10 inches away with said strap, so the whole system was floating at national grid voltages.
The computers controlling all this equipment never blipped even once and this particular search radar has been in use ever since.
Hazardous duty
While electricity can be hair-raising, there are plenty of other causes for fear in the IT workplace. Many of the scares we get come from the people we deal with—users, contractors, co-workers, and our bosses. And sometimes we are the most dangerous person to ourselves of all. AspirantFool tells of a simple mistake he made when he made an IT house call at a mental hospital:
The place is mostly okay, but there are armed guards on the metal detector at the front door, so to make things easier on myself, I always leave my phone and wallet locked in my glove box when I go in.
On the third floor, there's a big fire door that's never open with a big red sign on it: "AWOL Risk". I've just gotten called to third floor, and it turns out it's in that area, so I press the little buzzer at the door, and tell the nurse on duty that I'm with IT and I'm here to work on Dr. So-and-so's computer. She lets me in, and while I wade through a crowd of people that make me feel like I just walked into my great grandmother's nursing home, only younger and angrier, I notice that she's sitting in a glass-on-three-sides fishbowl in the middle of a common area, and give her a polite wave.
I get the doc's office at the end of one of the halls with a lot of unsettling looks, but otherwise without incident, and he's got a whole bunch of issues. I spend some time teaching him how to use his new office application, and then he goes off to an appointment. I finish up after about another half hour and start heading out. The same zombies are milling about in the common area, but the first nurse's shift must've ended, because there's a new person in the fishbowl. I foolishly walk to the door and try it, but it's locked - of course, "AWOL Risk". So I head over the fishbowl and ask the nurse to buzz me out, to which she replies, "I can't let you out, I don't know who you are."
Remember how I'd saved time and worry by leaving my phone and wallet in my car? Apparently they're pretty serious about the whole AWOL Risk thing. I'd only been on the job for a couple months, and, thanks to the handy address book stored on the company cell phone, hadn't bothered to learn any of my bosses' or coworkers' numbers. This facility, though it houses computers connected the university network, is in a state-run building filled mostly with state employees, so of course there's no university directory or similar available. I explained that I'd left my ID in my car, and asked if perhaps one of the guards could escort me while I went to get it, but nope, Nurse Ratched kept me locked in there for an hour until Dr. So-and-so came back from his appointment and vouched for me. On the plus side, I learned all about how the government worked to kill the steam engine with the advent of gasoline, and how UFOs don't come from outer space, they're actually launched from bases deep in the ocean run by "the Chinese or the Russians or whoever else is down there"
Sometimes, the worst threat to your network is the guy hired to be in charge of securing it, as Korpo tells here in this story of a Halloween 10 years ago:
I was working desktop support at a DOE facility. Some "Cyber Security Specialist" drone suddenly discovered that we had a major security problem, in that it showed the last username on the login screen to all our Win2k and XP desktops. “That’s half of our security,” he proclaimed. So at 2PM on Halloween, desktop support received an email indicating we had an emergency, mandatory, SMS (Microsoft System Management Server) push coming tonight and to alert the users to leave their desktops on. We did.
What got pushed was the entire HKLMSOFTWAREMicrosoftWindows NTCurrentVersionWinlogon key and all its values that the "Cyber Security Specialist" had just exported off his desktop XP machine with the "don't show last user" value turned on. Not just the one value, but the entire key and its many, many entries. The SMS folks were directed to push out that patch exactly as provided to every single desktop.
The XP desktops had major problems. The 2k machines, since they'd just been passed a bunch of values they didn't understand, came up to a blank wallpaper screen with no login box. Every. Single. One. More than 1000. None would get onto the network, none were usable.
We had to pull the HDD out of every one of those 2k machines, do the remote registry edit thing, fix a bunch of entries, and mount them back in the old boxes. Plus fix anything with the XP machines that may have broken.
The invisible contractor
Lee Hutchinson dug up a tale of a disappearing hired gun from a post he shared on Ars in 2002. It’s a classic, and worth retelling here:
One of the guys in the core sysadmin group left recently for greener pastures, and his replacement arrived for work on Monday. Most of the team is made up of contractors from $GIGANTIC_IT_CONTRACTING_AGENCY, so the core server team manager put in a request for another contractor with the correct skillset--in this particular case, we needed a level 20 HP-UX wizard. Said wizard was delivered on Monday.
I arrived like normal on Monday morning and saw that the new guy had arrived and was settling into the appropriate cube. His fellow contractors were clustered in with him, doing the standard Monday morning pre-work talking and helping him settle in. I amble over and shake hands.
The guy's hand is wet. Clammy, slightly cold, and damp. He's smiling, and appears fine, but wet hands. I get introduced, along with the inevitable pr0n joke ("...so if you need some disk space to hide your pr0n, he's the guy to ask! H4W H4W!"), everybody laughs, I tell the guy we're happy to have him and that we have plenty of work for him, and I head to my office.
Two hours later, I emerge from my first round of meetings and realize I can't remember the new guy's name. I'm starting to feel bad about it, because I'm going to be dealing with him later that day, as we've got a Superdome that needs configuring and I have to show one of my minions how to do SAN allocation with volume set addressing, because HP-UX rides the short bus to the data center. I get up and head out to see the guy again, frantically trying to remember his name, and decide I need help. My office is the last one in the row, so I stick my head into my neighbor's office.
"Dude, what the hell is the FNG's name? I can't remember."
Coworker looks up at me, his brows drawn down in puzzlement. "Why?"
"Because...I was going to see...if...he...why are you looking at me like that?" As I'm talking, coworker's expression changes to bemusement. He looks as if someone is standing behind me with a Nerf Cannon, but I can see my reflection in the window and no one's behind me. Coworker shakes his head.
"Guy split. Left like an hour ago."
"What, like...to go downstairs?" I am slow on the uptake this morning.
"No, like, he quit."
"But he'd only been here a half-hour!"
Coworker nodded. Apparently, upon arrival new guy had made a bunch of jokes to his fellow contractors about how working at a big company was going to be easy street, and that most of the stuff on his resume was fake and that he'd cruised through the interview to land this job so he could "just relax and get paid that government money". His fellow contractors had violently disabused him of the notion that the job was a blow off by laying out his starting project list, and then inviting him to the morning's non-optional project planning meeting to talk about him jumping into the backlog of HP-UX admin work that's been piling up.
At some point, the new guy excused himself as if he were going to the bathroom, left the secure area, and...vanished. The core server team manager got a call shortly after that from the contract company account manager, consisting of the account manager blubbering saying that the new guy had just called him up and quit over the phone. Account manager then made repeated apologies and promised to produce another, more better HP-UX admin at some point in the immediate future.
I've never had this happen before and am still kind of blown away by it. I suppose he thought he was walking into some kind of Dilbertesque fantasyland, where we all sit around and browse the web and maybe occasionally staple a document. Admittedly, working aerospace and DoD contracts, that's sometimes true, and we have our share of Dilbert moments. But I can't imagine what kind of idiot would actually try to pull that for real. Or, rather, I guess I CAN imagine it, since I've met him. Is this common? Anyone else have this happen to them?
TL;DR -- Guy lies on his resume, lands a contract job at a giant company, assumes it'll be a BS Dilbert-style blow off, then freaks out and blows when he finds out he's expected to work.
The task of Amontillado
Enlarge / Take note. Sean GallagherLee’s story reminded me of a situation when I was an IT contractor of sorts and wished I could have disappeared. Years ago, I ended up supplementing my teaching assistant income by working as a temp through a large national temp employment agency. When the agency found out about my technical skills, it started sending me out on all sorts of bizarre assignments—most of which were just for a day because some unlucky person had jury duty or a well-timed cold.
But the strangest one of them all was an assignment that brought me to a large office building in downtown Baltimore—the headquarters of a no-name federal IT contractor. The assignment call had said they needed someone to do a “database sort,” which I figured amounted to running a few SQL queries and generating reports.
I had never been more wrong in my life.
I showed up and was led by the company president—a middle-aged woman who asked me about my experience as we walked—past a server room full of AS 400s and other hardware. The company apparently had a contract to do digital indexes of microfiche of various important government documents, and it needed to produce a sorted output for a customer. And I was to be the one to produce that.
She took me to a desk set up in a hallway and handed me a stack of five-and-a-quarter-inch floppy disks.
Apparently, the “database”—of National Labor Relations Board grievance filings from Amtrak employees—was actually a series of document files. WordPerfect 4.2 document files. And they wanted the records alphabetized by employee last name and output to another series of WordPerfect files.
I died inside a little. But I did get about 40 billable hours out of it.
Life is horrible
Many readers wrote in about their personal tech support horrors not at work, but at home—driving three hours to plug in a USB mouse for someone, trying to help parents or grandparents understand the scary world of technology, or being frightened by things done by relatives with technology that risked life and limb.
There are a million stories of horror in the tech world, and we’ve just scraped the surface. Of course, the greatest horrors of all may be completely unspeakable. As David Crowell posted, “I just sit in my cube having my soul sucked out of me.”
So take those moments of horror in your personal tech life as a gift. Share your scary stories with others to teach them, if only by being a cautionary tale of what not to do with IT—but also be thankful that you’ve got something to hold onto during the rest of the long, soul-sucking grind that the corporate work week can be that reminds you of how much better off you are than that guy who put the RAM in backward, opened that phishing email, or renamed every user in the corporate database “JDoe” during a software test.
And have a happy Halloween.