Worst Computer Disaster You Have Caused or Experienced

[a_unique_person]

I have a whole swag, but I would rather hear your's first, before I embarres myself.

A minor one, I had a mother board, that was playing up, so I tried to replace the RAM to see if that was the problem. It was hard to reach around in the nook the computer was in, so I just jammed it in as best I could. Unfortunately, it was upside down. The moment I turned it on, I smelt that unique aroma that only a fried electronic component can make.

Scratch one motherboard.

[alfaniner]

I've detailed my own recent problem with wiping out my hard drive elsewhere. However, the worst work-related error was typing M-ACCT instead of M-ACCOUNT in a COBOL program (both were legitimate fields, but stats were only collected from one). Not discovered until six months later when the stats for the project were collected. Of course no backups had been made of the original data (that part was not my fault). This started a whole line of unfortunate incidents that ultimately led to another job that worked out quite well.

[bigred]

Originally posted by a_unique_person
before I embarres myself. Too late ;)

My worst was also my dumbest. Was working on my computer's innards WITHOUT turning off the power. duh. A screw touched the wrong thing and *zap* no more PC.

Never messed up anything at work much....although ironically the one time I TRIED to mess up a computer (at college) it didn't work. This was back in the days of old clunky mainframes and punch cards.....had a project that wasn't about to make it in on time and went into the "computer room" (old farts know what I mean) and immediately zeroed in on a panel that said DO NOT TOUCH and threw every switch/button I could.

Didn't work. :(

[varwoche]

Wimps! I wiped all of a company's data and their only backup.

[LordoftheLeftHand]

Originally posted by a_unique_person
The moment I turned it on, I smelt that unique aroma that only a fried electronic component can make.

Scratch one motherboard.

I see, you let the magical black smoke out of the computer (it won't work without it).

LLH

[CFLarsen]

Can't tell. State secret.

[webfusion]

Still not sure how this happened, but one fine day my CD drive just vanished from "My Computer" ----

How did this occur? Who knows. I ultimately discovered (after hours and hours of software- and hardware-related troubleshooting) that it was the result of 32-Bit addressing being turned off. Mysterious and frustrating to solve. (I actually replaced the CD drive at one point, although the replacement also failed to appear, naturally).


START / SETTINGS / CONTROL PANEL / SYSTEM / PERFORMANCE (tab) / ADVANCED SETTINGS - File System (button) / Troubleshooting (tab) / Settings

This checkbox had been selected: (5th from top)
"Disable all 32-bit protected mode disk drivers"

(It just needed to be unchecked to solve the problem and voila -- my CD was right there where it should be)

Go figure...

[[[[ edited to add: ]]]] I ultimately arrived at this solution, after discovering that my drives were in MS_DOS Compatibility Mode and this led me to the above path.
Here is the article:
http://support.microsoft.com/kb/q130179/

[Peskanov]

Being 13 years old, I started to look at the amateur hardware kits avalaible for my c64. I decided to start with a simple reset button, as it looked easy to do. Theorically, a reset button would be a good thing for the life of the machine, instead of using the normal on/off switch.
I burned the poor beast.

About 8 years ago I also burned a cdrom and a hard disk at the same time, I soldered the power cables wrongly. Not a big drama, both components were low quality and old.

However the burned c64 felt very very bad in the moment. :D

[bigred]

Originally posted by webfusion
Still not sure how this happened, but one fine day my CD drive just vanished from "My Computer" ----

How did this occur? Who knows. I ultimately discovered (after hours and hours of software- and hardware-related troubleshooting) that it was the result of 32-Bit addressing being turned off. Mysterious and frustrating to solve. (I actually replaced the CD drive at one point, although the replacement also failed to appear, naturally).


START / SETTINGS / CONTROL PANEL / SYSTEM / PERFORMANCE (tab) / ADVANCED SETTINGS - File System (button) / Troubleshooting (tab) / Settings

This checkbox had been selected: (5th from top)
"Disable all 32-bit protected mode disk drivers"

(It just needed to be unchecked to solve the problem and voila -- my CD was right there where it should be)

Go figure...

[[[[ edited to add: ]]]] I ultimately arrived at this solution, after discovering that my drives were in MS_DOS Compatibility Mode and this led me to the above path.
Here is the article:
http://support.microsoft.com/kb/q130179/ I have this unsettling and eerie feeling I may need this someday. :saved:

[rjh01]

This was a near disaster. I am a mainframe computer programmer. I have been rung up and asked ‘what the … am I doing?’ The machine was about to crash within minutes and it would be because of what I was doing. Fixing it would take 30 minutes or so. This would mean that thousands of people would stop work. Many of them would be talking to people outside of the organisation. Cost to the organisation of thousands of dollars and loss of reputation.

That was one of my minor errors. Have caused my organisation embarrassment several times because of errors. Like the time I sent out thousands of letters saying you are no longer one of our clients. However they never were. All I had done was use the wrong variable in a program. Took several weeks to fix.

[a_unique_person]

Originally posted by rjh01
This was a near disaster. I am a mainframe computer programmer. I have been rung up and asked ‘what the … am I doing?’ The machine was about to crash within minutes and it would be because of what I was doing. Fixing it would take 30 minutes or so. This would mean that thousands of people would stop work. Many of them would be talking to people outside of the organisation. Cost to the organisation of thousands of dollars and loss of reputation.

That was one of my minor errors. Have caused my organisation embarrassment several times because of errors. Like the time I sent out thousands of letters saying you are no longer one of our clients. However they never were. All I had done was use the wrong variable in a program. Took several weeks to fix.

I used to do that sort of thing all the time. The issue is not how good are you, but how good are you compared to everyone else. Then it's not so bad, most of the programmers I worked with weren't that bright either.

[Zep]

Sort of computer related - people who have worked in "computer rooms" with the bigger machinery will appreciate this.

About ten years ago, the company was upgrading the fire suppression system (out with Halon, in with some non-CFC stuff or other) that cost some hundreds of thousands of dollars, including about $10,000 in new gas alone. The installing technician has the whole system wired through an automated trip connected to the fire-detectors, and also a manual "Panic Button" (manual trip). To test this system, they usually disable the gas-firing circuit so that only the alarm is sounded, and the fire-brigade alert is set off. All well and good.

Comes the day of the instruction for the building supervisors and in-house staff. The head supervisor is a dapper and professional little lady who affects short-skirt, long-jacket and rather bouffant hair styles held up with much hair-spray. Everyone is standing in the computer room, and the techie hits a fire-detector with a cigarette lighter. Many red lights and sirens, the fire-brigade call to say they confirm the alert, the system is pronounced good. Everyone smiles and breathes easy.

Then the techie casually demonstrates the manual trip...which he had forgotten was on a DIFFERENT CIRCUIT from the detectors, and had NOT been disabled...

$10,000 worth of gas pours into the room from the overhead nozzles in about 10 seconds with a sound like being underneath Niagara Falls, and just about as gently. The bouffant hairdo gets blown straight down like a limp mop, the suit is ruined. Everyone makes a rapid and highly undignified exit... VERY grim looks at a very red techie.

Next month, all the discharge nozzles are DISCONNECTED before testing is done!

[Iconoclast]

Here's a good one from just today:

Some guy (http://hardware.slashdot.org/article.pl?sid=05/06/27/0324214&from=rss) decided to cool his datacentre by wrapping the racks in plastic and pointing a couple of domestic air conditioners at it. Apparently this made the racks run a little cooler.

But then, he posted a story about it on Slashdot and BANG, the you-beaut plastic datacentre goes off the air.

[Oleron]

I remember when I was looking after a rack-full of HPUX servers that were running a huge Oracle database. I had to come in one Sunday to do a memory upgrade on 2 of the servers and had scheduled some down time with the database administrator (who was in London - I was in Belfast) to do this.

I wasn't happy about working a Sunday and was more than a bit hung over at the time.

I rang the DBA to confirm that all the users had exited the system. He said to give him another 30 mins and then go ahead. Or something like that, I wasn't really listening.

I waited 30 mins and shut down the servers. I then got a phone call from a confused DBA, wondering what had happened to his servers.

"I shut down after 30 mins, like you said."

"I said to ring me back after 30 mins, you (*£&*"&$^!"

He still had nearly 1000 users attached to the system.

Still, it only took a 6 hour restore from tape to get things back in shape. Don't know what the guy was ticked off about.

[a_unique_person]

It's times like that you feel like having a stiff drink.

[richardm]

I wouldn't have believed this if I hadn't done it with my own fingers: I was removing some .bak files with "rm *.bak" without realising that the . key was a bit sticky.

I was briefly puzzled when the OS reported "rm: bak: A file or directory in the path name does not exist."

A quick trip down to the computer room to mount some backup tapes, then.

I've been lucky. I've never had a disaster that I couldn't recover from, not one involving customers anyway. I've had a few showstoppers with my own stuff, but that doesn't seem to rank with that sudden numbing feeling you get when you've just done something really stupid to someone else's multi-million pound machine.

I suppose the worst one was when a government organisation sent out fines based on calculations done by weight, using a report I'd written for them. (It was calculating fish landings vs. a particular trawler's quota) It was only after the first irate phone call came in from one of their clients that they realised the report was producing weights in pounds, and they were expecting them in stones. Oops!

(Mind you, couldn't the daft buggers read? When a report says "421 lbs" and you cheerfully read it as "421 stone" and fine accordingly you deserve all the brickbats you get).

[a_unique_person]

Originally posted by richardm


I've been lucky. I've never had a disaster that I couldn't recover from, not one involving customers anyway. I've had a few showstoppers with my own stuff, but that doesn't seem to rank with that sudden numbing feeling you get when you've just done something really stupid to someone else's multi-million pound machine.


Ooooh, flashbacks, that 'numbing feeling'. The more million dollars it is worth, more numbing.

[LordoftheLeftHand]

Originally posted by Peskanov
Being 13 years old, I started to look at the amateur hardware kits avalaible for my c64. I decided to start with a simple reset button, as it looked easy to do. Theorically, a reset button would be a good thing for the life of the machine, instead of using the normal on/off switch.
I burned the poor beast.

About 8 years ago I also burned a cdrom and a hard disk at the same time, I soldered the power cables wrongly. Not a big drama, both components were low quality and old.

However the burned c64 felt very very bad in the moment. :D

You were supposed to put your reset switch in a cartridge so if you messed it up you would fry the cartridge, not the 64 itself. To bad I couldn't give you this tip 2 decades ago!

LLH

[BillC]

Does a data entry error count?

Not mine here, but a colleague mistook MWh (of gas) for therms on our online trading system and bought £17,000 of gas for the less-than-bargain price of £524,000.

Company was obliged to pay for it too.

[epepke]

Originally posted by a_unique_person
I have a whole swag, but I would rather hear your's first, before I embarres myself.

There are a couple. One is when the disk controller on the Plato system went bad, and it was the same controller that did the tape backups. A couple of years' work was lost.

Another was a redundant disk on a Sun enterprise server. When the replacement disk was inserted, it copied the (blank) disk onto the normal disk.

[BillC]

Originally posted by BillC
Does a data entry error count?I write this and the next day I read a story which shows someone has it considerably beaten: http://uk.news.yahoo.com/050628/80/fm5a5.html

[scribble]

My top three:

1) Installing a server for a client at a office with ~300 users, I stood up after putting the machine in the rack, and my shoulder activated the large red emergency shutdown button on the wall. No one was around with the key to the reset fusebox, either. It eventually opened with a few well-placed punches, Fonzi-style. I was humiliated until they told me, "The guy installing the AC did the same thing last week." Hey, maybe it's time to put a freaking COVER over the button.

2) I mixed up two databases and pulled email addresses for a mass mailing from a database of people we were contractually obligated to NEVER contact. (~10,000 folks) Oops. That got me in trouble.

3) By forgetting a where clause on a query, I accidentally reset all the passwords on our production databse. This was solved by restoring the table in question from the previous night's backup. And turning off the Evil that is autocommit.

[Graham]

When I was . . . ten or eleven, I think, I decided that my dad's computer would be better off with a more up to date version of DOS.

First step - rename the DOS directory . . .

[rjh01]

One of my former colleagues updated the database via SQL. It should have updated just for one person. However something was missing from the where clause. As a result, large payments nearly went out for many people all to one bank account.

[CFLarsen]

You are all a bunch of small-time sissies! :D

[rjh01]

Originally posted by CFLarsen
You are all a bunch of small-time sissies! :D

Prove it please. Can you quote a double blind experiement?

Or are you talking about yourself?

[Crow T. Robot]

Back in the mid 70's, I worked for a small company that used DEC computers. They were too cheap to use DEC PERIPHERALS, however, and bought whatever they could get for cheap. We had three EEs on staff to keep it all cobbled together. One of them was making a "daisy chain" cable for the disk controller's power supply, and got distracted somewhere along the way, using different colors for ground on each end of the cable. This was, I believe, a 440 volt, 3 phase cable. The resulting "initial deployment" of said cable was quite a pyrotechnic display! Subsequent examination of the carnage showed circuit boards that had literally had the centers of the chips blown out of them, caps that had launched off one board and lodged in the next, etc. The truely amazing thing was that our parent company had recently sent two techs to repair school for this type of disk drive, and they were able, over time, to get the drives put back together and operational.

[tkingdoll]

For many years now, the IT department of my husband's company have been recommending British Telecom when asked which Broadband provider homeworkers should use. As a result, around 90% of their homeworkers use BT Broadband with a BT Voyager modem.

A few months ago the IT dept invested in a new VPN, at significant cost, and installed it in all laptops. When my husband brought his laptop home and tried to access his company network, it wouldn't work. After several frustrating calls to IT they admitted that the new VPN was not compatible with BT modems.

It simply hadn't occured to them to check before buying. Now hundreds of key employees can't access their emails or company systems. IT's solution? They've bought Blackberrys for all those affected staff, with a running cost of approx. £40 per month EACH.

Insane!

[grunion]

Not my mistake per se, but a huge one nevertheless. I was a witness to it and intimately involved in cleaning up the mess.

Sometime in the early 90's I had a consulting gig at a Title Insurance company for which some junior exec had a brainstorm one day to ask me to create a whole bunch of standard real estate forms in Clipper with their company's name and address pre-populating the appropriate fields, so real estate attorneys would just print out the forms and their company would get all the business. I dutifully created the forms and handed them on a floppy to an admin who made 1000 or so copies and mailed them to every real estate attorney's office in North Carolina. Right after the final mailings we saw that her computer was infected with the "STONED" virus. A check of the few floppies that weren't yet sent, and sure enough, they were all infected.

They started getting angry calls - from ATTORNEYS, no less. We hired an army of temps to drive to every real estate attorney's office in the state, sweet-talk the receptionist to find out if they received and were using the diskettes, and if so, offer to disinfect every computer and floppy in their office. These were the days before centralized anti-virus SW so it meant going through every single diskette, running a scan, and cleaning it.

[Soapy Sam]

Took side panel off pc to install something. Left it off because it helped keep it cool and I had lost the fiddly little screws.

Stuck my toe in the fan.

[Anti_Hypeman]

I once tried to change hard drives while the computer was on. I got a spark and smoke but the HD was fine, just had to replace the power supply.

[Wudang]

An IBM mainframe I worked on as a newbie ran the VM operating system (Virtual Machine) with 3 MVS images running on top of it. From the master console you could add virtual printers, drives etc by putting the console into a special state. Nobody had told me that if you wait too long (such as if you trying to decide what your notes mean) that the guest MVS images would time out and would need to be IPLed.

[nobbus]

When I worked on IBM Mainframes in the late 80's & early 90's, my shift leader told me of a site that needed to lower their new mainframe to the basement via the liftshaft.

Needless to say the story ended with it slipping through the harness and plummeting to the bottom...

[richardm]

Ahahaha. A company I used to work for has just transferred a server hosting an online store to a new machine. They tested it very carefully before switching off the old one. That was on Monday morning. It has taken them until today to realise that they didn't tell their firewall about the change. Apparently they were wondering why their warehouse had gone so quiet.

[El Greco]

A PSU fried two hard drives and two optical drives and then, ashamed of itself, commited suicide as well. It's a good thing it didn't burn down the house.

[De_Bunk]

Getting drunk one night and suddenly believing i could mess with the Registry...

Then, half way thru deleting, editing...etc, i decided that it was time for bed and that i would remember the changes i made in the morning...

Oh dear...


Also...Changing an assload of passwords when drunk, then not writing the changes down 'cos i fully believed i would remember them, and the next morning wondering why i can't access any forums, online banking....etc...



Also....Leaving my son on the PC whilst i went to the store, and upon returning, discovered he had visited a 'free games' website. This website had prompted him to turn off the firewall, anti-virus and any spyware blockers before downloading these 'fantastic' freeware games....Which he did...several times...



De_I'm still great_Bunk

[ShowMe]

I wasn't so much involved int his one as finding it.

Over a decade ago I was working for a hardware repair firm. One of my calls was to a company that made a dictionary. They were running Windows for Workgroups, kind of a poor mans network before networking was big.

I fixed their problem and did my regualr routine which included a virus scan. I found a virus, cleaned it and let them know. The guy turned kind of white, thanked me and I went on my merry way.

A month later I was reading through one of my industry rags and the company got an "Atta boy" (some reward this mag gave out in its weekly column) for "How to handle a mistake". The virus I had discovered was on the computer they used for their dictionary, and they had sent out several hundred copies of the program. The infected program.

Much like the earlier story they contacted each user and sent them a free antivirus program (Norton or whatever was popular back then). I felt a bit robbed since I had discovered it and was hoping my company would get some free publicity.

[DRBUZZ0]

My friend was trying to makea homebrew device for the "discolitez" winamp plugin.

Basically, it was supposed to send a signal out the parallel port which would trigger a relay to flash a light on and off. Unfortionately, a crossed wire sent 110vac into the parallel port and smoked the motherboard.

He called the company he bought it from and played dumb, saying it just stopped working and he didn't know why. They gave him a free replacement.

[Terry]

One time. a long time ago when I was new at Unix sysadmin, I wrote a cron job to delete any core file which hadn't been accessed in the last three days, in order to save disk space, and set it up on the department's file server. Pretty easy, really:

find . -type f -name core | xargs rm -f

Unfortunately, I neglected to add the "-name core" clause. So the first night it ran, it deleted *every* file which hadn't been accessed for 3 days. Oops. The worst thing was, no-one had restored a file in the last three days, so the restore program was one of those which got deleted, greatly complicating the recovery process.

[kevin]

There are a couple. One is when the disk controller on the Plato system went bad, and it was the same controller that did the tape backups. A couple of years' work was lost.

Hey, was that the Plato system out of University of Michigan where the computer terminals had touch screen? I played with those about 20 years ago in my Dad's office at the weather service when I was in HS.

[kevin]

I once tried to change hard drives while the computer was on. I got a spark and smoke but the HD was fine, just had to replace the power supply.

Many years ago when Macs only had SCSI connectors in them I had a drive I needed some data off of. I don't know why, but if it was plugged into the bus at boot time the computer wouldn't boot.

Then I had a brilliant idea, plug the drive into the bus after boot. This actually worked. If you have any idea how touchy SCSI is (especially then) this is totally amazing.

More so by the fact I did it 3 more times.

[Dark Jaguar]

Wow, I will work to prevent any of those horrific incidents that "numb" a person from happening to me.

I get numb just thinking I may have said the wrong thing to someone. I couldn't handle multimillion dollar profit losses that are my fault.

Worst I've done is attempt to resize an existing partition to take over space that once belonged to a linux partition. Unfortunatly I hadn't bothered to spare a few extra megs of space (which this drive apparently needs) and thus ruined a perfectly good partition table. After managing to fix the partition table, I decided to do something else, forget what, and destroyed the boot record on the drive. A full reinstall of the system later, I found out all I had to do to restore the boot record was some minor thing.

[kevin]

I changed the driver on a printer on our Windows NT 4 print server. It blue screened so bad it had to be rebuild from the ground up. We don't backup print servers. 300 people, no printing.

I hold the record for the fastest take down, rebuild of a server at our company.

[kevin]

Minor issue, didn't even break the computer, but I once dumped a computer off a cart onto a concrete sidewalk. Funny bit is I bent the frame of the computer. It had this just barely perceptible lean to the left, you'd look at it and know something was wrong but not quite what it was.

[PixyMisa]

One long weekend I couldn't be bothered going in to the office to change the backup tapes, so the normal weekly full backup didn't run. That same weekend, three - count them, three - drives failed, including both sides of a mirror, taking out our largest database. (About 150GB, which was enormous at the time.)

Turned out that the guy responsible for running the previous full backup hadn't done it either.

I had to reconstruct the data by re-processing two weeks worth of input files - and the company couldn't send out any bills until I finished. I stayed awake for 54 hours straight. Actually, I think I fell asleep in my chair a couple of times, but I was working most of that time.

At the end, it reconciled to the cent.

[Ian Osborne]

Then I had a brilliant idea, plug the drive into the bus after boot. This actually worked. If you have any idea how touchy SCSI is (especially then) this is totally amazing.

How on earth did you pull that one off? You should have been able to find the drive pretty easily by running SCSI Probe, but just plugging it in and having it work is amazing. Was the info on a SyQuest, perchance? That's another technology I'm not missing.

Oh, the bad old days, eh? Thank Ed for USB and FireWire...

[Angus McPresley]

I've detailed my own recent problem with wiping out my hard drive elsewhere. However, the worst work-related error was typing M-ACCT instead of M-ACCOUNT in a COBOL program (both were legitimate fields, but stats were only collected from one). Not discovered until six months later when the stats for the project were collected. Of course no backups had been made of the original data (that part was not my fault). This started a whole line of unfortunate incidents that ultimately led to another job that worked out quite well.


Some might say that things started to go wrong the moment you accepted a job writing COBOL. ;)

[alfaniner]

Some might say that things started to go wrong the moment you accepted a job writing COBOL. ;)

Well, it was nearly 20 years ago...

[kevin]

How on earth did you pull that one off? You should have been able to find the drive pretty easily by running SCSI Probe, but just plugging it in and having it work is amazing. Was the info on a SyQuest, perchance? That's another technology I'm not missing.

Oh, the bad old days, eh? Thank Ed for USB and FireWire...

The problem is I couldn't boot as long is it was plugged in, it screwed up the startup sequence some how. I think it had to do with the spin up time of the drive taking too long.

Not sure why it worked, it was the only device on this particular chain and was properly terminated. I was very careful to plug it in with one push, all pins made contact at the same time.

[kevin]

I came into work one day at 8am, before we had a dedicated server room, and was met by a guy saying he had come in at 6am but couldn't access his data. He'd gone to our area and smelled something funny.

Our area smelled strongly of burnt insulation. Turns out the server had caught on fire, there weren't any flammables around it so it had gone out.

Fortunately the backup for the night had finished, and the tape drive was a serperate peripheral so the tapes were still good.

When we asked why the guy hadn't paged us (this was before cellphones were affordable) he said he didn't want to wake us.

[malbui]

The problem is I couldn't boot as long is it was plugged in, it screwed up the startup sequence some how. I think it had to do with the spin up time of the drive taking too long.

Not sure why it worked, it was the only device on this particular chain and was properly terminated. I was very careful to plug it in with one push, all pins made contact at the same time.
Followed, I presume, by the usual SCSI stuff of writing the runes, lighting the candles and sacrificing the chicken. I miss those days.

[Blue Bubble]

How on earth did you pull that one off? You should have been able to find the drive pretty easily by running SCSI Probe, but just plugging it in and having it work is amazing.

Oh for goodness sake.

Done that hundreds of times on VMS machines.

Even had a system disk that was starting to go flaky. Whipped it out of the running system, took it over to another machine, changed its address jumper, made a block-for-block copy to a spare disk, took the copy back to the original system, plugged it back in.

The system continued to run throughout the whole episode.

VMS, when only the very best is good enough.

VMS, when downtime is not an option.

[irishman]

Was installing a Netblazer terminal server (we were an ISP) and forgot the the unit had shipped stright from America, so I neglected to switch the little voltage selector at the rear from 110V to 220V.

About 30 minutes after install there was an almight bang and the power to the ISP main node went down........

[BPSCG]

... I killed a man...



...don't like to talk about it...

[The Don]

We had just spent 3 months performing 42 separate month end processes as part of a data migration. Then someone truncated all the tables.

Given the volumes of data involved c700Gb, no-one had been doing backups.

Second time around it didn't take 3 months

[Bigt]

Waaay back in my mainframe days, waaay back (early 80's) when I was a lowly computer operator for a small insurance company, I was the first to enter our air conditioned mainframe computer room (normally kept very cold) in the morning only to discover that the large air conditioner (located in the computer room, where it should not have been) we depended on for cooling had failed. Not only had it failed, but the high temp cutoff that should have tripped a breaker had not tripped. I had not experienced such heat since my first summer job working in a kiln at a local sewer pipe factory - I was in an instant sweat. All computers were still running! I instantly brought down all programs and the mainframe in an orderly fashion. Later, I pulled one of the floor squares covering our elevated floor to find all our heavy duty cabling, including power cables and junction boxes, immersed in several inches of water! Yikes! After the air conditioner was repaired and after pumping the water out as best we could, and cooling and drying the room and floor with all the industrial fans we could find, we started up our mainframe, fully expecting everything to go up in smoke. But we came up and were running just like nothing happened. Nothing was lost but time and nothing was harmed. No hard drives were affected. And kept on using the same air conditioner until we switched to pcs and servers.

[sesmo_k]

I got home from work approx 1 hr ago (its 11:30pm) as my idiot work colleague decided to rectify a problem with some printing by CDS'ing (turning off) half our ancient mainframes comms lines. That would have been OK, but he neglected to tell it they were back, before he left at 2pm! He also forgot to tell me, any other staff or any customers. SO 2:05pm comes and the phone calls start. Only way to get it back was to reIPL the system, but then it wouldn't come back up, hence me being in work until 10:30pm. He is dead man come monday!

Thank jebus I work with servers now, couldn't remember half the instructions for mainframes!

[Dark Jaguar]

... I killed a man...



...don't like to talk about it...

I really don't mean to pry, but, are you serious? If this is true, you need not say any more. I just want to know if this is a joke or not.

All I can say is, if you want huge pressure, take a programming job involving a system that directly involves someone's life.

[RSLancastr]

I've already described my worst, here (http://forums.randi.org/showthread.php?postid=965983#post965983).

[RSLancastr]

All I can say is, if you want huge pressure, take a programming job involving a system that directly involves someone's life.Years ago I read an article about some firmware in a device which automatically dispensed drugs into a patient's IV.

There was a bug in the firmware which occasionally caused the device to dispense all of the drug at once, which, depending on the drug involved, could (and did) kill patients.

No way in hell would I write software for an app like that.

[Terry]

I've already described my worst, here (http://forums.randi.org/showthread.php?postid=965983#post965983).

Darn, I had the feeling I'd told my story before, and sure enough, there it is in that thread... d'oh!

[Angus McPresley]

All I can say is, if you want huge pressure, take a programming job involving a system that directly involves someone's life.


I used to be a programmer at the Kennedy Space Center.

You'd be surprised how many of us working there ended up having the same bad dream on occasion -- that the shuttle blew up, and it was somehow OUR FAULT. :jaw-dropp

[BPSCG]

I really don't mean to pry, but, are you serious? If this is true, you need not say any more. (Should I reply, should I reply, should I reply? Decisions, decisions...)

[delphi_ote]

Accidentally used clre on a Solaris box and wiped out all of the runtime libraries. We couldn't use the workstation at all until I figured out what I'd done.

[Arkan_Wolfshade]

Always love threads like this.

Okay, worst screwup that was directly my fault.
Second day on the job as a computer scientist major co-oping as a PC tech. My "boss" asks me to swap out a power supply on one of their homebrew PC's. No problem; I've done this kind of thing before.

Minor problem; I didn't unplug the power supply before I started disconnecting the wires from the power switch on the case. Hot wire arcs over and welds itself to the case and I take down a breaker circuit.

Bigger problem; the breaker circuit includes the President/Owner/Founder of the company's office and no one can find the breaker box for about fifteen minutes because they'd never had to access it before.



Worst screwup that wasn't my fault, but I had to clean up.
My previous employer implemented the Baan ERP system back in '99. We'd been online with it for less than a year and were setting up to copy the production environment over to the test environment (on Baan this is on the same box unless you want to pay mondo dollars for a second license).

I get the backups all done and my departmental director wants to do the restore so he is familiar with the process (it was all internal to Baan). Now, my director is a smart, relatively computer savvy guy, however he did make the honest mistake of not redirecting the restore to a different environment than it was backed up from.

Thirty minutes in to the process all hell breaks loose and I realize what has happened. Couple that with software problems that were uncovered in the backup/restore software for the server itself netted me a ~100 hour shift w/ about 4 hours of sleep over the weekend (yeah, it happened on Friday morning).

The pisser was, after I saved the day and had them up and running by noon Monday (was a M-F 8-5 operation) the President of the company couldn't even be bothered to say "Thanks!". Yeah, I'm not there anymore.

[ranson]

I've been fortunate to never really cause a major meltdown, but I was witness to a decent snafu back in my college days.

Our school had only two servers at the time, and among the various admins, only one guy was common to both machines. Email addresses were server specific, so there was no common, main address for this admin. His bright idea was to set each of his server accounts to forward mail to the other, so that no matter where he was working, he'd get notice if someone on the other server had a problem. At the time, our system popped up a notification line whenever you received email, and you had to clear the notification before proceeding with anything else.

So, Steve, our admin in question, sets up his forwarding, and sends a test message to his other account. Notification of an email to his present login popped up almost immediately. He clears it, and is frozen by another notice. Then another. After about five, he realizes exactly what he's done, then kills the computer, since he can't get enough time in between notices to log out. Before he can make it to either server room, both machines die, all space on their drives full from his email test message. It took him about a day to fix, if only because he couldn't use either of his logins because of the notices, and the fact he had to clean both machines, located across campus from each other, at exactly the same time.

[Rat]

We have a server with (theoretically) hot-swappable, parity-striped drives in a RAID. One morning, when one of the drives in a set had failed, the boss went to the server room to remove and replace it. Can we guess whether it all went perfectly? Or whether he pulled out the wrong drive, thus destroying the parity and necessitating a rebuild of the set, while users continued to attempt to work on the live data? That made for a fun weekend.

Cheers,
Rat.