'The email systems melted down. Everything went offline': Completely clueless employee emails 30,000 people about an apartment, spawning thousands of emails

Advertisement
  • 01
    Watch - 。 ( D "[] decided it'd be fun to hit reply all and in 48 point font reply, 'DON'T TELL ME WHAT TO DO!""
  • 02
    Font - r/talesfromtechsupport Posted by u/sfsdfd 3 32 Company-wide email + 30,000 employees + auto-responders
  • 03
    Font - I witnessed this astounding IT meltdown around 2004 in a large academic organization. An employee decided to send a broad solicitation about her need for a local apartment. She happened to discover and use an all-employees@org.edu type of email address that included everyone. And by "everyone," I mean every employee in a 30,000-employee academic institution. Everyone from the CEO on down received this lady's apartment inquiry.
  • 04
    Font - Of course, this kicked off the usual round of "why am I getting this" and "take me offa list" and "omg everyone stop replying" responses... each reply-all'ed to all-employees@org.edu, so 30,000 new messages. Email started to bog down as a half-million messages apparated into mailboxes.
  • 05
    Font - IT Fail #1: Not necessarily employees@org.edu quite reasonable - but granting unrestricted access to it (rather than configuring the mail server to check the sender and generate one "not the CEO = not authorized" reply). making an all- email address - that's That wasn't the real problem. That incident might've simmered down after people stopped responding.
  • 06
    Font - In a 30k organization, lots of people go on vacay, and some of them (let's say 20) remembered to set their email to auto- respond about their absence. And the auto- responders responded to the same recipients - including all-employees@org.edu. So, every "I don't care about your apartment" message didn't just generate 30,000 copies of itself... it also generated 30,000 * 20 = 600,000 new messages. Even the avalanche of apartment messages became drowned out by the volume of "I'll be gone 't
  • 07
    Font - That also wasn't the real problem, which, again, might have died down all by itself. The REAL problem was that the mail servers were quite diligent. The auto-responders didn't just send one "I'm away" message: they sent an "I'm away" message in response to every incoming message... including the "I'm away" messages of the other auto- responders.
  • 08
    Font - The auto-response avalanche converted the entire mail system into an Agent-Smith-like replication factory of away messages, as auto- responders incessantly informed not just every employee, but also each other, about employee status. The email systems melted down. Everything went offline. A 30k-wide enterprise suddenly had no email, for about 24 hours. That's not the end of the story.
  • 09
    Font - The IT staff busied themselves with mucking out the mailboxes from these millions of messages and deactivating the auto- responders. They brought the email system back online, and their first order of business was to send out an email explaining the cause of the problem, etc. And they addressed the notification email to all-employees@org.edu. IT Fail #2: Before they sent their email message, they had disabled most of the auto- responders - but they missed at least one. More specifically:
  • 10
    Font - adudeguyman It's always entertaining when this happens. I love reading the "godd it people, stop telling people to stop replying to all" emails
  • 11
    Font - trip74 Did she get her apartment though?
  • 12
    Font - TheMormegil92 Something very similar happened to my university last year, too. Though the auto- responders didn't set off each other, fortunately. So it was just mass panic between students. Hilarious, but ultimately mild.
  • 13
    Font - [deleted] We had one of the owners try and send an inappropriate joke to the VP of Sales, a guy named Andy. You know what comes before "Andy" alphabetically? That's right, "All". Worst part was the IT manager simply powered off the Exchange server in an effort to prevent people from seeing the email. It didn't work. And it toasted the information store. That happened a couple weeks before I started, and I'm so glad I missed it.
  • 14
    Font - PolloMagnifico Please... just be smarter than the comp... It warms my heart to know that computers are still too stupid to stop babbling incoherently to one another. I still have a chance of dieing before the robot uprising.
  • 15
    Rectangle - pag NeoPhoenixTE What did you do? Please tell me someone was playing a fiddle in the server room as the server burned down...
  • 16
    Rectangle - Michelanvalo Holy s that must have been a complete fuking disaster. Did any heads roll?
  • 17
    Font - ZA twcsata I don't belong here, but you guys are cool Wow, your email server DDoS'd itself. That is amazing. Glorious. A tale for the ages...and the single biggest disaster I think anyone has ever reported on here. Have an upvote just for the sheer awesomeness.
  • 18
    Font - tsukinon I can see sending a "Hey, you know you're hitting reply all, right?" after the first few. It's pointless, but I get it. Sending "Stop hitting reply all" after about 25 previous "Stop hitting reply to all" is futile. Sending it after 50 is an exercise in insanity. Replying all to tell people to stop relying to all? The only explanation is that the person either a) has the ability to fire everyone or b) has just reached the point where the only relief is to make others suffer more.
  • 19
    Font - David_Trest Ugh. I remember at my first job. Someone had an autoresponder (that basically said due to spam, this email address isn't active anymore, use <newone>) set up, and was getting spammed with a spoofed address that didn't properly understand spoof bouncebacks. What happened was the two kept on replying to each other and building up, impacting our queue until people starting complaining. We shut off that autoresponder (and any others used for a similar purpose) and purged the mail
  • 20
    Font - Isgrimnur We aren't down because we want to be! I was working for the US orange cell phone provider during the 2004-7 timeframe. We had a similar thing happen. If you remember those late-night tv ads for ringtones, to get them removed, we had to submit a form to the company and enter a response e-mail address. One of our geniuses used orange_us @orange.com and kicked off the nonsense. I counted at least 300 replies in my inbox from field workers to call center managers to salespeople. It
  • 21
    Font - Bugisman3 This happened to the cable internet company that was just starting out in Singapore back in the 90s. Some guy figured out that the email that was being used for system announcements were accessible to all subscribers too. Oops. I think the email storm didn't stop for weeks.
  • 22
    Font - 91 againer This will probably never see the light of day. I had this exact same situation happen when I was working at a major financial institution (think several overseas offices and quite possibly one of the biggest banks in the US). They sent out a mass email asking employees who were using a certain software package to remove the package if they weren't using it so they could free up some software licenses. The first email came in " I do not use this software". They hit reply to all.
  • 23
    Font - Email was slowed down to a crawl. You could hear everyone in the company talking about it. Mobile email, and the overseas email delivery started going offline. So me being the smart a diligent employee. Decided it'd be fun to hit reply all and in 48 pt font reply "DON'T TELL ME WHAT TO DO !". I knew when it was delivered, I could hear the laughter hit like a wave through every floor / pod area. My company messenger blew up, people were messaging me from other departments, sending me more
  • 24
    Font - Later that afternoon, my boss pulls me aside. He told me one of the higher ups saw my e-mail and thought I was being insubordinate. My boss at the time had a great sense of humor and gave me a wrap on the knuckles and sent me about my day. Two years later I left the company, but after the email incident, whenever I was introduced in meetings someone would inevitably say. "You're the don't tell me what to do guy!"
  • 25
    Font - bug-hunter We had a similar issue, but the guy called the help desk asking how to send to everyone. Helpdesk guy helpfully tells him how to select first person in address book, and shift click the last one. So it sent to every person and list in the address book. So in addition to >1 MB of headers, since this was in the days of dialup, each site's computers proceeded to choke on the message, causing it to corrupt in the inbox. Took almost a month to fully sort all the remove sites out. We
  • 26
    Font - [deleted] We had a retiring employee do something very similar. They sent this lovely long note about how much of a pleasure it was working for the company and how they will miss everyone, blah blah blah. They added the entire fting GAL (1000 employees) to their "To" line. ...Followed by a bunch of idiots replying to all with "best wishes!" Email went down for half a day. Thankfully it wasn't as bad as OP's story.
  • 27
    Font - o geared4war It happened at my work quite regularly before they figured it out. Didn't effect the auto reply thing but it was still great entertainment. Someone would send one. The replies started "take me off this list", "why am I getting this", GM - "Why is this happening?". Across an entire government transport agency. Over and over for DAYS. people would come back from their day off and instead of reading the full chain and get to the do-not-reply-to- these-frigging-emails message wou
  • 28
    Font - slinky317 What kind of mail server has the 000 replies go to everyone on the chain, and not just the original sender only?
  • 29
    Font - formerlyme0341 All e-mails can be funny. At a place I used to work (~200 employees) a disgruntled employee signed up all@domain.com for a bunch of dating sites and other spammy things. It was hilarious. Of course to stop the emails you had to login. Seemed like the IT department tried to block addresses as fast as they could but more just got signed up. It was hilarious.

Tags

Scroll Down For The Next Article