Tech employee working 50-hour-weeks makes massive mistake impacting 500 servers: 'Team is melting down... client's gonna bail'

• 
  via Miguelangel Miquelena

  Tech worker stares at computer in horror, realizing he's made an irreversible mistake. 

  _{The image does not depict the actual subjects of the story. Subjects are models.}

• When you work in network security, the security part is kind of a big deal! So this person was upset to discover that they'd made a huge mistake, and they explained what happened, adding that they're seeking advice. 

  They use a lot of technical jargon in their story, so let's just read through it and parse what happened with this whole debacle. 

  They share that they: 

  pushed unified vuln dashboard with live criticals to public github repo. team is melting down

  cannot even process what just happened. we have been grinding for weeks to unify vulnerability data from 12 different security tools into one dashboard.

• 
  via Getty Images

  Two tech workers have a chat in the server room. 

  _{The image does not depict the actual subjects of the story. Subjects are models.}

• They then name a bunch of programs that probably only the techiest of techs among us have heard of: “tenable, qualys, snyk, wiz." Okay. Sounds made up, but I'm sure they know what they're doing.

   They share that: 

  Apis pulling scans, risk scores, everything normalized into single panes so management stops yelling about tool sprawl. Finally got a demo view working Friday. Pulled all the feeds, built the unified queries, even added some fancy risk prioritization graphs. Excited as h*ll so i made a repo to share with the team over weekend.

  Sounds good so far, right? 

  They seemingly did what the boss wanted, and created a demo version with all the data input. 

  But then… disaster struck, just as the weekend began. While this person had their 48 hours of free time, private information became public information. Oh. No. 

• 
  via Getty Images

  Tech workers holding a serious meeting to discuss how to handle the situation. 

  _{The image does not depict the actual subjects of the story. Subjects are models.}

• Forgot to init as private. Pushed to my work GitHub account which is public by default because I use it for side scripts. Commit message was literally 'unified vuln view with prod feeds live check this out team'.

  So what they're saying here is that they used an account that is their personal work account, but they left it public for their own ease of access. And apparently, the message when you open this thing was that it was for the team to check out the live results. 

• Monday morning, Slack explodes. external vuln scanner picks up our repo, indexes it, and now our entire high med crit list from prod environment is scraped and showing in public searches. 

  It does not look good for this worker. 

  They explain that at the crack of Monday morning, their Slack messages are filling up because a vulnerability scanner discovered their mistake. The scanner indexed all of the sensitive information on there, which, as they explain, includes: 

  customer names, asset tags, cvss scores for unpatched stuff across 500 servers. 

  This is like… a very big mistake. 

  The tattletale vulnerability scanner told on them!

  One of our biggest clients' assets right there with 'immediate exploit' tags. Heart stopped when I saw it trending in some threat intel feed.

• 
  via Getty Images

  Boss questions team about what to do to secure their servers.

  _{The image does not depict the actual subjects of the story. Subjects are models.}

• They then explain that in their panicked state, they tried to take action. But it was already a bit too late (like, a whole weekend too late, unfortunately). Even after they deleted it, they realized that other people had already scraped the information, as had Google itself. The internet is forever. 

  Now. The fallout. 

  Team lead is furious, ciso [Chief Information Security Officer] looping in legal, clients getting calls.

  They add that they spent an entire morning doing all that they could to pull back the information, which essentially ruined the whole dashboard they worked on. They're now in a state of blaming themself for the whole thing, wondering how they could've made such a mistake. But perhaps their employer is to blame, because they've been working 50 hour weeks. 

  If you're working 10 hour days, you're bound to be pushing yourself to the brink. You have less time to sleep, less time to work out and eat healthy meals, less time to let your brain unravel as you watch TV with your partner or read a good book. And a lot of employers have their workers operating this way all the time. 

• 
  via Vitaly Gariev

  Employee working long hours sleeps at his desk.

  _{The image does not depict the actual subjects of the story. Subjects are models.}

• Here's what's going on now, as the company manages the immense fallout: 

  Still recovering data feeds without breaking prod scans again. anyone been through this kind of exposure. how bad is the fallout usually. clients gonna bail. need advice on disclosure or cleaning this up before it hits news. please tell me someone has a worse story or fix.

  Well, the post made its way to X, and it has numerous comments on Reddit, so safe to say it has breached containment. And with a story this big, affecting 500 servers, I mean… maybe this person should start looking for new work before the news of it all gets too big? Again, not to blame them, but they may want to just look forward instead of sitting around in this giant mess they made.

• 
  via Getty Images

  Workers brainstorm solutions to their tech problems

  _{The image does not depict the actual subjects of the story. Subjects are models.}

• A lot of people were shocked by the story… 

  
  Impressive-Toe-42

  I hope this isn’t real, for your sake. If it is, the last thing I would do is advertise it on Reddit and invite more people to come and find it.

   

  Theloneus-punk

  You need to take a break from the internet and read a book about operational security

   

  BigOrangeSky2

  Might want to publicly publish your resume next

• This person can't just wait around at their job and be known as “the guy who let 500 servers' worth of client information leak out.” There's probably going to be new rules created because of this guy. It's not looking good!

  Best of luck to this person, and let's all hope our data wasn't leaked out due to this breach. 

Tags