'I was like okay': Tech employee asked not to push meeting against his advice, costs the company thousands of dollars

Advertisement
  • 01
    Font - Want me to stop postponing meetings? Ok S OC So I work in a tech company, usually we have a software deployment every other Thursday. The team usually has a long meeting on Friday (2-3 hours) after deployment. However, usually we have some minor issues after deployment and I have to do a lot of monitoring and fixing, so I usually ask the team to push the meeting to Monday so we can stabilise the system first.
  • 02
    Font - A few weeks ago in the meeting, they pointed out how I'm always postponing the meeting, and we never have it in its set date which is Friday. I mentioned that usually the system isn't stable on Friday and I have to fix it. But they all agreed that we must stick to the schedule. I was like "okay".
  • 03
    Font - After 2 weeks, I attended the meeting on its scheduled time on Friday. It went on for like 3 hours. When we came out, there were hundreds of emails and tickets from the client, the servers were down for hours right at peak usage time. Our clients were and had lost tens of thousands of dollars during that time.
  • 04
    Font - The thing is, it was already the end of my workday, so my boss had to pay me a hefty amount for working on weekends, and twice the days in leave as a replacement.
  • 05
    Rectangle - unkiepunkie This is why should happen at the beginning of the week... 4 hr. ago deployments
  • 06
    Font - Moneia 4 hr. ago Or the meeting should be on a Monday when any issues have been spotted\tracked\work ed on.
  • 07
    Font - RedditAdminAreMorons. 4 hr. ago So why wasn't it ever suggested to permanently move the meeting to a Monday after everything has been tested, implemented, and found?
  • 08
    Font - Sinaneos OP. 4 hr. ago Because Friday is kind of a chill day, after deployment. So they wanna have the meeting then, and utilize Monday's hours for more work
  • 09
    Font - FunnySheep 4 hr. ago This is a good story, but it sounds to me your team is doing IT in an absolute terrible manner. That level of manual involvement is beyond absurd, there are huge problems.
  • 10
    Font - Sinaneos OP. 4 hr. ago We're still working on our processes. There are tests and all, but we still have cases where someone pushes something and it crashes the servers due to conflict with production data.
  • 11
    Font - flyingemberKC. 4 hr. ago edited 4 hr. ago You've failed to recognize that your boss is trying to get a bad deployment process fixed by making it a lot more expensive to roll out software that breaks everything for hours. He wants the system down, customers demanding refunds and you working overtime. Having a Thursday deployment should not be broken the next day. That's the sign of a horribly bad SDLC process.
  • 12
    Rectangle - argybargyargh. 4 hr. ago Mandatory long meeting the day after a deployment? Yeah, that outcome is predictable.
  • 13
    Rectangle - Ready-Strategy-863. 4 hr. ago Why do you have bugs to fix after the deployment?
  • 14
    Font - Sinaneos OP 4 hr. ago Bugs that show up in production that were missed on staging, or because of a very niche edge case.
  • 15
    Font - barjam 3 hr. ago. edited 3 hr. ago Every single software company I have ever worked at had a strict "no deploy on Friday" policy. Also if your deployments are anything other than completely uneventful, you are doing something very, very wrong. Our company does weekly deployments across multiple countries with thousands of clients. Deployments are automated and very rarely does anything go wrong.
  • 16
    Font - RealKenshino. 3 hr. ago Your tech company only has you on call? The server was down for hours just because you weren't around? Processes or not, deployment on Fridays or not. How is that a sustainable design? I cannot believe your clients hire you guys - it'd not pass any sensible audit with any of mine
  • 17
    Font - Sinaneos OP. 2 hr. ago Well it's a small company, and the resources are very scarce. I'm not a DevOps guy, but I had to work on it because the system was a mess and nobody was willing to do anything about it. I've asked a lot to hire or train someone to do DevOps to add some redundancy for this part. But they kinda want to cheap out because hiring an experienced DevOps engineer is not cheap.

Tags

Scroll Down For The Next Article