‘I had been sleeping in the office working nonstop’: Software administrator rescues company's data, only gets $625 and a below-market raise, so he quits

Advertisement
  • 01
    Computer
  • 02
    Font - Posted by u/Raucous Rat 1 day ago n That time I saved the company's data and got $625 for it.
  • 03
    Font - I [posted a few years back] (https://www.reddit.com/r/sys admin/comments/d1ss5w/ad min refuses to upgrade wind ows 7 and server/) and got a ton of great advice that helped me improve my professional behavior, so I thought I'd return with a story from my last few months at that company.
  • 04
    Font - So a year and a half after that post, we got hit with a ransomware attack. I don't know the root cause, but I do know that one of our higher- up employees had their login compromised and that we had an RDP server open to the internet. This was something I brought up to our sysadmin and CFO previously but no changes were made. Anyway,
  • 05
    Font - I was called into work at about 6am to help recover. The IT team for this company was me as DBA and one sysadmin and we had about 100 client PCs. The attack managed to encrypt our ERP databases, the latest backup, and about half of our client PCs. We were able to stop the spread pretty quickly, so I immediately started in on getting the
  • 06
    Font - database backups ready to restore. Oh man, was this ever a "trust but verify" situation. If you've read my previous post, you'll know there's a bit of animosity between me and the sysadmin. It had grown significantly since then when it came to database backups.
  • 07
    Font - About a year before the attack I realized I didn't have a good handle on our backups and didn't like that our backup strategy was just writing backups to a server drive which was then cloned to external hard drives, which were rotated out every day for each day of the week. Testing them was a big pain because I had to get a drive from
  • 08
    Font - another building, come back to the office building, go into his office, ask if I could connect a previous day's drive, connect the drive, mount it to the server, go back to my desk, then perform the test restore before going back into his office, reconnecting the current day's drive, then taking the backup I tested back to the other building. I'm
  • 09
    Font - a bit embarrassed to say that I barely tested these backups. I proposed via email that we start syncing a copy of our database backups to Backblaze. The sysadmin's first response to this was, "is that some kind of Chinese server farm?" I assured him that it wasn't and that I can set up a proof of concept, then he stopped responding.
  • 10
    Font - At this point I was tired of putting in a ton of effort to convince him of things just to then be ignored or shot down based on a whim, so I set up the POC and let it run.
  • 11
    Font - A couple weeks later, a user started complaining about slowness with the ERP. He immediately asked me if I had done the proof of concept, which I told him I did. He tore into me about how he can't do his job when I'm installing crap on his servers. The slowness was completely unrelated to the rsync job I had been running, but it didn't
  • 12
    Font - matter to him in the end. I took it to the CFO, who then brought in the sysadmin. We still couldn't come to an agreement to do this backup. The sysadmin insisted that we would need a vendor to come in to set up new hardware and software and we would end up spending tens of thousands of dollars on this and we just didn't have the
  • 13
    Font - budget for it. The CFO let the sysadmin leave and we discussed the details a bit further. In the end, he told me to just keep doing the backup (it was costing us <$10/mo for 800GB of databases before they were compressed and sent to Backblaze).
  • 14
    Font - Back to the hack. I started going through the external hard drives and found that every single one was unusable. Turns out the backups (a drive copy of the backup drive on the server) were failing for months because the drives were too small. The only backup we had available was the Backblaze ones. I was able to download
  • 15
    Font - the latest ones and had them ready to go within a couple hours, so at this point we just needed to reinstall Windows Server and the ERP application. He was having trouble finding a Windows Server 2008 install, so we had to go with either 2012 or 2016, I don't remember which.
  • 16
    Font - I do remember that we needed to change the drive partition from MBR to GPT because I had to show him how to do it. While we were installing the new OSes, I noticed that he was using licenses he had bought online for cheap. I have no idea if they were legal, but they were coming from the Netherlands which I think is a grey-area as
  • 17
    Font - far as reselling goes. Either uncomfortable with way I was it and I knew this company had the money for proper licenses. I made sure the CFO knew and left it at that.
  • 18
    Font - Fast forward 3 days, I had been sleeping in the office and working nonstop to get the client PCs back up. We didn't have any kind of automated deployment so I was going to each one, doing a fresh Windows install, and connecting it to the domain.
  • 19
    Font - We were finally operational again and we could start shipping orders out. I was feeling pretty smug with my backup and made sure senior leadership knew. A couple weeks later I was given my reward: $625 on my next paycheck and a small raise to only 30% below market rate for my role and experience.
  • 20
    Font - The sysadmin was given $1250 and a bigger raise to nearly double my salary.
  • 21
    Font - 2 months later I was putting in my 2 weeks notice and they countered with a just-below- matching salary and a $20k bonus offer, but only if I complete a list of projects they were looking to get done over the next year. I declined and
  • 22
    Font - have now been happily working at the new company for a couple years and now make nearly double what my top salary was at the previous place. So yeah, that's my story. Thank you for reading! Feel free to comment on or criticize what I did, it helps me understand how I can be better.

Tags

Scroll Down For The Next Article