Manufacturing engineer gets blamed for product failures, complies with orders to shut down key tests he warned against, and calmly watches failure rates skyrocket: ‘This was a terrible idea, so I argued against it’

Author image for Etai Eshet
Etai Eshet • Editor
Jul 23, 2025 6:00 PM EDT

Manufacturing engineers occupy that unique corporate position where they understand both theory and reality, making them dangerous witnesses when management decides to ignore physics in favor of wishful thinking and interdepartmental finger-pointing becomes more important than actual problem-solving.

This particular battle started when design engineers decided that a 20% failure rate must obviously trace back to the testing station, because blaming the new guy requires less paperwork than admitting design flaws. Their solution involved the corporate equivalent of closing your eyes during a horror movie: if you can't see the problem happening, it must not be happening.

Advertisement
  • Recently stumbled across this subreddit and remembered a story I thought you guys might want to hear. Unfortunately, my industry is kind of specific, so I will have to change some details and make some things vague to remain anonymous - but the core of the story is all there.
    via subwaysmoothie

  • You want me to disable half of my entire testing stage? Okay!

    "I fought this as much as I could for two weeks, before the order finally came down from my direct manager"
    via halfpoint

    The image does not depict the actual subjects of the story. Subjects are models.

  •  Never mind that the testing station mimicked real-world conditions, that the relay couldn't physically fail under those circumstances, or that other products used identical components without issue. Logic bows to hierarchy, and junior engineers learn quickly that datasheets and evidence carry less weight than senior assumptions and departmental politics.


     

  • Design engineering makes bonehead decision to force me to remove a critical half of the testing procedure for one of the products we build. That decision has wide-reaching effects and causes a different product to experience a 100% failure rate.
    via subwaysmoothie
  • This forces design engineering into firefighting mode for months trying to determine the cause. Years ago I worked as a junior manufacturing engineer for a certain company building certain, relatively complex products, and one of the stations | was responsible for was the first test station.
    via subwaysmoothie
  • We got the core mechanism and the electronic assembly of the products right off the production line, and I performed white-box testing to ensure everything looked right and worked properly before sending it to a different station for assembly and black-box testing.
    via subwaysmoothie
  • I'm trying to avoid talking about the specific testing methodologies used, hence "white box" and "black box" because that's the best way I can describe it without being more specific. White box refers to testing it with direct access to all the internal components, so you can measure all the different parts, verify that individual parts work correctly.
    via subwaysmoothie
  • Black box refers to testing it after the entire thing has already been put into an enclosure and you no longer have access to the internals, so really, you're just verifying that the thing works and does everything it's designed to do.
    via subwaysmoothie
  • Cheezburger Image 10533690112
    via anontae

    The image does not depict the actual subjects of the story. Subjects are models.

  • The product I'll talk about today - let's give it a codename of "azure" - belongs to the greater family of "blue" products, which were all very similar but may have had different configurations, slightly different parts, etc.
    via subwaysmoothie
  • "Azure" was a new product, and in the first few production runs, we saw failure rates of 20%+ at black-box testing (the station after mine). When the engineers dug into it, they found that one of the relays on the electronic control board was fused shut on those 20% of units that failed.
    via subwaysmoothie
  • For whatever reason, they turned to me (they *love* blaming my station) and said that my station was causing it. For a bit more background, my white-box testing station had powered-off and powered-on testing.
    via subwaysmoothie
  • Cheezburger Image 10533698048
    via halfpoint

    The image does not depict the actual subjects of the story. Subjects are models.

  • In powered-on testing, we turned newly built units on for the first time, which also meant that if anything was wrong with it that would make it fail when powered on, it would fail at my station. That's why I was always careful to make sure that the initial powered-off testing was thorough enough to cover as many of my bases as possible.
    via subwaysmoothie
  • So when I powered it on for the first time, the unit wouldn't blow up. The design engineering team apparently didn't believe that. They were convinced that when I powered on "azure" units for the first time at my station, the initial power surge was sending a big current spike through the relays.
    via subwaysmoothie
  • This was causing them to fail. Their proposed solution was to simply eliminate powered- on testing during white box testing. This was a *terrible* idea, so I argued against it:
    via subwaysmoothie
  • 1. The power supply in my test jig is set to be as closely matched as possible to the actual power supplies we send out with these units into the field. That means if there was a power surge that was causing the failures, it's a design issue and would be occurring with units out in the field if I didn't power it on during white-box.
    via subwaysmoothie
  • Design engineering team said that, no, it must be an issue with my tester, because they didn't believe there could be anything wrong with their design. 2. I pulled up the datasheet for that relay and showed that it was physically impossible for that relay to fuse, in the circuit configuration it was placed in, with the amount of voltage my test jig could supply.
    via subwaysmoothie
  • Apparently, the design engineers ignored that entire page of my report - they didn't think a junior manufacturing engineer's analysis was even worth looking at, and trusted their own assumptions more. 3. I had yet to see a single unit that was proven to have a good relay before my station and a fused one after my station.
    via subwaysmoothie
  • That would have been the concrete evidence I needed to believe that my station was fusing the relays. Design engineering said, "we don't need concrete evidence, we're sure this is what's happening". 4. If we disable powered-on testing, we'll lose a *lot* of test coverage.
    via subwaysmoothie
  • The design engineers just went, "whatever, we'll catch any issues at black-box". (This was a bad idea because our black-box tester, while it could tell us that the unit wasn't working, could not tell us *what part in the assembly* was causing it not to work.
    via subwaysmoothie
  • Units that failed white-box had a >70% successful repair rate; units that failed black-box had <10%, at least without going back through white-box.) 5. Finally, I argued that we had other products in the "blue" family that went through the exact same test jig, using the exact same relay in the exact same circuit configuration, and hadn't seen any issues before.
    via subwaysmoothie
  • That last argument I made was a big mistake. What I said was, "We have other 'blue' family products going through that same jig with no problems." What the design engineers apparently heard was, "We have other 'blue' family products going through that same jig, and they're all killing relays, and subwaysmoothie hasn't noticed yet because he's incompetent".
    via subwaysmoothie
  • They came back at me twice as hard. I fought this as much as I could for two weeks, before the order finally came down from my direct manager: as per directives from the design engineering team, all powered-on testing was to be dis ed from the test jig for *all "blue" family products*.
    via subwaysmoothie
  • Not just "azure". (For what it's worth, my manager was on my side for most of this, and only gave me the order to avoid any unnecessary trouble when it looked like the company leadership was going to get involved.)
    via subwaysmoothie
  • ed Well, fine. I went ahead and dis powered-on testing. As I predicted, all of the "blue" family products - "cyan", "turquoise", "cerulean", etc - started seeing 3x the failure rate at black box testing and we were now stuck with a bunch of units that we didn't know how to fix.
    via subwaysmoothie
  • But that's besides the point - how about "azure"? Same 20% failure rate. Nothing changed. Just as I said, my station wasn't killing the relays.
    via subwaysmoothie
  • So the design engineers went and took another three months figuring out what the actual cause of all of the relay failures was, which, as it turns out, was some flaw with the way the black box test was being run combined with some other part on the assembly that was underspec.
    via subwaysmoothie
  • I dunno specifics, I wasn't part of this conversation anymore. They spent a bunch of money and got it fixed, and never followed up with me saying "hey, looks like. you were right, it wasn't caused by powered-on testing at white box" - which, crucially, also means that I never got a directive to *re-enable powered on testing*.
    via subwaysmoothie
  • So we ran like that for a few months, me licking my lips all the time, because I knew what was coming and it was delicious. See, there was another product in the "blue" family that I'll call "navy".
    via subwaysmoothie
  • "Navy" was a bit of an oddball, because the client had some requirement that demanded microcontroller B be installed, as opposed to microcontroller A on all of the other "blue" family products. That was the only difference, which meant I used the same test jig for it.
    via subwaysmoothie
  • We sourced microcontroller A from a vendor who also pre-loaded it with the firmware we needed flashed on it. Years ago, we had also apparently done the same with microcontroller B. But the vendor for B that could preprogram them for us had shut down.
    via subwaysmoothie
  • We could not find a single other vendor who could preload the firmware for us. That's when we turned to internal solutions. Someone found out that the test jig at my station (then managed by someone else) had direct access to the microcontroller's programming interface.
    via subwaysmoothie
  • They developed a way to flash the firmware from my test jig. That meant we could now buy blank B units directly from the manufacturer, then flash it with the firmware ourselves. This was a great solution because blank B units were cheaper, flashing it during powered-on testing wouldn't add an extra step to our production process.
    via subwaysmoothie
  • It would just be a part of the white box testing step. Of course, flashing the firmware required the unit to be powered on. All of this happened years before I had joined the company, and before most of the current crop of design engineers were involved with this project.
    via subwaysmoothie

  • And so, his compliance unfolded beautifully: disable powered testing, watch failure rates triple across the entire product line, observe the original problem persist unchanged, then sit back while design engineers spent months chasing their own tails. The real masterpiece emerged when a different product required firmware flashing during the now-forbidden powered testing phase, creating a 100% failure rate that sent the entire department into firefighting mode. Months of investigation, dozens of theories, and zero solutions later, someone finally asked the right person the right question.

  • This was, in fact, documented, but all of these products had gone through hundreds of ECNs (basically formal engineering notifications that "something" has changed with the product) and nobody was reading through hundreds of them to familiarize themselves with the entire history of the product.
    via subwaysmoothie
  • When they demanded I disable powered-on testing on *all* "blue" family products, this microcontroller programming step for "navy" was also dis ed, meaning any "navy" units we built would make their way over to black box testing with a blank microcontroller.
    via subwaysmoothie
  • I knew, but that was only because I knew exactly what my test did. I also knew that this was documented in an ECN from 8 years ago. I was probably the only person involved that knew, and I didn't say a word.
    via subwaysmoothie
  • After several months of not building "navy", a new production run finally starts. The newly built units pass my white box, powered- off testing. They then made their way over to black box testing, and...boom, **100% failure rate.**
    via subwaysmoothie
  • What's more, this failure was a particularly tricky one, because as far as the black box tester was concerned, the unit *could not even turn on*. The reason was that the microcontroller in question held the firmware that controlled power delivery for the entire electronic assembly.
    via subwaysmoothie
  • Without that firmware, nothing worked. No lights, no output, no fans, no nothing. Which means they had a h I of a time trying to figure out what the issue was. That entire department went into firefighting mode, with everyone losing their minds over why this product we had produced for nearly a decade with high yields was suddenly failing at a rate of 100%.
    via subwaysmoothie
  • (Not me - I was happily running production for other products.) It dragged on for months, with the design engineers pursuing a dozen different leads and all of them fizzling out.
    via subwaysmoothie
  • Surprisingly, nobody ever approached me, because I guess their theory for everything was "white box powered-on testing kills units" and now that they had already asked me to disable powered-on testing, they had no other theories for how I might have been affecting things.
    via subwaysmoothie
  • As far as they were aware, there was no problem with white-box testing. I just sat back and pretended I wasn't even aware of what was going on while everything was melting down around me.
    via subwaysmoothie
  • I decided that the moment someone asked me about it, I'd reveal everything, so several months down the line when someone finally brought it up to me during casual conversation, I spilled it:
    via subwaysmoothie
  • Them: "Hey, did you hear about all the Navy units not turning on at black box?" Me: "Oh, no, I didn't. Could it be because we're no longer programming the microcontroller at white box?"
    via subwaysmoothie
  • Them: "What?" Me: "What?" Within a day, I was called into a meeting with *everyone* - my direct manager, his manager, a few of the other manufacturing engineers, quite a few program managers, the design engineering team, even a VP - and told to explain myself.
    via subwaysmoothie
  • Me: "Well, I was told in no uncertain terms to disable powered-on testing a few months ago, and microcontroller programming is part of that process. I assumed that when the call was made, everyone was aware of the implications of taking such a drastic measure.
    via subwaysmoothie
  • I figured you had found a new vendor to pre-program the microcontroller Bs or something." Design engineers: "You never told us!" Me: "Yes, but I couldn't describe *all* of the hundreds of potential new failure modes skipping powered on testing - would now introduce it would have taken me all week.
    via subwaysmoothie
  • The fact that white box programs the microcontroller during powered-on testing is documented in ECN #2082. Didn't you read that?" I got off scot-free from that meeting. However, this then led to the VP (who was a former engineer, by the way) investigating *why* the design engineers had called for powered-on testing to be dis ed, which revealed that:
    via subwaysmoothie
  • 1. They had ignored my well-founded, technically sound opinion, despite the fact that I was supposed to be considered the subject matter expert on white box testing, 2. Disabling powered-on testing *did not solve the issue*, 3. Once powered-on testing was dis ed, the failures at black box tripled, leaving us with a bunch of defective units that we didn't know how to fix, and
    via subwaysmoothie
  • 4. Once it became clear that the cause was not powered-on testing, they did not follow up and ask that it be re-enabled. (I got a bit of flak for this part too, but in the end the VP agreed with my viewpoint that if I dis ed powered-on testing and then heard nothing from the design engineers, I could assume the problem was resolved and had no need to follow up.)
    via subwaysmoothie
  • In the end, nobody got fired or anything, but a few members of the design engineering team did receive a reprimand for their behavior during this entire event. For the entire time I was with that company, they tiptoed around me and never falsely blamed me for any issues again.
    via subwaysmoothie

  • The reveal delivered that rare corporate moment when documentation trumps authority and physics defeats politics. Sometimes the best malicious compliance involves simply following orders exactly as written, then watching reality reassert itself with compound interest on every ignored warning.

Want More? Follow Us and Add Us as a Preferred Source on Google.

Tags

management boss workplace-stories hr manager bosses work PTO managers supervisor workplace Horrible Bosses work-life balance
Scroll Down For The Next Article