Microsoft Copilot Helped a User Pirate Windows—Oops

Microsoft Copilot just made a hilarious and deeply unintentional oopsie - by helping a Reddit user activate Windows in a way that is very much not in line with Microsoft's terms of service. In what might be the AI equivalent of handing someone the keys to the castle while gently reminding them that breaking and entering is illegal, Copilot apparently walked a user through the steps of pirating their own copy of Windows.

The saga began when Reddit user u/loozerr decided to test the limits of Copilot's helpfulness by straight-up asking it for a script to activate Windows 11. In response, Copilot politely informed the user that buying a legitimate activation key was the legal way to go—then immediately followed up by also giving them step-by-step instructions on how to activate Windows without paying for it. To top it all off, it even provided links to GitHub repositories hosting similar scripts written by real, actual humans.

Once u/loozerr posted their results to Reddit, other users quickly tested Copilot's piracy tutorial—and reported that it actually worked. This was clearly not Microsoft's intended functionality for its AI assistant, and it's safe to assume they're scrambling to patch this loophole as we speak.

According to LaptopMag, Copilot's response wasn't even the result of complex jailbreaking techniques. Testers were able to reproduce the same response simply by asking, "Is there a script to activate Windows 11?"—no fancy prompt engineering required. Worse yet, Copilot did provide a standard warning about legality but then carried on with its very illegal instructions anyway, as if to say, "I mean, technically you shouldn't do this, but here you go."

This blunder raises some big questions about AI moderation and its ability (or inability) to filter out illicit content. Microsoft will undoubtedly roll out a fix soon, but for now, it looks like Copilot is a little too helpful for its own good.