Dear Tech Companies: Please Stop Stepping on Your Own Dicks
Forcing users to decipher what you do with their data is a terrible idea.
This isn’t exactly a privacy disaster, as such. Instead, I’m going to discuss one of the catalysts that lead to privacy disasters (and in turn, give me ample fodder for blog posts). Tech companies keep stepping on their own dicks and then being surprised when it hurts.
It’s a problem that has grown in volume and velocity, and one that is glaringly obvious to me, but seemingly novel and strange to tech billionaires. The self-dick-stepping problem (or if you want to be more PG about it, the rake-stepping problem), can be caused in a variety of different ways, but most of the time the root cause when it comes to privacy-specific dick-stepping is either:
The company decided to release a product that is unnecessary/unwanted and creepy/invasive; or
The company released a potentially useful product in a way that is opaque, scary, or otherwise demoralizes users.
Take for example, this deeply concerning post that I came across just yesterday on LinkedIn:
As a veteran enthusiast of dystopian tech horrors, I figured I should do a little investigating before freaking out too much. Sometimes, after all, the internet freaks out over nothing. A little sleuthing confirmed that the poster was correct — buried deep in the bowels of Word’s settings (it took me five clicks before I found it) was a bold claim ‘Your Privacy Matters’ — followed by some text and a bunch of features that had me opted-in by default, including the elusive ‘Connected Experiences’ section.
Clicking on the links for these Experiences directed me to two very long pages — ‘Connected Experiences in Office’ and ‘Connected Experiences in Microsoft 365’,1 which mostly explained what Connected Experiences were, and included their own additional links to explore. None of which, of course, touched on the question mentioned by OP of whether Microsoft was using all our Word docs secretly to train Copilot. For my own amusement, I will be referring to Copilot hereinafter as Clippy LLM.
Each of the three dozen (!) Connected Experiences included its own link to further documents, except for two: Print and … Clippy LLM.
But these are just technical documents, I told myself. Surely, all would be made clear in the Terms of Service, Privacy Statement, and the ‘Data, Privacy, and Security for Microsoft 365 Copilot’ documents. Right? Right? Let’s check.
Microsoft Privacy Statement: “As part of our efforts to improve and develop our products, we may use your data to develop and train our AI models.”
Translation: We may use your data for training Clippy LLM.
Microsoft Terms of Use (from February 2022): “Use of Your Content. As part of providing the AI services, Microsoft will process and store your inputs to the service as well as output from the service, for purposes of monitoring for and preventing abusive or harmful uses or outputs of the service."
Translation: Microsoft may store this data, but they probably won’t use it for training Clippy LLM.
Data, Privacy, and Security for Microsoft 365: For organizational clients only, “prompts, responses, and data accessed through Microsoft Graph aren’t used to train foundation LLMs, including those used by Microsoft 365 Copilot”.
Translation: At least for organizations, Microsoft’s API for connecting all its services together won’t harvest your data. Microsoft defo will not be using it to train Clippy LLM.
Searching on Google also took me to loads of documents, and questions on the Microsoft Community pages where people raised the same concerns, with conflicting responses. For example, two different Microsoft Vendors said, yeah, probably, but also no? Meanwhile, this Microsoft Agent said, yeah. But another Microsoft Agent said ‘Nah’.
Finally, one commenter (himself a Microsoft solutions wizard) finally shared the goods that neither I, nor Google could find: A document called ‘Copilot Let’s Talk’, which appears to be a reasonably user-friendly explainer on Clippy LLM and how it integrates with the rest of the Microsoft Borg. The ‘What data is excluded from model training’ and ‘Does AI training apply to Copilot Pro…’ excludes “Users logged in with M365 personal or family subscriptions” those under the age of 18, commercial users, Microsoft 365 consumer users integrated within Microsoft 365 consumer apps (like Word, Excel, PowerPoint & Outlook), or users in certain countries (the EEA, Brazil, China, Israel, Nigeria, and a handful of small islands). 2
Guys: This is Not How You Do Things
All of this took a few hours of me researching, reading, and puzzling over documents, and many helpful comments on LinkedIn to piece together. And I’m fortunate in that I have a reasonably large and broad audience of readers on LinkedIn. And I’m extremely fortunate that a very patient Microsoft specialist came across my post and shared his insights. However, the same does not hold true for the vast majority of Microsoft users, who probably do not want to be involuntarily training Clippy LLM just because they use Word.
It’s no wonder that of course, hiding crucial details from your customers leads said customers to freak the fuck out, share their concerns on social media, which then spread like a pernicious STI around the internet, only to cause further reputational harm, incite my rage, and occasionally, lead to regulatory or market backlash.
Then the tech company is forced to backpedal, roll the technology back, explain itself to angry journalists/regulators/privacy folks like me, and promise to do better. Except they never do because tech companies …
Keep Stepping on Their Own Dicks
What’s so annoying, is that this is both predictable and avoidable. It’s predictable because companies keep creating solutions in search of a problem, foisting those solutions on their users (hello, Recall!), failing to treat their users like human beings, and then getting all Pikachu-faced when that approach lands poorly. But it’s also avoidable: They can learn from their previous mistakes and stop doing this.
In essence, tech companies keep stepping on their own dicks and then wondering why it hurts so much. The techbros decide, for seemingly arbitrary or opaque reasons, to release a potentially invasive product, or do something creepy with data, and then being very, very surprised when it backfires spectacularly.
The company justifies its repeated dick-stepping behaviors by telling itself that this time, the product will be cutting-edge, or create immense shareholder value, or worse, will guarantee or justify someone’s personal promotion up the corporate ladder.
But they don’t do any sort of sanity-check. They don’t actually investigate whether there’s a real product fit here, or whether their new software solves a legitimate problem or business need. They don’t rely on sources of organizational wisdom & perspective (like other product groups, legal & privacy teams, UX colleagues, or people representing diverse stakeholders). They don’t think through how to release the product in a way that empowers, rather than demoralizes users. They never talk to me.
Instead, it’s tunnel vision to the finish line. It’s the tech equivalent of running around blind, barefoot, and naked in a rake factory, and then wondering why rakes keep smacking them in the face.
Honestly, many of the Connected Experiences features look rather cool, and some look like they might even be useful. Lots of these will/do legitimately improve my Word experience. But Microsoft just … turned everything on by default, didn’t tell anyone what they did, and buried the choice modals deep in the bowels of the program. It shouldn’t take a user five clicks, hours of research, or a helpful Microsoft commenter, to turn something off or figure out what it does. Most critically, Microsoft was inconsistent about what Connected Experiences actually did, how these experiences worked, or whether they were sharing user data with Clippy LLM.
This was a prime example of Microsoft stepping on their own dicks, and then wondering why they keep getting yelled at on the internet and by regulators. But then again, maybe it’s not sufficiently painful enough for them to care. I mean, they haven’t reached US-government-levels of distrust, so…
To sum things up: Microsoft (and Google/Amazon/LinkedIn/Meta/Twitter): Nobody wants your shitty always-on AI/LLM tools, and if they do, the better approach is to show them the benefits and why your product is cool, rather than opting everyone in by default and then making it painfully hard to opt out. Stop treating your users like surprise rakes in a rake factory. Stop stepping on your own dicks.
Thank you for coming to my TED talk.
I still, to this day, have no idea where Microsoft Office ends and Microsoft 365 begins. Microsoft really does seem to refer to them interchangeably in product materials. I’m sure there is a distinction, but it really shouldn’t be this hard for me as a user to understand what that distinction is.
This is … a lot. You can figure out if you’ve been opted in or not by logging into Microsoft 365 and selecting Account > Privacy > Model Training (if it’s available, otherwise Account > Privacy > Copilot) and opting out.
Is this anatomically/biologically possible/probable? Defies the law of physics? Good read indeed! Kudos to Alexander Fleming.
Honestly, I can't ❤️❤️❤️ this enough, Carey.
The entire history of tech bros, Microsoft and empires in one very tidy post.
We'll have to talk about the movie rights some day...