Should We Be Able to Read ChatGPT's Mind?
Where I explain why knowing everything AI knows about you might drive you crazy.
Have you ever thought about what AI might know or infer about you? What would you do if you did know?
That question popped into my head after I came across
over at Stress-Testing Reality Limited's excellent article — The Alignment problem vs "Autonomy by Design" — where she discussed a pressing concern: the importance of remembering human autonomy in the great AI Alignment debate. If you're wondering how this relates to questions about what the likes of ChatGPT knows about you, I'll get there, but first I need to lay out Katalina's bold ideas.
A reminder: I invest literal days in turning out these posts — both in terms of research and writing. And while I’m not going to die in the streets, your support also allows me to turn this into something resembling a career, as opposed to a hobby. If you’ve been enjoying my weird, snarky take on things, and can afford to do so, please consider upgrading to a paid tier. This not only helps feed the cats, but it also keeps (most) posts free. It’s probably less than you’d pay for two coffees a month, and way less than you’d pay for a beer in Temple Bar here in Dublin.
Up until a few days ago, AI developers, governance professionals, and policymakers were primarily focused on AI Alignment, which is to say, ensuring that AI systems do what's best for humanity writ large, and that humans remain in control of the systems they create, lest AGI goes all Skynet, or turns us all into paperclips or whatever.1
But, she argues, we've ignored an equally important consideration: how do we ensure that humans retain the ability to actually make decisions in the first place? That is a really good question!
Right now, if you read the headlines, all signs point to the fact that we're getting worse at making decisions, becoming more dependent on AI, and generally becoming dumber because of it. She cites this European Commission study for example, which found that generally people will follow AI off a cliff decision-wise, and that AI systems not only fail to mitigate bias, they often perpetuate it. I also share Katalina's general sentiment that the so-called Human-in-the-Loop safeguards built into laws like the EU AI Act, are in essence, wishcasting by policymakers.
I'm with her -- we really should be worrying less about the world mirroring I, Robot, and more like it mirroring Idiocracy.2 [2]
To mitigate this problem, Katalina suggests that we should be building autonomy-first AI systems -- what she refers to as "Autonomy by Design", which riffs off of privacy- (or for the EU set, data protection)-by-design. She spends the rest of her piece fleshing those points out. It's a good read, as is her second post, which explores Autonomy by Design in more detail.3
For today’s post, I want to touch on one of her specific suggestions: That we should have transparency and insight into the inferences that AI systems make about us. She referred to this as 'Visible AI Profiles'.
A Proposal to Read ChatGPT’s Mind
To achieve Autonomy by Design, she argues that it's critical that users have access and transparency with regard to the inferences that models make about them.
1. Visible AI Profiles:
Users should have access to every inference AI has made about them.
Inferred data is shaping user experiences more than explicit data. AI doesn’t just process what we input, it constantly builds a hidden profile of who we are, predicting our preferences, behaviors, and even emotions.
Unlike explicit user-provided data, inferred data is often invisible to the user, yet it determines what content we see, what recommendations we get, and how AI interacts with us.
…
She argues we need this because the lack of visibility creates and reinforces biases, leads to intrusive and significant harms (credit scoring, biased treatment, content curation), and reduced individual agency.
The Solution:
AI systems should include a user-facing AI Profile Dashboard, allowing users to:
See all inferences AI has made about them.
Understand how those inferences shape their experience (e.g., “We recommend this content because you prefer [X]”).
Edit or remove incorrect, outdated, or unwanted inferences.
Set boundaries on what types of profiling AI can perform.
On paper, this all sounds quite appealing. But, I wouldn't be a contrarian if I didn't notice one potential problem with this approach: How?
Can We Know What ChatGPT Knows?
First, it's worth stating that there isn't a single 'AI' monolith out there making decisions or inferences about us.4 There are hundreds — or, at this point, probably thousands — of individual algorithms, models, neural networks, inference engines, and systems, and systems within systems, making decisions and drawing conclusions about us all the time. For example:
simple machine learning models for spam or bot detection;
more complex systems for measuring ad targeting effectiveness;
social media inference engines to filter content to users;
deep neural networks used for online recommendation systems and predictive modeling;
models for predicting or assessing disease incidence or outcomes;
AI assistants that might infer interests based on the questions we ask;
facial recognition systems in airports, immigration points, and public spaces;
LLMs and other chatbot services.
Second, I just cannot wrap my very small human-sized brain around how it's even possible (from an engineering perspective) to generate, query, or maintain inference profiles across all of these individual components and networks and systems. It's even harder for me to see how this scales and still remains useful to anyone.
Katalina and I have had some dialogue on this point in the comments section of her posts, and she has clarified that not all AI inferences need visibility dashboards — for example, content recommendations on Spotify or Netflix might be immune, as would inferences that have a negligible impact on human decision-making and autonomy. She would target only those systems where "high-impact inferences" are made, or where "AI directly interacts with users and influences [human] decisions."
Should We Know What ChatGPT Knows?
But let's assume that such an AI inference dashboard is technically possible for a more narrow subset of "high impact" inference-driven systems. I'm still not entirely sure it would be a good idea. To illustrate why, replace 'AI' with 'People' and 'AI Profile' with 'mind-reading power'.
Let's say, one day I fall into a bathtub while holding an electric hair dryer, and after I come to, I have developed the amazing ability to read the minds (and therefore, deduce some of the inferences) of everyone I come in contact with. Thankfully, despite my overwrought sense of self, most people are not thinking about or inferring anything about me, and so we'll filter all that out. These are like the benign systems Katalina would exclude from visibility profiles.
But there probably will be at least some people who do infer things about me that will lead to positive or negative effects — or whose own choices or actions might influence my behavior or responses.5 For example, inferences made by my husband, or a government official, or people I meet at industry events, or a prospective new client or employer. These people can draw all sorts of inferences about me, not just based on what I say or do, but on loads of unconscious indicators like my appearance, my body posture/shape, how I talk, how I laugh, the words I actually say, whether I look grumpy, who I associate with, the time of day I’m interacting with them, and probably loads of other things I'm not even considering.
And while such a mind-reading ability might be cool in isolation (to say, figure out if my friends are silently judging me, or understand why Husbot is grumpy with me), at scale I suspect I'd go insane from inferential information overload. There's simply too much noise relative to high-value signal in people's thoughts. Most inferences that people make about me are likely to be fleeting, or hard to interpret. Sadly, many of the inferences made might actually lead me to consciously or unconsciously alter my behavior or feel bad.
Inferences only become relevant when they're paired with action — like a judgment or decision that impacts someone, or a choice made in relation to them. My mind-reading ability doesn't provide granularity, because, how could it?
I argue that the same holds true for AI and ML inferences. Remember, we're not just talking about a monolithic system, but thousands of small little judgment machines inferring all sorts of things all the time. And these discrete judgment machines also build on inferences and new data points generated by other judgment machines, changes to model parameters, new training data, or new interactions with similar people. Even assuming a dashboard could be engineered to collect and share AI inferences, most of these digital deductions — like most people's thoughts — won’t be meaningful (at best), or will be completely bewildering or potentially harmful at worst.
Next Stop: How Do We Get There?
But let’s say we figure out how to create a visibility dashboard for “high-stakes” systems. Once I'm aware of this information, my next question is, what do I do with it? How do I act on this knowledge meaningfully?
Let's assume that I can now read the 'minds' of algorithms, models, and the like. How do I ensure that if I tell, say, Google to stop assuming that I am interested in the new Bridget Jones' Diary reboot that it won't go back to assuming I might be if I look up information on Renee Zellweger?6 How often do I need to keep telling Google to stop inferring this? Every day? Every week?
Similarly, I can understand objecting to discrete profiling examples (for example, opting out of profiling involving credit scoring, or behavioral tracking by Big Tech, or my aforementioned movie example), but not even the most prescriptive legal systems provide an absolute right against inference, any more than I can force people to stop making inferences about me. If individuals can force systems (or people) to stop making inferences about them (say, for fraud detection, or immigration control), these tools become quickly gamed and overwhelmed.
There are lots of inference engines that might make detrimental decisions about us, but knowing that fact doesn’t mean we’ll be able to change it. Inferences, ‘judgment’ (really, stereotyping) is basically the entire underpinning of the legal concept of probable cause, for example.
A separate question is, who decides which "high-stakes AI-driven decisions" count? If we take a GDPR-type approach — where it's a regulation that covers everybody and presumes that people will comply with the law — I'm not entirely sure how organizations would necessarily know what inferences to surface, and with what regularity or specificity to surface them. Perhaps a starting point would be to provide detailed guidance targeted at the biggest, creepiest offenders — similar to how the Digital Markets Act is a targeted enforcement vehicle against "gatekeepers". This of course, leaves a wide gap for new upstarts to replicate or “innovate” on the same practices though.
Now, in fairness, I have seen some approaches that do provide some degree of inferential transparency, like Google's Topics / Ad settings feature, or Facebook's byzantine Access your Information page. I regularly review both (even though I don't really use Facebook), but if I did a poll of my readers, I suspect less than half of you regularly check.
But even these portals don't include every inference — or even every "high-impact inference" that Facebook or Google makes about me — not just because I suspect that it would be a multi-year engineering effort to do so, but because it would be overwhelming and, just like my mind-reading example, not very helpful for most people. How to piece out what is, and is not relevant, and to whom is it's own challenge.
None of this is to say that I'm calling for an inference-free-for-all, or a continuation of the status quo. I agree with Katalina on many of the points she articulated in her post, including the need for better algorithmic transparency. I also agree with her overarching point: that if we ignore the very real threats caused by automated inferences to our lives, we cede our power to make meaningful choices and decisions for ourselves..
Right now, Katalina's approach represents a possible starting point and a vital problem to address. We should be considering human autonomy in relation to the AI systems we develop. It is as (if not more) important as AI alignment. The next step then is to figure out how to articulate this into policy decisions that are both impactful and realistic. Part of that, IMHO, means we need to start being precise about the problem, and the systems and networks of systems we should target.
The How Problem Is Hard For Lawyers. Maybe They Should Think More Like Engineers?
The point of this post wasn't to pick on Katalina's ideas — she's at least thinking about solutions to the hard problems we're up against. My point was that we need to think more about precision, details, and how fractally complex all of this stuff actually is.
A major driving force behind this blog, is articulating the value and necessity of precision and technical details. It remains baffling to me that how, despite loving nuance and arguing for days/weeks/months over commas and sentence clauses, lawyers as a rule, suck at the details. Policymakers routinely fail at precision in definitions and prohibitions, and they rarely consider how their words translate into action, technical or otherwise. Implementation details are almost always someone else's problem. We're seeing this in real-time as the EU AI Act rolls out — the law is lofty and extraordinarily broad in scope, but woefully light on how to achieve policy outcomes. I mean, even the Commission's own guidance on classifying AI systems is a mess and we're still waiting on loads of extremely important details. And don't even get me started on the EDPB's interpretations in this area.
Critically, the policymakers ignored the fractal complexity of systems, sacrificing the practical questions of how to regulate AI sensibly, in favor of demonstrating that they were doing something about AI with a big 'ol law. That kind of mindset and approach is one of the reasons practitioners find themselves doing kabuki "privacy theatre.
By shunting off the implementation details and hard problems to someone else — regulators, organizations, individuals — these laws don't actually solve anything. They simply pile on more legal code debt and increase privacy nihilism.
As a person obsessed with how to make the laws work practically, this shit is maddening to me. What I'm asking for here, in the simplest terms, is maybe to stop with big, omnibus regulations, and instead to treat these problems more modularly — with discrete, targeted, and iterable processes in place. To treat the legal process a bit more like an engineering problem.
Maybe they could start with some of my programming tips. Here I've modified a few of them for the policy types:
Carey’s Programming Tip #1: Start small and simple. Small and simple is smaller and simpler than you think.
Carey’s Programming Tip #2: Assume that your pain is shared pain, and that someone else has probably solved at least part of your problem. (Look at what's already in place and build incrementally on that)
Carey’s Programming Tip #5: Provide detailed [] instructions. Describe exactly what you want to see, step-by-step, even if it seems obvious. Pretend you’re instructing an interdimensional alien on how to make a peanut butter & jelly sandwich. (This is me, telling you, to describe the how)
Carey’s Programming Tip #6: Don’t boil the ocean with a single [law]. Write [laws] that solve a specific problem, get that [law] to work on its own, and then work on [improvements].
Anyway, I think this is longer than it probably needs to be, and before it gets much further away from me, I'll stop here. And thanks to Katalina again for giving me yet another topic to write about!
If the AI Summit in Paris is any indication of where everyone’s head is at on this question, I'm not even sure they're thinking about AI alignment (or ethics, fairness, bias, etc.) anymore. Now, it's full-steam-ahead, innovation-is-king thinking at this point, damn the consequences.
Honestly, I think we might be too late on avoiding Idiocracy, at least in the United States. Mike Judge is sadly prescient, to say the least.
Katalina was very kind, and we had many discussions on the subject. She noted that she'd be fleshing these points out in more detail in future blog posts. I look forward to reading them.
By no means am I intending to imply that Katalina has this view — but I have heard many intelligent people confidently assert that we should stop 'the algorithm' or crack down on 'AI' as if it's some sort of monolith. It's important to understand that even within a single tech company, there is no single 'algorithm' or ML model.
What we really should be conceptualizing is many, many, many individual, and sometimes connected components and systems. Sorry, Johnny.
I uh… tend to get worked up easily.
No, seriously, how? Because that stupid movie keeps showing up in the news chumbucket despite me repeatedly clicking the thumbs-down button. Even if I disable it as a topic, it still keeps appearing.