Update: Kevin Beaumont on his Double Pulsar blog, added some very useful additional context on how Recall works at a technical level, and the information security implications of Microsoft’s approach. I discussed many of the problems he identified (non-optionality, exploitability by adversaries like hackers/governments, the fact that there’s no filtering of … anything). What I did not know was that in addition to recording everything you do on your machine as an OCR’d screenshot, it is also writing that text into an easily searchable (and grabbable) SQLite database in the user’s folder.

Guys. I cannot begin to express how absolutely insane and bad that is. I have added Kevin’s observations below as DPIA risks in bold. He also has some very helpful suggestions for how to disable this abject nightmare, so his post is worth reading.

This week, Microsoft graced the world with yet another tech idea that comes straight out of a Black Mirror episode: an always-on, always-recording life-logging tool that takes screenshots of everything you do on your computer. But now with AI to find things!1

Here’s what Microsoft had to say:

Search across time to find the content you need. … With Recall, you have an explorable timeline of your PC’s past. Just describe how you remember it and Recall will retrieve the moment you saw it. Any photo, link, or message can be a fresh point to continue from. As you use your PC, Recall takes snapshots of your screen. Snapshots are taken every five seconds while content on the screen is different from the previous snapshot. Your snapshots are then locally stored and locally analyzed on your PC. Recall’s analysis allows you to search for content, including both images and text, using natural language. Trying to remember the name of the Korean restaurant your friend Alice mentioned? Just ask Recall and it retrieves both text and visual matches for your search, automatically sorted by how closely the results match your search. Recall can even take you back to the exact location of the item you saw.

This is one in a very long list of ideas that indicates two things:

1. Some things are better left as half-baked ‘philosophical’ ideas cooked up in dorm rooms when you’re high with your friends;

2. Tech project teams clearly like reading/watching dystopian sci-fi, but continue to ignore the ‘cautionary tale’ aspects of said genre entirely.

Recall (they really should have just called it Torment Nexus 1), is on by default. Any indicia of it being around is only present if you activate it with a Windows key, or were involved in the initial setup. It records everything that you do on your PC, unless you explicitly either shut the thing off, or manually exclude certain apps or websites.

Thankfully, it does filter private browsing activity and websites, but only if you’re using Edge (or to a lesser extent, a Chromium-based browser — get fucked, Mozilla users!) Users must manually add websites in Recall settings, because nothing exemplifies ‘user choice’ and control like adding to cognitive load in the UX:

But fear not: Even if you managed to disable things and block website monitoring, Microsoft still records things if you accidentally button-smash the ‘launch Recall’ feature. As a Windows user, I have accidentally button-smashed so many things that this strikes me as less a possibility, and more an eventuality.

Snapshots are stored forever, with the text recorded into a plaintext SQLite database in the user’s AppData/CoreAIPlatform folder.2 At least until you run out of disk space, and then it follows a first in, first out process. Data is also encrypted at rest using Bitlocker or Device Encryption, which is better than nothing. You do not need to run as admin to see this file. I cannot even begin to explain again, how bad this is.

There’s no indicator (as far as I can tell) showing users whether Recall has been activated. There’s also no logic built in to prevent storage of things like financial or credential information in a snapshot, sensitive materials like your nudes or private conversations. Though, of course, it being a Microsoft product, there are controls against storing DRMed materials.

So, needless to say, if you sit down to use someone else’s computer to say, check your bank account or whatever, congrats, they now have access to everything. They won’t be able to see that Kindle book you’re reading though. As Kevin notes on his blog, the actual database “compresses well, several days working is around ~90kb. You can exfiltrate several months of documents and key presses in the space of a few seconds with an average broadband connection.”

About the only good thing about this version of the Torment Nexus, compared to a previous version of this that was released with far less fanfare in Windows 10 (h/t to themudman.bsky.social for pointing that out!), is that everything is stored & processed locally instead of being sent to the cloud. At least for now.

Unsurprisingly, regulators immediately noticed, and the ICO, which is the UK’s data protection regulator, announced days after launch that they had initiated an investigation. I’m sure Ireland as Microsoft’s lead regulator, as well as Germany, France, and other countries data protection regulators will soon follow. Update: According to Cianan Brennan over at the Irish Examiner, the DPC is allegedly looking into it.

Alright, so that’s the background. Let’s get to the fun part (for me at least) of the Privacy Disasters series: pointing out the risks that Microsoft should have identified if they’d done any sort of data protection impact assessment (DPIA) before launching this eldritch horror.

The DPIA that Microsoft Should Have Done

First, a little legal reminder of why this needs to happen, as this wasn’t apparently clear to anyone at Microsoft.3 Under Article 35 of the General Data Protection Regulation (GDPR), a controller4 must perform a data protection impact assessment when certain “high risk” processing activities occur. The Irish DPC helpfully provides a list of high-risk processing activities. For purposes of the Recall product, Microsoft probably should have done a DPIA given that the processing its doing is high-risk for one or more of the following reasons:

• Using innovative new technological or organisational solutions (like AI and creepy always-on life-logging tools);

• Processing used for the purpose of systematically monitoring, tracking or observing individuals’ location or behaviour (like what websites they visit, what apps they use, and who they communicate with);

• Processing that concerns vulnerable data subjects (for example, children, folks with limited or impaired capacity, people in abusive situations);

• Undertaking large-scale processing of personal data (for example, law enforcement would love this information to go after baddies, employers to track employees, and domestic abusers to stalk their victims).

The Recall tool also arguably processes personal data on a large-scale for a purpose(s) other than that for which it was initially collected, especially if the tool is on by default and the privacy notice fails to identify every processing activity that Recall can be used for. The large-scale in this case is looking at Recall in aggregate (it’s going to be on by default on all CoPilot+ laptops apparently unless users disable it).5

Now that I’ve got the why DPIA out of the way, let’s look at some of the questions and risks that I’d probing the product team & engineers about. That is, after I got all the swearing out of the way:

• Transparency: How are users being informed of the processing that’s happening here? Is it buried in a 20-page privacy notice that nobody is going to read? What about users on shared machines like family computers, or shared terminals at a public library? Right now it looks very much ‘one and done’, which means Recall is likely to catch a whole lot of users off-guard.

• Consent of users/third parties: Currently, users are only notified when a fresh install of Windows 11 occurs. If someone else installs Windows for you and selects the default options, have you, as the actual user provided informed, meaningful consent? If I’m in an abusive relationship and my abuser cheerfully acts as my tech support and turns this on by default to monitor my search and website activity, what then?

  What about third parties? If I use Recall, anyone I interact with online (on chat, or in a video) has no clue they’re being recorded. While screenshots and video logging exist today, the frequency and likelihood of any specific interaction being recorded and accessed again is tiny. Most people aren’t taking screenshots of every single change at 5-second intervals. That frequency gets much higher when we’re talking about every Windows 11 system. As of 2019, Windows is installed on over 800 million devices, and currently (as of April 2024) represents 30% of the entire OS market share in the world, with Windows 11 making up 26% of that. That’s an awful lot of devices potentially recording everything.

• Proportionality: This leads in nicely to my next concern. Recall is monitoring almost everything a user does on their PC. Every chat. Every website click. Every draft email.

  Our collective expectations of privacy are that most people we chat with aren’t invasively recording our every comment, utterance, or intimate moment. We assume that our early, roughly drafted emails disappear into the ether after we send the final, polished piece. We assume a modicum of control over what we share with others. If we share something or post something, we usually can delete it later. Or we can set a timer to delete messages after a certain period of time (like on Signal or SnapChat). But these assumptions no longer hold with an always-recording and instantly searchable life-logging tool like Recall.
  
  Also, deleting the physical contents being ‘recalled’ does not delete the stored details in the database. That sticks around indefinitely.

• Children: What about kids? Most data protection laws, including the GDPR, have heightened standards for consent in relation to children. And while many kids have learned the value of deleting web browser activity, they may not know that mom & dad turned this thing on and are monitoring everything they do on the family computer, like whether they looked for information on sexual health, bullying, gender identity, spoke to a friend about family abuse, recorded a suggestive selfie, or did other things they probably don’t want the parental units to know about.
  
  Windows is also popular with schools, and I don’t even want to begin to imagine how Recall would be used in that situation.

• Expanding Purposes: Microsoft is effectively installing a keystroke-logging / spyware tool on everyone’s CoPilot+ enabled machine. While Recall currently processes and stores things on-device (including AI processing, yay), what assurances do users have that it will stay that way? What if Microsoft strikes a really neat deal with OpenAI or an advertiser, that shifts processing or storage to the cloud in order to ‘improve user experience’ or ‘provide more targeted ads’?
  
  What if Microsoft is compelled by the US or an adversarial government to monitor criminal behavior, report ‘grooming’ behavior, or target members of a disfavored group, like journalists?

• Surveillance & Retaliation: Recall isn’t just for end-users — Microsoft envisions this tool being widely used by businesses as well. For the enterprise, policies are set at the group level, which means that individual employees won’t be able to opt out and disable always-on workplace monitoring, though they may be able to delete snapshots, and filter websites & apps. It’s honestly a bit unclear. Once again, this creates a ‘consent’ issue. The regulators are pretty clear that the employment relationship is lopsided and imbalanced making reliance on consent a non-starter in most employment contexts.

  Remember that draft email I mentioned in the proportionality section? Now imagine that it’s an early draft of an email you wrote to your boss because he ticked you off that day. How many of us have written an initial draft email in anger or frustration, only to sit on it for an evening, pet a cat, cool down and draft something more measured and reasonable in the morning? I doubt it’s just me, although I do have rage issues.
  
  But now, if you’re at work and Recall is on, your boss will also be able to find out that your seemingly-measured response started out as an angry, profanity-laden rant where you insulted your bosses’ mom, and told him he sucked as a human being. Your initial frustration (which was never sent, mind you!) might cost you your job.

• Exfiltration & Data Breaches: Kevin went into great detail about the threats from exfiltration of data, and how easy it is. “I have automated exfiltration, and made a website where you can upload a database and instantly search it.” While he hasn’t made this live, what this means is that if he figured it out, a whole load of people will be able to easily do the same once this hits go-live.
  
  There is also an API for searching user activity, and third party apps can plug in to ‘enrich’ or view stored data.
  
  As Kevin observed we’re going to see an explosion in data breaches — both in frequency (because it’s on-by-default and most people won’t know), and severity.
  
  ”… if people have used a Windows device with Recall to access the service/app/whatever, hackers can see everything and assemble data dumps without the company who runs the service even being aware. The data is already consistently structured in the Recall database for attackers.

  
  So prepare for AI powered super breaches. Currently credential marketplaces exist where you can buy stolen passwords — soon, you will be able to buy stolen customer data from insurance companies etc as the entire code to do this has been preinstalled and enabled on Windows by Microsoft.”

• Compliance Hell: That leads me to a related concern — all of this newly collected, stored, and searchable data means a whole lot more compliance hell. If businesses think access, rectification and deletion requests suck ass now, multiply that by every single snapshot, on every employee’s device, for every single change ever made. And let’s not even get started on litigation holds!

• Controllership: This, IMHO is huge, and is likely to be overlooked until it creeps up in a lawsuit somewhere or gets called out by a regulator. Right now, it appears that Microsoft assumes that there’s no real GDPR or data protection issues at play because Recall is under the ‘control’ of end users/individuals, and processing activities done by individuals are generally out of scope for GDPR purposes since they are considered a ‘purely personal or household activity’.6

  Ignoring the business use case I mentioned above, I think there’s an interesting question of whether the use of Recall by an individual user is in fact always a ‘purely personal or household activity.’ The GDPR explains (in Recital 18) that the ‘household exception’ applies when an individual is processing (aka, doing stuff with) personal data for personal reasons, with no ‘connection to a professional or commercial activity.’
  
  The intent of the exception (which was drafted in the early days of the internet) was pretty simple: the EU didn’t want to make every type of private communication or use of personal data a giant assache. The Commission reasoned that nobody wanted to require Joe Random User to post a transparency notice or set up a data processing agreement when he emailed grandma, or posted his group selfie on Twitter. And no regulators wanted to enforce against Joe Random, because that would suck.

  But the household exemption isn’t absolute, and over the years, there have been a few cases clarifying when it applies (correspondence, and communications with friends) and when it doesn’t (CCTV recording of public spaces outside of a home, publication of personal details of others on a blog). It turns out, that the ‘purely’ bit in ‘purely personal or household activity’ is really important, actually. It’s not enough, in other words, that processing is done for a personal reason — it has to be only for a personal or household use. When you start adding others to the mix and sharing that information, things get messy.

  One aspect that I think damns Microsoft here (and potentially many individuals who use Recall) is that Microsoft envisions that app developers may create and interface with Recall to provide enhanced user experiences — for example, the ability to jump back into a task that has been ‘recalled’. I’m not sure that an interface that allows say, some random Microsoft app to pull up your Recall logs qualifies as a purely household activity anymore, especially if that application is doing different things with the data (like harvesting it for content/AI training).

  Nor is the deeply troubling use-case that Evacide reminded me of: what happens when a domestic abuser uses Recall to stalk their partner? Did the European Commission really intend to legitimize or at least exempt stalking as a household activity?
  
  And if the household exception doesn’t hold, does that make us all controllers? Imagine trying to help your neighbor with an access request. Or going through mom’s Recall snapshots to delete every time she said something nasty about Timmy down the street. Or filing a data breach notification when someone manages to break into your machine.

There’s more I could include here, but I’m at the email limit. Suffice to say, Recall, as it exists now, represents a privacy disaster. Maybe it could be improved. Certainly Microsoft could do better. As it stands though, I suspect Microsoft will face lots of regulatory ire, and it’s appropriate. They should have talked to me first.

Subscribe now

Leave a comment

Share

My body has a bad habit as of late. Despite taking melatonin, I consistently wake up at about 4:50am every morning. I’m not up-up, but I’m up enough that I can mindlessly scroll on my phone, where I catch up on all of the latest privacy disasters, discourse and outrage that occurred during the four hours of sleep I was able to manage. Yesterday, I was greeted with this on my Twitter timeline:

The video features four vignettes of Gen Zers living their best lives, wearing what looks like a slightly thicker and more glowy Apple AirTag around their necks1 The first scene features a 20-something woman, exerting herself on a solitary hike in the woods, the next a frustrated guy playing a video game with trash-talking friends, the third a woman messily eating a falafel in her non-descript apartment, and the fourth is what I can only describe as an extremely awkward date on someone’s rooftop.

And regardless of whether the characters are alone or with other actual humans, all are sporting this goofy-looking pendant, known as the ‘Friend’ (or possibly ‘Emily’?) that is recording in real-time their lives and feeding all of that to a large language model (LLM).

The noble goal of this always-on friend, according to the device’s founder, Avi Schiffmann in a recent Wired article, is to stave off loneliness by offering AI-based companionship. I liken it to what happens if the AI chatbot Replika had a three-way with Microsoft’s Recall, and the failed Humane pin.

According to the Wired piece:

The Friend purely offers companionship. It’s meant to develop a personality that complements the user and is always there to gas you up, chat about a movie after watching it, or help analyze how a bad date went awry. Not only does Schiffmann want the Friend to be your friend, he wants it to be your best friend—one that is with you wherever you go, listening to everything you do, and being there for you to offer encouragement and support. He gives an example, where he says he recently was hanging out, playing some board games with friends he hadn’t seen in a while, and was glad when his AI Friend chimed in with a quip.

There’s very little information about the pendant, which I will be referring to as the Friendant on the friend.com website, beyond the fact that it connects to the user’s mobile device via Bluetooth, that it’s always listening and processing what it hears, it only works on iOS devices for now and communicates by sending chat messages, presumably in a dedicated Friend.com app. Also that it’s $99 and won’t be available until Q1 2025.

Also, in a move that feels very reminiscent of early 2000s pre-crash internet exuberance, Schiffmann told the Wired folks that he purchased the friend.com domain for a cool $1.8 million, after raising $2.5 million in VC-backed funding over two rounds.

Now, I’m not a venture capitalist or anything, but I’ve been around the block a bit. Hell, I even lived in the Valley for a few years and if I learned anything, it’s that if you’re going to build a successful product, you really should be laser focused on the product itself. That means prioritizing product development (programming & UX), identifying a good market fit, and offering a catchy tagline on why people should actually buy the thing you’re hawking. If you’re a seasoned founder, you might also be thinking sensibly about longer-term risks, like potential legal & regulatory issues, especially if you’re creating some sort of ‘disruptive tech’.

What you should not do is spend most of that budget on a fucking domain name.

A Privacy Notice That Isn’t

With the product details out of the way, let’s get to the good stuff — me lambasting the Friendant’s privacy notice.

To start, the Privacy Policy Notice is, unsurprisingly, scant on details. The controller (MyTabAI) includes the barest information on processing and use of personal data. What little is there only touches on information collected through the website, or through pre-orders via the Stripe link. The notice says nothing about what the Friend device itself will be collecting, how it’s sharing this with Claude.ai, how long data is stored on device (or on Anthropic servers), what the context window is, how it somehow remembers user context, etc.

And guess what guys. If you were wondering if the notice mentioned anything at all about sensitive data, well, rest assured, it does.

We do not process sensitive information. All personal information that you provide to us must be true, complete, and accurate, and you must notify us of any changes to such personal information.

Uh huh.

It’s clear that Schiffmann and his team were laser focused on having a beautiful, albeit potentially legally-problematic design, registering the cleverest domain name they can, and launching it on World Friendship Day (30/07/2024). They were far less concerned with the annoying legal and data protection technicalities, or those pesky questions around autonomy, choice, consent, etc.

Funnily enough, the notice also gives away the fact that before Schiffmann started calling it the Friend(ant) in mid-May, he was referring to the device as the Tab. There’s even a YouTube video with that name that was released 9 months ago.2 That YouTube video, btw, is both illuminating and deeply, deeply cringe.

In it, Schiffmann describes the Friend (nee Tab) as a ‘context aware’ device, that ingests the contents of the wearer’s life so that they don’t have to provide context every time. It’s unclear how that happens exactly, beyond the fact that the Friend is constantly listening (for up to 15 hours) and shunting queries to Anthropic AI’s Claude 3.5 large language model. Between jokey scenes where Schiffmann is reading a book on ChatGPT prompts, he volunteers that he’s spent some months developing custom prompts, presumably so that Claude can provide vaguely relevant responses to everything the device is listening to.

But Wait… We Have Competing Disasters!

I need to stop and take a beat before I activate Angry DPIA mode to inform everyone reading that this is actually the second AI-based wearable pendant called Friend that is available on the market right now, because this is the cursed world we now live in.

This other Friendant is an open-source jobber created by Nik Shevchenko, a ‘Thiel Foundation Fellow’ (vomit) and CEO of BasedHardware. This Friendant is more about traditional productivity, like the Humane & Rabbit R1 wearables, not staving off loneliness like Schiffmann’s creation. Despite offering more whizz-bang AI features, including syncing with all the apps and storing and processing locally on-device, this Friend is available for pre-order for the low, low price of $70, with a Q3 2024 delivery date.3

The BasedHardware Friendant has a slightly better privacy notice, but it’s still pretty bad. Recorded audio is processed using Deepgram for transcription, and transcriptions are vectorized using Pinecone. Only the vector representations are stored, and all storage is on-device. Allegedly, ChatGPT also runs on-device as well, and it still manages to boast a 24-hour battery life somehow.

Truthfully, this sounds like privacy-washing bullshit. Especially if you understand how these services work and how computationally expensive all of this is. Oh, and there’s also this contradiction in terms:

None of the data is transmitted to our servers or any third-party servers, except for the services mentioned (Deepgram, Chat GPT, Pinecone), which also operate locally.

I mean, look at this thing? It’s the size of an anal suppository, and you mean to tell me that it’s got all of that complex GPU and storage-hungry stuff running all the time and the battery life is better than my phone, and somehow this doesn’t overheat to the point of catching fire?

Anyway, I haven’t looked at the code yet, because I’m tired. Maybe it does do all of that, IDK — here’s the Github link if you’re curious. To his credit, at least Shevchenko open sourced his work so it can be assessed more easily.

Finally, I would be remiss if I didn’t mention that Shevchenko, the founder and CEO of a real-life company hoping to secure actual VC money was mad enough about Schiffmann poaching the Friend name that he posted a diss track on Twitter, as well as challenging Schiffmann to fight him. In light of this, I would suggest that Shevchenko rename his product Frenemy. But this is not legal advice.

If Only They Hired Me

Now then, on with the fun that you’re all waiting for. Namely, what these lads would have considered if they’d been smart enough to hire me as a consultant before releasing these things into the world. Hell, here’s the kinds of questions any spy-wearable should probably look into at least. Again, not legal advice — it’s just basic common sense.

For the sake of my sanity, I’m going to refer to the Schiffmann’s wearable as the Friendant, and Schevchenko’s wearable as the Frenemy, because

First, a quick reminder. Under Article 35 of the General Data Protection Regulation (GDPR), a controller must perform a data protection impact assessment when certain “high risk” processing activities occur. The Irish DPC helpfully provides a list of high-risk processing activities. For purposes of these wearables, a DPIA is a good idea because it seems likely that both trigger at least one of these conditions apply:

• use of innovative new technological or organizational solutions;

• systematic monitoring of publicly-accessible areas on a large scale (since users will be wearing them everywhere);

• processing of sensitive data of a highly personal nature (if you’re gonna treat it as your bestie and wear it around constantly, it’s going to pick up all sorts of juicy details about you and anyone you interact with);

• collection of personal data about vulnerable individuals (this one depends heavily on context. It’s less a problem if the founders stick to targeting Bay Area GenZers, much more of a concern if it targets grandparents with Alzheimer’s or kids).

Now that I’ve got why a DPIA is important out of the way, let’s look at some of the questions and risks that I’d want answers to.

• Transparency: There is absolutely no transparency here in the privacy notices. While the Frenemy at least goes into some detail, it’s still all very vague, and I doubt either of the privacy/transparency statements meet the letter and spirit of most US state privacy notice laws, much less the rigor of the GDPR. How are users being informed of the processing that’s happening here? Where is the data being stored? How is it being shared with Anthropic / OpenAI? What are you guys doing with all that data? Is it going overseas? Is it being secured? What’s the lawful basis?

• Consent of users/third parties: Currently, only the wearer consents. Because these wearables are always on, the only way I as an innocent bystander can object or opt-out from being recorded by someone wearing these pendants is by asking them to turn it off and hoping they do, walking away, or ripping the damn thing off their neck and chucking it into oncoming traffic. I can see some regulators getting mighty miffed about that. And that presumes I know that these pendants are always recording spy-wear, not just weird jewelry.
  
  And how does withdrawal of consent even work here? Say I’m fine with this at first, but I decide later I don’t want your Friendant to know about all the deep thoughts I shared with you. How does the owner of the Friendant/Frenemy delete that information? Can users purge information stored on Anthropic/OpenAI servers?

• Proportionality: Both wearables monitor and record every interaction, every intimate conversation, and all those deep thoughts shared out loud. As I’ve mentioned before, most of us are still not cool with having our lives constantly being recorded all the time. Notwithstanding the ubiquity of CCTV and facial recognition in public spaces, I think many people would still freak out if they knew that their casual conversations with someone were being recorded and shared with Anthropic or OpenAI, these guys, and god-knows-who-else, all for a little user convenience. Just look at how we collectively reacted to Google Glass. Or how quickly Recall was withdrawn by Microsoft.

  
  Remember, it’s not just recording the wearer’s voice and personal information, but any voice and details shared by those interacting with the pendant-wearer.
  Just imagine you’re a 20 or 30-something person, going out to some douchey bar in the Bay Area looking for love. You spy a cute potential paramour in the distance, who seems to be wearing a shiny piece of jewelry that glows weirdly for no reason. Now you have to wonder if what they’re wearing is some sort of repurposed AirTag, or if it’s a Friendant talking shit about you or sharing snarky observations at your expense. Who wants that?

• Children: What about kids? Most data protection laws impose heightened standards for consent in relation to children. With the rise of laws specifically designed to protect kids’ privacy gaining traction in the US and elsewhere (including the Kids Online Safety Act and the Children And Teens Online Privacy Protection Act, which recently passed with overwhelming support in the US Senate), having an always recording wearable seems like a recipe for regulatory scrutiny. And everybody wants to protect the children, so that’s something that crosses party lines.

• Exfiltration & Data Breaches: While I haven’t looked at the Frenemy’s code base (yet), I suspect that security was … not exactly a priority for either of these companies. Given that both interact via Bluetooth via mobile devices, and Bluetooth is vulnerable to eavesdropping and other man-in-the-middle attacks, I could see someone finding an exploit in a matter of weeks here. Just like people found with Recall. It’s a goldmine for criminals.
  
  Points to Frenemy for allegedly keeping some of this data on-device, which is better. Minus points for writing all of this in very vague and contradictory language, though.

• Controllership: Holy shit is this one a dumpster fire. So, ordinarily data protection laws treat data that normal people use for personal reasons differently than say, data collected by Meta or Google. That makes sense: Nobody wants to police Grandma’s collection of photographs of her grandkids, you recording your friend doing something stupid, or all those dick pics sent between people on WhatsApp. Generally speaking, if it’s not processing for a professional or commercial use, all the baggage of data protection laws generally will not apply.4
  
  But these personal use exceptions are usually construed narrowly. It’s not enough that processing is done for a personal reason — it has to be only for a personal or household use. When you start adding others to the mix and sharing that information, things get messy. In the EU for example, if you say, set up a CCTV camera to film all your neighbors for your personal jollies, that might be a use for personal reasons, but it’s not a personal or household activity under the law. In fact, a 2014 decision by the Court of Justice of the European Union said as much:
  
  To the extent that video surveillance … covers, even partially, a public space and is accordingly directed outwards from the private setting of the person processing the data in that manner, it cannot be regarded as an activity which is a purely ‘personal or household’ activity …5
  
  I see no reason to believe that this would be different just because it’s a life-logging device that only records audio. That makes life fun both for the creators of these tools AND anyone foolish enough to wear them. If anyone needs help with handling data subject requests, HMU.

• Two-Party/All-Party Consent Laws & Eavesdropping: Many countries have some form of eavesdropping law on the books. While many target telephone communications and most only require a single party to consent, some apply to electronic communications generally. According to the Reporters Committee for Freedom of the Press, 11 states have what are known as two-party, or ‘all-party’ consent requirements for recording, including California, Delaware, Florida, Illinois, Maryland, Massachusetts, Michigan (at least for recordings made by a third party who is not involved in the conversation), Montana, New Hampshire, Pennsylvania and Washington. Other states (including Oregon and Missouri) require all-party consent for in-person recordings of conversations.
  
  While laws vary on how consent is obtained and whether implicit consent suffices, the assumption is that recording must be overt and obvious enough to the other people being recorded, and can’t look like an innocuous piece of jewelry around someone’s neck.
  
  This is more likely to impact the wearer than the Friendant/Frenemy creators, but imagine the lawsuits that would occur if Bay Area techbros start get arrested for recording their dates or whatever without consent. Or cops. Lol.

• The Data Act: There’s a separate and very interesting question about whether these sorts of devices (or other AI spy-wearables) might qualify as connected products under Article 1 of the Data Act. Obviously, more questions would need to be asked — namely, what kind of product data and related service data is collected. Interestingly, I don’t know if this triggers much additional review under the AI Act.

• Various Biometric laws: Both Texas and Illinois have fairly robust biometric privacy laws that protect individuals from having their biometric data (including voiceprints) collected for commercial purposes. If I was doing a more robust DPIA, I’d want to interrogate whether and how voice recordings of participants are used. Clearly some processing is occurring. The answer is how much and for what specific purposes.

Yeah, so this isn’t an exhaustive assessment, and it’s theoretically possible that some or all of these concerns have been addressed and the founders simply haven’t informed anyone yet. It’s possible, but in the way that time-travel is possible, or me winning a Ms America pageant is possible, which is to say, not freaking likely.

So yeah. Maybe one or both of the Friendant/Frenemy creators will reach out and we can dig into how these things aren’t the stuff of dystopian nightmares. Hope springs eternal.

But for now, I won’t be holding my breath. As always, if you’ve got a thought/observation or think I missed something, leave a comment or send me an email.

Leave a comment

Share

Share Privacat Insights

Subscribe now

So there’s lots of terrible tech out there, and loads of it relies on AI. I simply don’t have the spoons to process, much less respond to all of them. Still, sometimes a shitty idea is so very extra bad that I feel morally obligated as a public servant of privacy to share why the inventors are not only staggeringly wrong, but why the product they have released is toxic to humanity. And today ladies, and particularly gentlemen, I’m going to talk about the sheer unmitigated disaster that is Calmara.

CW: There will be terrible dick-related jokes throughout, and a cat.

According to its website, Calmara is an app that uses AI to detect whether your soon-to-be-lover is laden with a sexually transmitted infection or not. It’s basically hot dog/not hot dog, but for penises and STIs. It goes like this:

1. You (or the person you’re planning to have sex with) download the app.

2. You take a picture of your penis (or their penis, if you get their explicit but totally unverified consent) and upload it on their app.

3. Some pre-processing occurs (to control for lighting and other confounding issues).1

4. That image gets sent to … AWS somewhere, and an AI model assesses whether the image is full of STIs or not.

The site claims a 90% accuracy rate, with no cited sources, and is full of cutesy language that tries to make it seem cool and secure. However, there’s no actual data on what’s going on under the hood. Oh and it’s available in the EU (and everywhere else on earth, apparently) because absolutely nobody sat these lads down and told them that maybe this might run into legal/privacy/ethical/cultural/pornography/CSAM issues. After all, who needs a buzz-kill lawyer, when you’ve got AI, amirite?!?!

The Cofefe Calmara Privacy Policy (ugh) and Terms and Conditions of Use indicate that they clearly never bothered talking to anyone familiar with HIPAA, or data protection laws generally. By the looks of it, they avoided interacting with a lawyer entirely. Their terms of use statement is littered with the kind of language that lay people think will cover their ass but usually don’t. For example, the always fun, ‘The Service is intended for informational purposes and is not a substitute for professional medical advice’ and obligations to obtain explicit consent, and a full release of liability if the user fails to do so: ‘HeHealth Inc. will not be liable for any claims, damages, or legal actions resulting from the submission of images without proper consent, including but not limited to privacy infringements or violations of any law.’

To test this, I downloaded the app, and … yeah this is not how any of this works.

It’s the legal equivalent of magic pixie dust, and if I were the California Attorney General, I’d be hopping on this D(isaster) like a porn star.2

The Calmerror Calmara website also claims that there is an age verification process, and despite my effort to try this out with cat pictures, I never saw it.

Also, their TOS and FAQs include a general disclaimer that absolves them of liability if an underage person uploads an image. Guys, in many jurisdictions, there is no get-out-of-jail-free-card for child porn, and an underage person’s peen is probably gonna get you into trouble, no matter what the TOS says.

Their privacy policy is equally bad, and it doesn’t include any details on how images are used (beyond to deliver services and to ‘enhance and innovate our service offerings’), their legal reasons for processing data beyond the image itself, how long images are kept, how these very intimate images are secured, how data is shared and with whom. The sum total of their sharing statement is:

To provide you with seamless service, we share your information with service providers and partners who assist in service operation, including data hosting, analytics, marketing, payment processing, and security. These collaborations are vital for a seamless and secure service experience.

Given that the only personal data/information they admit to collecting is images, cookies, and ‘user behaviors’ and their statement is extremely vague about what they do with this data, it seems entirely reasonable to assume that they are mass-blasting dick pics and everything else with all parties involved. Congrats guys, your dicks might end up on marketing collateral!

Also, their FAQ claims that because they use AWS to store and do the AI magicks, that makes their app HIPAA compliant.

A Momentary Interlude While I Drink and Check Out Their IP

The company behind Calimari Calmara trades under the name HeHealth Inc.3 HeHealth, which launched in 2022, offers a flashier version of the same technology, called Hehealth.ai, which is basically a paid version of the app, with a bonus doctor’s review. No idea about which one of these dudes is actually reviewing the images, or whether any are actually licensed/certified in proctology, but hey, somehow that’s slightly better than the free Calzone Calmara version, which is doing medical diagnosis entirely by AI. According to the HeHealth app, 31,129 people have allegedly scanned their weens and uploaded them to an AWS server somewhere.

HeHealth.ai is based on patented technology, which can allegedly detect 10 different STIs according to the website (though the patent and pre-print validation study only seem to show detection for genital warts, herpes eruption, cancer, candidiasis, and syphilis). It uses a trained AI model that is augmented with a small number of actual healthy/unhealthy penises (<240), and potentially synthetic images, if I read the patent details correctly.

I’ll admit, I’m happy that there’s science behind it. But that doesn’t stop this whole affair from being a massive privacy disaster.

Lads: Just Go See a Doctor

Guys. Fellas. Gents. I get that sometimes the urge to get down may overwhelm better judgment. That privacy and data protection might not be top of mind when you’re getting ready to get dirty. Still, don’t let your little head get ahead of your bigger one on this. If you’re worried, or if things don’t look right down there, just go see a doctor.

Right now, based on everything provided on the Calmara and HeHealth websites, I’m unconvinced that they’re thinking of your interests. Based on what’s being presented, this app looks rushed and hoping to cash in on the AI hype. While it may be well-meaning, it seems to lack any consideration for the kinds of things I would expect to see out of a legitimately-compliant mobile health app, which is what it should be, because this app is offering you medical advice. Specifics and details are light. Testing seems limited and it took me effort to find it. Everything is far too cutesy, and really, do you want cutesy when you’re uploading images of your dick to god-knows-where?

Hell, I’m not even convinced that the AI can do much more than detect if your peen is not exactly right (whatever that means). For (literal) fuck’s sake: just go see a doctor.