Thoughts on the Law, Building a GPT, and Tsunamis of Change
Thoughts on the Law, Building a GPT, and Tsunamis of Change
I play with toys to process deep thoughts, y'all.
I've been having a bit of fun with ChatGPT's new 'Create a GPT' feature. Right now, it's only available for paying subscribers.
I'm in the process of writing a piece musing on consent, the law, and artificial intelligence, and wanted to summarize some of the law review and scientific literature that's already been written, find trends and topics to explore, and generally wade through what are oftentimes overly verbose pieces. After a whole bunch of trial and error (enough to say, take up the time for reading 2 of the 9 or so articles I found), I created a niche, specific purpose Summarizer GPT. It's pretty good at at least drawing out repeated themes, core concepts and keywords, and distilling down the author's main points.
For the sake of transparency, I dumped the (slightly cleaned up) output of one of the law review articles I read (From Transparency to Justification: Toward Ex Ante Accountability for AI, by Gianclaudio Malgieri and Frank Pasquale) here. There's also a direct link to the citation.
Building on what I've learned, I also created a more general Case Summarizer GPT. It's very similar to the same tool I created using GPT-3.5 on the command line, which I wrote about here.
Since it's using GPT-4 and the Code Interpreter feature, it's able to process cases and large documents and spit out results far more consistently and accurately. I ran the recent CJEU case Gesamtverband Autoteile-Handel eV v. Scania CV AB (Case C‑319/22), issued 9 November, did the same rough level of cleanup and highlighting/markdown magic. You can find a copy here.
If anyone wants access, let me know and I'll share a link.
Updating the Case Summarizer
Over the course of the project (which again, was about a day's worth of work) I wanted to share what I think has improved and what's still annoyingly frustrating about using these summarizers for legal research.
The Good
This helps to solve the 'too long; didn't read' / 'not enough time in the day' problem. Compared to reading dense, almost Dickensian tomes (most of the law review articles clocked in at well over 40 pages with hundreds of footnotes, with one hitting 127 pages and over 500 footnotes), GPT's summarization gives me context, highlights points or novel issues to think about, and identifies the specific pages to review. For example, in the Malgieri and Pasquale article, GPT surfaced their proposed 'unlawfulness by default' approach, which requires AI makers to justify and prove their systems are lawful and compliant ex ante rather than post hoc.
It provides an easy way to highlight and integrate concepts for later analysis. I suck at programming, but I've gotten passably good at thinking a bit like an engineer. It turns out that this ability is very useful when it comes to structuring prompts for GPT. For instance, I use a 'second brain' tool known as Obsidian. Obsidian has a ton of powerful features, among them the ability to link concepts together easily and represent them in a variety of different ways. To harness that power for creating graphs and linkages in Obsidian I added this instruction to the bottom of my turn-by-turn instruction set:
Use Markdown formatting, including double brackets [[]] around each term. You should ask for clarification if needed, but try to fill in any missing details yourself. After a little bit of tinkering (mostly trying to get the graph tool to work properly, which is a separate post of hell), I get outputs like this:
What's interesting is that I asked my Summarizer GPT to surface not only a list of specific pre-defined concepts (AGI, AI, inference, consent, etc.) but also to identify and tag concepts explored by the authors that I hadn't thought of. So, in the Malgieri and Pasquale article, 'licensure model', and 'ex ante accountability' were identified as future topics worth exploring. This happened for most of the articles I read.
It's improving and hallucinating less. The Summarizer and Case Summarizer GPTs actually turned out reliable citations. This is a marked improvement over past iterations, particularly when it came to case summaries. In the GPT 3.5 days, if you asked it for a list of referenced cases, even in the document you provided, it was just as likely to spit out hot garbage gibberish as valid cases cited by the court. I had to ask for the cases specifically, but it reliably grabbed them and provided them in a list, along with a summary of the key elements discussed.
The Bad
Copyright issues. I admittedly did not ask for permission here. To mitigate some of the legitimate copyright concerns, I did turn off the feature that allows the articles to be incorporated into GPT's learnings. Additionally, data uploaded using the Advanced Data Analytics/Code Interpreter engine only stay on the OpenAI system for a maximum of 3 hours. Copyright is, and will remain, a hard problem, and a valid concern particularly for the law review articles and published authorship generally. It's a hard issue -- on the one hand, there's an irony in analyzing law review articles explicitly touching on consent, inference, and privacy rights using a Big Tech tool with a pretty cavalier opinion about these issues. But, conversely, its important to recognize that by using these tools, I (and others) actually can become aware of and incorporate, share, or build on the ideas and thoughts the authors are trying to put out into the world in the first place. The SSRN link for the law review article I mentioned notes that only 700 people have downloaded the article since it was published in May 2022. I suspect significantly less read it through. A well-trained, accuracy improved GPT could lead to more discovery, more interest, and greater understanding. For example, it's reasonable that if I didn't speak English, I could add a requirement to translate and summarize the article instead -- a feature not yet available in any PDF software I'm aware of. FWIW, I still hold that court cases, legislation, regulations and the like are and should be treated as public, common goods and therefore fair game for the likes of ML tools.
Many iterations. None of this came straight out of the box. It took quite a bit of refinement, tweaking, and yes, some manual editing. My Summarizer GPT didn't find, for example a few key terms like 'unlawfulness by default', and sometimes flagged silly things like URL or the author's name, particularly when I was just starting to fool around. Prompt engineering takes a lot of trial and error.
It still misses important stuff. My Case Summarizer GPT missed some pretty big concepts that I knew were present in the Scania case -- for example, the court's Third Point of Law covering when a VIN number is personal data and when it's not. It mentioned the conclusion (that VIN data could be personal data in certain contexts), but completely omitted all the interesting elements, like the comparison to Breyer and Inspektor v Inspektorata regarding the conditions where data is linked to a natural person. That's really important! Fortunately, it did provide some context clues (like the very brief mention of personal data as part of the Third Question) which gave me the ability to quickly jump to the relevant section and read it directly.
It still hallucinates: The tricky thing here is that I think it hallucinates less obviously. The takeaway remains the same: treat GPT like a very inexperienced 1L who's interning 3 weeks into law school. Don't trust, do verify, but at least their first pass might save you a little time.
On Technical Tsunamis and Outmoded Legal Protections
We're in strange, turbulent times, and truthfully, I feel like a ground shift is, or will soon, take place in terms of how we as a society grapple with what may represent a new technological age. Right now, it feels like there's a lot of enthusiasm in just trying to hold back what feels like a giant tsunami of change and disruption with the legal equivalent of sandbags, wooden barrier walls, and prayer. Even the new laws still look at the problems in the same old ways, with models constrained to working at human scale -- e.g., licensure, more consent and transparency, or through meaningless process improvements like audits and "explainability."
To continue the tsunami analogy, I don't think we should resign ourselves to passively being washed away, but I don't think piling on the legal equivalent of more sandbags is going to cut it here. This may be a situation where technical solutions are necessary to fix technical problems. Or one where, like the dawn of the Industrial Age, we need to revisit certain concepts with fresh eyes.
Until then, I'll keep reading (with a little help from my GPTs) and thinking about these problems. One problem I hope to talk about soon, once I can successfully dump it from my brain, is how AI systems steer you towards the 'optimal' path. What does it mean to give meaningful, freely-given, informed consent in a world where you're being manipulated and nudged into decisions that often are the best, and most optimal ones?
NB: I have disabled the paid subscription option for the time being. I felt really bad charging people when I wasn’t turning out content. Now that I’m no longer up to my eyeballs in travel and work things, I hope to change that. But if you like what I had to say here and feel generous, I always appreciate the support.