From Frustration to Automation: How I Built a LLM-Powered Legal Case Summarizer

Programming generally scares me, but with the combined power of ChatGPT, Python, Obsidian and annoyance, I managed to write a pretty rad legal case summarizer anyway.

I’m going to tell you a little secret about myself: I am annoyed easily.

I realize that for some of you, this won’t be very revelatory. For example, my poor, long-suffering husband, friends & family, as well as some former colleagues and lovers, all know that my fool-suffering threshold¹ is basically zero:

But, I make up for my seemingly endless low-level rage with the world by forcing myself to learn things that help me to be less annoyed. I believe the experts call this ‘growth,’ or maybe life hacking.

One way I do this, is by trying to understand why a seemingly annoying thing is the way it is. Understanding why often helps me to come to terms with what otherwise feels like arbitrary bullshit and unnecessary fool-suffering. Sometimes, of course, the answer is that shit is really complicated, and I then am less annoyed because I at least understand that there are reasons.

For example, I recently had a bit of a tiff with a certain department within the Irish government. In short, they told me that my birth certificate was not authentic and that I should just send them a real one instead. Critically, they did not tell me why my birth certificate, which has been widely recognized by other governments as being legitimate, was deficient. They only said that I needed to conjure up one that met their undisclosed standards.

Now, a normal person might have just requested a new birth cert and hoped for the best. But, I am not normal, so instead, I wrote an angry letter with specially-crafted words like ‘irrational and unreasonable’ and ‘contrary to the principles of natural justice.’ And guess what? My letter worked. Suffering avoided.²

But my low fool-suffering-threshold doesn’t end with people and opaque bureaucracies. It also extends to other, more mundane things, especially tasks that are repetitive or boring, or consume lots of my time but still need to get done. Especially, if I think said tasks can be automated in some way.³

Other people have also come to this conclusion, and for the most part, they become programmers or engineers. But, for like, a dozen different reasons, I have come to realization that despite my deep appreciation for automation and engineering, I do not have the aptitude, energy, or physical or mental capacity to become a programmer or engineer, any more than I could become a world-famous concert pianist. I have spent almost three decades of my life trying to learn both the piano and programming, and while I can passably read and play music, I cannot make my brain understand or remember how to generate the types of magic incantations necessary to get a computer to do what I want.

And so today, I’m going to tell you a story. This is a story about how I used LLMs to help me program and unblock what has been a persistent problem for many years, and in so doing, has made me a bit less annoyed with the world. Maybe even happy?

There are Too Many Cases, and Not Enough Time to Read Them All

When ChatGPT burst on the scene in 2022, a fire went off in my brain. I saw an immediate opportunity to solve a major pain point in my life. Namely, there are dozens of dense, but highly consequential legal opinions, judgments, and guidance materials put out into the world all the time.⁴ I want to read and make sense of everything that comes out but I cannot, because I also like sleeping, and spending time with cats, and actually going outside.

As I mentioned above, I am not a programmer by temperament, so using an LLM to help me write some code seemed like a perfect solution to what had been a years-long blocker for me. To test the waters, I asked ChatGPT to write a very simple Python script for me that would read some text, store that information in memory, make a request to ChatGPT with my prompt + the text, have ChatGPT perform its LLM magic, and send back a response that was displayed in my terminal. The first text samples were small, and the first prompt was very broad: ‘Please summarize this document.’

Carey’s Programming Tip #1: Set low expectations & start small and simple. Small and simple is smaller and simpler than you think. You’ll thank yourself later.

After a few false starts and some back-and-forth haggling with ChatGPT (at the time, ChatGPT 3.5), the program actually worked! It could read the text, send it off to the OpenAI servers to be summarized, and respond with an outputted summary.

The next step was to actually get it to do this for a case. The thing I quickly learned is that LLMs have a finite amount of space for doing what’s asked of them. This is known as the context window, and for ChatGPT 3.5, the context window was around 16,385 tokens. Each token is about 4 characters of text, so we’re really talking about 65,540 characters, which is 10,083 words, or roughly 46 pages of text. That seems like a lot until you realize that this context window covers everything from the prompt, text to be analyzed, reasoning/processing, and output.⁵

A visualization of OpenAI’s context window for the o1 model.

Exceeding the context window was an early and persistent problem, and it usually led to one of two results. If I was lucky, I’d get a very short, usually truncated, crappy summary. But usually, I’d just get an error saying that I’d exceeded the token limit. Whoopsie.

“Surely,” I thought to myself, “some for-reals programmer has come up with a solution to this problem.” And someone had. To get around the context window problem, I’d need to have my program break the document up into smaller chunks of content, summarize each chunk individually, store the chunked summary in a file somewhere, and repeat. Once ChatGPT had summarized the last chunk, it would then need to go back and summarize that file where the stored chunk summaries were, and save that to a file.⁶

Carey’s Programming Tip #2: Assume that your pain is shared pain, and that someone else has probably solved at least part of your problem.

This is, sadly, a joke cover, but nearly every programmer I know actually does this when they can’t figure something out.

The author had also helpfully put his code up on Google Collab, which is a great environment for learning how programs actually work, even if you’re a non-programmer like me. It’s an interactive environment where you can run and tinker with discrete code slices individually to really see how things work and break them in a gentle and easily observable way.

Iterate, Test, & Repeat, Until the Heat Death of the Universe.

Once I had the chunking problem mostly sorted, I moved my program out of the Collab environment. That was a fun exercise in frustration, because I learned that once I moved code out of the safety of Google Collab, into the lawless wasteland that is my Windows desktop running Ubuntu Linux under the hood, I would have to install many, many things to make the computer do what the Google Collab environment did automatically.

I will not explain everything I had to do, but I will share two Python-specific problems I faced:

1. There are different versions of Python, and stuff written in the older version of Python usually do not work in the new Python without varying degrees of effort. And it almost never works going backwards, because, fuck you that’s why.

2. A fresh Python install is fairly bare-bones⁷, and my scripts required a number of packages to run properly. But it’s Python, so that means I also would need to install all the package-specific dependencies. And sometimes, because of #1, those dependencies can’t really be satisfied because no one has updated the library for like 7 years. Welcome to dependency hell.

Carey’s Programming Tip #3: Be prepared to enter dependency hell at least once during your journey. Learn to process the stages of grief quickly.

Once I escaped dependency hell, ChatGPT and I got back to work. Now, unlike for reals programmers like my husband, I did not really have a software development environment. My ‘development environment’ consisted of a combination of Notepad++ in Windows, and the Ubuntu command line which I used to test/run my code. I remember many times where Husbot would wander into my office, either to bring me a motivational beer or remind me to eat, or to offer assistance when ChatGPT and I had really gotten stuck. He would peer over my shoulder, frown, and silently judge me for my banjanxed setup, and my sheer incompetence when it came to basic software engineering practices.

Take for example, what I called ‘version control’. My approach consisted of saving slightly different versions of whatever I was working on using delightfully unhelpful names like CJEUCaseSummarizer.py, CJEUCaseSummarizerOld2.py, CJEUCaseSummarizerv3_USE_THIS_ONE.py, , etc. Eventually, even I got confused as to which version did what, or which version was working. Oh, and I did this on different laptops, for reasons.⁸

Similarly, my commenting process was also unforgivable, and it still needs work, if I’m being honest. Adding descriptive comments in code is valuable, both to provide clarity to others on why specific choices were made, but good comments are also helpful when it comes to looking back at your own code, when months or years have passed and you no longer remember why you did that thing. So, for example, instead of ‘add UTF-8 support’ try something like

# Open and read the file. Add UTF-8 encoding to avoid the always annoying Unicode ''charmap' character maps to <undefined> error' when processing cases like Gäñă v. Ḍèëç.    

Carey’s Programming Tip #4: Your future self will be eternally grateful for having a good version control process and leaving descriptive, useful comments and not shit like #LOL I Fixed the Thing.

Fix Yo' Prompts.

Even after I got my program up and running and it was turning out halfway decent summaries, I still had to contend with another problem: the generated summaries were still pretty bad.

Some of this was a byproduct of the model (ChatGPT 3.5 was notorious for turning out garbage) + my poor programming skills, but I also realized that my early prompts did not provide the LLM with enough structure and detail on what I actually wanted to see. Asking the LLM to ‘summarize this case’, with no additional context or clarification consistently led to very generic, unhelpful answers. I’d forgotten a very important programming tip:

Carey’s Programming Tip #5: Provide detailed prompt instructions. Describe exactly what you want to see, step-by-step, even if it seems obvious. Pretend you’re instructing an interdimensional alien on how to make a peanut butter & jelly sandwich.