When OpenAI released its chatbot, ChatGPT, in November 2022, it seemed to take over the internet. Every day someone was demonstrating what the language processing tool was capable of—including almost passing the bar exam.
Chicago-Kent College of Law Professor Daniel Martin Katz, along with a colleague, proved that OpenAI’s GPT-3.5 software is almost capable of passing the bar exam. ChatGPT is the popular free demo version of a fuller program, TEXT-DAVINCI-003, better known as GPT-3.5.
In his paper, “GPT Takes the Bar Exam,” Katz and co-author Michael J Bommarito III outline the process for teaching GPT-3.5 not just to take the Multistate Bar Exam (MBE), but for it to pass two portions—evidence and torts. He says a full pass is right around the corner.
“We say in the paper that in 0–18 months it will pass the bar exam, I’ll stick by that,” he says, noting that three years ago the GPT-2 could not even process the question prompts effectively.
GPT-3.5 answered 50.3 percent of the questions on the MBE right. Humans get about 68 percent of the answers correct.
Now that these technologies are improving so rapidly, Katz says chatbots will likely change the way that people live their lives and do their jobs sooner than many of us expect.
But, Katz adds, even though the tools can help, they’re not a replacement for a licensed attorney. GPT can still “hallucinate,” or give answers that sound intelligent, but are actually nonsense. There are bugs, and Katz says there needs to be a “human in the loop.”
“You shouldn’t use these tools with no human oversight, but a person using this has a jetpack now,” Katz says.