A brief that reads cleanly and a brief that will survive a judge's scrutiny are two different documents. Generative AI has gotten very good at producing the first. It has done nothing to guarantee the second, and the gap between the two is exactly where hallucination risk lives.
Every firm running AI on real matters, whether it's contract review, due diligence, or demand letter drafting, eventually runs into the same question: how do you know the output is actually correct, rather than simply confident-sounding? That question deserves a real answer, and this piece gives one without resorting to fear of the technology.
What a Hallucination Actually Looks Like in Legal Work
In a legal context, a hallucination isn't a glitch that announces itself. It's a case that doesn't exist, a real case cited for a holding it never made, a quote attributed to an opinion that says something else, or a statute cited to the wrong jurisdiction or the wrong year. The output is grammatically perfect and formatted correctly. It simply isn't true.
That distinction changes what "checking the work" has to mean. A typo is easy to catch on a read-through. A fabricated citation with a plausible case name, a real-sounding reporter number, and a holding that fits the argument isn't. It passes a skim. It sometimes passes a first read by an associate under deadline pressure. It doesn't pass a judge's clerk who pulls the case.
Why Confident Language Creates False Reassurance
Large language models are trained to produce fluent, decisive prose. There is no built-in mechanism that makes the model hedge more when it is less certain, and no visible signal that separates a well-supported statement from a fabricated one. The tone is the same either way. That is the core problem with treating a model's confidence, or the polish of its output, as a proxy for accuracy: the two aren't connected.
This is why "the AI sounded sure" isn't a verification step. A firm's workflow has to treat every AI-drafted citation, quote, or factual claim as unverified until a human traces it back to a real source. Confidence in tone is a writing quality. Accuracy is a research quality. Legal work depends on the second one.
The Scale of the Citation Problem, Measured
This isn't a hypothetical risk. Law professor Damien Charlotin maintains a public database tracking court cases in which a filing contained AI-fabricated or misrepresented legal citations. As of September 9, 2026, the tracker had logged 2,035 such cases across more than 50 countries and territories, with the United States, Canada, and Australia accounting for the largest shares.
The breakdown inside that number is worth sitting with. Pro se litigants without counsel account for the largest single group, at 1,171 cases. Lawyers and law firms, not just self-represented parties, are responsible for 810 of the logged incidents, and judges themselves account for 31 more. Case law citations account for 1,829 of the logged problems, the largest single subject category, ahead of legal norms, exhibits, and doctrinal citations combined. This has already reached deep into represented practice. It isn't confined to unrepresented filers experimenting with a chatbot.
What a Fabricated Citation Actually Costs
Courts have already started responding, and the sanctions are real. In March 2026, the U.S. Court of Appeals for the Sixth Circuit sanctioned two attorneys in Whiting v. City of Athens after their briefs contained more than two dozen fake citations, along with a misrepresentation of the district court record. The panel found the attorneys had cited cases that didn't exist, that lacked the language attributed to them, or that didn't support the propositions cited. Worth noting precisely: the court's show-cause order asked the attorneys to disclose whether generative AI or a ghostwriter had produced the citations, but the sanctions themselves rested on a narrower, arguably more useful principle, that no filing may contain a citation a lawyer hasn't personally read and verified, regardless of how it was generated. The court didn't need to prove AI caused the errors to sanction the conduct, which is exactly the point: a verification workflow has to catch a fabricated citation whether a person or a model produced it.
The court ordered them to reimburse the opposing side's attorney fees, pay double costs under Federal Rule of Appellate Procedure 38, and pay $15,000 each in punitive sanctions, calling the underlying conduct an abuse of the adversary system rather than mere drafting carelessness.
Regulators are responding too. In March 2026, the State Bar of California's ethics committee opened public comment on amendments to the state's Rules of Professional Conduct that would make AI verification a binding, disciplinary duty rather than a best practice. The proposed comment to Rule 3.3 requires lawyers to "verify the accuracy and existence of cited authorities, including ensuring no cited authority is fabricated, misstated, or taken out of context, before submission to a tribunal," and a parallel amendment to Rule 1.1 requires lawyers to "independently review, verify, and exercise professional judgment regarding any output" generated by AI tools used on a client matter. That is the direction the profession is moving: verification is turning from a best practice into a binding duty.
Purpose-Built Legal AI Tools Are Not Automatically Immune
It would be convenient if switching from a general chatbot to a legal research platform solved the problem outright. It doesn't, on its own. Stanford and Yale researchers ran a benchmark study, with queries executed between March and April 2024, testing leading AI-powered legal research tools against real legal questions. One of the most widely used tools hallucinated more than 17 percent of the time and produced accurate, well-grounded answers about 65 percent of the time, the same benchmark of legal research questions behind both figures.
That is a real improvement over general-purpose chatbots tested in the same study, but it's nowhere near zero. An unverified answer from any AI research tool, purpose-built or not, is still a draft that hasn't been checked yet.
The lesson isn't that legal AI tools are unreliable in some disqualifying way. It's that the tool a firm chooses matters less than the verification workflow wrapped around it. A well-governed system built on a strong model, with human review and source tracing built into the process, will consistently outperform an ungoverned deployment of the best model on the market.
Building Verification Into the Workflow, Not Bolting It On
The firms getting this right aren't asking associates to double-check AI output out of general caution. They design the workflow itself so that every AI-assisted deliverable, whether it is an NDA review, a due diligence summary, a closing checklist, or a first-pass deposition outline, produces output a human can trace back to its source document or citation before it moves forward. The AI system surfaces the underlying clause, case, or contract language it relied on alongside its conclusion, and a qualified reviewer signs off before anything reaches a client or a court.
That is a design decision, not a training slide. It has to be built into how a matter intake tool routes work, how a document review workflow flags low-confidence extractions for a second look, and how a closing checklist gets assembled so nothing skips human sign-off because it looked finished.
Where Governed AI Fits
This is exactly what governed AI is designed to solve. North OS is built around the principle that AI-generated work product has to be traceable: every output is tied back to the source material it drew from, so a reviewing attorney can verify a citation or a contract clause in seconds rather than re-researching it from scratch. North Deploy builds the firm-specific controls, approval steps, and workflow routing around that foundation, so verification is built into how work moves through the firm rather than a step someone has to remember to add. The skills library extends that same governed, source-traceable approach to firm-specific playbooks, from contract review standards to matter intake criteria, instead of leaving each practice group to build guardrails from scratch.
What to Require Starting Now
Don't wait for a sanctions order to build a verification workflow. Firms should require that every AI-generated citation, quote, or factual claim in a client-facing or court-facing document be traced to its source before it goes out, the same standard California is moving toward making mandatory. Verification should be a workflow requirement enforced by the system rather than a professional courtesy left to individual judgment under deadline pressure.
Firms that already have an AI adoption plan in place should revisit it with this lens specifically: where does an AI-drafted citation or factual claim currently reach a client or a court without a documented human check against the original source? That gap is the one worth closing first. For a broader look at building that kind of implementation plan firm-wide, see North's practical guide to firm-wide AI implementation.
Sources: AI Hallucination Cases Database, Damien Charlotin (accessed September 9, 2026); Whiting v. City of Athens, U.S. Court of Appeals for the Sixth Circuit opinion (March 13, 2026); Lawyers Citing Nonexistent Cases Ordered to Pay Opponents' Attorney Fees, Double Costs, $15K Fine Each, Reason/The Volokh Conspiracy (March 14, 2026); Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Magesh, Surani, Suzgun, Dahl, Manning, and Ho, Stanford Law School (fieldwork March-April 2024); Proposed Amendments to the Rules of Professional Conduct Related to Artificial Intelligence, The State Bar of California (March 2026).