Skip to Content

Lessons Learned from Using Generative AI

Lawyers share what works, what doesn't and where professional judgement matters most

A person holding an open laptop with "Learning" on the screen.

This is the fourth of six articles in our series on GenAI use in practice.  We thank our contributing counsel for their responses: Denise Brunsdon (DB), Codie Chisholm (CC), Adam Drew, KC (AD), Katrina Edgerton-McGhan (KEM), Jennifer Hawkins (JH), Jivanya Reddy (JR), Ryan Robertson (RR), Marcus Sixta (MS), Dylan Snowdon (DS), Laura Warner (LW), and Jenna Weinkauf (JW).

Law Matters asked: “What have you learned from your engagement with GenAI tools?”

RR: The tools have improved remarkably quickly.  I’ve learned that they can be useful for spotting issues and testing drafts, but their confident answers still need careful verification.

MS: “AI slop” is a real problem. Most of the content generated by AI can be recognized for what it is. AI has a formulaic and obvious writing style, at least at this stage of development. AI generated content should be used as a starting point and generally must be heavily edited and customized to achieve some level of authenticity, otherwise it will be rejected by the reader.

LW: That they are hit and miss. That they are “yes people” and need specific instructions to remain neutral. That their work must be checked and evaluated thoroughly.

KEM: I think AI is a good search and compile engine. It is excellent at taking a prompt like “can you find where witness X said something about Y,” and identifying where that appears in the evidence in a way that does not require inefficient keyword searches.  The tool is a good “devil’s advocate” – every time I have used it to test my argument, it comes up with ideas that I have not thought of that improves the overall product. It can augment my practice without taking over the primary responsibilities for the file or work product.

I also found it a bit intimidating to start to use. I needed a bit of hand holding to get started, but now I use it daily.

JR: GenAI can sound very confident even when it is wrong. It is useful for organizing ideas and identifying issues, but it still requires careful review and professional judgment.

JH: Using GenAI is still time consuming: it takes time to thoroughly anonymize the notes, finesse our prompts, and edit and fact check the resulting summaries of evidence.

AI can make mistakes, such as mis-quoting a participant or focusing on an irrelevant portion of an interview. As such, caution is required.

JW: AI output is unpredictable.  I have found it too unreliable on sourcing to use it in preparing any materials or arguments.  GenAI saves time on low-complexity tasks, but it is no replacement for a qualified counsel who understands the nuances of the law and the local market.

DS: Answers from GenAI tools are often confidently inaccurate or technically correct but problematically misleading.  As a result, it is difficult for junior counsel unfamiliar with an area of law to determine when an error has been generated.

An additional concern about using AI systems within a law firm is that the primary use of today’s tools appears to have the goal of replacing initial research and drafting that is completed by junior lawyers.  However, if junior lawyers are not conducting that research, then they are not being provided with the experience necessary to become knowledgeable and capable senior counsel.

DB: AI reinforces for me that good judgment matters even more when the tools get faster. I find that the use of AI for non-legal tasks can streamline practice organization, leaving more mental energy for the work that requires legal judgment: deciding when and how to use AI, drafting prompts, and completing human review and revision.

CC: I have had to think differently about tasks to break them into their elements and identify appropriate GenAI tools.  Different GenAI tools are better suited to different tasks, and the way you segment the elements of the task significantly affects the quality of the result.

I’ve learned that efficiency isn’t really the goal although GenAI often helps us be more efficient. We need to think about and use the tools to become better lawyers. The real upside is when you use the tool to generate a higher quality work product or identify an issue or argument for further review.

No one tool can do everything, so you must understand the strengths and limitations of what’s available and modify your practice accordingly.

You must stay on top of the newsletters and training because the tools are changing so quickly. What was true 3-6 months ago may not be true any longer.

AD: AI tools seem primed to assist careful lawyers with their practice. On the other hand, the speed with which they produce output and the “confidence” with which they answer questions seem primed to lead careless lawyers into serious errors.