Gen AI Cut Lawyers’ Drafting Time in Half, UK’s Ashurst Says

Ashurst lawyers using generative AI were able to write the first draft of briefs in about half the time it took them on their own, the UK-based law firm found in an in-depth trial of several artificial intelligence tools.

The Silver Circle firm set out to create its own framework for testing legal tech tools because of “a lack of anyone else having done that,” Tara Waters, a London-based partner and the firm’s chief digital officer, said. It gives the firm a basis for evaluating AI tools going forward, she added.

Law firms have been grappling for more than a year with the challenges posed by generative AI. In the absence of standard evaluation tools, they are relying on feedback from their attorneys and working with outside consultants and vendors to address the AI shock. Ashurst took formal quantitative approach to its trials, and then took it a step further by making the results public.

Ashurst conducted an experiment from November to March to see how much AI helped its lawyers do their jobs, looking at factors like time saved, accuracy, and quality of output.

“Ultimately, this all feeds into our investment processes,” Waters said. “We want to make sure that if we were pushing hard to invest fast, and potentially invest reasonably large quantums, that we had all of the data that we could possibly have to hand to support our recommendations.”

The Ashurst trial didn’t involve any client data or work product, but it was broadly based. The firm tapped more than 400 staff to participate, including lawyers from all levels and practice areas, as well as business staff. Once it moves past testing AI tools, it intends to begin using the technology for client-facing work, Waters said.

It tested three products: One tool specifically designed for lawyers, one more general-purpose AI tool, and a tool more narrowly designed for a specific legal task. The firm said it couldn’t share the names of the tools tested.

Some of the testing was “blind,” with a four-lawyer panel judging UK case summaries prepared by humans and AI, without knowing which was which.

The firm’s findings included:

• Using generative AI tools, lawyers saved 45% of the time they said it would otherwise have taken them to write a first draft of a legal brief—about 2.5 hours saved per briefing draft, the firm said in a release. They also saved 80% of the time it would have taken them to draft UK corporate filings “requiring review and extraction of information from articles of association"; and 59% of the time it would have taken them to draft reports about industries and sectors using company filings.
• Output produced by generative AI was at least as accurate, or legally correct, as a human lawyer’s first draft in 67% of cases. Human output had higher average scores for accuracy, and the AI content had greater variations than human-created content in how accurate it was.
• The panel was unable to identify half of the AI-created output as coming from AI, but it made no mistakes on identifying human-created content.

“I personally believe the market is still quite nascent” for generative AI legal tech tools, Waters said, adding that she expects a quality improvement as the second- and third-generation tools come to market.

Now Ashurst is thinking about whether it wants to continue using third-party tools or build its own, and how it can differentiate itself from its competitors. “If we all buy the same tools, we’re all in the same boat,” Waters said.

As for billing, Waters said she expects AI’s direct impact on legal fees to be two to five years away. It will take time for the industry to have sufficient data on generative AI’s time savings, and adjust billing accordingly.

Even farther out are changes in how the industry hires, and the model of a firm’s pipeline to move employees from associate to partner, Waters predicted.