AI-Powered Legal Tech Is a High-Stakes Buy. Here’s How to Vet It

For in-house legal ops pros trying to decide what tech to buy, there’s a checklist to review before spending company money.

It’s a core job function for the professionals, many of whom essentially serve as their legal department’s chief operating officers. And trying to decide on a contract lifecycle management or e-discovery tool comes with higher stakes than buying a stapler or printer.

“If they’re selling you the Cadillac, it’s probably the Camry, and you should lift the hood up,” said Judy Lao-Klinker, director of legal operations at Prologis, a real estate investment trust.

Picking the right legal tech means asking vendors the right questions, comparing notes with their existing customers, and potentially test-driving the software. Here’s what legal ops professionals in interviews and panelists at LegalWeek said to keep on your checklist:

Due Diligence

As with any big purchase, doing your homework is the first step to evaluating legal tech tools. Researching vendor information security practices, how long they’ve been in the field, and what’s needed to train new users are some of the key questions.

Data privacy is a pressing concern, especially as more vendors are using generative artificial intelligence and legal professionals worry their data is being used to train large language models like ChatGPT.

The AI boom adds another level of consideration. New technologies require additional training, so buyers need to consider how long it will take their team to get up to speed on a new tool.

The vendor-vetting process should involve team leaders with different fields of expertise, including information security, information technology, compliance, risk and ethics, Gregory Harvey, a data privacy lawyer, said during a LegalWeek panel.

“Doing this as a legal department helps model this for the rest of the organization,” said Harvey, senior counsel for data privacy and cybersecurity at National Grid Plc, an energy company.

Above all, legal ops leaders should identify specific tools that address specific needs, Jessica McDonald, manager of legal operations at XPO, said on the panel with Harvey. Adding vendors can bring additional risk, so buyers need to weigh the benefits they’re getting against any possible downsides.

“Don’t go out with the concept of ‘I need AI because everyone’s talking about AI,’ ” McDonald said.

Benchmarking

Gen AI in legal tech may still be in its infancy, but it’s been around long enough for some third parties to begin measuring its accuracy and effectiveness. A study conducted by the startup Vals AI released earlier this year examined tools built by Harvey, Thomson Reuters, LexisNexis, vLex and Vecflow that are being sold to law firms and corporate legal departments.

(Bloomberg Law also makes tools that use artificial intelligence, but it didn’t participate in the Vals study.)

Another study published by the University of Minnesota Law School and the University of Michigan Law School compared how vLex’s Vincent AI tool enhanced the work of law students.

Lawyers welcomed the quantitative insights provided by the studies, which they said brought much-needed insight to the legal AI market. But there are limits to how much insight those studies can really bring, according to Iria Giuffrida, a William and Mary Law School professor. She commended the authors for their contributions, but said law is both an art and a science, and distilling outputs into right and wrong answers can blur some artistic and human elements.

“You judge effectiveness in the context of litigation,” Giuffrida said in an interview. “You measure it by success.”

Customer Reviews

To capture that real-world output, tech buyers should also look at how well a tool performed at other companies of similar size.

“The real test comes with actual performance live with customers,” Brandon Gleklen, an investor at Battery Ventures, said at a LegalWeek event.

Demonstrations are great—and benchmarking has some value—but the best way to get an idea of how well something actually works is to hear about it from colleagues, legal ops professionals said.

Vendor websites tout customer reviews to show off their successes.

Pilot Programs

Finally, running a pilot program or a proof of concept trial can be a good check that a tool works for your company.

JPMorgan Chase was able to walk away from a tech tool after running a proof of concept trial, said Jamal Brown, the financial giant’s executive director for legal operations and knowledge management. The bank started using the tool after the vendor, which he didn’t name, gave a fantastic pitch.

But after a few months, things weren’t going well, and JPMorgan decided to ditch the tool, Brown said.

“If we had continued to try to hit that nail over the head with the same hammer, it would have just been catastrophic,” he said.