Keep Judges and Lawyers Out of Legal Predictions, Tech CEO Says

Welcome back to the Big Law Business column, written by me, Roy Strom. Today, we talk with the CEO of a legal prediction software company about why he thinks judges and lawyers shouldn’t be evaluated in predictive modeling.

Not too long ago, I wrote about a tech start-up that says it knows the best real-life lawyer for appeals of U.S. patent infringement cases.

Sure, the tool itself, called Ex Parte, was interesting. Using technology to predict legal outcomes is a potential game-changer. But there was another story: Some of the lawyers the tool said were the best fit for real cases felt uneasy about the whole thing.

Maybe lawyers don’t want to live up to the pressure of the algorithm. Or maybe they don’t like the concept of singling out a “best lawyer” for a particular case. Maybe, coming from a computer, the individualized predictions felt too brazen. “What does this algorithm know about me?” lawyers asked.

This is how lawyers feel, but how would the layman react?

Everyone knows the lawyer you hire matters. And, yes, different judges have different views. But if an algorithm someday quantifies precisely just how important those factors are in run-of-the-mill legal disputes, there’s a good chance that the general public will also feel uneasy about the legal system’s fairness.

That is one reason why Benjamin Alarie, a University of Toronto Faculty of Law professor and CEO of legal analytics company Blue J Legal, argues to keep the individual tendencies of judges and abilities of lawyers out of the algorithms that he has said will one day eliminate “99 percent of legal uncertainty.”

“The legal system is supposed to operate the same for everybody regardless of the judge you get or the lawyer working on your case,” Alarie said in an interview.

Alarie and his co-founders at Blue J Legal, fellow University of Toronto professors Albert Yoon and Anthony Niblett, have argued machine learning can predict how courts will decide legal disputes more cheaply and accurately than humans. They have predicted a “legal singularity” and spelled out how Blue J Legal is already predicting outcomes in Canadian tax disputes.

Today, Alarie says his company’s algorithms are, on average, about 90 percent accurate for predicting the outcomes of tax disputes. The company’s customers include the Canadian government and nine of the 10 largest accounting firms in Canada, Alarie says.

Rather than analyzing already-filed cases, as Ex Parte does, Blue J Legal’s software allows users to build a case by answering 15 to 30 questions about their tax dispute.

Some analytics companies are focused on understanding procedural aspects of particular courts—how long judges take to rule or how frequently they side in favor of a plaintiff. Bloomberg Law’s own Litigation Analytics product functions in a similar way.

Alarie likes to say his company, which recently launched a U.S. tax law product, is capturing “the merits” of the case and making predictions off of that.

Alarie admits that including information on individual judges or lawyers could make the tool more accurate, but he believes it would only make a slight difference. And the company has other ethical reasons to keep that information out of its models. Alarie referenced a recent ban on judicial analytics in France as a sign that judges are reluctant to have their work pored over by analytics firms.

“If you feel like cases are being decided on irrelevant grounds, then that is going to tend to undermine the public’s confidence in the rule of law,” Alarie said.

Some proponents of judicial analytics have said that showing judges how they compare to their peers will encourage them to move toward the mean and will help eliminate biases. Alarie thinks that can be accomplished without analyses of individual judges.

In a 2018 presentation Alarie predicted that algorithms like those Blue J Legal uses in tax and employment law would be expanded to bring “absolute clarity to every area of the law, everywhere and on demand” based on cases’ merits.

But that would not come without growing pains. Alarie said that when AI drives a fuller understanding of the impacts of the legal system, it would be akin to waking up with a hangover after throwing a party in your home.

“You look around and in the light of day everything is a complete mess. I think that is very likely to happen when we start having [mature] legal technology,” Alarie said. “We will look around and go, did anybody know that this was the pattern of consequences associated with our current legal system? Did anyone actually realize that things were as messy as they actually are?”

Worth Your Time

On Bankrupt Law Firms: More details have emerged on the failure of LeClairRyan. A Chapter 11 bankruptcy filing shows the firm entered into a loan agreement in 2017 that the lender said was in default by mid-July this year. It has people thinking about what a recession might mean for teetering firms.

On Law Firm Mergers: The much-discussed transatlantic tie-up between Allen & Overy and O’Melveny & Meyers was called off in a surprise announcement that some are anticipating will lead to defections from the U.S.-based O’Melveny.

On White Collar Practices: Nixon Peabody hired a former acting ICE director as the law firm diversifies its white collar practice (including a focus on qui tam cases) to better align with Trump administration prosecution preferences.

On Big Law Prosecutions: A jury found former Skadden, Arps, Slate, Meagher & Flom partner Greg Craig not guilty of scheming to mislead federal officials about his work for a Russia-aligned regime of Ukraine’s government.

That’s it for this week! Thanks for reading and please send me your thoughts, critiques, and tips.