Cheating Scandal Aside, New Remote Bar Looks a Lot Like Old One

America’s budding lawyers aren’t cheaters. They just need help taking online tests.

At least, that’s the opinion of bar exam authorities in the biggest states and the company that created the remote proctoring software used during the first-ever online bar exams of the pandemic era. Despite thousands of students being initially flagged for possible cheating and then exonerated, bar authorities in California and New York say they don’t plan any major fixes this go around.

“In many ways, we’ve kind of been at the mercy of how the technology is developing,” said Aaron Taylor, executive director of the AccessLex Center for Legal Education Excellence, a nonprofit that works to improve access to law school. “It’s unfortunate, because this is such a high-stakes exam on so many levels.”

Despite congressional demands that test-monitor ExamSoft explain “steps that your company has taken to protect the civil rights of students,” the biggest changes planned in California and New York—which are among more than 30 states holding remote bar exams Feb. 23-24—amount to this: better preparing test-takers to understand how the technology works.

“There were not widespread technical problems on the October 2020 New York Remote bar examination and most of the issues that required a call by an examinee to software support were resolved on exam day and the candidates were able to take the entire exam,” said John McAlary, executive director of the New York Board of Law Examiners.

In California, the exam “flagged” more than 3,000 of the 9,000 test takers for problems, and after a closer look, the bar sent 429 of those people “Chapter 6” notices that they were being investigated for possible rule breaking. In the end, only 47 test-takers were implicated.

“Given the volume of applicants that applied for this exam, 3,000-plus flags was not alarming,” said California State Bar Director of Admissions Amy Nuñez.

Congressional Concern

But critics don’t agree. Six Democratic senators including Elizabeth Warren of Massachusetts, Cory Booker of New Jersey, and Richard Blumenthal of Connecticut, wrote ExamSoft Chief Executive Officer Sebastian Vos demanding to know what changes the company would make after reports that “students—particularly students of color and with disabilities—have faced alarming issues in using the software, been locked out of tests, or wrongly accused of cheating.”

In a response to the lawmakers, Vos denied any major problems, and said ExamSoft investigated claims of systemic software glitches regarding test takers’ race or gender.

“It would be a key priority to resolve any such issues,” Vos wrote, “but we have found none.”

Modifications to Next Tests

Nuñez said the California bar this time will offer mock exams that more closely reflect the software interface used in the exam, particularly for the performance test portion, meant to mimic legal challenges a lawyer might face at work.

The bar also will better explain the process for those who get Chapter 6 notices, “specifically the difference between being issued a notice versus having a notice affirmed,” Nuñez said. “The notices can be modified to spell things out more clearly to applicants.”

And California bar officials are reviewing survey responses from October as well as from takers of the “baby bar” exam in June—given mostly to students completing their first year of law study at specific schools—to see if other tweaks make sense.

ExamSoft has improved how in-test attachments can be viewed, said Britt Nichols, the company’s chief revenue officer. ExamSoft will also now include support for Apple devices using new processors and is implementing other user experience improvements, said Nici Sandberg, associate director of marketing content and communications.

‘It Was Shocking’

One woman who took the bar exam in California in October and got a Chapter 6 notice in December said the accusation “was insulting. It was shocking.”

The notice claimed the woman’s eyes may have been outside of webcam view for a prolonged period during the test.

“My first thought was, ‘Hey, I didn’t do that! What is this?’” said the lawyer, who has worked as an attorney for an AmLaw 100 firm and now wants to be able to practice in California. She spoke on condition that she not be identified for fear of career harm.

Her second thought: the potential implications if California officials made her notify state bar “character and fitness” offices that she had committed a possible misdeed—a fate she was spared when her lawyer stepped in.

That attorney, Erin Joyce of Pasadena, Calif., said most Chapter 6 notices sent to her clients never should have been issued and that the allegations against them resulted from faulty technology. Those include one notice sent to an Hispanic test taker who was told that the facial recognition software used by ExamSoft deemed him “not present” during exams he completed. The man was later exonerated.

ExamSoft said it needed more information to properly respond to the allegation, but said the situation seemed out of the norm.

“It does seem highly irregular, especially since humans from the clients review each file before making such determinations and the client is always the one making final determinations around any anomalies,” Sandberg said in a statement.

How It Works

Here’s how ExamSoft’s testing software works: A student takes a self-portrait with the program as a baseline image, and completes practice exams to become familiar with the software. On exam day, the student takes another selfie and begins the exam.

While the student is taking the test, ExamSoft records audio and video. After the test ends, the student’s encrypted test and audio-video file is uploaded to a secure server.

The audio-video file is then run against an artificial intelligence tool that flags potential anomalies—which can range from leaving the test-taking area to placing food or a drink within view.

A student’s eyes must stay within camera view throughout the 60- to 90-minute sections of the test, which in California in October totaled 9.5 hours over two days, and this month will increase to 12.5 hours for prospective lawyers.

A human proctor separately reviews the recording for any potential anomalies.

The jurisdiction using the software, like a state bar, then does its own review of the recordings and ultimately makes the call about any further investigation.

“ExamSoft makes no adjudication decisions about breaches in academic integrity,” said Vos.

After the California Bar reviews a flagged video, applicants still suspected of wrongdoing are sent Chapter 6 notices that they’re being investigated, and that they have a right to respond in writing.

The bar then conducts a subsequent, more in-depth review of the video, before affirming or dismissing the allegations.

In New York, where 5,150 remote tests were given in October, neither the bar nor ExamSoft would provide details about how many people were initially flagged.

But 13 candidates were the subject of an investigation, said McAlary. Nine have been cleared of wrongdoing, and the other four are pending, he said.

Facial Recognition Bias Concerns

Facial recognition software has been shown to disproportionately misidentify Black and Brown people, said Jason Kelley, a strategist at the Electronic Frontier Foundation, a digital civil liberties group.

ExamSoft doesn’t own the facial recognition software it uses, and Vos said: “Because a human’s always reviewing that blindly—they have no idea what the AI said—that helps us take away that bias” against test-takers of color, those with disabilities, or those wearing religious garments who may be flagged.

Nuñez defended the California bar, noting that October was “our first high-stakes exam conducted online.”

“The technology enabled us to give the exam during the pandemic, ensuring that people’s path to licensing was not obstructed or delayed further,” she said in a written statement. “The alternative would have been to halt the exam, which would have had a negative impact on the legal community at large.”