OpenAI Pushes Prompt-Hacking Defense to Deflect Copyright Claims

OpenAI Inc. is making manipulation of ChatGPT central to its defense against a wave of copyright lawsuits from publishers and authors—accusing some of “prompt hacking” and seeking vast troves of documents to learn exactly what questions they fed into the artificial intelligence tool to yield the outputs underlying their claims.

Access to the black box of plaintiffs’ chatbot prompts could enable OpenAI, usually named as a co-defendant with Microsoft Corp., to bolster its assertions that outputs regurgitating protected works are “highly anomalous results” only possible through tens of thousands of attempts and “deceptive prompts.”

The discovery strategy could allow OpenAI to stockpile prompts it says underscore the manipulation argument, potentially strengthening its case that training AI tools using copyrighted material isn’t infringing under fair use doctrine—or at least limiting damages if jury verdicts go against it—multiple intellectual property attorneys said.

The strategy is likely geared toward potential trials in more than a half-dozen lawsuits, and the company’s word choice could be a “rhetorical effort” to sway jurors by deflecting from the copyright claims, said Justin Hughes, a professor at Loyola Law School.

“Prompt hacking is incredibly clever rhetoric,” Hughes said, because in the “hipster and tech community, you say ‘hack’ and it sounds OK. But to the broader audience, like jurors, you say ‘hack’ and it sounds bad.”

Soon after New York Times Co. brought its lawsuit alleging the chatbot near-duplicated published stories, OpenAI hit back with accusations the newspaper paid someone to “hack” ChatGPT models to generate outputs resembling old articles.

The AI behemoth has accused three other plaintiffs of manipulating ChatGPT to produce examples supporting their copyright claims, and sought broad discovery from four total parties about their pre-suit investigations.

OpenAI’s strategy took a hit when a group of authors led by comedian Sarah Silverman mostly escaped its request. A California federal judge on Aug. 8 ruled they had to turn over only the prompts, outputs, and account settings related to examples their complaint cited, not for all of their queries. But similar requests remain pending in three other cases, including a motion to compel the Times to turn over documents before the US District Court for the Southern District of New York.

How the Times was able to generate its examples is “a central question” in the case because they “appear to be the fruits of Plaintiff’s prolonged and extensive efforts to manipulate the ChatGPT LLMs,” OpenAI told Judge Sidney H. Stein.

Diverting attention to hacking claims or how many tries it took to obtain exemplary outputs, however, avoids addressing most publishers’ primary allegation: AI tools illegally trained on copyrighted works.

“If it takes 100 tries to get one, but the one is still evidence that the model has, through statistical relationships of words, has effectively memorized the New York Times article or memorized two pages of the Silverman book, it doesn’t matter how many times it took,” Hughes said.

OpenAI didn’t respond to requests for comment.

Developing Manipulation Claims

OpenAI notified the US District for the Northern District of California in September 2023 it planned to seek discovery in Silverman’s suit regarding “all efforts” by plaintiffs to prompt ChatGPT to suggest outputs similar to their works.

After the authors amended their complaint in March 2024 to include examples of ChatGPT outputs summarizing parts of their writing, OpenAI accused them of manipulating the chatbot.

OpenAI is seeking similar documents in suits brought by the Authors Guild and the publishers of the New York Daily News, Chicago Tribune, and other news outlets. It made similar manipulation allegations against Intercept Media Inc. and Daily News LP.

Intercept disputed the need for regurgitation examples when OpenAI moved to dismiss its case in April, citing a lack of specificity.

“When the New York Times made such allegations, Defendants accused them of nothing less than computer hacking,” the news outlet’s opposition said.

Judge Jed S. Rakoff of the Southern District of New York ordered Intercept in June to amend its complaint to rectify “the seeming lack of specificity.” Three weeks after it did, OpenAI accused it of “prompt manipulation.”

That tactic seems to be geared less toward disputing the legal merits of plaintiffs’ copyright claims and more to trial strategy, said Knobbe Martens partner Mark Lezama.

“You want to be able to show the jury ‘Look, the plaintiffs here, they’re trying to sell you something that doesn’t quite live up to what they’re promising,’” he said. “‘What they haven’t told you, for every time they got a response they liked there were 100 or 1,000 other prompts they inputted that generated responses they didn’t like.’”

Sandra Aistars, a professor at George Mason University’s Antonin Scalia Law School, said plaintiffs querying AI chatbots isn’t nefarious. “They’re simply seeking the information that any plaintiff would seek in order to bring a lawsuit,” she said.

Since they’ve gotten the cold shoulder when seeking information about how AI developers trained their models, Aistars said, “they have no choice, I think, but to query the system itself in order to get the evidence that they need in order to assert a claim under the copyright laws.”

Arguing Fair Use

OpenAI’s effort may also be aimed at supporting its argument that using copyrighted works without permission to train ChatGPT qualifies as fair use, Hughes said.

One of the four fair use factors is how it may affect the market for the protected works. Plaintiffs want to show the model is generating, and is prone to generate, text that could be used as a substitute for their works, he said.

“OpenAI, in turn, wants to prove this is a novel occurrence, this is extremely rare, this is something done by a hack by the plaintiffs,” Hughes said.

Showing how rare occurrences of seemingly copied outputs are could demonstrate the models are designed to—and usually do—avoid regurgitating content, said Robert Brauneis, a professor at the George Washington University Law School.

If AI companies prove it’s really hard to make their tools produce essentially similar content, a court might say, “it really is a fair use because it’s an absolute rarity, and maybe only when intentionally induced to do so will the model output something infringing,” Brauneis said.

Even if a court or jury decides OpenAI’s actions don’t qualify as fair use, evidence showing how infrequent regurgitations are could be used to minimize damages—which can range from $100 to $150,000 per article—or prevent an injunction, said Vanderbilt University Law School professor Daniel Gervais. OpenAI could argue that an injunction isn’t needed because average users can’t use its system how the plaintiffs did, he added.

Looking Ahead

Plaintiffs have fought OpenAI’s ChatGPT testing discovery requests. The Times called the prompt-hacking allegation “as irrelevant as it is false” in its March dismissal opposition.

“OpenAI’s true grievance is not about how The Times conducted its investigation, but instead what that investigation exposed: that Defendants built their products by copying The Times’s content on an unprecedented scale—a fact that OpenAI does not, and cannot, dispute,” it said.

The Times’ Aug. 9 filing noted the Silverman case decision, in which plaintiffs successfully argued the methodology and data used to test the chatbot in preparation for their lawsuit qualified as opinion work product and turning it all over would improperly give insight into attorneys’ opinions.

OpenAI previously said it sought prompt examples and collaboration from the Times before the lawsuit to help prevent duplicative outputs.

Aistars said such efforts don’t matter once alleged infringement has already happened.

“Those are all great things that they might do after the fact, but that does not resolve the infringement to begin with,” she said.

Still, Hughes expects OpenAI to keep pushing the strategy.

“All you need is one judge to disagree with the Northern District of California, and then you’ve got a pair of cards,” he said. “For the next judge, you just say ‘pick one.’”