OpenAI built an LLM that hacks its other LLMs

OpenAI built an internal model called GPT-Red to attack its own models, according to MIT Technology Review. That's a shift from red-teaming as a scheduled audit to red-teaming as a running part of how OpenAI ships, per the report.

GPT-Red trains through self-play. An attacker LLM tries to break a target model, the target defends, and the loop repeats, sharpening the attacker's tactics over many rounds.

The focus is prompt injection. OpenAI says the process caught a novel fake chain-of-thought attack, one its human red-teamers hadn't seen before. Training against it produced what OpenAI calls its most defended release yet.

GPT-Red won't ship publicly. It's an internal tool that supplements, not replaces, human red-teamers, per the report.

An attacker model already found an exploit that trained security researchers missed. That should worry anyone who assumes human review catches every sharp edge before ship.

The catch

The report doesn't say how GPT-Red's discoveries get triaged or whether false positives are a problem. It also doesn't say how much compute the self-play loop costs against a human red-team engagement. Those numbers would say whether this scales to smaller labs or stays an OpenAI-only advantage.

Watch whether other labs disclose similar internal tooling in the coming months. Right now, self-play red-teaming looks like a moat only OpenAI has built.