Jailbreak Resistance and Red-Teaming in Practice

What a Jailbreak Actually Is

A jailbreak is a prompt specifically crafted to make a model bypass its safety training and produce a response it would normally refuse. This is distinct from prompt injection, covered in Prompt Engineering Advanced: injection is about untrusted content being mistaken for instructions, while a jailbreak is a direct attempt, by the user themselves, to talk the model out of its own trained restrictions. The two can combine in practice, but they’re different mechanisms worth keeping distinct.

Academic taxonomy work on jailbreaks — surveyed in Rao et al.’s 2023 formalization of known jailbreak methods — documents several recurring categories at a conceptual level: role-play framing (asking the model to adopt a persona described as having no restrictions), hypothetical or fictional framing(wrapping a request inside a story, thought experiment, or “what would a character say” setup), and educational or research framing (presenting a harmful request as existing purely for study purposes). This page describes these categories conceptually, as a field of study — not as working prompts, since the point here is understanding the defense landscape, not providing an attack toolkit.

Source: Rao et al., “Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks,” arXiv:2305.14965

Red-Teaming: Trying to Break It Before Users Do

Red-teaming means deliberately attempting to break a system — surfacing jailbreaks, harmful outputs, and other failure modes — before it’s deployed, rather than discovering them in production from real users. Both major labs treat this as a standard, published part of shipping a model, not an optional extra.

Anthropic’s foundational paper, “Red Teaming Language Models to Reduce Harms,” tested red-teaming across multiple model sizes and training approaches, and released a dataset of 38,961 red team attackspublicly for others to analyze. A key finding: models trained with RLHF (reinforcement learning from human feedback) got harder to successfully red-team as they scaled up in size, while other training approaches didn’t show the same improvement.

Source: Anthropic, “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”

More recently, Anthropic’s Constitutional Classifiers research describes over 3,000 hours of red teaming against an early classifier-guarded model, reporting that no red teamer found a universal jailbreak able to extract harmful information at a level comparable to an unguarded model across most target queries. Deploying the classifiers came with a measured cost, reported precisely rather than glossed over: an absolute 0.38% increase in production-traffic refusals, and 23.7% additional inference overhead.

Source: Anthropic, “Constitutional Classifiers: Defending Against Universal Jailbreaks across Thousands of Hours of Red Teaming,” arXiv:2501.18837

OpenAI publishes its own, separate approach: an external Red Teaming Networkof outside experts, used alongside internal testing. OpenAI’s GPT-4o system card reports over 100 external red teamers across 29 countries speaking 45 languages, specifically testing (among other things) resistance to jailbreaks.

Source: OpenAI, “OpenAI Red Teaming Network” · OpenAI, “OpenAI’s Approach to External Red Teaming”

A Structured Guideline: Building Defense in Depth Into a Real System

Putting the Intermediate page’s layers together with red-teaming gives a concrete, ordered guideline for a production system — five named layers, roughly in the order a request and response actually flow through them:

1. 1. Input validation and filtering

   The first checkpoint, before any request reaches the model. Blocks known-bad patterns and malformed input cheaply — but only recognizes what it was built to recognize, so it is never the only layer.

2. 2. System-prompt guardrails

   Explicit instructions shaping intended behavior. Cheap and effective for ordinary use, but — as covered on the Intermediate page — not a hard security boundary, since a crafted input can override instructions delivered in the same channel.

3. 3. Real-time classifier / safety-layer checks

   A dedicated model or classifier, separate from the main model, screening exchanges for jailbreak attempts as they happen — the layer Anthropic’s Constitutional Classifiers research describes in production detail below.

4. 4. Output filtering

   Checking the model’s actual response before it reaches the user — catching harmful content or leaked sensitive data regardless of how the model arrived at producing it, as covered on the Intermediate page with the PII-masking example.

5. 5. Red-teaming before deployment, and monitoring after

   Deliberately trying to break the system before real users do, then continuing to monitor production traffic for new attack patterns after launch — since a system red-teamed once at launch time can still face novel attacks discovered later.

No single layer here is sufficient on its own — that repeats a theme from the Intermediate page deliberately, because it’s the entire justification for building five layers instead of picking the one that sounds strongest and stopping there.

The Cost/Reliability Trade-off of Stricter Guardrails

Guardrails aren’t free to tighten. Anthropic’s own measured numbers above show this precisely: reducing jailbreak risk came bundled with a real, non-zero increase in refusals on ordinary production traffic, plus additional compute cost per request. This is a documented, general tension in the field — stricter filtering catches more genuine attacks, but it also flags a larger share of legitimate requests that merely resemble a blocked pattern.

Move the slider below to see the qualitative shape of this trade-off. The specific labels are illustrative, not measured data from any particular system — the point is the direction and shape of the curve, which is real, not any specific number:

There is no setting that eliminates both risks at once. The practical response isn’t searching for a mythical zero-trade-off setting — it’s picking a strictness level deliberately for the specific application’s risk tolerance, and using the layered, named defenses above (rather than one blunt filter cranked to maximum) so that tightening one layer doesn’t have to mean over-refusing everywhere at once.

Fun Fact

Anthropic’s Constitutional Classifiers research reports that its early prototype classifier initially over-refused harmless queries and used excessive compute — meaning the very first version of a defense built specifically to fight the strictness trade-off still had to be iterated on to actually reduce it, rather than just moving the problem around.

Test Yourself