AI Red Teaming: Evasion Attacks, Jailbreak Testing, and Proving a Model Is Safe

Every AI system ships having passed its tests. That is exactly the problem: functional tests ask "does it do the right thing when asked properly?": a question attackers never ask. AI red teaming asks the opposite question, and it is the discipline that separates a model you hope is safe from one you can show is safe. CompTIA SecAI+ Domain 2 expects you to know both the attacks and the method. Here's both.

Why normal testing misses everything

Traditional software testing works because software is deterministic: same input, same output, and a test that passes today passes tomorrow. Machine-learning systems break all three assumptions. The same prompt can produce different outputs; behavior depends on inputs no test suite enumerates; and the model's failure modes were never designed, they emerged from training.

That leaves a gap with a name. A model's operational envelope is where it was tested; the attack surface is everywhere else. Red teaming is the systematic exploration of everywhere else.

Evasion attacks: fooling a model at inference

The classic adversarial attack is evasion: modify the input, not the model, until the classifier is wrong. The canonical demonstration is an image altered by a perturbation invisible to a human that flips a confident classification entirely. It works because a model's decision boundary is not the human one; it is a high-dimensional surface with edges no one designed, and evasion is the search for the nearest edge.

For security teams the stakes are concrete: malware crafted to slip past an ML classifier, spam shaped to score as benign, a face or plate that a recognition system misreads. The exam cares that you can distinguish the flavors:

The transfer property is the one that surprises people. Adversarial examples generated against a model you can access frequently work against one you can't, because models trained on similar data learn similar boundaries. Secrecy buys less protection than it feels like it should.

It helps to see why this is hard rather than sloppy. A classifier draws a boundary through a space with thousands of dimensions, fitted to the data it saw. Almost all of that space contains inputs no training example ever occupied, and the boundary's shape out there is essentially unconstrained: an artifact of the fit rather than a decision anyone made. Evasion is simply the search for the shortest path from a correctly-classified point to that unconstrained region. The attack isn't exploiting a bug in the code; it's exploiting the geometry of how learning works, which is why patches don't apply and robustness has to be trained in.

That distinction matters for how you talk about findings. When a red team reports an evasion result, the fix is rarely "correct this input." It's retraining, ensembling, adding a detection layer, or accepting a documented risk: decisions with cost and schedule attached, which is why the report has to reach engineering leadership and not just the model owner.

Jailbreaks: evasion for language models

For LLMs the same idea wears different clothes. A jailbreak doesn't perturb pixels; it reframes the request until the model's refusal behavior stops applying. The recurring families are worth knowing by name because they generalize across models:

• Role-play and persona framing: "you are an actor playing a character who explains…" The model's safety training is attached to contexts, and a fictional frame is a different context.
• Instruction hierarchy attacks: text asserting higher authority than the system prompt ("system override", "developer mode"), exploiting the fact that a system prompt is not a security boundary.
• Encoding and obfuscation: base64, leetspeak, translation, token splitting. Safety filters often key on surface patterns; the model still understands the payload.
• Multi-turn escalation. Nothing in any single turn trips a filter; the conversation as a whole walks the model somewhere it would have refused in one step. This is the family automated scanners most often miss.
• Payload splitting: the request is assembled from fragments the model concatenates itself.

The method: red teaming as a process, not a vibe

Anyone can poke at a chatbot for an afternoon. What SecAI+ tests is whether you can run this as a repeatable engagement with results you can act on:

Two details separate a useful engagement from theater. First, scope must define failure in business terms before testing starts: "produces disallowed content" is weak; "discloses another customer's data, executes an unapproved transaction, or generates advice we'd be liable for" is testable. Second, findings must escalate. A model saying something rude is a finding; a model saying something rude and being wired to an email tool is an incident waiting to happen. Chaining is where red teaming earns its budget, and where it meets agentic AI risk.

Automation and humans do different jobs here. Automated harnesses replay known jailbreak corpora and fuzz variations at a scale no person can match. That's your regression net. Humans find the novel framing, the domain-specific abuse case, the multi-turn path that no corpus contains. Run the automation continuously in CI; bring humans in for depth before major releases.

Measuring the result without fooling yourself

A red-team report that says "we found 14 jailbreaks" is nearly useless: 14 out of how many attempts, against which behaviors, at what severity? Mature programs report an attack success rate against a fixed, versioned attack set, broken out by category, so the number moves for real reasons and can be compared release over release.

Three traps distort that number, and the exam-relevant instinct is to distrust a clean result:

• Overfitting to the test set. Patch the exact prompts the red team used and the score improves while the vulnerability class remains. Always retest with held-out variants of the same technique.
• Judging with the model. Using an LLM to grade whether output was harmful is scalable and standard, but the judge is itself manipulable and has its own blind spots. Sample and human-verify.
• Non-determinism. A prompt that fails once may succeed on the third attempt; single-shot testing systematically under-reports. Run each case repeatedly and report the rate, not a binary.

The pattern to internalize: absence of evidence is not evidence of absence. A passing red-team result means your current techniques didn't break it: not that nothing will.

Where this sits in a real program

Red teaming is not a pre-launch ritual; it's a control that runs alongside others. Before release it validates the safety claims you plan to make publicly. In CI it acts as regression testing, catching the model update or prompt tweak that silently reopens a fixed issue. In production it becomes monitoring: the same attack signatures your red team wrote become detections, so probing attempts surface as an alert instead of a post-incident discovery.

That last handoff is what turns testing into defense. The output of an engagement should not just be a document. It should be new test cases in the pipeline, new detection rules in the SOC, and where a finding can't be fixed, a documented risk acceptance with an owner. Which is exactly how governance (SecAI+ Domain 4) expects the loop to close.

What you can actually do about it

Adversarial robustness has no silver bullet, and the exam rewards saying so. What exists is layering:

Underneath all of it sits the honest framing the exam wants: robustness is a risk-reduction exercise, not a solved problem. You are raising the attacker's cost and shrinking the blast radius, not eliminating the failure mode. That's also why continuous monitoring matters as much as pre-release testing: a technique published next month will work against the model you shipped last month.

Red teaming is where SecAI+ Domain 2's testing and attack-analysis objectives meet. See how attackers reach the model in prompt injection explained, what they do once the model can act in securing agentic AI, or work the whole domain with our SecAI+ Domain 2 study guide.

#SecAIplus #AISecurity #AIRedTeaming #AdversarialML #Jailbreak #LLMSecurity #EvasionAttacks #MLSecOps #CompTIA #CyberSecurity