In Depth
Ordinary QA checks that a system does what it's supposed to. Red-teaming checks what it does when someone tries to make it misbehave. The two are not interchangeable: an agent can pass every happy-path test and still hand over confidential data to a cleverly worded prompt, or produce fluent fabrications the moment it's pushed past the edge of its knowledge. Those failures only surface when you go looking for them adversarially.
Good red-teaming is systematic, not a one-off brainstorm of nasty prompts. It works from a map of how the specific agent can fail (what it has access to, who talks to it, what it's authorized to do) and probes each path. The OWASP Top 10 for LLM Applications gives a useful checklist of the categories to cover, including prompt injection, sensitive-information disclosure, and overreliance on model output. The output isn't a pass/fail stamp. It's evidence: here are the inputs that made the agent fail, here's how often, here's how bad.
And red-teaming isn't done once. A model update, a new tool the agent can call, or drift in the world it operates in can reopen a failure mode you'd closed. Mature teams treat it as a recurring control rather than a launch checkbox.
What It Looks Like
A team red-teaming a clinical-summary agent doesn't just ask it to summarize a chart. They feed it charts with contradictory entries to see if it invents a resolution, embed instructions inside the note text to see if it follows them, and request summaries of records the user shouldn't be able to access. One probe reveals that a particular phrasing makes the agent confidently include a medication that isn't in the source. That finding becomes a fix and a documented control, caught in a test instead of in a patient's record.
Why It Matters For AI Vendors
Red-teaming converts unknown risk into known, fixable risk, and that pays off in a few places. It hardens the product, so fewer failures reach production. It produces the evidence enterprise buyers now demand in security review, where they increasingly want to see red-team results rather than just hear that testing happened. And it informs insurability: an underwriter can price an agent whose failure modes have been probed and bounded far more confidently than one that's a black box.