How ATR Detection Benchmarks Actually Work: Every Number, Per Corpus

There is no single ATR precision number, and any benchmark table that gives you one is lying to you. ATR reports each result against a specific corpus, with the corpus size named, and always labeled as Layer 1 deterministic-rule behavior. On NVIDIA garak the Layer 1 rules hit 97.2% recall over 650 in-the-wild jailbreak samples. On a self-built PINT-format corpus of 850 samples they hit 99.7% precision at 63.6% recall. On HackAPrompt's 4,780 samples they hit 69.6% recall against a 28.6% baseline. Those are different numbers because they are different tests. This post walks through every one of them so you can reproduce them and decide for yourself.

ATR (Agent Threat Rules) is an independent, MIT-licensed detection standard. The corpus is 680+ rules across 10 categories (nearly 690 as of July 2026), shipped as the npm package agent-threat-rules. PanGuard is built on ATR and operates it, the way a vendor operates Sigma or CVE. Everything below is the Layer 1 deterministic rule engine: pattern rules that run in milliseconds, not an LLM judge.

Why every number is per-corpus, not engine-wide

A jailbreak corpus and a benign-traffic corpus measure two different things. Recall answers "of the real attacks, how many did we catch?" Precision and false-positive rate answer "of what we flagged, how much was actually benign?" You cannot average these across corpora and get a meaningful figure, because the base rate of attacks is wildly different in each. A blended "99.7% precision" headline hides which corpus it came from and what the attack density was. So ATR refuses to publish one. Each benchmark below names its corpus, its size, and whether it is measuring recall or precision.

garak: 97.2% recall on 650 in-the-wild samples

NVIDIA garak is an LLM vulnerability scanner. Its corpus here is 650 in-the-wild jailbreak samples. Against that set, ATR's Layer 1 rules catch 97.2% of the attacks. This is a recall number: it tells you the rules see real jailbreaks that are actually circulating, not just textbook examples. It does not, on its own, tell you the false-positive rate. That is a separate measurement on a separate corpus, covered below.

PINT-format: 99.7% precision, 63.6% recall on 850 samples

This is a self-built corpus of 850 samples, assembled from deepset/prompt-injections and Lakera Gandalf. It is PINT-format, but it is not Lakera's official private PINT set, so treat it as an internal reproducible benchmark rather than a third-party leaderboard result. On it, ATR Layer 1 rules reach 99.7% precision at 63.6% recall. Read that pair honestly: when these rules fire, they are almost always right (99.7% precision), and they deliberately stay quiet on roughly a third of the samples (63.6% recall) rather than guess. That is the enforce-lane tradeoff in miniature. High confidence, narrow reach.

HackAPrompt: 69.6% recall vs a 28.6% baseline, 100% precision on 4,780 samples

HackAPrompt is a peer-reviewed adversarial dataset (EMNLP 2023), 4,780 samples of prompts written by humans specifically to break guardrails. This is the hardest corpus of the four. ATR Layer 1 rules reach 69.6% recall here at 100% precision, against a 28.6% baseline. Two things matter. First, 69.6% more than doubles the 28.6% baseline, which is the real signal. Second, precision holds at 100%: on this adversarial set, when a rule fires it is never wrong. Recall below 70% on a deliberately adversarial corpus is not a flaw to hide. It is the honest ceiling of deterministic pattern matching against human red-teamers, and it is exactly why the standard documents its known evasion techniques instead of claiming full coverage.

SKILL.md: 100% recall, 97% precision, 0.2% FP on 498 samples

The SKILL.md corpus is 498 real-world samples of the agent skill files ATR was built to scan. This is the home-turf test, and the numbers reflect it: 100% recall, 97% precision, 0.2% false-positive rate. Every malicious skill in the set is caught, almost nothing benign is flagged. This is the corpus that matters most for the actual product surface, because scanning a skill or an MCP server before you install it is what pga scan does.

The benign gate: why false positives are lane-based, not one number

False-positive rate is where a single headline number does the most damage, so ATR gates every rule against 65,000 benign samples and reports FP by lane, not blended. In the enforce lane, false positives run about 0.24%. In the hunt lane, which is the default, they run about 9%. There is no single blended FP figure, and saying otherwise would be dishonest. The reason two lanes exist: enforce is for when a false alarm is expensive (blocking a legitimate agent action), so it is tuned tight. Hunt is for surfacing candidates a human will triage, where missing a real attack costs more than a noisy flag. Same rules, two operating points. Quote the lane or quote nothing.

Reproducibility

None of these are marketing numbers. The rule corpus ships as npm install -g @panguard-ai/panguard, and the standard itself is MIT-licensed and public, so the rules that produce these results are the rules you can read. The four attack corpora are named above (garak, the PINT-format build from deepset + Gandalf, HackAPrompt, SKILL.md) and the benign gate is 65,000 samples. When you cite an ATR benchmark, cite it with its corpus and its lane. A number without a corpus is not a benchmark. It is a slogan.

FAQ

Does ATR have a single overall precision or false-positive rate?

No. Precision and FP depend entirely on the corpus and the operating lane. ATR reports each per corpus, and false positives are split into an enforce lane (~0.24%) and a hunt lane (~9%) measured against 65,000 benign samples.

Why is HackAPrompt recall only 69.6% when garak is 97.2%?

They are different corpora. HackAPrompt is a deliberately adversarial dataset of human red-team prompts, so it is far harder. The honest ceiling of deterministic pattern matching there is 69.6% recall, still more than double the 28.6% baseline, at 100% precision.

Is the PINT-format result the same as Lakera's official PINT benchmark?

No. It is a self-built 850-sample corpus in PINT format, assembled from deepset/prompt-injections and Lakera Gandalf. It is not Lakera's official private PINT set, so treat it as an internal reproducible benchmark.

Are these Layer 1 rule numbers or full-engine numbers?

All figures above are Layer 1 deterministic rules only, the pattern rules that run in milliseconds. They are reported that way on purpose so the deterministic floor is measured separately from anything else.

Read the standard and reproduce the numbers yourself: install with npm install -g @panguard-ai/panguard, then run pga scan <target> against a skill or MCP server before you install it.