BENCHMARK RESULTS
Public benchmark results for 836 ATR rules
Every result below names its data source, method and the ATR version that ran it. Where a result cannot be re-run, its entry says so.
All precision / FP figures are per-corpus Layer 1 measurements, not engine-wide claims. Recall figures are hunt-lane (the default). The lane-based benign-gate false-positive rates we used to quote were withdrawn on 2026-09-19 pending re-measurement; there is no single engine-wide FP number.
Garak (NVIDIA jailbreak corpus)
NVIDIA garak is an open-source LLM red-teaming framework. We ran ATR v4.1.5 against its in-the-wild jailbreak set. The figure is hunt-lane (the default, all rules); the enforce lane, used when blocking is on, catches far fewer. It is partly closed-book: some ATR rules were written from this corpus, so it is not an out-of-sample result.
Recall
92.3%
Sample size
650 samples
Layer
Regex only (no LLM second opinion)
Lane
hunt (default)
ATR version
v4.1.5
Reproduce
npx tsx scripts/eval-garak-inthewild.ts (in agent-threat-rules repo)SKILL.md (ATR internal benchmark corpus)
Labeled corpus of 498 AI agent skills from ClawHub, OpenClaw, and Skills.sh: 32 malicious, 466 benign. The benign samples are also the set ATR rules are tuned against, so this is an in-sample check, and its precision and false-positive figures are not cited. Recall is hunt-lane.
Recall (hunt lane, in-sample)
100%
Sample size
498 samples
ATR version
v4.1.5
Reproduce
npx tsx src/eval/skill-benchmark.ts (in agent-threat-rules repo)PINT (self-built PINT-format corpus)
A self-built 850-sample corpus in PINT format, assembled from the deepset prompt-injections dataset plus Lakera Gandalf. This is NOT Lakera's official PINT benchmark, whose corpus is private, and is not attributable to any third party. It is effectively a prompt-injection-family score, not a measure of overall coverage. Recall is hunt-lane.
Recall
65.41%
Precision (PINT-format corpus, self-built)
100%
Sample size
850 samples
Layer
Layer 1 — regex only
ATR version
v4.1.5
How to read these numbers
Precision / FP here are Layer 1 (deterministic-rule) measurements on this specific corpus, not an engine-wide figure. The lane-based benign-gate false-positive rates were withdrawn pending re-measurement.
Reproduce
npm run eval:pint (in agent-threat-rules repo)Wild Scan (published agent skills, 2026-04)
Scan of 101,280 published AI agent skills and MCP servers from five public sources (OpenClaw, ClawHub, Skills.sh, Hermes Agent, and an MCP server crawl), run 2026-04-13 with scan engine v2.0.0 (113 ATR rules, not today's rule set): 1,434 flagged. A flag is a rule match, not a verdict, so no confirmed-malicious count is given, and the record keeps paths and rule ids, not file contents. Not a labeled benchmark: there is no ground truth, so it has no recall or precision.
Skills scanned
101,280
Flagged
1,434
ATR version (scan engine)
v2.0.0 (113 rules)
Reproduce
Not re-runnable: the scan record (data/full-scan-v2-2026-04-14.json in agent-threat-rules) keeps paths and rule ids, not content.HackAPrompt cluster mining
ATR run against the HackAPrompt EMNLP 2023 competition corpus (4,780 deterministic samples). Result: 69.6% recall. Closed-book: ATR mined rules from this corpus's misses, and the 28.6% baseline is ATR's own recall before those rules. The corpus is attack-only, so precision and false-positive rate are mathematically undefined on it and are not reported. Methodology and rule additions documented in the public engineering blog.
HackAPrompt recall
69.6%
ATR recall before these rules
28.6%
Sample size
4,780 deterministic
ATR version
v3.5.0
Reproduce
npx tsx src/eval/run-hackaprompt-benchmark.ts (in agent-threat-rules repo)Want to run ATR on your corpus and publish the results? Open a PR at Agent-Threat-Rule/agent-threat-rules. We add your benchmark to this page with full attribution.
Reviewed by Adam Lin · Last reviewed 2026-05-12