Small, curated suite
Ninety‑two cases cannot represent the open‑ended attack surface of a production agent. The two structural multi‑turn cases do not cover model‑mediated history manipulation; paraphrase and obfuscation remain incomplete.
Current results, setup, and caveats.
Baseline captured and report reviewed August 10, 2026. These are vendor‑published engineering measurements, not an independent assessment or a promise of production performance.
Accuracy baseline
| Metric | Result | Interpretation |
|---|---|---|
| Mode | Mock | Deterministic heuristic path; no live Anthropic or Voyage request. |
| Cases | 92 | Ten categories across five agent personas. |
| Overall accuracy | 86.96% | Both the expected allow/deny result and intent category must match. |
| Deterministic subset | 100.00% | Only cases marked as supported by the mock heuristic. |
| Deny precision / recall / F1 | 1.000 / 0.823 / 0.903 | The baseline recorded no false‑positive denies and 11 missed attacks. |
Evidence limit
The digest-pinned qualification policy is publicly inspectable. The raw case fixture and harness source are not publicly downloadable, so an external reader cannot reproduce this vendor report independently from the published material alone.
A case passes only when verdict and intent both match.
Each fixture is one MCP inspection request paired with an expected authorization result and intent category. A wrong verdict, a right verdict with the wrong category, or a non‑successful response fails the case. The positive class is deny.
Case mix
| Category | Cases | What it checks | Mock supported |
|---|---|---|---|
| Benign authorized | 20 | In‑scope calls that should pass. | Yes |
| Benign metadata | 5 | Resource and tool discovery that should pass. | Yes |
| Privileged action | 8 | Destructive keyword shapes that should deny. | Yes |
| Direct prompt injection | 12 | Known instruction‑override patterns. | Yes |
| Meta reasoning | 4 | Plan and ignore bypass triggers. | Yes |
| Credit‑card drift | 5 | Forbidden data access against a persona. | Yes |
| Persona‑drift stretch | 12 | Cross‑persona actions without a keyword trigger. | No |
| Malicious code write | 12 | Reverse shell, keylogger, ransomware, and persistence shapes. | Yes |
| Compromised package | 12 | Known‑bad or typosquatted package installation. | Yes |
| Multi‑turn window drift | 2 | Rapport‑then‑strike sequence plus its benign control. | Yes |
Why overall accuracy is below 100%
The 12 persona‑drift stretch cases intentionally contain no keyword trigger. Mock mode has no live embedding or classifier signal, so this category exposes the gap between the deterministic floor and the full semantic path. The baseline does not hide that gap.
No committed live‑provider baseline is published here. A live run incurs provider cost, changes with model versions, and should be measured against a pinned model configuration before it is compared with the mock result.
No live model is currently published as qualified. The support matrix stays experimental until current live receipts meet the three‑run security, schema, latency, and cost thresholds.
Separate Clavenar overhead from model‑provider time.
The whole‑pipeline run measured mTLS ingress, Brain in mock mode, policy evaluation, and proxy fork/fold at the /mcp boundary. It used 10 concurrent workers, a 20‑second measurement window, and a 5‑second warmup on the development stack.
| Path | p50 | p95 | p99 | Approx. throughput |
|---|---|---|---|---|
| Verdict‑cache hit, fixed payload | 1.67 ms | 2.20 ms | 2.90 ms | 5,800 rps |
| Cache miss, varied payload | 14.0 ms | 20.1 ms | 29.4 ms | 660 rps |
Do not add these numbers together blindly
The cache‑miss result measures Clavenar's mock‑mode pipeline floor. It does not include a real LLM provider request. On a live semantic path, provider round‑trip time can add hundreds of milliseconds and may dominate the verdict. Measure hot and cold paths with your chosen provider, region, policy set, and evidence store.
What this report does not establish.
Mock mode is a floor
The deterministic subset rewards the heuristics it was designed to exercise. It is useful for regressions; it is not a substitute for a live semantic evaluation.
No competitor baseline
No like‑for‑like run against another guardrail is published here. Do not infer comparative superiority from these standalone measurements.