Auditing and testing LLM agents.
Auditing an LLM agent means being able to replay what it did, re-score it with judges you choose, and confirm that every published number regenerates from raw files. Gubernaut publishes its own record on those terms: transcripts and four-family judge panels with SHA-256 provenance, the extraction scripts, and a replay cockpit that steps through the sealed runs turn by turn.
What makes an LLM agent auditable?
An agent is auditable when three things are recoverable after the fact: what it received, what it decided, and under what conditions it answered. Sampled text alone gives only the last of those, and second-hand. A deterministic control layer adds a decision trace that replays identically, so an incident review can diff behaviour instead of arguing about it.
What should an audit ask for?
An audit should ask for the raw transcripts of both arms verbatim, the judge panels with provenance hashes, the scripts that regenerate every published number, a pre-registration dated before data collection, and the nulls stated beside the headline. Gubernaut publishes all five under CC-BY-4.0.
01
Raw transcripts, both arms, verbatim. Without the ungoverned arm there is no comparison, only an assertion.
02
Judge panels with provenance hashes. SHA-256 per file, so a later edit changes the hash and becomes visible.
03
The scripts, not just the numbers. Extraction and statistics code that turns raw panels into every published figure.
04
A pre-registration dated before collection. Design, scoring protocol and analysis fixed in writing before any data existed.
05
The nulls and the failure modes. A record with no null and no disclosed limit has usually been filtered rather than measured.
How do I replay a run?
The replay cockpit on the research page steps through the sealed runs turn by turn, both arms side by side, with the controller telemetry as recorded at generation time. The cockpit is a recorded run replay with no live API, so what it shows is the record rather than a fresh generation.
How do I re-score the record with my own judges?
Re-scoring runs against the published transcripts. Every reply ships verbatim with SHA-256 provenance, and the extraction scripts that turn raw panels into published numbers are in the public repository. A different panel or a different rubric can be applied to the same replies and compared against the sealed matrices.
How do I test the governor against my own probes?
Install the proxy, point your client at it, and run your own adversarial sequence through both arms.
What does the release gate cover?
The release gate covers concurrency, fail-closed behaviour and cross-implementation parity before a version ships. Version 1.0.1 recorded no posture cross-contamination under 240 concurrent mixed requests, no fail-open leaks, and bit-exact agreement between the Rust core and the Python reference.
| Gate | What it asserts | Result at 1.0.0 |
|---|---|---|
| Concurrency | 240 concurrent mixed hostile and benign requests | No cross-contamination, no dropped requests, upstream hit 120/120 |
| Fail-closed | Neither SDK reaches the upstream when the proxy is down | Canary upstream recorded zero hits in every dead-proxy case |
| Parity | Rust core against the Python reference | Bit-exact, checked against golden traces in the repository |
| Edge soak | 10,000 ticks in workerd, a Node worker and Node main | Bit-exact, flat memory |
Source: HARDTEST v1.0 release gates, 2026-07-24.
What does the published record actually show?
Across 4 frontier model families the regulated arm was calmer in 15 of 16 generator×judge cells by sign (11/12 off-diagonal), 13 of 16 at p<.05, with homeostatic recovery replicating in 4/4 families. One cell, GPT generating and Gemini judging, returned a flat null at −0.04.
Source: Pre-registered cross-family evaluation. Paper: arXiv 2607.24339, DOI 10.5281/zenodo.21303518.
What should an auditor be sceptical of?
Any control claim that reports a headline and nothing else. The Gubernaut record carries a flat null at −0.04, three cells short of p<.05, five documented failure modes, and a sensor-recall limit on calmly worded hostility, all published in the same release as the headline and reproducible from the same files.