Contents
Purpose and reporting standard
This report sets out the empirical evidence for TELOS. It keeps observations apart from interpretations and discloses the development and calibration conditions. It reports the recorded reproduction outcomes and their evidence, and it states which claims are still open.
We apply five controls:
- Each detection result names its scorer, benchmark and sample size, and gives false positive context where available.
- We report zero observed failures as an observed count.
- A benchmark integrated into a runner counts as reported scored evidence here only when its result is reported and admitted to this report.
- We label each development result as development evidence.
- We describe a cryptographic integrity check as integrity evidence.
System evaluated
Constitutive fidelity is the measured correspondence of submitted work to the human-declared specification, using the versioned Purpose Anchor in force. The results here are about particular tests and scorer profiles.
The retained agentic benchmark paths use AgenticFidelityEngine through their benchmark adapters. Each result has to state what was measured and which scorer profile was used. The archive note covers the archived prompt-level scoring configuration.
The published datasets and result artifacts let you inspect the reported outcomes.
Calibration and experimental design
The retained agentic results are reported development evidence. Each claim needs its own scorer, population, calibration and evaluation provenance.
This report includes a statistical value only when the available artifacts establish where it came from.
Tolerance bands
The deployer sets tolerance bands around the Purpose Anchor, and they set how tight a deployment runs. You read a measurement as a position against the configured bands.
Archived prompt-level evaluations and withdrawals
AILuminate, HarmBench, MedSafetyBench, the internal California SB 243 set and XSTest were evaluated with earlier prompt-level scoring harnesses. The January 2026 suite contained 2,100 prompts and the July suite contained 2,550. On 13 September 2026, the zero-miss detection results from the four adversarial harnesses were withdrawn, along with the 0.14 and 0.12 percent bounds derived from them. Those harnesses returned a detected classification on every branch and could not record a miss. We do not treat the reported 74 percent false-positive figure as a measured result because the process that produced it has not been located. XSTest's benign-prompt comparisons are historical and sit outside the zero-miss withdrawal.
The original run records and signed capsule remain unchanged, and the withdrawal also applies to its four adversarial result rows. The nine-row historical re-check included four adversarial rows that were later withdrawn, two archived XSTest rows and three agentic-benchmark rows. The agentic subset is reported separately below. Archived sources and withdrawal notices stay on record for accountability: AILuminate, HarmBench/MedSafetyBench (10.5281/zenodo.21629088), SB 243 (10.5281/zenodo.18370504), and XSTest. Retained notices cover the other affected archived evaluations.
External agentic benchmarks
The historical AgentDojo result is withdrawn as a supported result.
The table below lists 1,329 scored examples across two benchmark families: 977 from PropensityBench and 352 from AgentHarm.
Scroll horizontally to read all columns.
| Benchmark | Scored examples | Reported result | Important qualification | Artifact |
|---|---|---|---|---|
| PropensityBench | 977 scored task examples, sampled from a reported population of 5,874. | 100% reported detection success rate | Development evaluation using the historical scorer. Benign or false-positive rate: not reported for this run. | Zenodo |
| AgentHarm | 352 | MiniLM: harmful-case classification 145/176 (82.39%); benign preservation 78/176 (44.32%). Mistral-embed: harmful-case classification 176/176 (100%); benign preservation 0/176 (0%). Historical harness classifications. | The Mistral profile's higher harmful-case classification rate coincides with zero benign preservation; these are different configurations. | Zenodo |
The AgentHarm configurations produced different harmful-classification and benign-preservation results. These are cross-profile measurements.
July 2026 agentic subset of the historical re-check
The preserved historical run contains nine rows. This section reports its three agentic-benchmark rows, and the archive note accounts for the six prompt-level rows. Original receipt bytes and dispositions are unchanged.
| Benchmark | Recorded outcome | Evidence limit |
|---|---|---|
| PropensityBench | REPRODUCES, reported 100 percent on 977 examples | Self-attested projection |
| AgentHarm | DRIFTED: MiniLM profile did not reproduce the Mistral-profile headline | Compare harmful classifications beside benign preservation |
| AgentDojo | COULD-NOT-RUN in this re-check | Historical supported-result claim withdrawn |
The nine original projections remain available at https://github.com/TELOS-Labs-AI/telos-verify/tree/b60917f695a56d5c66520aa6b8db73fd88de5831/vectors/valid as historical integrity evidence. They carry integrity hashes, and TELOS signatures are in the Record of Trust.
Internally scored benchmarks outside the reported detection totals
Two more benchmark suites have runner integrations (nominal inventories 2,000 and 1,054 items). They are excluded from the reported detection totals.
Scroll horizontally to read all columns.
| Benchmark | Examples | Status |
|---|---|---|
| Agent-SafetyBench | 2,000 | Earlier internal scoring was excluded from the reported evidence. The preregistered frozen-version evaluation on 2026-11-15 is now reported: engine 866a0ea, strict operating point, independent scoring, all 1,780 eligible items; FAIL at the preregistered bar. Engine remains UNCALIBRATED. See roadmap, Calibration, for the figures and scope. |
| InjecAgent | 1,054 | Internally scored; excluded from the reported evidence |
| Total | 3,054 | Excluded from reported detection totals |
Evidence inventory and provenance
Scroll horizontally to read all columns.
| Evidence class | Count | What it supports |
|---|---|---|
| External agentic scored examples | 1,329 | Reported development examples from the two-family table above. |
| Integrated; internally scored, outside the reported detection totals | 3,054 | Runner coverage and future evaluation capacity |
| Published Record of Trust | 31 native rows, 14 published | Signed commissioning and the receipt digest for the final live run in this snapshot, checkable offline |
| Internal pytest count | Unverified in the available publication artifact | Engineering activity |
Reproducibility links
Scroll horizontally to read all columns.
| Dataset or result artifact | Link |
|---|---|
| PropensityBench | 10.5281/zenodo.18562833 |
| AgentHarm | 10.5281/zenodo.21629991 |
Interpretation
The retained agentic-benchmark records describe classifications and comparisons, and each one is specific to its profile.
Recommended next validation phase
We recommend an independent, publication-grade study. It should be preregistered before threshold tuning and include:
- A frozen Manifest, boundary corpus, embedding model, scorer version, thresholds, and decision policy.
- A development set separate from a sequestered held-out test set.
- Independent labels with documented adjudication and inter-rater agreement.
- Sensitivity, specificity, precision, recall, false positive rate, calibration, and confidence intervals at every published operating point.
- Domain and model transfer tests without post-hoc recalibration, followed by a separately reported recalibrated condition.
- Adaptive red-team evaluation with stated access levels.
- End-to-end outcomes, including task success, user interruption burden, time to resolution, and harm avoided.
- Reproduction by an organization that did not design the scorer.
Bottom line
The retained agentic research is development evidence. When you report it, put harmful-case classifications beside benign-case preservation and keep the scorer profiles distinct.