Public DocsPublication

Validation Report

TELOS stands for Telically Entrained Linguistic Operating Substrate. It is the system for constituting agents under purposes people define, and for measuring fidelity to those purposes.

Evidence, Calibration, Reproduction, and Claim Boundaries

October 2026 · TELOS AI Labs Inc.

Contents

Purpose and reporting standard

This report sets out the empirical evidence for TELOS. It keeps observations apart from interpretations and discloses the development and calibration conditions. It reports the recorded reproduction outcomes and their evidence, and it states which claims are still open.

We apply five controls:

  1. Each detection result names its scorer, benchmark and sample size, and gives false positive context where available.
  2. We report zero observed failures as an observed count.
  3. A benchmark integrated into a runner counts as reported scored evidence here only when its result is reported and admitted to this report.
  4. We label each development result as development evidence.
  5. We describe a cryptographic integrity check as integrity evidence.

System evaluated

Constitutive fidelity is the measured correspondence of submitted work to the human-declared specification, using the versioned Purpose Anchor in force. The results here are about particular tests and scorer profiles.

The retained agentic benchmark paths use AgenticFidelityEngine through their benchmark adapters. Each result has to state what was measured and which scorer profile was used. The archive note covers the archived prompt-level scoring configuration.

The published datasets and result artifacts let you inspect the reported outcomes.

Calibration and experimental design

The retained agentic results are reported development evidence. Each claim needs its own scorer, population, calibration and evaluation provenance.

This report includes a statistical value only when the available artifacts establish where it came from.

Tolerance bands

The deployer sets tolerance bands around the Purpose Anchor, and they set how tight a deployment runs. You read a measurement as a position against the configured bands.

Archived prompt-level evaluations and withdrawals

AILuminate, HarmBench, MedSafetyBench, the internal California SB 243 set and XSTest were evaluated with earlier prompt-level scoring harnesses. The January 2026 suite contained 2,100 prompts and the July suite contained 2,550. On 13 September 2026, the zero-miss detection results from the four adversarial harnesses were withdrawn, along with the 0.14 and 0.12 percent bounds derived from them. Those harnesses returned a detected classification on every branch and could not record a miss. We do not treat the reported 74 percent false-positive figure as a measured result because the process that produced it has not been located. XSTest's benign-prompt comparisons are historical and sit outside the zero-miss withdrawal.

The original run records and signed capsule remain unchanged, and the withdrawal also applies to its four adversarial result rows. The nine-row historical re-check included four adversarial rows that were later withdrawn, two archived XSTest rows and three agentic-benchmark rows. The agentic subset is reported separately below. Archived sources and withdrawal notices stay on record for accountability: AILuminate, HarmBench/MedSafetyBench (10.5281/zenodo.21629088), SB 243 (10.5281/zenodo.18370504), and XSTest. Retained notices cover the other affected archived evaluations.

External agentic benchmarks

The historical AgentDojo result is withdrawn as a supported result.

The table below lists 1,329 scored examples across two benchmark families: 977 from PropensityBench and 352 from AgentHarm.

Scroll horizontally to read all columns.

Benchmark Scored examples Reported result Important qualification Artifact
PropensityBench 977 scored task examples, sampled from a reported population of 5,874. 100% reported detection success rate Development evaluation using the historical scorer. Benign or false-positive rate: not reported for this run. Zenodo
AgentHarm 352 MiniLM: harmful-case classification 145/176 (82.39%); benign preservation 78/176 (44.32%). Mistral-embed: harmful-case classification 176/176 (100%); benign preservation 0/176 (0%). Historical harness classifications. The Mistral profile's higher harmful-case classification rate coincides with zero benign preservation; these are different configurations. Zenodo

The AgentHarm configurations produced different harmful-classification and benign-preservation results. These are cross-profile measurements.

July 2026 agentic subset of the historical re-check

The preserved historical run contains nine rows. This section reports its three agentic-benchmark rows, and the archive note accounts for the six prompt-level rows. Original receipt bytes and dispositions are unchanged.

Benchmark Recorded outcome Evidence limit
PropensityBench REPRODUCES, reported 100 percent on 977 examples Self-attested projection
AgentHarm DRIFTED: MiniLM profile did not reproduce the Mistral-profile headline Compare harmful classifications beside benign preservation
AgentDojo COULD-NOT-RUN in this re-check Historical supported-result claim withdrawn

The nine original projections remain available at https://github.com/TELOS-Labs-AI/telos-verify/tree/b60917f695a56d5c66520aa6b8db73fd88de5831/vectors/valid as historical integrity evidence. They carry integrity hashes, and TELOS signatures are in the Record of Trust.

Internally scored benchmarks outside the reported detection totals

Two more benchmark suites have runner integrations (nominal inventories 2,000 and 1,054 items). They are excluded from the reported detection totals.

Scroll horizontally to read all columns.

Benchmark Examples Status
Agent-SafetyBench 2,000 Earlier internal scoring was excluded from the reported evidence. The preregistered frozen-version evaluation on 2026-11-15 is now reported: engine 866a0ea, strict operating point, independent scoring, all 1,780 eligible items; FAIL at the preregistered bar. Engine remains UNCALIBRATED. See roadmap, Calibration, for the figures and scope.
InjecAgent 1,054 Internally scored; excluded from the reported evidence
Total 3,054 Excluded from reported detection totals

Evidence inventory and provenance

Scroll horizontally to read all columns.

Evidence class Count What it supports
External agentic scored examples 1,329 Reported development examples from the two-family table above.
Integrated; internally scored, outside the reported detection totals 3,054 Runner coverage and future evaluation capacity
Published Record of Trust 31 native rows, 14 published Signed commissioning and the receipt digest for the final live run in this snapshot, checkable offline
Internal pytest count Unverified in the available publication artifact Engineering activity

Scroll horizontally to read all columns.

Dataset or result artifact Link
PropensityBench 10.5281/zenodo.18562833
AgentHarm 10.5281/zenodo.21629991

Interpretation

The retained agentic-benchmark records describe classifications and comparisons, and each one is specific to its profile.

We recommend an independent, publication-grade study. It should be preregistered before threshold tuning and include:

  1. A frozen Manifest, boundary corpus, embedding model, scorer version, thresholds, and decision policy.
  2. A development set separate from a sequestered held-out test set.
  3. Independent labels with documented adjudication and inter-rater agreement.
  4. Sensitivity, specificity, precision, recall, false positive rate, calibration, and confidence intervals at every published operating point.
  5. Domain and model transfer tests without post-hoc recalibration, followed by a separately reported recalibrated condition.
  6. Adaptive red-team evaluation with stated access levels.
  7. End-to-end outcomes, including task success, user interruption burden, time to resolution, and harm avoided.
  8. Reproduction by an organization that did not design the scorer.

Bottom line

The retained agentic research is development evidence. When you report it, put harmful-case classifications beside benign-case preservation and keep the scorer profiles distinct.