Validation Report

Purpose and reporting standard

This report isolates the empirical evidence supporting TELOS. It separates observations from interpretations, discloses the development and calibration conditions, identifies results that were reproduced with a different embedding model, and states which claims remain open.

The report follows five controls:

1. A detection result is reported with its scorer, benchmark, sample size, and false positive context where available.
2. A zero observed failure result is not described as proof of zero risk.
3. A benchmark integrated into a runner is not counted as a scored benchmark unless a result is reported.
4. A development result is not described as held-out validation.
5. A cryptographic integrity check is not described as proof of authorship or trusted time unless the associated key and time controls are independently established.

Claims under test

System evaluated

The principal historical tests used a three-tier scoring configuration derived from declared purpose, domain boundaries, and prohibitions. The public telos-mcp v0.1 repository does not expose that calibrated production scoring system. Its current public scoring path is a synthetic prototype. The validation runners, protocol, production boundary data, and exact scoring configuration used for the reported experiments are held by TELOS Labs and are available under NDA.

This disclosure matters for interpretation. The published datasets and result artifacts support inspection of the reported outcomes, but they do not yet permit a fully independent reconstruction of the production measurement engine from public materials alone.

Calibration and experimental design

Thresholds were tuned on development suites. No held-out split was used for the principal internal adversarial evaluations. The results should therefore be read as development and re-verification evidence, not as an unbiased estimate of performance on an unknown deployment distribution.

This report includes only statistical values whose provenance can be established from the available artifacts. It therefore states no p value, power estimate, effect-size statement, AUC comparison, or Wikipedia-centroid accuracy delta.

Internal and public adversarial evaluations

January 2026 evaluation

The January suite contained 2,100 adversarial prompts:

The historical three-tier TELOS configuration reported 0 undetected attacks in 2,100 prompts. Using the conventional rule of three, a rough 95 percent upper confidence bound for the undetected rate is approximately 3 / 2,100, or 0.14 percent. This bound assumes independent and identically distributed Bernoulli trials, an assumption that benchmark prompts do not fully satisfy.

The same evaluation reported these attack success rate baselines:

The 0 percent TELOS row is a self-scored development result. It is not an independent third-party certification.

July 2026 re-verification

The July suite contained 2,550 adversarial prompts:

The re-verification reported 0 undetected attacks in 2,550 prompts. Under the same rule-of-three approximation, the 95 percent upper confidence bound is approximately 3 / 2,550, or 0.12 percent. This result is evidence about the tested suite and scorer. It does not establish a universal deployment failure rate.

Aggressive point and false positives

On the 50-prompt California SB 243 set, the aggressive operating point reported 0 undetected attacks and a 74 percent false positive rate. This demonstrates the central threshold tradeoff: very high sensitivity can be obtained at a level of overflagging that may be operationally unacceptable.

The 74 percent figure must remain attached to the aggressive-point result. Reporting the detection result without it would materially misstate the evidence.

Benign-use calibration with XSTest

XSTest was used to measure exaggerated safety behavior on 250 benign prompts.

This result supports H5 for the tested benign set. It does not by itself show that the healthcare-specific configuration preserved the original attack detection rate because adversarial detection was not rerun under that configuration.

External agentic benchmarks

The scored external agentic evidence comprises 1,468 examples across three benchmark families.

The AgentHarm rerun is especially informative. A change from the published Mistral embedding path to MiniLM reduced the reported rate to 82.39 percent. The result supports the whitepaper’s architectural argument that the measurement engine is a calibrated system, not a model-independent formula whose performance transfers automatically.

The AgentDojo result also illustrates why detection cannot be the only outcome. Perfect reported injection detection accompanied by 0 percent benign pass on the evaluated benign cases indicates that a system can appear safe while being unusably restrictive.

July 2026 governed re-verification

The headline external-agent results above are the January 2026 published figures. A later governed run under observation addressed nine published result rows:

The MiniLM run used local scoring because the run’s declared purpose prohibited metered APIs. The two embedding profiles are not interchangeable. The 82.39 percent result is therefore a cross-profile comparison and a non-reproduction of the published headline profile, not evidence of a time-based degradation in the same frozen system.

The nine unsigned re-verification receipts are available in the commit-pinned telos-verify vector directory at https://github.com/TELOS-Labs-AI/telos-verify/tree/b60917f695a56d5c66520aa6b8db73fd88de5831/vectors/valid; they preserve the published row values and dispositions as self-attested integrity evidence, not independently validated run records.

Internally scored benchmarks excluded from reported evidence

Two additional benchmark suites completed internal scored runs, but their results are excluded from the scored validation evidence reported in this document:

Out-of-scope and drift proof of concept

A governance benchmark based on CLINC150 and MultiWOZ reported:

These are proof-of-concept results and should not be combined with the adversarial safety totals.

Evidence inventory and provenance

Reproducibility links

Interpretation

The evidence supports a narrower claim than a conventional security guarantee. It shows that a configured TELOS measurement approach can identify many tested departures and attacks, and that its behavior changes materially with the boundary corpus, threshold, domain configuration, and embedding model.

The evidence does not yet show:

• independently reproduced production performance;
• stable performance across unknown deployment distributions;
• an acceptable sensitivity-specificity tradeoff for every domain;
• causal improvement in downstream agent behavior;
• resistance to adaptive attackers who know the scoring system;
• regulatory sufficiency;
• trusted receipt authorship or time in the public prototype.

Recommended next validation phase

A publication-grade independent study should be preregistered before threshold tuning and should include:

1. A frozen Manifest, boundary corpus, embedding model, scorer version, thresholds, and decision policy.
2. A development set separate from a sequestered held-out test set.
3. Independent labels with documented adjudication and inter-rater agreement.
4. Sensitivity, specificity, precision, recall, false positive rate, calibration, and confidence intervals at every published operating point.
5. Domain and model transfer tests without post-hoc recalibration, followed by a separately reported recalibrated condition.
6. Adaptive red-team evaluation with access levels stated explicitly.
7. End-to-end outcomes, including task success, user interruption burden, time to resolution, and harm avoided.
8. Reproduction by an organization that did not design the scorer.

Bottom line

The current research is meaningful development evidence, particularly because it exposes false positive costs and embedding-model dependence. It is not yet sufficient for a claim of universal or independently validated safety. The strongest defensible position is that TELOS has a testable measurement architecture with promising benchmark results, transparent known limitations, and a clear path to stronger validation.

Related documents

• Whitepaper
• Technical method
• Record of Trust protocol
• Regulatory alignment map