Accuracy Evidence File
The Accuracy Evidence File is a three-week, fixed-price service from ETT. We check a set of your AI outputs against the documents they came from and show, for every claim, whether the source supports it, contradicts it or never mentions it, and what the output left out. It runs on Analect.
AI output is going into decisions. Nobody can show it was right
Summaries, reports, case notes, letters and briefings are drafted by AI and signed by people. Checking them by hand is slow and uneven, and a single accuracy score means nothing without saying what it was measured against.
Regulators, auditors, insurers and clients are starting to ask the same question: how do you know? And the hardest failure to catch is the fact that was left out. An output can be right on every claim it makes and still miss the one that mattered.
- Spot checks that cover a fraction of the output and leave no record behind.
- “Accurate”, with no statement of what it was checked against.
- Omissions nobody measures, because nobody is looking for them.
Every claim, checked against the document it came from
- 01
Claims
Each output is broken into the claims it makes.
- 02
Checks
Two independent models check every claim against your source.
- 03
Verdicts
Each claim gets a verdict, with the passage it rests on.
The source says so. We quote the passage that supports it.
The source says otherwise. We quote the passage that contradicts it.
The source never mentions it. Reported as unsupported, not as wrong.
Our two checking models disagree. Left for a person, never settled by a tiebreak.
And what was left out. The material facts in your source that never made it into the output are reported on their own, and never averaged into accuracy.
Accuracy and completeness, reported separately
“No contradictions found” is not a statement that the output is true. Every certificate says what it does not certify.
Does your source contradict anything the output says?
- No contradictions found
- Every claim was supported or left unsupported. None was contradicted.
- Contradictions found
- At least one claim is contradicted by a passage of your source.
- Indeterminate
- The checks did not settle, so no grade is issued.
- Not certifiable
- Not enough source to check against, or no claims that could be counted.
Did the output leave out anything material?
- Nothing material dropped
- Every material fact we settled, within the agreed scope, appears in the output.
- Material facts dropped
- At least one material fact in your source is missing from the output.
- Completeness indeterminate
- Too few facts settled to give a figure, so none is given.
- Not measured
- Completeness was not measured on this output, and the certificate says why.
What a certificate records
One certificate per output. It is written so a reviewer, an auditor or a client can follow every verdict back to the passage it rests on.
One register across the whole set: outputs checked, both results for each, claim totals with every denominator stated, every contradiction in full with its source quote, every omission, the method, the limits and the model versions used. Delivered as PDF and as a machine-readable file, with your team's own logins to the working file for the length of the engagement.
- The output and its source
- What was checked against what, with what produced the output recorded as you stated it.
- Each claim
- As taken from the output, so a reviewer can see exactly what was checked.
- Its verdict
- Supported, contradicted, unsupported or disputed. Disputed claims are left for a person.
- The source quote
- The passage each supported or contradicted verdict rests on.
- Omissions
- Material facts in the source that the output left out, counted separately from accuracy.
- Both results
- Accuracy and completeness, each with its denominator stated.
- Method and limits
- The models and versions used, the thresholds applied, and what the certificate does not certify.
- Digests and a reference
- Of the exact source and output it was issued against. Anyone holding the certificate can check on our verification page that we issued it, that it has not been withdrawn, and that their file is the one it covers.
Three weeks, with your reviewer on every dispute
The three weeks start when we have the outputs and their sources.
- Week 1
Scope
We agree the set: which outputs, which sources and what matters most. We certify three outputs together first, so your reviewer sees the format before the full run.
Yours: Provide the outputs and their sources. Name a reviewer.
- Week 2
Certify
We run the full set and go through every contradiction and every disputed claim with your reviewer.
Yours: Two to three hours of reviewer time.
- Week 3
Deliver
The Evidence File, and a readout for your sponsor and your risk or compliance lead.
Yours: A sixty-minute readout.
From you
- Up to 30 AI outputs, such as summaries, reports, answers or letters, and the source documents each one was produced from.
- For each output, what produced it: the system or model, the date and the purpose or prompt. We record these as you state them and label them as stated.
- Pasted text, PDF or Word, up to roughly a 200-page source and a 40-page output each. Scanned images need converting to text first.
- A reviewer who knows the subject, for two to three hours.
Where it goes
- We never keep your source documents or the outputs themselves. Each check reads them and lets them go.
- We keep the certificates: the claims taken from each output, the source quotes each verdict rests on, and the material facts listed from your source, which you can switch off. They sit in a private store only named people at your organisation and at ETT can open.
- The checks run on models from Anthropic and OpenAI, reached through Vercel's AI Gateway under their commercial API terms. Our sub-processor list is available on request.
What the Evidence File is not
Not a certificate of authorship
It certifies an output against a source. It does not certify which system wrote the output, or that the source itself is right.
Not a single score
You get verdicts, their denominators and the quotes behind them, because a percentage without its denominator cannot be checked.
Not a black box
The checks use AI models and say so. Every verdict sits beside the passage it rests on, and the thresholds are printed on the certificate.
Not a replacement for professional review
It shows your reviewers exactly where to look.
One fixed fee covers
- Up to 30 outputs certified against their sources, with omissions
- The Evidence File as PDF and as a machine-readable file
- A reviewer session, and a readout for your sponsor and your risk or compliance lead
- 30 days' access to the working file after delivery
- Optional: more outputs in packs of 25, on the same terms and in the same file
What we only know once we see your documents
- Which of your formats extract cleanly as text, and which need converting first
- Which facts in your sources count as material for this set of outputs
- Who reviews the disputed claims, and how quickly they can turn them round
About the Accuracy Evidence File
What counts as an AI output?
Anything an AI system drafted from source documents: summaries, reports, case notes, letters, briefings or answers. Each output needs the source documents it was produced from.
How is a claim checked?
Each output is broken into its claims, and two independent models check every claim against your source. A supported or contradicted claim quotes the passage it rests on. A claim the source never mentions is reported as unsupported, not as wrong, and a claim the two models disagree on is marked disputed and left for a person.
Do you keep our documents?
No. Each check reads the source and the output and lets them go. We keep the certificates: the claims, the quotes each verdict rests on and the material facts listed from your source, which you can switch off.
Does a clean result mean the output is true?
No. “No contradictions found” means no claim was contradicted by your source. It does not certify that the source is right, or which system wrote the output, and every certificate says so.
Can a client or auditor check a certificate themselves?
Yes. Each certificate carries a reference and the digests of the exact source and output it was issued against. Anyone holding it can check on our verification page that we issued it, that it has not been withdrawn, and that their copy of the file matches.
How large can the documents be?
Up to roughly a 200-page source and a 40-page output each, as pasted text, PDF or Word. Scanned images need converting to text first.
Start with three outputs
A thirty-minute scoping call to choose the outputs and sources and agree what matters most. Then we certify your first three together, so your reviewer sees the format before the full run.