Analect Proof
Analect Proof tests, on your own documents, whether a model answering from retrieved passages does as well as the same model reading the whole document, and counts the input each route used. Its verdict has three states: holds, does not hold, or not shown. It says not shown when a run did not measure enough to tell.
On the platform, with access set up by ETT Last reviewed by George McKenna, Chairman & CTO, ETT.
Does sending the model less still get the same answers?
What goes in
- A document, uploaded or from your library.
- The models you want tested.
- Optionally, your own questions and expected answers, up to 40.
What comes out
- A verdict for each model: holds, does not hold, or not shown, with the interval it rests on.
- The input each route used, counted, beside the result.
- For a not shown, how many more questions it would take to settle.
- An export of every question, every answer and every score.
How is it measured?
- Questions are written from the whole document, not only its opening pages.
- Trap questions, whose answer is not in the document, test whether a model declines rather than invents.
- Figures are graded by arithmetic, not by a model's opinion.
- Two judges score every answer, and a verdict that depends on which judge is asked cannot read holds.
- Plain search at the same budget runs beside it as a control, so the method has to beat something.
- A floor is measured for what each model already knew without the document.
What does Analect Proof not do?
- It does not promise a saving before it has measured one. The result is about your documents and nobody else's.
- It does not give one overall score or put models in order. Each model gets its own verdict on each kind of question.
- It does not turn a small sample into a pass. A run that measured too little reads not shown.
Where does Analect Proof fit in an AI pipeline?
Beside it, not inside it. Analect does not sit in the path of your AI requests, and you do not change your models or tools to use it. Analect Proof works at the point marked below.
- 01
Before you build
What the work should cost, and on which model.
- 02
Prepare the documents
What is in them, what they cost to hold, who may see them.
- 03
Choose what the model is given
Whether sending less still gets the same answers.
Analect Proof works here.
- 04
Run
Your own models and tools. Nothing of Analect's sits here.
- 05
Check the output
Whether each claim is supported by its source.
- 06
Prove it over time
Evidence an auditor can follow, for one system over a period.
How do we start using Analect Proof?
Analect Proof is on the platform. ETT sets up access and runs the first test with you on your own documents. Accounts are set up by ETT, and each account sees only the workflows its work needs.
Also under what your AI is fed: Analect Explorer, Analect Governance, Analect Store. All workflows
About Analect Proof
What does not shown mean in Analect Proof?
That the run did not measure enough to say either way. A verdict of holds needs the evidence to rule out a loss, and does not hold needs a loss to be shown. Anything between the two is reported as not shown, with the number of further questions it would take to settle.
Is Analect Proof a tool for evaluating retrieval?
It answers one question about retrieval: on your own documents, does a model given retrieved passages answer as it does when it reads the whole document, and how much input does that take. Plain search at the same budget runs as the control.
Do we need to write our own questions?
No. Analect Proof writes questions from the whole document, including trap questions whose answer is not there, and reads the document's own figures as a key graded by arithmetic. Your own questions and answers are optional, and are graded the same way.
See it on worked material, then on yours
Thirty minutes with the team that runs Analect: the AI Spend Forecast and the Accuracy Evidence File on worked examples, and what either would look like on your bills or your outputs.