Skip to content
What your AI is fed

Analect Proof

Analect Proof tests, on your own documents, whether a model answering from retrieved passages does as well as the same model reading the whole document, and counts the input each route used. Its verdict has three states: holds, does not hold, or not shown. It says not shown when a run did not measure enough to tell.

On the platform, with access set up by ETT Last reviewed by George McKenna, Chairman & CTO, ETT.

The question it answers

Does sending the model less still get the same answers?

What goes in

  • A document, uploaded or from your library.
  • The models you want tested.
  • Optionally, your own questions and expected answers, up to 40.

What comes out

  • A verdict for each model: holds, does not hold, or not shown, with the interval it rests on.
  • The input each route used, counted, beside the result.
  • For a not shown, how many more questions it would take to settle.
  • An export of every question, every answer and every score.
Built to be checked

How is it measured?

  • Questions are written from the whole document, not only its opening pages.
  • Trap questions, whose answer is not in the document, test whether a model declines rather than invents.
  • Figures are graded by arithmetic, not by a model's opinion.
  • Two judges score every answer, and a verdict that depends on which judge is asked cannot read holds.
  • Plain search at the same budget runs beside it as a control, so the method has to beat something.
  • A floor is measured for what each model already knew without the document.
The limits

What does Analect Proof not do?

  • It does not promise a saving before it has measured one. The result is about your documents and nobody else's.
  • It does not give one overall score or put models in order. Each model gets its own verdict on each kind of question.
  • It does not turn a small sample into a pass. A run that measured too little reads not shown.
Where it sits

Where does Analect Proof fit in an AI pipeline?

Beside it, not inside it. Analect does not sit in the path of your AI requests, and you do not change your models or tools to use it. Analect Proof works at the point marked below.

  1. 01

    Before you build

    What the work should cost, and on which model.

  2. 02

    Prepare the documents

    What is in them, what they cost to hold, who may see them.

  3. 03

    Choose what the model is given

    Whether sending less still gets the same answers.

    Analect Proof works here.

  4. 04

    Run

    Your own models and tools. Nothing of Analect's sits here.

  5. 05

    Check the output

    Whether each claim is supported by its source.

  6. 06

    Prove it over time

    Evidence an auditor can follow, for one system over a period.

How to get it

How do we start using Analect Proof?

Analect Proof is on the platform. ETT sets up access and runs the first test with you on your own documents. Accounts are set up by ETT, and each account sees only the workflows its work needs.

Also under what your AI is fed: Analect Explorer, Analect Governance, Analect Store. All workflows

Questions

About Analect Proof

What does not shown mean in Analect Proof?

That the run did not measure enough to say either way. A verdict of holds needs the evidence to rule out a loss, and does not hold needs a loss to be shown. Anything between the two is reported as not shown, with the number of further questions it would take to settle.

Is Analect Proof a tool for evaluating retrieval?

It answers one question about retrieval: on your own documents, does a model given retrieved passages answer as it does when it reads the whole document, and how much input does that take. Plain search at the same budget runs as the control.

Do we need to write our own questions?

No. Analect Proof writes questions from the whole document, including trap questions whose answer is not there, and reads the document's own figures as a key graded by arithmetic. Your own questions and answers are optional, and are graded the same way.

See it on worked material, then on yours

Thirty minutes with the team that runs Analect: the AI Spend Forecast and the Accuracy Evidence File on worked examples, and what either would look like on your bills or your outputs.

Book a demo