What Should an Independent AI Evaluation Report Show?

Short answer: A useful independent AI evaluation report explains what was tested, how it was checked and what the result can support. For answer-level evaluation, it should connect each finding to the question, the complete answer and the applicable reference. It should also disclose limits, explain the reviewer's role and say what happens next. A pass label without that information gives a buyer little to examine.

Independent evaluation can cover many different things. A model-safety evaluation, a review of organisational controls and a check of answers against business records are not interchangeable. Start by asking which one the report actually provides.

This guide focuses on factual and procedural answers produced in use, including answers from staff copilots and customer assistants. It is a buyer's checklist, not a prescribed reporting standard or a claim that any provider qualifies under a particular audit law.

Start with the scope, not the headline result

The report should identify the system or service, the task and the setting being evaluated. Look for the relevant market, language and capture dates. If the reviewer knows the configuration, it should be recorded. If it was not exposed, that limitation should be stated.

The question set matters as much as its size. Were questions selected by the buyer, generated by the evaluator, drawn from permitted records, or chosen because another tool flagged them? Different selection methods support different conclusions.

A set chosen to investigate suspected errors can establish particular discrepancies. It cannot, without a suitable design, establish the error rate across all answers. A large number alone does not fix a biased selection method.

Ask whether the report's headline says more than its scope allows. "The tested answers matched the supplied procedure" is a narrower conclusion than "the system is reliable".

A finding should show the comparison

For a consequential finding, you should be able to see what the assistant was asked and what it actually answered. A full response may contain a qualification that an isolated extract misses.

The report should identify the reference used, its applicable version and the clause governing the answer. The reference date and the answer date serve different purposes. A current page can show what is published now without proving what was published when an older answer was captured.

The reviewer should explain the discrepancy in plain terms. Did the answer change a condition, contradict a figure or fail to answer the question? The explanation needs to show why the difference matters for that case.

Not every buyer should receive every raw record. Access may be restricted by privacy, confidentiality or security obligations. The report should explain how an authorised reviewer can examine supporting evidence, rather than treating withheld data as proof that nobody may question the conclusion.

An example of a traceable finding

The following example is invented. It is not a client finding or a real service commitment.

Question: "I sent my request yesterday. When should I expect a response?"

Reference: "We aim to respond within ten working days after receiving all required documents."

Answer: "You will receive a response within ten days of your first message."

Finding: The answer turns a target into a guarantee, drops "working", and starts the period from the first message rather than receipt of the required documents.

Limit: This comparison concerns that response and that procedure. It does not establish how often the assistant makes the error, or why its wording changed.

A buyer can understand and challenge this comparison. "Deadline error: high risk" would not provide the same basis for review. Classification can help prioritise work, but it should not replace the evidence.

Ask how the evaluation was carried out

The method should describe how questions and references were selected, how complete answers were captured and how disputed cases were resolved. If automated tools contributed, ask what they did and where human review entered the process.

Using a different AI model as a judge does not by itself establish an independent conclusion. You still need to know the judgment criteria, the evidence available to it and how its mistakes are checked.

Independence also concerns the relationship around the work. Ask who commissioned and paid for the evaluation, whether there are relevant conflicts, and whether the assessed party can alter an adverse conclusion. A process should allow factual corrections and a response without allowing an inconvenient result to be silently rewritten.

Those are questions for the provider to answer, not a claim that a payment arrangement is automatically improper or that this checklist establishes legal auditor independence.

Eight questions to ask your provider

Use these alongside the practical checklist accompanying this guide:

1. Scope: Which system, tasks, languages and dates does the report cover?
2. Selection: How were the questions and answers chosen?
3. Evidence: Are complete responses retained, and how can an authorised reviewer examine them?
4. Reference: Which source versions apply, and were conflicts resolved?
5. Conditions: Were exceptions, starting points and eligibility requirements checked?
6. Method and independence: Who reviewed the findings, using what criteria, and with what relevant relationships disclosed?
7. Limits: What could not be tested or confidently determined?
8. Follow-up: Who owns the next action, and what will be re-tested?

This is Lawnise's recommended buyer checklist. NIST's voluntary AI Risk Management Framework supports documenting evaluation methods and unmeasured risks. It does not prescribe the report format above.

How Lawnise brings the evidence into a report

The checklist describes what a buyer should ask for. Lawnise's reporting workflow helps bring selected findings and their supporting records into a report, rather than leaving someone to reconstruct the comparison in a separate document.

The existing report we inspected includes captured answer extracts, reference facts and source links, observation dates and evidence identifiers. Its scope section explains the checks and what is outside the report. These fields make a finding easier to inspect and trace back to the platform; they do not turn a selected set into a measure of overall AI accuracy.

Automatically assembled content still needs review. The person preparing the report has to check the selection, explanation and limitations before sharing it. The sample report describes itself as "an AI-output observation and governance aid, not a legal or regulatory finding." A draft is not a sealed final report or a legal audit conclusion.

Platform example: this excerpt comes from an existing draft report about public AI answers, not an internal-assistant evaluation. It shows source validation, governance confirmation and evidence rendering. Client identifiers are excluded. This illustrates the workflow, not a guarantee that every report satisfies every item in the buyer's checklist.

View the report-method excerpt at full size

Keep limitations and follow-up visible

A limitation is useful when it identifies what it prevents you from concluding. "Limited sample" is less informative than an explanation that only one workflow and one language were checked. Missing evidence should be stated, not buried beneath a confident summary.

The report should also distinguish a proposed fix from a verified result. An owner can update a procedure or configuration, but the affected answer still needs checking after the change. Preserve corrections to the evaluation itself with a dated explanation.

For public reporting, our guide to reading an AI accuracy barometer explains a related boundary: verified examples are useful, but they are not a population accuracy score. A confidential evaluation should be equally clear about what its evidence can and cannot support.

The report does not need to be long to be useful. It needs to let the people responsible for the system understand the finding, inspect its basis and decide what to do. The strongest question to ask a provider is often the simplest: can you show me how you reached this conclusion?

To discuss the scope of an independent factual and procedural evaluation, talk to Lawnise.