How to Read a Lawnise AI Answer Accuracy Barometer

Short answer: Each Lawnise barometer reading covers one market and one sector, from one capture of public AI answers. It reports a small number of individually checked findings, each with the question, what the answer said, the source it was checked against and the limits of the finding. A reading is not an accuracy rate, a ranking of AI systems or a trend. Read it finding by finding, and keep each finding's qualification with it.

What a reading covers

Our barometers cover AI answers about banking and insurance in Singapore and Malaysia. Each page holds one market and one sector, such as Malaysian insurance, and is refreshed in place with a new reading. The ASEAN overview brings the pages together.

The "How we checked" section near the top tells you four things:

• the market and sector, for example Malaysian insurance;
• the month of the capture, when the questions were put to public AI systems;
• the systems included in that capture, named only as the set tested;
• any coverage limitations, for example where a system's capture failed or was incomplete.

Three different dates can appear on a page, and they mean different things. The capture month tells you when the answers were collected. "Updated" or "Last reviewed" records the review date shown for that version, not the original publication date. Each reference records when its source was accessed. A finding describes the answer as captured and the source as checked, not necessarily what either says today.

A coverage limitation is not a finding. If a capture failed or a system returned no usable answer, that limits what the reading can say. It is not counted as a wrong answer, and a reading makes no comparison between systems on the basis of an incomplete capture.

How a finding is checked

Automated checks flag answers that may conflict with a source. A flag is a lead, not a finding. Review also includes answers the automated checks did not flag; detector labels do not decide the findings. For each featured case, the full answer is read alongside the question and compared with the relevant official or published source: a regulator's page, a scheme's rules, or an institution's own policy or disclosure. The review also separates a wrong statement from an omission, a qualified estimate and a harmless difference in wording. We describe this in more detail in how Lawnise verifies AI answers.

Each finding on the page usually follows the same pattern:

1. What was asked. The question is described in plain terms. The exact prompt is not always reproduced.
2. What the answer said. The part of the answer that matters, not every sentence in it.
3. What the source says. The clause, rule or published commitment the answer was compared with, usually with a reference.
4. What the finding does not mean. Most findings include a sentence that sets their limit. A finding about a dispute limit may say an exception is not automatic. A finding about a complaint route may say referral does not guarantee an outcome. That sentence is part of the finding, and it is often the most important one.

If you quote a finding, keep its qualification with it.

What cleared answers and source notes tell you

A reading is not only a list of errors. Two kinds of note help you judge it.

• Correct handling. Each reading records answers that handled the same points correctly, and flags that did not hold up against the source and were cleared. These notes are there so the findings aren't read as the whole picture.
• Source limitations. Where a source would identify a private institution, it is described without the name, and the original is kept in our internal evidence record. Where a reading relies on a source with a known limitation, for example on when a page was captured, the finding says so rather than overstating what was checked.

What a reading cannot establish

A barometer reading is deliberately narrow. The featured examples are selected for review, not drawn as a random or complete sample. So a reading cannot tell you:

• how often AI answers are wrong across all questions or all answers;
• which AI system is better or worse. We do not attribute findings to a provider or rank systems;
• whether an error persists or becomes more common over time. The published findings are selected observations, not a controlled month-to-month comparison. Each reading stands on its own;
• anything about a named bank or insurer. Private institutions are not named. Public schemes and regulators are, because their rules are public.

That is why each reading describes itself as single-month and directional, and not a ranking.

Reading history, references and corrections

Below the current findings, the Reading history section keeps earlier readings on the same page. The previous reading is open and older ones are collapsed. Each earlier entry keeps its own findings, its checked-and-cleared notes and its limits, and is not rewritten to fit a later month. If a correction is needed, it is recorded openly rather than silently changed.

The References list links public sources and shows when each was accessed. Withheld sources are described but not linked.

If you think a finding is wrong, or that it identifies an institution when it should not, you can request a correction. Our methodology sets out how research is governed, including how corrections are handled.

Why we publish readings this way

It would be easy to turn these checks into a single score. We don't, because a score would suggest a measurement the readings are not designed to make. A verified example with its qualification is more useful to the people who rely on it: a risk owner can check the same point against their own published facts, and a reader can see exactly which condition was missed. The same idea is behind our explainer on why a correct source doesn't guarantee a correct AI answer.

If you'd like to understand what public AI is saying about your own institution, checked the same way, we're glad to talk.