Why Multilingual AI Validation Is More Than Translation

Short answer: Translation asks whether wording has been carried into another language faithfully. Multilingual AI validation also asks whether the answer is correct for the user's question, the governing reference and the local context. A fluent answer can still change a deadline, lose an exception or describe the wrong procedure. Those are errors of meaning and application, not simply awkward translation.

An English-language test tells you what happened in the English cases you checked. It does not establish how an assistant answers questions in another language. If staff or customers use those other languages, their actual questions need a place in the evaluation.

This applies to internal knowledge tools as well as customer-facing assistants. A staff member checking a local procedure may mix everyday language with an English product name. Checking only a polished English version of that question can miss what the person actually asked.

Fluent language does not establish a correct answer

Imagine a procedure that says a response is targeted within ten working days after all required documents arrive. This is an invented example, not a published service standard.

An answer in any language could read naturally while saying "ten days after you contact us". The prose might be excellent. The starting point and the type of day have changed.

A language reviewer can assess whether the phrasing is natural and whether a translation is faithful. The factual reviewer still needs to compare the answer with the procedure. A good translation of an inaccurate answer remains inaccurate.

The same distinction applies when an answer is generated directly in another language rather than translated from English. There may be no intermediate translation to inspect. The question is whether the final answer kept the rule and its conditions.

Research such as MEGA, a multilingual evaluation study published in 2023, evaluates language models across languages and tasks. Its historic results are not a measure of today's assistants or of Lawnise's performance. The practical point here is narrower: language is part of what an evaluation covers, not something an English result settles automatically.

Test the language people actually use

An organisation should identify which languages are used for the particular task being tested. The answer may be used by a customer, a service team or a staff member trying to apply a procedure. Those are different contexts, even when the underlying policy is the same.

Ask people familiar with the work to help prepare realistic questions. A literal translation of an English test question may be useful for a controlled comparison, but it should not be the entire test set. Local terms, abbreviated names and mixed-language phrasing can affect what the question means.

For example, a user might retain an English service name inside a Malay question. A reviewer needs to know whether that name refers to a specific channel or is being used loosely. Replacing it with a general translated term before checking could remove the distinction the answer needed to preserve.

This is a reason to test the wording people use, not a claim that mixed-language questions always produce errors. A test should document which language combinations it covered and which it did not.

Establish which reference governs the answer

Where an organisation publishes multiple language versions of a document, check their versions and effective dates. Do not assume they contain the same provisions because they share a title.

If the versions disagree, ask the responsible owner which applies to the case. Keep the disagreement in the review record. A reviewer should not silently choose the version that makes an answer look right or wrong.

Where the governing record is in one language and the answer is in another, the comparison needs both subject knowledge and language competence. A translated extract can help someone read the evidence, but it should remain linked to the original clause. The review must be able to return to that clause if a qualification is disputed.

This is also why local context matters. Translating a regulator's name does not establish that its rules apply in the user's market. A familiar term can describe a different scheme elsewhere.

Keep the original evidence alongside the explanation

Retain the exact question and complete answer in their original form. Keep the relevant conversation where it changes the meaning. Then identify the sentence or passage being checked, with its governing source.

An English explanation of the finding can be useful for a wider review team. Label translated quotations as translations, and distinguish the reviewer's explanation from what the assistant actually said. Do not replace the original evidence with a tidy English summary.

If the reviewer cannot confidently determine the meaning, mark that limitation and seek suitable review. Guessing is not an acceptable substitute for language competence. An uncertain interpretation should not become a confident adverse finding.

Five checks beyond translation

These checks are the accessible text version of the practical figure accompanying this article:

1. The rule: does the answer state what the applicable reference says?
2. The conditions: did eligibility requirements, exceptions and starting points survive?
3. The quantities: are figures, units, dates and time periods attached to the right requirement?
4. The local terms: do named services, schemes and routes mean the same thing in this context?
5. The question: does the answer address the user's actual task and market?

In the invented deadline example, "ten working days" addresses quantity, while "after all required documents arrive" supplies the starting condition. The answer needs both. Recognising a familiar number is not enough to clear it.

This is an answer-level comparison, not a complete assessment of everything a system can do in a language. The wider enterprise-assistant testing guide explains how to keep questions, references and review outcomes together.

How Lawnise handles language in an answer check

The language of an answer is part of its evidence, not just a presentation choice.

Lawnise's language-aware factual checking path uses the detected language and mixed-language context when analysing a response. In supported configurations, the factual check is instructed to examine the original-language wording, while the finding explanation is reported in English. The evidence view retains the answer alongside its language context and the explanation of the issue.

That lets a reviewer return to the actual wording instead of relying only on an English paraphrase. The languages and checks available depend on the configured system. Detecting a language is not the same as establishing that a particular check can run for it.

Platform example: these two views of the same record show a submitted Malay answer with its detected language label, then the English explanation used to examine its treatment of the deposit-protection limit. PIDM provides separate limits for eligible Islamic and conventional deposits; that distinction is what the reviewer needs to consider. [3] This is an example of the workflow, not evidence of accuracy across Malay-language answers generally.

View the Malay answer at full size

View the finding explanation at full size

State what the language review can support

A useful evaluation records the languages and tasks tested, the kinds of questions used, and the reference versions. It also explains who reviewed the meaning and what remained uncertain.

Do not describe a system as accurate "in Malay" or "in Chinese" on the strength of a small selection of answers. That phrase implies much broader coverage than the record can show. Equally, a discrepancy in one language is not evidence that the language is generally less reliable than another.

NIST's voluntary AI Risk Management Framework connects evaluation with deployment context and recognises the need to document limits. Language belongs in that context when it affects how users ask questions and understand answers.

The result is useful when it makes the boundary clear: these questions, in these languages, were checked against this record. What remains outside that boundary should be visible too.

For help defining a factual and procedural evaluation in a multilingual setting, talk to Lawnise.