Who Checks Your Local LLM's Answers? What a Spot Check Taught Us to Decide Before Launch

Who checks the answers of a local LLM that you run in-house? Unlike cloud AI, you run both the model and its settings yourself, so checking the quality of its answers is also up to you. Even when it passed its tests before launch, checking the answers that go out every day in production tends to be left for later.

Our own in-house AI assistant, Mr.AI, was no different. Mr.AI is a local LLM running on our own GPU server. So we built a spot check that pulls some of the answers already delivered to users and verifies the facts in them. The aim is to find hallucinations, meaning plausible statements that are not true, in production answers.

Measured on questions people had really asked, about one in five answers contained a clear error. On top of that, 8 in 10 of the first check's failures were false alarms, and it missed two thirds of the errors. This article covers how the check works, what changed after we fixed it, and what a company adopting a local LLM should decide before ordering. The figures come from our records as of 11 November 2026.

Without a way to check, errors after launch stay invisible

There are three stages at which an LLM's answers can be checked.

The first kind of test is not enough on its own, because it looks only at fixed questions. Research has reported that the behaviour of an LLM service can change substantially in a short time under the same name. A check at answer time scores every answer and blocks the bad ones, but it cannot look deeply without keeping users waiting.

With a local LLM, you have to provide these yourself. Some cloud services, such as Amazon Bedrock, come with an answer-time check built by the provider. A model you run yourself does not come with one. You also swap models yourself, so each swap means checking that the settings and the answers still match.

To learn about errors in production, you need a way to look at delivered answers afterwards, taking time. Our own AI described an internal mechanism that did not exist, and gave a monthly cost more than 20 times too high. Until we built the check, we did not know how often this was happening.

It matters most where the AI's answers are used directly for decisions and actions.

• The AI answers customers or employees directly
• It searches internal documents and data and answers with amounts, quantities and dates
• It not only answers but also acts, registering or sending something

In these settings, one error becomes a wrong order or wrong guidance as it stands. Errors cannot be brought to zero, so a way to measure how often they happen, and keep reducing them, has to be part of the system.

How it works: the checker never sees the delivered answer

We reasoned that a grading AI that reads the answer and is asked "is this correct?" gets pulled along by how the answer is written. So the checker is not shown the delivered answer. It answers again from the same material, and we compare the two.

• Record the evidence. We save the results of the tools the AI called while writing the answer, such as search, exactly as they were at that moment
• Take a sample. When URLs or numbers appear in the answer but in neither the question nor the evidence, the chance of sampling goes up
• Re-answer without showing the answer. The checker receives only the question and the recorded evidence
• Compare claim by claim. We split the answer into factual claims and judge the claims containing values that are not in the evidence
• Feed the result back. A kind of answer that keeps failing is sampled more, and "verify facts with tools" is added to the instructions for generating answers

Answering verification questions independently is known as Chain-of-Verification, and splitting an answer into claims as FActScore. Our check applies these to answers after delivery. The checker also runs on our own GPU, so answers and evidence can be checked without sending them to an outside AI.

What we measured: at first the check itself could not be trusted

We had the assistant answer 40 real questions from our internal chat again, with the production settings, and put every answer through the check. Then we compared every claim, failed or passed, with the evidence one by one, because without looking at the passes you cannot see what was missed. We also changed one number in correct answers and counted whether the check found the planted error.

Had we looked only at the fail count, the check would have seemed to be working well. The errors it found included an answer that guessed a website's purpose from the text of its URL without opening the page, and one that listed more than 40 attachment file names that did not exist.

There were two reasons the check did not work well.

• The check did not know the AI's memory. Mr.AI remembers what it is told about projects and team members and uses it in answers. The check had recorded only tool results, so correct claims based on memory were marked "no evidence"
• The checker shared the same assumptions. More than 80% of the misses were a joke and an old model name left in the system prompt, the instructions the AI reads first. We had not updated the prompt when we swapped the model. The checker ran with the same prompt, gave the same errors, and the two answers agreed

An error written into the system prompt is not something a check should catch; it is something to fix in the prompt itself.

Results after fixing and measuring again

We fixed the prompt and the check in two rounds on the same day. We corrected the self-description in the prompt, added the AI's memory to the evidence, and had formulas written in an answer recomputed by a program.

When the assistant answered again with the fixed prompt, the share of answers containing a clear error fell from about one in five to under one in ten. Removing the source of an error works better than finding the error with a check.

Half of the remaining errors were miscalculations. A monthly cost that should have been about 590 yen was given as about 13,000 yen. The AI judge passed it, and the check that recomputes formulas with a program found it.

We also compared it with a conventional check in which the grader is shown the answer. The hidden-answer check found 67% of the real errors; the shown-answer check found 22%. On errors that were just a changed number, the two were level at 94% and 97%. We cannot yet say the gap is the effect of hiding the answer alone, and sampling in production has only just started.

What a company adopting a local LLM should decide before ordering

From this measurement, we suggest settling six points before you order a local LLM build or a system with one built in. Asking a development company "how will you check the answers after launch?" is where the answers differ.

• Whether the evidence behind each answer is recorded. If the search results, prompt and memory the AI read are not kept, nothing can be verified later
• Who checks, and how often. At first a person has to look at both the passes and the fails of the check itself
• Whether calculations of amounts and quantities are left entirely to the AI. Formulas can be recomputed by a program
• Whether there is a procedure for fixing the prompt when an error is found. In our case the fix to the prompt did not reach existing conversations
• Whether there is a procedure for reviewing the prompt and the tests together when the model is swapped. Most of our misses came from here
• How far the check may reach outside. A checker that uses tools will open external pages

Building these in at the design stage means less rework than adding them later, because recording the evidence is part of the process that writes the answer.

When you consult us, we first confirm what can be recorded. We then run the same measurement as in this article on your real questions, and report the share of errors, their causes and the order in which to fix them.

FAQ

Can a spot check be added to an AI that is already running?

Yes, if the system can record the evidence behind its answers, such as search results. With an off-the-shelf service that cannot record it, what can be done is limited. We start by confirming what can be recorded.

Isn't a check on every answer enough?

No. A check at answer time stops obvious errors before delivery. How much it misses can only be learned by sampling and examining answers after delivery.

How much does a spot check add to the cost?

It depends on how much is recorded, the share sampled and how much a person reviews. It is common to check everything while usage is low and narrow it to a sample as usage grows. In the estimate consultation we propose a scope that fits your use.

Can a spot check eliminate hallucinations?

No. A spot check tells you where errors happen and how often. They only decrease once you act on that, by fixing the prompt or adding tools.

Summary

Unless you have your own way to check delivered answers, a local LLM's errors stay invisible. In our local LLM, about one in five answers contained a clear error, and by fixing the check and the prompt we brought that under one in ten. The check itself can only be trusted once both its fails and its passes have been reviewed.

When we estimate a local LLM build, or a system with one built in, we include how the answers will be checked: recording the evidence, the check, and human review. Whether you are about to adopt a local LLM or are concerned about the answers of one already running, ask a project manager through our free estimate consultation. Our local LLM build and implementation page describes how we work, from hardware selection to operations.

Related articles and pages

• Local LLM build and implementation: hardware to operations
• Local LLM case study: cloud AI or local, feature by feature
• AI agent case study: Mr.AI takes requests to production
• Delegating work to AI: scope, checks and fixing failures
• Local LLM hardware requirements and recommended models
• Improving a local LLM agent: what a harness can and can't change

Sources

• Amazon Web Services, "Use contextual grounding check to filter hallucinations in responses", Amazon Bedrock User Guide
• Lingjiao Chen, Matei Zaharia and James Zou, "How is ChatGPT's behavior changing over time?", 18 July 2023
• Shehzaad Dhuliawala et al., "Chain-of-Verification Reduces Hallucination in Large Language Models", 20 September 2023
• Sewon Min et al., "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation", 23 May 2023