How Far Can a Harness Improve a Local LLM Agent?

"Can you build an AI agent on a local LLM?" We hear this more and more in local LLM consultations.

You can. We built our own AI agent on a model that runs on one GPU (RTX 5090), and from July to October 2026 we evaluated and fixed it more than ten times against the same tasks. It now handles small, well-scoped fixes from our own development requests.

Our model comparison is in 10 local LLMs compared on Japanese business work. This article is about the system around the model, the harness. The harness made the agent ten times faster, but the model set the ceiling on how capable it was. All figures were measured in our own environment.

What a harness is

An AI agent works through a task by itself, planning steps, running commands and editing files. The model is the part that thinks. The harness is everything around it: it hands the model tools, builds the instructions, runs the action the model picks, returns the result and decides when to stop.

Our harness takes one action at a time. A fast role without reasoning normally drives the work, and a slower role with reasoning is called only when failures pile up. Both roles use the same model. We built our own harness to fit our tools, rules, permissions and logs.

How we evaluated

After every change to the harness, we compared it with the previous version on the same tasks.

• Our own tasks: 12 general checks (file handling, calculation, asking about a vague request and more) and up to 13 coding tasks. Code grades them by test results and the files produced. We also recorded steps, tokens and time
• A public benchmark: Terminal-Bench. We ran the same model in the standard harness (Terminus-2), which the Terminal-Bench team built for comparing models and which has a single tool, a terminal, and in ours
• Real development requests: the agent solved past requests from the same starting point, and we compared its changes with those of the cloud AI (Claude)

What changed: speed improved tenfold

The first version planned before every action and always checked at the end. The 12 general checks took 1,224 seconds in total, and on a vague request the planning role decided the content itself instead of asking.

The second version let the fast role drive and called the reasoning role only in cases such as two failures in a row. It passed all 12 checks in 123 seconds in total.

Removing small snags helped too. The agent kept typing a bare "python" and failing, so we made the right Python available from the start, and eight coding tasks went from 133 to 83 seconds.

What didn't change: the public benchmark score

On Terminal-Bench (78 of the 80 tasks in terminal-bench-core 0.1.1), the same Qwen3.6-27B scored 51.3% in the standard harness and 46.2–48.7% in ours, however much we improved it. On the first run it solved only one of the first 11 tasks, because of long commands timing out, transient network errors and a work log that grew past what the model could read. All were harness bugs, and we fixed them before comparing.

Ours won on exploration, data processing and code fixes, where checking each result before the next step paid off. The standard harness won on tasks that need a long sequence of commands typed in the right order.

When we read the 42 failed tasks one by one, only 6 had run out of steps. In 28, the agent reported "done" and was wrong.

• It formatted its own output, checked it itself and judged it correct. The grader looked at the unformatted form
• It tried only normal inputs, never edge values or invalid input

It did check, but it checked against its own assumptions rather than the task's acceptance criteria.

The version with a fix for each cause reached 48.7%, solving 7 new tasks while losing 5. We also built a shell that keeps its state between commands, but the score did not rise, so it is off by default. Among the tasks the final version failed, the largest group was fixes we had added that did not work (14 tasks). The gaps between versions stayed within run-to-run noise (3 to 4 tasks). We concluded that with this model, the harness alone is unlikely to pass 60%.

Added mechanisms caused new failures

Reading the work logs one by one turned up failures on the harness side.

• When long output was cut in the middle, the model filled the gap with plausible content, for example writing "3.22.4" as "v32.4". Saying that output was cut and asking it to reread prevented this
• The checking AI reversed "konnichiwa" wrongly and sent back the correct answer. The checker is the same model, so it makes the same kinds of mistakes
• A file-write action missed its target path, and a misleading "that is a folder" error made the agent repeat the action ten times. Saying exactly what arrived and what was missing fixed it
• Raising the output limit to 16,384 tokens let a repeating fragment run to the limit and wasted 241 seconds. We now stop repetition when detected and keep the limit at 8,192
• A bug in a feature we added made the file-editing tool empty files. The agent restored them from its own backup and finished, so the pass result hid the bug; we noticed only because steps doubled to 108

Real development requests: small fixes can be delegated

On small fixes it produced nearly the same change as Claude. On larger ones it stopped at the smallest change that made the symptom go away. It took about three to six times as long. Stating the layer to fix and the tests to add in the request makes up for this. On one real request it changed four files including two tests, passed all 1,971 existing tests and opened a pull request. Once, running as a background service, it stopped because a development tool (pnpm) was missing, so evaluate the agent the way it will run in production.

Swapping the model raised the ceiling

In August we kept the harness and swapped Qwen3.6-27B for Qwen3.8-27B. On the general tasks, output tokens fell to about 40% and each task took roughly a half to a quarter of the time. Muse Glimmer 30B, tried in October, passed the same number of tasks and took about 2.2 times as long per task on coding and about 1.2 times on general tasks (comparison article). The Terminal-Bench paper reaches the same conclusion: model selection is usually more important than the agent scaffold for performance.

So we keep the harness easy to move to a new model, and measure every new model again on the same tasks.

Tips for evaluating an agent

• Build tasks from your own work and grade them in code, not by impression
• Record steps, tokens and time, not just pass or fail. A broken agent can still finish
• Run the same version of a public benchmark several times. Noise can be larger than the gain from a change
• Read failures one by one before adding anything. The cause we assumed (too few steps) was not the real one (wrong but reported done)

Limits of this evaluation

• Our task set is small, and each Terminal-Bench version ran once. A gap of a few tasks is within the noise
• Terminal-Bench was measured on its early task set (terminal-bench-core 0.1.1), not the current version
• Only a few development requests were compared, and Claude's times cover its whole handling of each request
• The results reflect our harness and our environment (one RTX 5090, models quantised to 4-bit)

FAQ

Can you build an AI agent on a local LLM?

Yes. We run an agent on Qwen3.8-27B, which fits one 32 GB GPU, that passes all of our own tasks (13 coding tasks, 12 general checks), and we hand it small development fixes.

Should we use an off-the-shelf harness or build our own?

On the public benchmark, the standard harness beat ours. Measure with an off-the-shelf harness first, and build your own once you know what your tools, rules, permissions and logs need.

Can a better harness match a cloud AI?

No. A harness mainly changes speed, stability and how failures are handled. The ceiling on ability is set by the model.

Can a local agent take on development work?

Small, well-scoped fixes, yes. For larger ones, state the layer to fix and the tests to add in the request, and plan on human review.

Where do we start with evaluating an agent?

List about 10 tasks you want to hand over and define, for each, a pass condition that code can check.

Summary

The harness made our agent ten times faster. The public benchmark score, though, did not rise however much we improved it, and swapping the model is what raised the ceiling. Build the harness for speed and stability, and choose the model for ability.

We cover AI agents for your work from design to evaluation tasks, model and hardware selection, and operation (AI development support). If you want to hand work to AI without data leaving your company, ask a project manager through our free estimate consultation. Questions as early as "we want to see how much we can hand over" are welcome.

Related articles and pages

• Local LLM build and implementation: hardware to operations
• 10 local LLMs compared on Japanese business work: how to choose and delegate
• Muse Glimmer 30B vs Qwen3.8-27B on the same agent evaluation
• AI agent case study: Mr.AI takes requests to production
• Delegating work to AI: scope, checks and fixing failures

Sources

• Terminal-Bench, "Terminal-Bench" (measured with terminal-bench-core 0.1.1)
• Merrill et al., "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces", arXiv, 17 January 2026
• vLLM, "vLLM", GitHub
• Qwen, "Qwen3.8-27B", Hugging Face, released August 2026
• Meta, "Muse-Glimmer-30B", Hugging Face, released August 2026