How a Small Open Model Beat a Frontier LLM: the Rise of Harness-Maxxing
Why Qwen, DeepSeek and MiniMax can look better than Claude on benchmarks and still not feel better when developers use them.
Qwen3.6-27B tied Sonnet 4.6 on Terminal-Bench 2.0: 59.3 against Sonnet 4.6’s 59.1. A 27-billion-parameter open-weight model that can run on local Mac Minis, levels with a frontier commercial one.
Then, an independent researcher achieved even more mind-blowing results. They pushed the Qwen3.6-27B to 90.0 on SWE-bench Verified, using Anthropic's Claude Code CLI. Meanwhile, Sonnet 4.6's published score was 79.6.
Newly released DeepSeek V4 Flash appeared to be outperforming Anthropic's flagship models on Terminal-Bench. V4 Flash is a genuinely sparse model: 304 billion total parameters, with a fraction of them active on any given token. The July release reports 82.7 on Terminal-Bench 2.1. Claude Code with Opus 4.8 and Sonnet 5 scores 78.9 and 74.6 respectively on the official leaderboard. The direct comparison with Opus 4.8 reported by V4 Flash puts Opus 4.8 only slightly above with 85.0 points
MiniMax-M3 also scored 80.5% on SWE-bench Verified, above Sonnet 4.6. Its official evaluation record says it used Claude Code as the scaffolding, ran on MiniMax’s internal infrastructure, and averaged four runs.
And yet, does anyone even remember those models being that good?
Fable 5 is good. GPT-5.6 Sol is good. Sonnet 4.6 was good. But Qwen3.6-27B was only 2.4 points behind Sonnet 4.6 on the official SWE-bench Verified scores, and then 10.4 points above it in the engineered run. But was it really that good?
That does not match the empirical experience of developers. Anecdotes on X are weak evaluation data, though there are many of them that stated that the small Qwen is nowhere near the frontier.
An interesting story happened in Qwen’s github. In Qwen's issue tracker, a developer set out to reproduce the model card's 53.5% on SWE-bench Pro and got roughly 28% with a bash-only agent. Adding a str_replace file-edit tool took it to 50.7%, a confidence interval that contains the published figure. One specific tool in the harness increased the performance by almost double.
Which means that all those numbers are about a harness and not about the model.
In fact, the model itself does not mean that much in those evaluations.
Large language models are still next-token predictors. They can output text, code, plans and structured requests to use tools. Basically, they just yap. They have no agency by themselves. A model cannot open a file, run a command, preserve a task across sessions, ask permission before sending an email or check whether its code works.
What is a harness?
“LLM Y scores X on Terminal-Bench 2.1!”
This sentence tells you absolutely nothing.
The right sentence would be: LLM Y in Claude Code scores X on Terminal-Bench 2.1 at high effort. That is how the official leaderboard actually records a result: an agent, a model, an effort level. Another meaningful sentence: Moonshot published the Kimi Code harness, and Kimi K3 in Kimi Code scores 88.3 on Terminal-Bench 2.1 at max effort.
Look at what the footnote on Moonshot’s repo concedes. Kimi K3 is evaluated with the Kimi Code harness, while the competing columns report the best score across harnesses.
And when Artificial Analysis ran the same K3 on the same benchmark through their own harness, it scored 85.0, 3.3 points less.
When we evaluate the performance of an agentic system (system! not model!), we need to specify both harness and model.
So what exactly is a harness?
Microsoft calls it the scaffolding that turns a language model into an agent. A model by itself can only generate output. The harness drives it, executes tools, manages context, applies permissions and keeps it moving towards completion. LangChain’s Vivek Trivedy puts it as an equation: Agent = Model + Harness, where the harness is “every piece of code, configuration, and execution logic that isn’t the model itself.”
An agentic harness consists of a loop, tools, context, policy and persistence around the model
1. The harness runs the loop
It calls the model, receives a proposed action, executes or rejects that action, returns the result to the model, and decides whether to continue or stop.
2. The harness holds the tools
It decides whether the agent gets a shell, a file editor, a browser, GitHub, a database or an email sender. It also writes the tool descriptions and schemas that help the model select and use the right tool.
3. The harness manages context
It assembles the system prompt, user request, project instructions, selected files, conversation history and tool results. When the context becomes too large, it decides what to compact, prune, summarise or store elsewhere.
4. The harness applies policy
It permits, denies, sandboxes and asks for approval. A prompt can request that the agent refrain from sending an email. A permission rule can block the action until a human approves it.
5. The harness keeps going
It persists the task state, tracks progress, resumes after interruption, retries failed operations, delegates to subagents and decides what counts as complete.
I did not derive this list from Microsoft’s documentation, but it lands in the same place: loop, tools, context, approval policy, progress to completion.
The model depends on the harness for every job it gets.
There are also two layers of harness. Birgitta Böckeler draws this distinction as builder harness and user harness; I find inner and outer easier to say, but the layering is hers.
The inner harness is what the vendor built: the loop, basic tools, context management, orchestration and permission system inside Claude Code, Codex, Copilot, Kimi Code or OpenClaw.
The outer harness is what the user builds: project instructions, skills, examples, MCP servers, tests, validators, hooks and permission rules.
Böckeler divides the outer harness into guides and sensors. Guides act before the model does something: instructions, examples, skills and good tool descriptions. Sensors act afterwards: tests, linters, schema checks and reviewers that tell the agent what it got wrong.
For practical use I would add a third part: caps. A cap is a boundary the model cannot talk its way past. Guides advise and sensors report, but neither can stop anything. Permissions or infrastructure enforce a cap.
The model sits inside an inner harness, surrounded by an outer harness of guides, sensors and caps
How harnesses impact the performance
Terminal-Bench 2.1 shows the impact very clearly, because it reports models together with the harnesses that ran them and the effort level each used.
The same model produces different Terminal-Bench 2.1 scores in different harnesses
The same GPT-5.5 model scored 83.1% in Codex and 78.0% in Terminus 2, both at xhigh effort: a 5.1-point difference.
The same Opus 4.7 model scored 68.9% in Claude Code and 66.1% in Terminus 2, both at max effort: a 2.8-point difference.
Become a paid subscriber to AI Realist and unlock:
20% off Optimise Your AI Stack - live, hands-on, a small group
full access to your personal AI Realist workspace — all materials, slides, games, and events in one hub.
yearly paid subscribers get access to interactive AI Realist courses: watch module 1 of Optimise Your AI Stack for free here
full archive of the articles
direct chat with the response within 48 hours
https://msukhareva.substack.com/subscribe
