The Harness Matters More Than the Model

Theme of this issue: Reading through what landed in my feed this week, I kept bumping into the same idea from different angles. Not a coordinated thesis. Just four people working on different problems, and the pattern that emerged is hard to ignore.

🎯 Hook I spent the week reading. Mario Zechner on why he built Pi. Addy Osmani on Agent Skills. antirez on ds4.c. Sidhesh Badrinarayan on Eval-Ops.

Different problems. Different angles. Same undercurrent.

The model is a token generator. The harness — workflow, inference engine, evaluation loop, guardrails, process — is where the real leverage is.

🔥 Hot Take 💬 My take: The AI coding discourse is obsessed with the wrong question. Everyone asks "which model is best?" The better question is "what harness are you running it in?"

Reading across these pieces, here's what I see:

Mario's story: Claude Code's harness changed daily. The model was fine. The instability was in the tooling around it. He built Pi to own the full stack.

Addy's insight: Models skip the invisible parts of engineering — specs, tests, scope discipline. His Agent Skills project encodes those as workflows the model can't talk its way out of.

antirez's bet: General inference runtimes don't optimize for agent workflows. His one-model engine treats KV cache as a disk citizen and ships ready-to-use agent configs because that's what the use case demands.

Sidhesh's argument: Most teams "vibe check" their agents with n-gram metrics that measure word overlap, not intent. G-Eval with structured rubrics catches what ROUGE/BLEU miss.

None of them set out to prove the same point. But reading them together, the pattern is clear: the model is the least differentiated part of the stack. The harness — how you run, constrain, evaluate, and integrate it — is where quality lives.

I've been writing flow.md files in every component before coding. That's a harness choice, not a model choice. The practice of documenting before generating — that's where the quality comes from.

📰 Top Articles

1. Mario Zechner (Pi Creator) Interview — The Good Startup TL;DR: Mario reverse-engineered Claude Code when it became unstable, built Pi for full control over his harness. Key insight: most "model degradation" complaints are actually about harness churn — Claude Code's team changes the system prompt, features, and behavior daily. He hasn't seen model degradation in Pi because the harness stays stable. Also: open-weights (Kimi K2.6, DeepSeek V4) are closing the gap. His linguist wife 5x'd her output using agents for internal tools. Architecture > syntax is the new reality.

My take: This is the most honest AI coding interview I've seen. Mario isn't selling a vision — he built a tool because the existing one broke. The "harness stability" point is the one nobody talks about. Teams switching models chasing "better intelligence" are often cargo-culting. The stable harness + good enough model beats unstable harness + frontier model every time.

2. Agent Skills — Addy Osmani TL;DR: 20 markdown skills across 6 SDLC phases (Define → Plan → Build → Verify → Review → Ship). Core design principle: process over prose. Workflows with checkpoints and exit criteria beat 2,000-word essays on best practices. Most innovative feature: anti-rationalization tables — pre-written rebuttals to excuses agents use to skip steps ("this task is too simple for a spec" → "acceptance criteria still apply"). 27K stars in days.

My take: The anti-rationalization tables are genius and immediately stealable. LLMs are excellent at rationalizing shortcuts. Pre-writing the rebuttals is cheaper than debugging the result. I'm adding this pattern to my AGENTS.md tomorrow. Also: verification as non-negotiable exit criterion — "seems right" is never sufficient. This maps directly onto my flow.md practice.

3. ds4.c — antirez (Salvatore Sanfilippo) TL;DR: Narrow Metal inference engine for DeepSeek V4 Flash only. 284B MoE model, 2-bit asymmetric quants (routed experts at IQ2_XXS/Q2_K, everything else untouched), fits 128GB MacBooks at 21-27 t/s generation. Key innovation: KV cache as disk citizen — SHA1-keyed, survives restarts, saves at cold/continued/evict/shutdown points. Ships ready-to-use configs for Pi, OpenCode, Claude Code. Antirez claims DS4 Flash thinking is proportional to problem complexity (1/5 the tokens of other models).

My take: The disk KV cache approach is a genuinely new idea from someone who's been thinking about storage systems for decades (Redis creator). Most inference engines treat KV cache as RAM-only. antirez argues fast SSDs change the tradeoff — and he built the proof. Also: the narrow bet ("one model, tested against official logits, agent-ready") is the same philosophy Mario used building Pi. Opinionated > general-purpose when the use case is clear.

4. Eval-Ops — Sidhesh Badrinarayan TL;DR: Most teams "vibe check" their agents — manual transcript reviews, console logs, outdated string matching. ROUGE/BLEU/METEOR measure n-gram overlap, not semantic intent. The State Retention Dilemma: user says "no store credit," agent refunds to Visa (correct). ROUGE says FAIL (zero word overlap). Human says PASS (intent matches). Solution: G-Eval (LLM-as-Judge) with Chain-of-Thought reasoning and structured rubrics. Framework: DeepEval turns evaluation into unit tests.

My take: The State Retention Dilemma is the clearest explanation I've seen of why traditional metrics break for agents. The CoT-as-debug-log idea is practical — when an eval fails, you get a trace of exactly which reasoning step tripped. This is the evaluation layer of the harness. Without it, you're flying blind. With it, you catch regressions before they ship.

🛠️ Tools & Releases agent-skills (MIT) — 20 markdown skills for Claude Code, Cursor, Gemini CLI, Pi, OpenCode. Slash commands: /spec, /plan, /build, /test, /review, /ship, /code-simplify.

ds4.c — DeepSeek V4 Flash inference engine for macOS Metal. q2 GGUF for 128GB MacBooks, q4 for 256GB+. OpenAI + Anthropic API compatible.

DeepEval — Python framework for LLM evaluation as unit tests. G-Eval as objects, immutable criteria, CoT tracing.

💡 Dev Tip of the Week Steal the anti-rationalization table pattern.

Addy Osmani's agent-skills repo includes tables in every skill that look like this:

Excuse Rebuttal "This task is too simple to need a spec" Acceptance criteria still apply. Five lines is fine. Zero lines is not. "I'll write tests later" Later is the load-bearing word. There is no later. Write the failing test first. "Tests pass, ship it" Passing tests are evidence, not proof. Did a human read the diff?

Why it works for humans too: Most engineering decay isn't bad intent. It's people accepting plausible-sounding justifications for skipping the parts they don't feel like doing. Write down your team's common excuses and the rebuttals. Put it in your AGENTS.md or your wiki. It will save you arguments and catch the next tired Friday-afternoon shortcut.

🤔 Community Question ❓ What's one pattern you've added to your harness (AGENTS.md, flow.md, eval suite) that caught a regression your model would have missed?

📌 What I'm Learning / Building This week confirmed something I've been feeling for months: the model is the least interesting part of the stack.

I've been refining my flow.md practice — every component gets a planning doc before a line of code is written or generated. The docs capture: purpose, data flow, state, edge cases, test scenarios. AI then scaffolds the implementation and tests from the spec. I review every line.

The surprising insight from this week's reading: I should add anti-rationalization tables to my flow.md practice. When the agent argues "this component is too simple for a flow.md," I need a pre-written rebuttal. Small practice, big leverage.

Also: I'm tempted to try ds4.c for local agent work on a Mac Studio. The disk KV cache approach and agent-first design philosophy align with everything else I've been reading about harness quality.

📎 Sources for This Issue https://addyosmani.com/blog/agent-skills/

https://github.com/antirez/ds4.c

https://hackernoon.com/the-era-of-vibe-checking-ai-is-over-welcome-to-eval-ops

https://www.youtube.com/watch?v=sqtX2OmgOF0

https://arxiv.org/abs/2303.16634 — G-Eval paper