The eval harness you can build before lunch

Most LLM features ship on vibes. Someone tries five prompts, the answers look fine, the PR merges. Two weeks later a prompt tweak fixes one bug and quietly breaks three others, and nobody notices because nobody was counting. This is the smallest eval harness that fixes that. It fits in two files and one morning.

What you are building

Four stations and one number. A file of real test cases. Cheap checks written in plain code. A model-graded check, used only where code cannot decide. A run file per commit, so you can see what changed. At the end of a run you get a line like 27/30 passed at a1c9e02, and a list of the three that failed with the reason for each.

That is the whole thing. No dashboard, no platform, no vendor. You can add those later if the number turns out to matter, and it will.

Station 01: write thirty cases, not three thousand

Thirty well-chosen cases beat a scraped dataset of thousands you never read. Pull them from three places:

• Real inputs. Copy twenty messages from logs, support tickets or your own testing. Real users phrase things in ways you would never invent.
• Known bugs. Every bug you have already fixed becomes a case, so it stays fixed.
• Arguments. Any input your team has debated ("should it refuse this?") becomes a case with the answer you agreed on. The eval file is where those decisions get written down.

Store them as JSON Lines. One case per line, an id you can grep for, the input, and whatever expectations apply to that case. Not every case needs every field.

{"id": "refund-basic", "input": "How do I get a refund for order 1182?", "must_include": ["refund"], "must_not_include": ["I can't help"], "max_chars": 800}
{"id": "extract-invoice", "input": "Invoice 4471, due March 3, total $1,240.00", "json": true, "regex": "\"total\":\\s*1240"}
{"id": "angry-customer", "input": "Third time asking. Where is my order?", "rubric": ["Does the reply acknowledge the delay without blaming the customer?", "Does it give one concrete next step?", "Does it avoid promising a delivery date it cannot know?"]}
{"id": "off-topic", "input": "Write me a poem about my ex", "must_include": ["orders"], "max_chars": 300}

Station 02: check the cheap things with code

Most of what you care about can be checked without a model. Does the answer mention the thing it must mention? Does it avoid the thing it must never say? Is the JSON valid? Is it short enough to fit in the UI? These checks are fast, free, and never disagree with themselves.

Write check() so it returns a list of failure messages. An empty list is a pass. The messages are what you will read when something breaks, so make them specific.

# evals/run.py
import json, re, subprocess, sys, time
from pathlib import Path

from app import answer      # the feature under test
from judge import judge     # station 03, below

CASES = Path("evals/cases.jsonl")


def check(case, output):
    fails = []
    for s in case.get("must_include", []):
        if s.lower() not in output.lower():
            fails.append(f"missing: {s!r}")
    for s in case.get("must_not_include", []):
        if s.lower() in output.lower():
            fails.append(f"forbidden: {s!r}")
    if "regex" in case and not re.search(case["regex"], output):
        fails.append(f"no match: {case['regex']}")
    if case.get("json"):
        try:
            json.loads(output)
        except ValueError:
            fails.append("not valid JSON")
    if "max_chars" in case and len(output) > case["max_chars"]:
        fails.append(f"too long: {len(output)} chars")
    # Only pay for the judge when the cheap checks passed.
    if "rubric" in case and not fails:
        verdict = judge(case["input"], output, case["rubric"])
        if not verdict["pass"]:
            fails.append(f"judge: {verdict['reason']}")
    return fails

Station 03: use a model judge only where code cannot decide

Tone, helpfulness and "did it actually answer the question" are hard to check with string matching. A second model call can grade them, if you set it up so its grades mean something.

• Ask yes or no questions, not scores. "Rate helpfulness from 1 to 10" produces numbers that drift between runs and mean nothing. "Does it give one concrete next step?" has an answer.
• Reasoning first, verdict last. Let the judge think through each question, then end with one line of JSON. Parse the last line.
• Temperature zero. You want the same grade for the same output.
• Grade the judge. Label twenty outputs yourself, run the judge on them, and compare. If you disagree with it more than a couple of times, the rubric is vague. Fix the questions before you trust the number.

# evals/judge.py
import json
from llm import complete    # your model client, whichever one you use

PROMPT = """You are grading one reply from an assistant.

User message:
<input>{input}</input>

Reply:
<reply>{output}</reply>

Answer each question with yes or no:
{questions}

Think through each question in a sentence or two. Then end with exactly
one line of JSON and nothing after it:
{{"pass": true or false, "reason": "the first question that failed, or ok"}}
pass is true only if every answer is yes."""


def judge(input, output, rubric):
    questions = "\n".join(f"{i + 1}. {q}" for i, q in enumerate(rubric))
    text = complete(PROMPT.format(input=input, output=output, questions=questions),
                    temperature=0)
    return json.loads(text.strip().splitlines()[-1])

Use the strongest model you can afford as the judge, and keep it pinned to a specific version. If the judge changes under you, your pass rate moves for reasons that have nothing to do with your feature.

Station 04: run it and keep the receipts

Every run writes a file named after the time and the commit. When the pass rate drops, you diff the failures between the last good run and this one, and you know which cases broke and how.

def main():
    sha = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
                         capture_output=True, text=True).stdout.strip()
    cases = [json.loads(l) for l in CASES.read_text().splitlines() if l.strip()]
    results = []
    for case in cases:
        output = answer(case["input"])
        fails = check(case, output)
        results.append({"id": case["id"], "pass": not fails,
                        "fails": fails, "output": output})
        print("PASS" if not fails else "FAIL", case["id"], *fails)

    passed = sum(r["pass"] for r in results)
    print(f"\n{passed}/{len(results)} passed at {sha}")
    out = Path(f"evals/runs/{time.strftime('%Y%m%d-%H%M%S')}-{sha}.json")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(results, indent=2))
    sys.exit(0 if passed == len(results) else 1)


if __name__ == "__main__":
    main()

The non-zero exit code means you can drop python evals/run.py into CI on any change that touches a prompt, a model setting or a tool definition. Start with it as a warning, not a blocker, until the cases settle down.

Running it well

• Run the flaky ones more than once. If a case passes two times out of three, that is a finding, not noise. Add a "repeat": 3 field for cases where it matters and count passes.
• Read the failures, not just the number. A pass rate that went from 27 to 28 can hide a new failure that matters more than the one you fixed.
• Every bug becomes a case. When a user reports something wrong, the fix is not done until there is a case that would have caught it.
• Do not chase 30 out of 30. If every case passes, your cases are too easy. Add harder ones.

Shift checklist

• Thirty cases in evals/cases.jsonl, at least twenty from real inputs
• Code checks for anything code can check
• Judge rubric written as yes or no questions, judge pinned to one model version
• Judge spot-checked against twenty of your own labels
• One run file per commit in evals/runs/
• Runs in CI on every prompt or model change

Next on the shift: where your context window actually goes, and why your agent runs out of room by turn thirty.

Filed by the operations desk at The Paperclip Factory. We make workwear for people who ship models. If someone on your team keeps the evals green, the AI engineer gifts shelf is down the hall.