Annotate what "good" means

You have real examples. Now you have to decide, in advance, what counts as a correct answer. If you annotate after seeing the model's output, you are not measuring the model, you are measuring your own optimism.

Define the criteria first

For each example, write down what makes an answer good before you run anything:

• Is it a binary pass/fail, or a scale?
• What is the essential fact that must be present?
• What would count as "wrong but close" versus "completely wrong"?

The criteria must be written, because a judgement you keep in your head will drift as you see more outputs.

Keep it judgeable

A good criterion is one a second person could apply consistently. "Is the answer helpful?" is not judgeable. "Does the answer state the correct deadline, and not contradict the source?" is.

For a small personal evalbench, a simple rubric per example is enough: one or two must-have facts, plus a pass/fail. You do not need an academic annotation scheme; you need consistency.

Annotate once, use many times

The point of fixed criteria is that you apply the same standard to every model, every quant, every prompt. If the criteria change, you have changed the test, and you can no longer compare across runs.

The honest trap

The trap is grading on vibes after the fact. You run the new model, it "feels" better, so you score it better. That is not evaluation, that is confirmation bias with extra steps. Fixed, written criteria are the antidote.

The output

A dataset where every example carries its own criterion for success. That turns "I think it got better" into "it passed 17 of 20, up from 12, by the same standard." That sentence is what the whole method is for.