Replay and re-evaluate

You have a dataset, criteria, and a first set of scores. The method only becomes powerful when you loop: change one thing, replay the whole benchmark, re-evaluate against the same criteria, and compare. This loop is the entire point of the evalbench.

The loop

1. Make one change: a different model, a different quant, a different prompt, a different runtime setting.
2. Replay the same dataset through the same pipeline.
3. Re-evaluate with the same written criteria.
4. Record the new scores next to the old ones.

That is it. The magic is that after a few loops you have a table of "what actually helped" instead of a pile of hunches.

What this loop buys you

• Confidence. When a change makes the number go up and stay up, you trust it. When it does not, you revert without regret.
• Cheap experiments. Because you can replay, you are free to try things, which is how you find the good ones.
• A defensible story. "I tried these twelve configurations; this one won by these numbers" is a claim you can make to a client or a boss.

The rules

1. One variable per run. If the score moves, you must know why.
2. Same dataset, same criteria. Changing the test mid-way invalidates the comparison.
3. Keep losing runs too. The negative results are as valuable as the wins; they are what stop you from repeating mistakes.

The honest warning

The loop will sometimes show that your exciting change did nothing. That is not a failure of the method; it is the method working. Most changes do not help, and knowing which is exactly the advantage you are building.

Replay and re-evaluate until the loop is a habit. That habit, more than any model, is what turns someone who "wired a RAG" into someone who can prove it works.