Version the dataset and pipeline

An evaluation you cannot reproduce is not an evaluation, it is an anecdote. The whole point of a personal evalbench is that you can run it again after every change and trust the comparison. That requires versioning both the data and the code that runs it.

Version the dataset

Your examples are data, so treat them like data:

Keep them in a versioned file (a JSON or a set of files), not in a notebook or a chat.
Every change to an example is a change to that file, with a commit message that says what changed and why.
Freeze a version when you start a comparison, so "before" and "after" use the same examples.

If you cannot point to the exact dataset that produced a number, the number is not reproducible.

Version the pipeline

The pipeline is everything that turns an example into a score: the prompt, the model, the quant, the runtime settings, the scoring code. Version it the same way:

Keep the prompt in a file, not in your head.
Record the model, quant and settings with every run, not "the usual setup."
Commit the scoring code, so "score" means the same thing run to run.

Change one thing at a time

The core discipline of the method:

To test a new model, keep everything else identical and change only the model.
To test a new quant, change only the quant.
To test a new prompt, change only the prompt.

If you change three things at once and the score moves, you do not know which one caused it. One variable per run.

The output

A run produces a result you can re-read later: which dataset version, which pipeline version, which scores. When someone (or future-you) asks "why did we choose this model," the answer is in that record, not in memory.

Reproducibility is not bureaucracy. It is the difference between expertise and a feeling, and it is exactly the thing that lets you stand in front of a client with numbers instead of enthusiasm.