A real, anonymized dataset

Every serious evaluation starts with data, and the biggest mistake is starting with a toy. If you evaluate on examples that do not look like your real work, the results will not predict your real work. This is the foundation of your personal evalbench.

Start from real work

Collect actual examples of what you do: the documents you process, the questions you answer, the reports you draft. The closer to production reality, the better. A benchmark of synthetic examples measures a different thing than your job.

Anonymize before anything else

If the data is real, it is likely sensitive. Anonymize it before it goes anywhere, including into your own tooling:

Replace names, addresses, identifiers and any other PII with consistent placeholders.
Keep the structure intact: the anonymized text should still be realistic, so the model faces the same difficulty.
Treat the anonymization as irreversible: the original and the mapping should not live together where the model can see them.

This is non-negotiable for regulated or confidential work, and it is good practice everywhere. Your evalbench must never leak.

What a good eval example looks like

A single example is a task plus a way to judge the answer:

the input (a document, a question, a prompt),
the expected output or the criteria for "correct,"
ideally, why it is in the set (what it is testing).

A dataset of such examples, even a few dozen, is worth more than hundreds of unjudgeable ones.

Keep it small and real

Do not aim for a thousand examples on day one. Aim for twenty real ones that you can judge by hand. A small, honest, real dataset that you can iterate on beats a large, fake one you cannot trust.

The rest of this chapter is about turning this dataset into a pipeline you can rerun and a score you can defend. But it all starts here: real work, anonymized, judgeable.