Lesson 10 · Part 1. How language models work · 12 min

Evaluation

Goal. Explain why you need a held-out test set, read a score table, and know what a score does not tell you.

Why you need it

In classic app work you write a test, and it passes or fails. A language model is different. Its output changes with the prompt, the temperature and the model. Sometimes it looks right and is wrong. You cannot trust a few demos. You need a test set: a list of inputs with the correct outputs, and a script that scores the model's answers.

For our task, a good answer is a JSON object. A scoring script can compare it with the correct JSON, field by field. Our scorer reports these numbers.

exact match. The share of phrases where every field is correct. It is strict: one wrong field fails the phrase.
recall. Of all the fields that should be in the answer, how many did the model produce.
precision. Of all the fields the model produced, how many were right.
Invented fields. Fields in the answer that the user never asked for.
Unparseable. Answers that are not valid JSON at all.

Why the test set must be held out

A held-out test set is a set of examples that you never use for training. You also never use it to adjust your prompt in a way that copies its answers.

Imagine a student who sees the exam questions before the exam. A perfect score then shows memory, not understanding. A model is the same. If a test phrase is also in the training data, the model may simply remember it.

This risk has a name: overfitting. An overfitted model scores very high on its training examples and poorly on new input. The held-out set is how you detect it.

In our project we took these steps:

We trained on synthetic phrases, and we checked that no training phrase equals any test phrase. We found 0 overlaps.
We removed training phrases that were nearly identical to the test phrases.
We wrote a second, separate test set of 40 messy phrases by hand, with typos and mixed constraints, to check that the score is not only good on the first set.
We scored every approach, including the old regex builder, on the same phrases with the same scorer.

What we measured

Approach	Exact match, 100 held-out phrases	Exact match, 40 messy phrases
Regex builder in the app	25%	2%
Qwen2.5-1.5B, 4 examples in the prompt, no training	30%	28%
Qwen2.5-1.5B + LoRA, round 1	73%	78%
Qwen2.5-1.5B + LoRA, round 2	90%	88%
Qwen2.5-0.5B + LoRA, round 1	54%	62%
Qwen2.5-0.5B + LoRA, round 2	76%	88%

All runs are on the Mac, one run per phrase. Round 2 helped both model sizes on both test sets. The score also depends on the test set: the regex builder got 25% on one set and 2% on the other. The 25% is from the October code. The app's current regex code scores 28% on the bench and still 2% on the 40. The trained models move too: three training seeds of the round-2 0.5B scored 76%, 82% and 92% on the bench.

What the numbers do not tell you

Be careful with these results. These are the limits.

We used one run per phrase, so differences of a few points are noise. On 100 phrases, about 9 points is normal.
One person wrote the 100 test phrases and the 40 messy phrases. The second set is easier in some ways, so treat it as an optimistic second opinion.
The training labels in round 2 were written by language models and were not checked by a human. Some labels are debatable.
The model obeys fake instructions hidden in the user text. On a small attack set the round-2 1.5B followed 33 of 56, and our rule-based guard did not hold on new wordings (lesson 20). That set is not part of the score tables.
We measured battery drain only once, on the iPhone, and the result is rough (lesson 36). We did not measure it on Android.

A score measures one thing on one test set. It is not a promise about real users. That is why Part 2 shows how to look at the wrong answers, not only the total.

Putting it together

Here are the ideas of Part 1 in one list. Each one maps to a decision in your app.

Idea	What you do in the app
Tokens	Count tokens, not characters. Expect more tokens for non-English text and JSON.
Next-token loop	Stream tokens to the UI. Measure TTFT and tokens per second.
Weights are read-only	Treat the model as a data file. Fine-tuning gives a new file or an adapter.
Context window	Cap the history. Check that the system prompt, the input and the answer fit.
Temperature	Use 0 for extraction. Use a higher value only for creative features.
Quantization	Ship a 4-bit or 8-bit model. Check the file size and the RAM.
Prompts and fine-tuning	Start with a prompt. Fine-tune when the prompt is not enough.
Held-out test set	Score every change on examples the model has never seen.

Check yourself

Why must the test phrases not appear in the training data?

A phrase has nine correct fields and one wrong field. What does exact match say?

The regex builder scored 25% on one test set and 2% on another. What does that show?

Recap

Use a held-out test set and a script that scores every change. Do not trust a few demos.
Exact match is strict. Also look at invented fields and unparseable answers.
Our scores are one run each, with about 9 points of noise and test phrases written by the authors.
Part 1 in one line: tokens, a loop, read-only weights, a context limit, temperature, quantization, help from prompts and fine-tuning, and a held-out test.
Next lesson: the task in detail, from a sentence to filters.