Creating Evals to Validate LLM Training

In a previous article I showed how to teach a local model a new domain through a combination of llm training and RAG. In this article I will discuss my strategy for defining evals to ensure the model is performing correctly.

Project

Just as a quick refresher, my previous article describes how I trained a local model (qwen 3.5 4B) to act as a travel advisor for a fictional subway system. The main idea is to rely on continued pretraining (CPT) to teach the model how to reason about the subway system. But to add support for real-time updates, I combined the internalized knowledge with RAG.

As a visual reference, I have added a picture of the fictional subway map below:

Evals

One of the key components in determining if a model is performing well is to add comprehensive evals around it. I designed my evals around the following two requirements:

I need to measure if the model can generalize well, and answer questions not previously seen during training

I need to measure the model’s accuracy level as a baseline for future improvements or regressions

In my eval solution I ended up with three metrics:

1. Exact Match
2. Partial Match
3. Semantic Match

Exact Match

When possible, exact match is likely the best way to measure success since it allows you to make simple comparisons between model output and some reference data. This is by design a strict comparison since the match must be exact. In some cases, this may be unrealistic, but I will show a technique that worked well in this project:

Initially my model was returning subway itineraries in the form of English sentence. This presented a challenge to exact match comparisons since two correct sentences can vary greatly but still represent the correct answer. As a solution to this, I decided to pivot my solution to instead output the subway itinerary in the form of a structured json schema. In some case this is not an option, but the legs of a subway journey lend themselves well to this format.

I have included a sample of a predicted subway journey consisting of three legs below:

Depending on the strength and size of your model, json output might work out of the box, but smaller models may need some help. In my case I decided to fine tune the model to reliably output the subway itineraries in the form of json.

For the actual fine tuning I am using Unsloth with Supervised Fine Tuning (SFT). The only goal of SFT in this case is to teach the model to output responses in the desired format. I have included a link to the SFT corpus here in case you want to have a look.

The SFT corpus defines entries following the Alpaca format, which is more of a Q&A format, but I did not want to rely on SFT for teaching knowledge, only format. One of the key reasons I want to avoid using SFT to train knowledge is that it usually leads to Q&A memorization that doesn’t generalize well to unseen questions. Instead, I want the subway routing knowledge to continue to come from CPT training.

I experimented a bit with SFT settings, but in the end, I ended up having success with a lightweight LORA setup. I have shared the Unsloth SFT config here. Post SFT I tested the model by running an eval set of 604 routing questions.

Results:

As I have reported previously, model performance is very good. Post CPT and SFT I was able to get the exact match evals up to 98% from 573/604 correct results.

Partial Match

573/604 results are an exact match, but what happened to the remaining 31 questions? In all 31 cases the model provided a partially correct subway route, but one or more legs were missing. It is useful to measure partial successes since it creates a baseline for later improvements or regressions when retraining.

I have added a summary of partial results below. This table shows the partial correctness scores for the 31 route predictions that failed the exact match test.

Semantic Match

The third category is semantic matching, which is less common in a schema-based response, but I noticed a few cases during testing.

In this experiment I noticed that the model would occasionally output superfluous route legs (e.g. an unnecessary but correct intermediate station during a journey). While these additions make the route more verbose, it doesn’t interfere with the factual accuracy of the full route since it’s just additional information. I decided to capture this nuance in the evals, even though the final solution didn’t contain any semantic match cases.

Eval Set

As I mentioned above, the full eval set consists of 604 routing questions. None of these eval cases appear exactly in either CPT or SFT training corpuses. There is, however, some overlap between some route pairs in multi-part journeys (i.e. an itinerary that contains multiple sub journeys).

Out of the 604 eval questions, 457 questions are completely held out with no overlap. 137 eval questions have partial overlap with sub routes defined in the CPT corpus, but only 10 eval questions have full overlap

There is no overlap with the routes defined in the SFT corpus.

Based on this I have included the performance of the respective eval questions below. As you can tell, performance is good regardless of overlap. The table covers both raw exact correctness as well as adjusted correctness with partial matches included.

Held-out evaluation results

Conclusion

Overall, the evals tell a good story regarding the trained qwen 3.5:4B model’s ability to act as a travel advisor in this fictional subway domain. As always, the code is available on Github in case you are interested in checking it out.