Over the last few months, I have been working on an article series that shows how to teach a local llm a new domain. The project spans the entire life cycle from continued pretraining of the llm to enriching the trained knowledge through RAG, and picking a comprehensive eval strategy to evaluate performance. This article will offer a summary view of my learnings and achievements along the way.

Domain

Just to recap, the premise of this project was to teach a local llm a new domain by training on a fictional subway system. The key goal was to come up with a training dataset that would generalize well enough to build a travel advisor, capable of recommending subway routes.

In order to achieve this, the main requirement was to avoid a solution that relies on memorization of specific station pairs since this is known to generalize poorly to station pairs not encountered during training.

As a visual illustration I have included a picture of the fictional subway system below:

Awesomeville subway map showing the Green, Blue, and Gold lines
Awesomeville’s three subway lines connect at Central Station. Select the map to view it full-size.

Phase 1 – Designing the Training Data

The first phase of the project was to come up with generic training dataset that meets the requirement of generalizing well to held-out eval cases. The initial article walks us through the process of selecting LORA continued pretraining (CPT) as the training strategy, as well as the iterative approach to designing the training data.

The process of designing a robust dataset was fairly time consuming, but in the end, I arrived at a solution that performed well against the held out eval samples. Out of 105 eval cases, only two tests were failing. The failures were analyzed and attributed to a known issue, but overall, this was a very good starting point for phase 1.

Note: The local llm used for this experiment is QWEN 3.5 4B with Unsloth for LORA continued pretraining (CPT).

Phase 2 – Comparing RAG and Continue Pretraining

The second article digs into an interesting question where I compare the performance of RAG vs. CPT. The premise of the experiment is to compare the performance characteristics of internalizing the knowledge (CPT) vs. real-time reasoning from passing the subway map in the prompt (RAG).

As expected, the experiment showed that the primary difference is not accuracy, but performance. CPT showed a clear latency advantage from 2-3 times faster inference performance compared to RAG.

Phase 3 – Combining RAG and Continued Pretraining

The third article picks up from the previous RAG vs. CPT discussion, but from a more practical perspective. Comparing RAG to CPT from a performance perspective is interesting, but RAG vs. CPT is typically not the right question. Instead, you are more likely to encounter RAG and CPT as complementary solutions. The third article focuses on exactly that by exploring how to combine RAG and CPT to get the best of both worlds in the same project.

The article explores how RAG can enrich the travel advisor by adding support for real-time announcements like station closure and cultural events along the subway line. Basically, the article shows how CPT can continue to reason across stations, but by adding RAG, the travel advisor can factor in ad-hoc announcements that can’t be known during training.

Phase 4 – Designing Comprehensive Evals

The final article describes how I designed the eval solution for measuring performance along the way. As mentioned previously, the key goal was to come up with a solution that generalizes well. A key component of this is to come up with an eval dataset made up of questions not seen during training.

Overall performance for the model is very good (high 90%s), but a key part of the article is digging into the general strategy for doing evals. The article also offers analysis of specific failure categories.

My favorite part of this article is showing how fine tuning the model further can help the eval process. By fine tuning the CPT trained model using Unsloth SFT I am teaching the model to output the responses in the form of a strict json schema. The clear benefit of this is that evals can rely mostly on a strict exact match strategy for comparing model responses to reference answers.

It’s important to point out that the SFT fine tuning process is only there to influence the response format and not teach the domain knowledge itself. I make this point since QA style SFT training tends to lead to memorization, which I stated earlier is very undesirable. SFT is applied on top of the CPT trained model, but I made sure to avoid any overlap between SFT routing examples and eval questions.

Conclusion

In the end I am very happy with the performance of the final model. Through multiple iterations I was able to get a well performing model that demonstrates that CPT can be used to teach a model a new domain. I have put the code on Github in case you are interested in having a look.