In a previous article I showed how to teach a local model a new domain through continued pretraining (CPT). Specifically, I trained the model to act as a travel advisor, capable of navigating a fictional city’s subway system. As a continuation of this experiment, I wanted to explore how a RAG based implementation would compare. The local model used for this is experiment is QWEN 3.5 4B.

RAG vs. CPT

Even though this is a comparison between RAG and CPT for a particular task, it is not my intention to present this as two competing solutions where you pick one or the other. More realistically you would probably see CPT augmented by RAG in many cases to get the best of both worlds. RAG is generally the most flexible solution since no training is required, and your training data won’t get stale, but there are some interesting tradeoffs to consider.

Quick Recap

The original project used CPT to train a local model to act as a travel advisor, providing travel routes for a fictional subway system with up to four line transfers. The previous article showed a successful outcome where the model had generalized the subway map and made it part of its internal knowledge. As an illustration, I have added a snapshot of the subway map below:

In this new experiment I swapped out CPT for a simple RAG prompt. A sample of a combined system and user prompt can be found below:

Baseline

The CPT training data is the same as in the previous article with the only difference that I upgraded to QWEN 3.5 from 3.0.

I also created a new eval dataset of 100 test cases consisting of route combinations not seen by the model during CPT.

The baseline is using the CPT trained model without thinking enabled.

As before, the results are quite good with an eval score of 93/100 and an average execution time of 1.3s for full subway route recommendations. Enabling thinking brings the score up to 98/100, but at the expense of performance at an avg execution time of 8.7 seconds.

RAG

For the initial RAG implementation, I used the original QWEN 3.5 4B base model with the simple RAG prompt mentioned earlier in the post. Performance was reasonable at an eval score of 80/100. However, performance was much slower at an average execution time of 4.97 seconds with thinking disabled. Enabling thinking would likely improve accuracy, but the performance degradation made it impractical since the runtime degraded to 60 seconds.

The initial prompt used the original subway station names, but I learned during CPT that the model responds better to an internal representation using synthetic names. The synthetic names encode both line membership and ordinals in the name, which makes journey reasoning easier for the small model. See illustration below for the specific mapping. Note: The synthetic names are only used internally and are not user-facing.

As expected, using synthetic names helped both accuracy and performance. The new performance values improved to eval score: 88/100 and avg execution time: 3.4s. At this point I would say accuracy between RAG and CPT is close. We could probably make it even closer by improving the RAG prompt.

Performance is where I see the biggest difference though. This is interesting since the retrieval step is intentionally made very simple as a simple file read. I think the interesting part here is quantifying the difference in performance between internalizing the knowledge (CPT) vs. doing on-the-fly reasoning from the RAG content.

Below is a summary table of the performance characteristics across all 4 experiments.

Experiment runtime summary

I have included the repo on Github in case you are interested in checking it out.