In a previous article I did a comparison between RAG and Continued Pretraining (CPT) from a performance perspective. Even though the experiments showed that CPT has a clear latency advantage, we shouldn’t infer from this result that CPT is the “better” choice. In fact, RAG vs CPT is usually the wrong question, and you are more likely to see RAG and CPT combined in the same project to get the best from both options. In this article I will demonstrate how to configure a model with a combined CPT and RAG strategy.

Experiment

Again, we are back in Awesomeville, working with my travel advisor llm (qwen 3.5 4B), but this time we want to add support for real time travel announcements to the model.

In my previous article I showed how I used Unsloth to train the model to learn and reason about Awesomeville’s fictional subway system. As a quick refresher, I have added the complete map in the illustration below.

Awesomeville subway map showing the Green, Blue, and Gold lines
Awesomeville’s three subway lines connect at Central Station. Select the map to view it full-size.
CPT

As my previous experiments showed, CPT was a very successful strategy for learning the subway map, setting the model up well for recommending subway routes. Through held-out eval examples the model proved that it is able generalize the station graph well, without relying on memorization of specific routes.

CPT is great for learning general data that doesn’t change frequently (e.g. subway maps), but what happens when you need to incorporate dynamic data? Retraining often is both expensive and time-consuming, so we need a better approach to augment the stable CPT knowledge. This is where RAG comes into the picture to pull in fresh data.

RAG

To demonstrate support for real time updates, I have incorporated four concreate examples where the board of tourism in Awesomeville is making important announcements that are relevant for subway travelers and tourists:

  • There is a concert today at Westgate station
  • Heritage theater is announcing a new production of Hamlet
  • East Harbor station will be closed today
  • There is a brand-new Egyptian exhibit displayed at Bright Mill Museum

All these are examples of announcements that are impractical to cover in CPT training since the data changes at a moment’s notice. Instead, we should augment the model pipeline by incorporating a retrieval step to inject facts, not covered by training.

Combined CPT and RAG

To test CPT and RAG together in the pipeline, I have come up with a set of 4 demo questions. In order to answer these questions, the model needs to rely on internalized subway graph knowledge combined with dynamic station-to-event mappings provided through RAG.

See list of questions below:

  • Give me directions from Lake Harmony to the concert tonight.
  • I am at North Terminal. How do I get to the brand-new Egyptian exhibit?
  • What subway route should I take from South Gardens to see the new production of Hamlet?
  • From University Commons, how do I reach the station travelers must continue to because of today's subway station closure?

As you can see from these questions, the destination station is not found directly in the question. Instead, the model needs to rely on RAG to inject the announcement before recommending a route.

Let’s dig into one of the questions in more detail:

“From University Commons, how do I reach the station travelers must continue to because of today's subway station closure?”

This question is intentionally vague to make it more challenging, but by combining retrieved knowledge with internalized graph knowledge, I will show how the model arrives at the correct answer.

Let’s take a look at the system prompt and user prompt below:

System Prompt
Reason internally before answering and verify every required fact. Return a clear ordinary-English answer, not JSON, code, a schema, or Markdown. For a subway journey, name each line. State explicitly when a transfer occurs and name its station. Include only facts supported by the trained Awesomeville data; never guess or substitute an unsupported station, line, attraction, or access point. A route leg begins at the requested origin or the previous interchange and ends at the next interchange or requested destination. Do not include stations before the origin or after the destination. City-announcement context follows. Use it only when it is relevant to the question. Treat relevant announcements as current, authoritative facts and do not contradict them. # East Harbor Station Closure Public service announcement: **East Harbor Station is closed today**. Travelers who would normally exit at East Harbor must remain on the train and continue to the next station.
User prompt
From University Commons, how do I reach the station travelers must continue to because of today's subway station closure?

As you can tell, RAG provides details about the subway closure, but the statement is imprecise since it doesn’t mention what the “next” station is. However, since the model knows the subway graph from CPT training, we get the correct answer below:

Take Blue Line from University Commons to Central Station. Transfer at Central Station to Gold Line. Continue to Sunrise Point.

Not only does the model automatically know that Sunrise Point is the station right after East Harbor on the Gold Line, but it also knows to transfer from the Blue Line at Central Station.

Github

Again, if you are interested in having a look at the project, I have included the full source code on Github.