Follow-Up Questions in RAG: Query Condensation and Rewriting for Chat

People do not talk to a chatbot in complete, self-contained questions. They say 'how much is it?', 'and for kids?', 'what about Saturday?', 'explain in detail'. Each of those makes perfect sense in context and none of them makes sense alone. A retrieval system that searches on the literal words of the latest message will return nothing useful for 'explain in detail', because the words carry no topic. This is one of the most common causes of a chatbot that seems to forget the conversation. The fix is to turn each follow-up into a standalone question before searching: query condensation, sometimes called query rewriting. This guide explains the problem, the main techniques from simple rules to model-based rewriting, the risks, how to combine them, how to test on real conversations and how Crescelo approaches it.

The problem in one example

If retrieval sees only 'And with a cut?' it may find passages about haircuts, not the colour price. If it sees only 'Explain in detail' it may find nothing. The conversation supplies the missing context; the retrieval step has to use it.

Why it matters more in chat

• Chat messages are short and elliptical.
• Pronouns and references are constant: 'it', 'that one', 'the second option'.
• Visitors change topic without warning.
• Follow-ups are often the most valuable turns, where the visitor is closest to a decision.

A retrieval system that ignores history fails exactly when the visitor is engaged.

Technique 1: rules that add the previous topic

The simplest approach detects that a message cannot stand alone and prepends the previous user question. Signals include very short messages, pronouns, openers such as 'and', 'what about', 'how about', 'why', and phrases like 'in detail' or 'more'. It is instant, free and predictable.

Technique 2: model-based rewriting

A small language model reads the conversation and writes a standalone question, and sometimes several alternative phrasings. It handles subtle references and topic changes better than rules. The costs are an extra model call, added latency and the risk that the rewrite adds facts that were not in the conversation. Mitigations: use a small fast model, keep a strict timeout, cache results, instruct it to change as little as possible and fall back to the rules if it fails.

Technique 3: use both

A robust pipeline applies the cheap rules first and, where enabled, a bounded model rewrite second, searching with both the condensed query and the original and fusing the results. If the rewrite fails or times out, the rules still work. Crescelo follows this pattern: it resolves follow-ups using the previous question by default, and can optionally use a small model to write a standalone question and alternative phrasings, each searched and fused, with a short timeout and caching so a failure falls back to the rules.

What history to include

• The last one or two user questions are usually enough.
• Include the assistant's last reply if the visitor refers to something it said ('the second option').
• Do not include the whole conversation: it dilutes the query and costs tokens.
• Reset context when the visitor clearly changes topic.

Cap the amount of history, and prefer the most recent turns.

Broad and overview requests

Some messages are not follow-ups but broad requests: 'tell me everything', 'what do you offer?', 'give me an overview'. Searching on those words finds little. A good system recognises them and retrieves a spread of passages across the knowledge base instead of a keyword match. Crescelo treats these as overview questions and pulls a representative set of passages. See how to review your chatbot's conversations for spotting such questions in your own logs.

Risks and how to control them

How to test on real conversations

1. Collect twenty real multi-turn conversations, or write realistic ones.
2. For each follow-up, note the passage that should be retrieved.
3. Run retrieval with the raw message, with the rule-based condensation and with the model rewrite.
4. Measure how often the right passage is in the top five for each.
5. Read the failures: was the topic wrong, the rewrite invented or the passage missing?
6. Repeat after any change to the rules, the model or the prompts.

Follow-ups are often underrepresented in test sets. Make sure yours includes them; our own retrieval evaluation showed that follow-up questions were the weakest category, which is exactly where condensation earns its keep.

What the visitor should see

Condensation is invisible, and it should be. The visitor asks 'and for kids?' and gets a sensible answer about kids for the same service, without repeating themselves. If the assistant is unsure what they mean, it should ask a clarifying question instead of guessing. See the questions every service business chatbot should answer well.

Common mistakes

• Searching on the raw follow-up and blaming the knowledge base.
• Passing the whole conversation to the search, which drowns the topic.
• Rewriting every message even when it stands alone.
• Never testing multi-turn conversations.
• Letting a rewrite change the visitor's language.

Frequently asked questions

See a follow-up in action in the playground, read hybrid search for RAG and see the AI assistant feature.

Further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.): The original research paper behind retrieval-augmented generation.
• Google Search Central: creating helpful, people-first content: What Google looks for in useful pages.