Reranking results

Topic: Improving Retrieval Accuracy with Re-ranking

**I. Problem with Hybrid Retrieval**

• The existing hybrid retrieval approach (combining Vector Index and BM25) still contains weak points and "rough edges" in its retrieval process, even after adding complexity.

**II. The Re-ranking Technique**

• Definition: Re-ranking is a post-processing step added after the initial search results are retrieved.
• Goal: To improve the accuracy of the search pipeline by reordering the initial set of results based on their true relevance to the user's query.
• Process Flow:
• Run the initial search (Vector Index + BM25) and merge the results.
• Pass a subset of these merged results to a Large Language Model (LLM), such as Claude.
• The LLM is prompted to review the user's question and the retrieved documents.
• The LLM returns a list of the documents in order of decreasing relevance (i. e. , the most relevant document is placed at the top).

**III. Implementation Details**

• Functionality: The re-rank function is called automatically by the retriever after the initial search is complete.
• Prompt Structure: The prompt provides the user's query and the found documents (often formatted in XML).
• Efficiency Improvement: To avoid the inefficiency of having the LLM copy the full text of every chunk, it is more efficient to ask the LLM to return unique Document IDs, which are then used to establish the correct order.
• Structured Output: Techniques like Assistant message pre-fill and stop sequences are used to ensure the LLM returns well-formatted JSON.

**IV. Trade-offs**

• Pros: Significantly increases the accuracy of the search pipeline (e. g. , correctly prioritizing highly specific, relevant sections).
• Cons: Increases the overall latency of the search pipeline because it requires an additional call to the LLM.

Takeaways

• Re-ranking is a post-processing step applied after initial retrieval (like Vector Index + BM25) to improve search accuracy.
• The process involves passing a subset of retrieved documents to a Large Language Model (LLM) to reorder them based on their true relevance to the query.
• Efficiency is improved by instructing the LLM to return unique Document IDs rather than the full text of the chunks.
• The primary trade-off is a significant increase in search accuracy versus increased pipeline latency due to the additional LLM call.