Reranking results

Study Notes: Improving Retrieval Accuracy with Re-ranking

**I. Problem with Hybrid Retrieval**

• The existing hybrid retrieval approach (combining Vector Index and BM25) has limitations and "rough edges. "
• Even when queries are highly specific (e. g. , mentioning a specific team or incident), the initial retrieval may fail to place the most relevant document at the top.

**II. The Re-ranking Technique**

• Definition: Re-ranking is a post-processing step added after the initial retrieval process.
• Goal: To improve the accuracy of search results by reordering the documents based on their true relevance to the user's query.
• Process Flow:
• Initial Search: Run Vector Index and BM25 → Merge results.
• Re-ranking Step: Pass a subset of the merged results to a Large Language Model (LLM, e. g. , Claude).
• LLM Task: The LLM is prompted to return a list of the most relevant documents in order of decreasing relevance.

**III. Implementation Details**

• Re-rank Function: This function is automatically called by the retriever after the initial search results are obtained.
• Prompt Structure: The prompt provides the user's original question and the set of documents found by the initial search.
• Efficiency Improvement (Using IDs):
• Instead of asking the LLM to return the full text of every chunk (which is inefficient), the system assigns unique Document IDs to each chunk.
• The LLM is instructed to return only the ordered list of these IDs.
• Technical Implementation:
• Uses assistant message pre-fill and stop sequences to ensure the LLM returns well-formatted JSON.

**IV. Trade-offs and Evaluation**

• Pros: Significantly increases the accuracy of the search pipeline (e. g. , correctly prioritizing the most specific and relevant section).
• Cons: Increases the overall latency of the search pipeline because it requires an additional, time-consuming call to the LLM.

Takeaways

• Re-ranking is a post-processing step used to improve search accuracy by reordering documents based on their true relevance after initial retrieval.
• The process involves passing a subset of initial search results to a Large Language Model (LLM) to determine the correct order of relevance.
• Efficiency is improved by instructing the LLM to return only unique Document IDs rather than the full text of the chunks.
• Re-ranking significantly increases search accuracy but introduces higher latency due to the additional LLM call.