A multi-search RAG pipeline

Hybrid Search Architecture & Result Merging

• Goal of Hybrid Search: To combine the strengths of different search methodologies (e. g. , Semantic Search via Vector Index and Lexical Search via BM25 Index) into a single, improved search pipeline.
• The Retriever Class:
• This new class acts as the central orchestrator.
• It receives the user's query and forwards it to all contained search indexes.
• It receives the results from each index and is responsible for merging them.
• API Standardization:
• The successful implementation relies on all individual search indexes (Vector Index, BM25 Index, etc. ) having an identical Public API.
• Required methods include add_document and search.
• This standardization allows the Retriever to wrap and integrate new search functionalities easily.

Reciprocal Rank Fusion (RRF)

• Purpose: RRF is the technique used to combine and rank results coming from different search methodologies.
• Process:
• Gather all search results and their corresponding ranks (position in the search output) from every index into a single table.
• Apply the RRF formula to calculate a final score for each text chunk.
• Sort the final results based on this calculated score (greatest to least).
• RRF Formula: For a given rank column, the score contribution is calculated as:
• One over (one plus the rank number).
• Example: For Rank 1, the term is one over (one plus one). For Rank 2, the term is one over (one plus two).
• The final score is the sum of these terms across all rank columns.

Benefits of Hybrid Approach

• Combining multiple indexes (e. g. , Vector + BM25) significantly improves search accuracy and relevance compared to using a single index.
• The modular design allows developers to author and implement each search index in isolation, making the system highly extensible.

Takeaways

• Hybrid Search combines different methodologies, such as Vector Index (semantic) and BM25 Index (lexical), to improve overall search accuracy.
• The Retriever class acts as the central orchestrator, receiving queries and merging the results from all contained search indexes.
• Successful integration requires all individual search indexes to maintain an identical Public API, including methods like add_document and search.
• Reciprocal Rank Fusion (RRF) is the technique used to combine results, calculating a score contribution as one over (one plus the rank number) for each index.
• The modular design allows developers to implement and extend each search index in isolation, enhancing system extensibility.