A Multi-index RAG pipeline

Hybrid Search and Result Fusion

**1. The Need for a Retriever**

• Goal: To combine different search methodologies (e. g. , Semantic Search and Lexical Search) into a single, unified search pipeline.
• Implementation: A new class, the Retriever, is created to wrap the individual search implementations (like VectorIndex and BM25Index).
• API Consistency: The successful integration relies on both underlying search implementations having an almost identical public API (e. g. , methods like add_document and search).
• Function: The Retriever receives the user's query, forwards it to all contained indexes, and then merges the results.

**2. Reciprocal Rank Fusion (RRF)**

• Purpose: A technique used to combine and merge search results from different methodologies.
• Process:
• All search results from all indexes are collected into a single table.
• The rank (position) assigned by each individual index is recorded for each document.
• A specific formula is applied to calculate a final score for each document based on its ranks across all indexes.
• The documents are then sorted based on this final score (greatest to least).
• Benefit: RRF allows the system to leverage the strengths of multiple search techniques, leading to more relevant and accurate results than any single method alone.

**3. Benefits of the Hybrid Approach**

• Modularity: Each search index is implemented as a separate, isolated class.
• Extensibility: Because the Retriever relies only on a consistent API, new search functionalities can be easily added to the pipeline without modifying the existing code.
• Improved Accuracy: Combining semantic and lexical search techniques significantly improves the quality of the search output (e. g. , retrieving the correct document even if the vector search initially misses it).

Takeaways

• Hybrid Search unifies diverse methodologies (e. g. , Semantic and Lexical) into a single, cohesive pipeline.
• The Retriever class wraps individual search indexes, relying on consistent public APIs to forward queries and merge results.
• Reciprocal Rank Fusion (RRF) merges results by calculating a final score for each document based on its rank across all contributing indexes.
• This modular design allows for easy extensibility, enabling new search functionalities to be added without modifying existing code.