BM25 lexical search

RAG Pipeline Improvement: Combining Semantic and Lexical Search

**I. Limitations of Semantic Search**

• Semantic search, while powerful, can sometimes produce surprising or irrelevant results (corner cases).
• The system may prioritize chunks based on general meaning, even if the specific, critical terms are absent (e. g. , retrieving a general "Financial Analysis" section instead of a specific "Cyber Security" incident).

**II. Lexical Search (Classic Text Search)**

• Lexical search operates by breaking down a user's query into individual words (tokens).
• It searches for text chunks that explicitly contain these matching words.
• Goal: To provide a balance between the conceptual understanding of semantic search and the precision of exact word matching.

**III. BM25 (Best Match 25) Algorithm** BM25 is a common method used to implement lexical search in RAG pipelines.

• Process Steps:
• Tokenization: Break the user query into separate search terms (e. g. , removing punctuation and splitting by spaces).
• Term Frequency Counting: Count how often each search term occurs across all available text chunks/documents.
• Weight Assignment: Assign relative importance to each term. Terms used rarely across the entire dataset are given higher importance (higher weight), as they are likely more significant.
• Scoring: Identify the text chunk that uses the higher weighted terms more frequently. This chunk is considered the best match.

**IV. Integration Strategy**

• The final goal is to merge the results from two separate search systems:
• Semantic Search: Provides conceptual relevance.
• **Lexical Search (e. g. , BM25):** Provides precise keyword matching.
• By merging these result sets, the pipeline aims to achieve the best outcomes of both methods.

Takeaways

• Semantic search can fail to retrieve specific, critical terms, even if the general meaning is correct.
• Lexical search breaks down queries into tokens to find text chunks that explicitly contain those matching words.
• BM25 assigns higher importance (weight) to terms that appear rarely across the entire dataset.
• The best RAG pipelines integrate semantic search (conceptual relevance) and lexical search (precise keyword matching) results.