BM25 lexical search

RAG Pipeline Improvement: Hybrid Search

• Limitation of Semantic Search:
• While effective, semantic search can fail in "corner cases," returning irrelevant results (e. g. , Section 3 Financial Analysis) even when the desired information is present elsewhere (e. g. , Section 10 Cyber Security).
• The goal is to improve result accuracy and relevance.

• Hybrid Search Strategy:
• Implement two search systems running in parallel:
• Semantic Search: Uses embeddings and a vector database (current method).
• Lexical Search: A classic text search that breaks the query into individual words to find matching text chunks.
• Merging Results: The final step involves merging the result sets from both the semantic and lexical systems to achieve a better balance of search quality.

• Lexical Search Technique: BM25 (Best Match 25):
• BM25 is a common algorithm used for classic text search within RAG pipelines.
• BM25 Process:
• Tokenization: The user query is broken down into individual search terms (e. g. , removing punctuation and splitting by spaces).
• Term Frequency Counting: The algorithm counts how often each search term appears across all available text chunks.
• Weight Assignment: Terms are assigned relative importance (weight). Terms used frequently across all documents are considered less important; terms used infrequently are considered highly important.
• Scoring: The system identifies the text chunk that uses the higher weighted terms more often, determining the best match.

• Key Advantage of BM25:
• It prioritizes rare, specific terms (like "incident 2023") over common, generic terms (like "what" or "happened"), leading to more targeted search results.

• Next Steps:
• The final stage of the pipeline involves integrating and merging the results from the Semantic Search store and the Lexical Search (BM25) store.

Takeaways

• Hybrid Search combines Semantic Search (using embeddings) and Lexical Search (classic text matching) to improve result accuracy and relevance.
• Lexical Search often utilizes the BM25 algorithm, which tokenizes queries and assigns weights based on term frequency.
• BM25 prioritizes rare, specific terms over common words, leading to more targeted search results.
• The final stage of the RAG pipeline involves merging the result sets from both the Semantic and Lexical search systems.