BM25 lexical search

Problem with Semantic Search Alone

• Semantic search can miss relevant results in edge cases
• Example: searching "incident 2023 q4011" returned irrelevant sections (financial analysis) instead of highly relevant ones (cybersecurity section)
• Need to improve search result quality and relevance

Hybrid Search Solution

• Combine two parallel search systems: semantic search + lexical search
• Merge results from both systems to get balanced, comprehensive output
• Leverages strengths of both approaches

Lexical Search: BM25 Algorithm

• Classic text-based search method (Best Match 25)
• Common technique used in RAG pipelines

BM25 Process (High-Level Overview)

• Tokenization: Break user query into individual terms
• Remove punctuation and split by spaces
• Example: "what happened with incident 2023 q4011" → ["what", "happened", "with", "incident", "2023", "q4011"]

• Term Frequency Analysis: Count how often each term appears across all text chunks
• Frequent terms (like "a", "the") are common across documents
• Rare terms (like specific incident numbers) appear infrequently

• Assign Relative Importance: Weight terms based on frequency
• Rare terms get higher weight (more important for search)
• Common terms get lower weight (less discriminative)

• Rank Chunks: Find text chunks using higher-weighted terms most often
• Prioritizes chunks with rare, specific search terms
• Returns most relevant results first

Implementation Details

• Both semantic and lexical search systems use similar API structure (add_document, search functions)
• Next step: merge results from both systems into unified output

Takeaways

• Semantic search alone can miss relevant results in edge cases; hybrid search combining semantic + lexical (BM25) search improves result quality
• BM25 lexical search tokenizes queries, analyzes term frequency across documents, weights rare terms higher, and ranks chunks by relevance
• Hybrid approach leverages strengths of both semantic and lexical search systems with parallel execution and merged results