The full RAG flow

RAG Pipeline: Detailed Process Study Notes

**I. Pre-processing Phase (Indexing/Setup)**

• Chunking: The source document is broken down into smaller, manageable pieces of text (chunks).
• Embedding Generation: Each text chunk is converted into a numerical representation called a vector (an embedding).
• The embedding model assigns scores to specific dimensions (e. g. , Medical, Software Engineering) to represent the chunk's topic relevance.
• Example: A chunk might score 0. 9 for Medical and 0. 4 for Software Engineering.
• Normalization: This step scales the magnitude (length) of each vector to 1. 0. This is often handled automatically by the embedding API.
• Storage: The generated and normalized vectors are stored in a Vector Database, which is optimized for storing and comparing large lists of numbers.

**II. Query Phase (Retrieval)**

• Query Embedding: When a user submits a query, it is run through the exact same embedding model used during pre-processing to create a query vector.
• Vector Search: The query vector is fed into the Vector Database to search for the stored vectors that are most similar in nature.
• Similarity Calculation: The database uses mathematical calculations to determine the relationship between the query vector and every stored vector.

**III. Key Mathematical Concepts**

• Cosine Similarity:
• Calculates the cosine of the angle between the query vector and a stored vector.
• Range: -1 to 1.
• Interpretation: A result close to 1 indicates high similarity.
• Cosine Distance:
• Calculated as 1 minus the Cosine Similarity.
• Interpretation: A result close to 0 indicates high similarity.

**IV. Final Generation Phase**

• Retrieval: The vector database returns the text chunk(s) with the highest similarity score to the user's query.
• Prompt Construction: The final prompt sent to the Large Language Model (LLM) includes both the original user query and the most relevant retrieved text chunk(s).
• Response: The LLM processes the combined prompt to generate a final, informed answer.

Takeaways

• The RAG pipeline converts source text into numerical representations called vectors (embeddings) and stores them in a Vector Database.
• When a query is submitted, it is also converted into a vector to enable a similarity search against the stored data.
• Similarity is calculated using metrics like Cosine Similarity, where a result close to 1 indicates high relevance.
• The final Large Language Model prompt is constructed by combining the original user query with the most relevant text chunks retrieved from the database.