Introducing Retrieval Augmented Generation

Retrieval Augmented Generation (RAG) Study Notes

**I. Core Concept**

• RAG: A technique used to allow a Large Language Model (LLM) to answer specific questions based on information contained within a large, external document.
• Goal: To overcome the limitations of LLMs when dealing with massive datasets that exceed the model's input capacity.

**II. Problem: Handling Large Documents**

• Challenge: How to extract relevant information from a very large document (e. g. , 100 to 1,000 pages) and feed it to the LLM for specific querying.

**III. Method 1: Direct Prompt Injection (Naive Approach)**

• Process: Taking the entire document text and placing it directly into the LLM prompt along with the user's question.
• Limitations (Downsides):
• Hard Limit: LLMs have a maximum input size; very long documents will cause an error.
• Reduced Effectiveness: LLM performance decreases as the prompt length increases.
• Information Overload: The LLM may struggle to identify the exact information needed within a massive amount of text.

**IV. Method 2: Retrieval Augmented Generation (RAG)**

• Process: Instead of feeding the whole document, the document is pre-processed, and only the most relevant snippets are included in the prompt.
• RAG Workflow:
• Pre-processing (Chunking): The large document is broken up into smaller, manageable pieces (chunks).
• Retrieval: When a user asks a question, a search mechanism finds the chunks most relevant to that specific query.
• Generation: Only the relevant chunks are included in the prompt, allowing the LLM to focus its attention.
• Advantages (Upsides):
• Focus: LLM focuses only on relevant content.
• Scalability: Works effectively with very large documents and multiple documents.
• Efficiency: Smaller prompts lead to faster processing time and lower operational costs.
• Challenges (Downsides):
• Complexity: Requires significant pre-processing steps (chunking and searching).
• Search Mechanism: Requires defining and implementing a robust way to find relevant chunks.
• Context Guarantee: There is no guarantee that a retrieved chunk will contain all the context needed for a complete answer.
• Chunk Definition: Requires evaluation to decide the best way to split text (e. g. , equal portions vs. splitting by headers).

Takeaways

• Retrieval Augmented Generation (RAG) allows Large Language Models (LLMs) to answer questions using external documents, overcoming input capacity limitations.
• The RAG workflow involves three steps: Pre-processing (Chunking the document) → Retrieval (Finding relevant snippets) → Generation (LLM uses snippets to answer).
• RAG is more scalable and efficient than direct prompt injection because it focuses the LLM's attention only on relevant content.
• Key challenges include the complexity of the pre-processing steps and the necessity of implementing a robust search mechanism to find relevant chunks.