What Happens When Your RAG System Retrieves the Wrong Documents?

What happens when your RAG system retrieves the wrong documents? Or when the retrieved context isn't enough to answer the question?

A traditional RAG pipeline usually doesn't think twice. It follows a fixed path: Retrieve → Generate → Answer

But real world AI applications aren't that simple. Sometimes the system needs to search again with a different approach. Sometimes it needs to evaluate what it retrieved, verify whether an answer is actually supported, correct its own mistakes or ask a human for help before proceeding.

That's where Agentic RAG comes in. Instead of treating retrieval as a fixed step, Agentic RAG gives the system the ability to reason about the retrieval process and decide what to do next.

Why Does Agentic RAG Matter?

As RAG applications grow more complex and work with larger document collections, simply retrieving the top-k documents isn't enough to produce accurate answers. An agentic approach lets a RAG system:

1. Decide what information to retrieve

2. Reason over the retrieved context

3. Evaluate whether the context is relevant

4. Verify whether an answer is supported

5. Retry or refine retrieval when necessary

6. Escalate to a human when confidence is low

The goal isn't complexity for its own sake it's building RAG systems that are adaptive, robust and reliable instead of blindly following a fixed pipeline.

What I Built

This has grown into a 10-notebook series on LangGraph from the two core ideas every agent depends on up to a full set of production style RAG architectures.

Free (4 notebooks) the fundamentals:

• LangGraph Starter — state, reducers and an LLM node

• Conditional Routing — the router mechanic every agent decision is built on

• Agentic RAG — a hand built tool calling agent that decides per question whether to retrieve, which knowledge base to search and whether it needs a second lookup (multi-hop)

• ReAct Multi-Hop Agentic RAG — the same problem solved a second way, rebuilt on LangChain's prebuilt create_agent loop instead of a hand wired graph

Advanced (6 notebooks) — one architecture per failure mode:

• Corrective RAG — grades every retrieved chunk and falls back to a live web search to patch gaps before answering

• Adaptive RAG — routes each question to a vectorstore or the web up front, then self-checks the final answer for hallucination and relevance, looping back if it fails

• Human-in-the-Loop RAG — same retrieve/grade/generate machinery, but a real person approves, revises or rejects at two checkpoints instead of an LLM deciding alone

• Multi-Agent Supervisor RAG — a supervisor routes each question to one of three specialist subgraphs then critiques the answer and can escalate to a web specialist

• Parallel Fan-Out RAG — decomposes a multi-part question into sub-questions, answers them concurrently via LangGraph's Send API then aggregates the results

• Plan-and-Execute RAG — plans every step up front, executes them one at a time, and a replanner decides whether more work is needed before synthesizing the final answer

Every notebook is fully commented and markdown-explained top to bottom, runs on Groq's free tier API for the LLM and uses local BAAI/bge-m3 embeddings (sentence transformers) no embedding API key needed.

Free Version (4 notebooks): https://chandula7.gumroad.com/l/AgenticRAGFundamentals Advanced Version (6 notebooks): https://chandula7.gumroad.com/l/Advanced_RAG_LangGraph_Patterns

If you're building RAG applications this is a step toward moving beyond Retrieve → Generate and toward systems that can Retrieve → Reason → Verify → Correct → Answer