How to Build Agentic RAG with LangGraph

Build an agentic RAG workflow that routes, retrieves, validates, and answers queries.

This guide is for developers who want to move beyond basic retrieval augmented generation and build a system that can refine a query, choose the right source, validate the answer, and handle multimodal inputs. After following the steps, you will have a working agentic RAG prototype with routing, tool use, and answer checks.

We will use LangChain docs and the LangChain GitHub repo, plus LlamaIndex docs and the LlamaIndex GitHub repo, and optionally LangGraph docs and the LangGraph GitHub repo for orchestration.

Before you start

• Node 20+ or Python 3.11+ installed
• An OpenAI API key or another LLM provider key
• A vector database account or local vector store, such as Chroma or FAISS
• Access to at least one data source: documents, API, or web search
• Git installed for cloning starter code
• Basic familiarity with embeddings, retrieval, and prompt templates

Step 1: Set up the agentic RAG project

Your first outcome is a local project skeleton that can run an LLM, call tools, and store retrieved context.

Create a new app, install the core dependencies, and add environment variables for your model key and retrieval backend.

mkdir agentic-rag-demo && cd agentic-rag-demo
npm init -y
npm install langchain @langchain/openai @langchain/community dotenv
# or use Python if you prefer

Verify the setup by running a minimal script that loads your key and prints a test response. You should see a successful model call, not an authentication error.

Step 2: Index source documents

Your second outcome is a searchable knowledge base that the agent can query instead of guessing from the prompt alone.

Load your documents, split them into chunks, generate embeddings, and store them in a vector index. If you are using LlamaIndex, this is where you connect files, PDFs, or web pages to the retrieval layer.

// Pseudocode
loadDocuments();
splitIntoChunks();
embedChunks();
storeInVectorDB();

Verify the index by running a similarity search for a known phrase. You should see the most relevant chunk returned with a matching source reference.

Step 3: Add query routing logic

Your third outcome is an agent that decides where each query should go before retrieval starts.

Build a routing step that classifies the request as document lookup, web lookup, API lookup, or direct answer. This mirrors single-agent RAG routing and gives you a clean entry point for more advanced multi-agent flows later.

if (needsRealtimeData(query)) routeTo("api-tool");
else if (isInDocs(query)) routeTo("vector-search");
else routeTo("web-search");

Verify the router by testing three query types. You should see different routes chosen for a policy question, a product question, and a real-time request.

Step 4: Orchestrate tool use and retrieval

Your fourth outcome is a multi-step workflow that can refine a query, retrieve context, and call tools when the first answer is not enough.

Use an agent loop or LangGraph state machine to support query refinement, source selection, retrieval, and follow-up tool calls. This is the core agentic behavior: the system can plan, act, inspect the result, and try again if needed.

state = {
  query,
  refinedQuery,
  source,
  context,
  draftAnswer,
  validationResult
}

Verify the orchestration by asking a complex question that requires two hops, such as a document lookup plus a live API check. You should see the agent chain multiple steps instead of returning the first retrieved snippet.

Step 5: Validate the final answer

Your fifth outcome is a response checker that rejects weak or unsupported answers before they reach the user.

Add a validation pass that compares the draft answer against the original query and retrieved context. If the answer is missing evidence, the agent should fetch more context, rewrite the response, or return a safe fallback.

if (!isSupported(draftAnswer, context)) {
  draftAnswer = refineAndRetry(query, context);
}

Verify validation by testing an ambiguous prompt and a grounded prompt. You should see the system ask for more information or produce a cited answer only when the evidence is sufficient.

Step 6: Expand to multimodal inputs

Your sixth outcome is an agentic RAG pipeline that can handle text, images, and real-time inputs in one workflow.

Add specialized tools for OCR, image captioning, or live data fetches, then route each input type to the right agent. This is where agentic RAG becomes more flexible than traditional RAG because it can adapt its retrieval strategy to the source type.

// Example tool set
textRetriever();
imageCaptionTool();
realtimeApiTool();

Verify the extension by submitting one text query, one image-based query, and one time-sensitive query. You should see each input type produce a different retrieval path and a context-aware final answer.

Common mistakes

• Routing everything to the vector store. Fix: add a source classifier so real-time or API-backed questions do not use stale documents.
• Skipping validation. Fix: compare the final answer with retrieved evidence and retry when support is weak.
• Adding too many agents too early. Fix: start with one router and one retriever, then split into specialized agents only after the workflow is stable.

What's next

Once the basic pipeline works, deepen it with memory, parallel sub-agents, and evaluation sets so you can compare routing accuracy, answer quality, and latency across different agent designs.