Building RAG on LlamaIndex?

LlamaIndex RAG returning the wrong chunks? We bring it back.

LlamaIndex is the quickest way to stand up a RAG pipeline and the most forgiving place to make bad chunking decisions. When retrieval returns the wrong passages, the node parser blows up on your PDFs, or the vector store drifts out of sync with the source documents, we go through the indexing step by step and get the answers back to something you can trust.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

llamaindex.log

still failing

What it is

LlamaIndex at a glance

LlamaIndex is an open-source framework for building retrieval-augmented generation (RAG) and agent applications on top of large language models. Alongside LangChain, it is one of the two most widely adopted frameworks in the space, and its focus is deliberately narrower: data ingestion, indexing and retrieval done properly.

The library covers document loaders for most common sources, a rich set of node parsers for chunking, a storage abstraction over Pinecone, Weaviate, Chroma, pgvector, Qdrant and most other vector stores, and query engines that assemble retrieval, reranking and response synthesis. There is a hosted platform (LlamaCloud) for teams that would rather not run the ingestion stack themselves.

Type
RAG and agent framework for LLM applications
First released
2022 (originally GPT Index)
License
MIT
Maintained by
LlamaIndex, Inc.
Typically used for
Document ingestion, chunking, vector indexing, retrieval, agentic RAG

The honest version

LlamaIndex pros and cons

No technology is all upside. This is where LlamaIndex earns its place and where it tends to cause trouble.

Where LlamaIndex is strong

  • Deep focus on retrieval qualityNode parsers, hierarchical indexes, recursive retrieval and reranking are first-class parts of the library rather than afterthoughts bolted onto chat completion calls.
  • Wide vector-store supportPinecone, Weaviate, Chroma, pgvector, Qdrant, Milvus, Elasticsearch and many more are supported out of the box with consistent APIs.
  • LlamaHub for connectorsHundreds of community loaders cover Notion, Confluence, Google Drive, S3, Salesforce and the long tail without having to write custom scrapers.
  • LlamaParse for messy documentsThe hosted parser handles PDFs with tables, images and awkward layouts better than most open-source alternatives, which saves a lot of custom pipeline work.
  • Agent workflowsWorkflows and query engines give you deterministic control over multi-step reasoning without the full chain abstraction tax.
  • LlamaCloud for the ingestion stepTeams that just want a hosted index without running Pinecone, embeddings jobs and parsing workers can offload the whole pipeline.

Where LlamaIndex bites

  • API churn is realServiceContext was replaced with Settings, module paths have moved between llama_index.core and plugin packages, and older tutorials go stale within months.
  • Chunking is still the hard partThe library gives you excellent parsers; choosing the right one for your documents is judgement work, and the default settings rarely survive contact with real PDFs.
  • Response synthesis can hallucinate quietlyThe tree-summarise and compact strategies will produce fluent answers even when retrieval returned nothing relevant, so eval harnesses are not optional.
  • Cost model is implicitAn ingestion run that embeds every page of every document can produce a surprise bill, especially on OpenAI embeddings at scale.
  • Metadata is easy to loseFile paths, page numbers and section titles fall out of node metadata quietly if the parser is swapped, which breaks citations downstream.
  • Observability is bring-your-ownTracing agent and query-engine runs needs Langfuse, LangSmith or an OpenTelemetry integration; nothing is built in.

Sound familiar?

Common LlamaIndex problems we fix

The LlamaIndex problems that bring people to us, and what you have once each one is fixed.

Node has no embedding

Upsert fails because nodes are not embedded

An ingestion pipeline that forgot to run the embedding transform, or a cache that returned pre-embedding nodes from a previous version. The vector store rejects the write cleanly, and nothing is indexed.

Pipeline producing embedded nodes

QueryEngine: no nodes retrieved

Retrieval returns nothing for sensible queries

Usually a similarity threshold set too high, metadata filters that no node matches, or an index that was populated under a different embedding model to the one the query is using.

Retrieval tuned to the real corpus

input exceeds max_tokens

Chunks are too big for the embedding model

A node parser configured with a chunk size the embedding endpoint cannot accept, or a document that produced one oversize chunk because the splitter could not find a sensible break.

Chunks that fit the model's window

metadata schema mismatch

Vector store rejects writes after a schema change

A new field in node metadata that was not declared in the target index schema, or a type change that the vector store will not accept without a rebuild.

Schemas aligned on both sides

pypdf: incorrect startxref

Document loader stumbles on bad PDFs

A scanned PDF with no text layer, a malformed cross-reference table or an encrypted file with an unexpected cipher. The default loader logs a warning and silently returns almost nothing.

Documents parsed cleanly or re-routed

ReActAgent: Could not parse output

Agent loop cannot parse the model's thought

A model that drifted from the Thought and Action format the ReAct agent expects, often after a model upgrade or a slight prompt change. The agent falls back to error paths or loops.

Agent robust to real model output

IngestionPipeline: cache mismatch, re-embedding

Cache is not hitting, embeddings burn through budget

The pipeline's content hash changed because a transform was added, moved or re-ordered, so the cache invalidates and every node is re-embedded from scratch on the next run.

Cache stable across ingests

SubQuestionQueryEngine: empty response

Sub-question decomposition returns nothing

The decomposed sub-questions do not match any node in the index, usually because the planner used different vocabulary to the corpus. Needs query transforms or a hybrid retriever.

Sub-questions landing on real data

What we do

LlamaIndex help, from chunking to agents

Call us for one bad retrieval, or let us design the whole RAG pipeline end to end.

RAG rescue
Retrieval that returns the wrong passages, response synthesis that hallucinates over empty context, or indexes that silently drift from source documents.
Document ingestion
PDFs, HTML, DOCX and Notion dumps parsed with the right loader for the content, with metadata and page numbers preserved for citations.
Chunking strategy
Node parsers chosen against your documents rather than the default, with eval runs to prove the change made retrieval better and not just different.
Vector store integration
Pinecone, pgvector, Qdrant and Weaviate wired in with proper schema, retention and reindex paths for when the embedding model changes.
Agent workflows
Workflows and query engines designed around deterministic tool use rather than open-ended ReAct loops that go off the rails.
Managed Tech
Continued operation of the ingestion pipeline, embedding budget watching, retrieval quality tracking and upgrades when LlamaIndex releases change module paths.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running LlamaIndex alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

LlamaIndex questions

Something else? Call+1 737 727 7327

Why is my LlamaIndex RAG returning irrelevant chunks?

Retrieval tends to fail for one of four reasons: the chunks are too small or too large for the question, the embedding model does not match what was used at index time, the metadata filter excludes the right documents, or a similarity threshold is set too conservatively. Fix the one that applies, and run an evaluation set to confirm the change actually improved recall.

Should we use LlamaIndex or LangChain for RAG?

LlamaIndex is the stronger pick when the core job is document ingestion, chunking and retrieval quality. LangChain earns its keep when the application is dominated by agent orchestration, tool chaining and workflow logic across many steps. Teams routinely combine them, with LlamaIndex owning the index and LangGraph or Workflows owning the agent on top.

How do we keep LlamaIndex embedding costs under control?

Use the IngestionPipeline cache so unchanged documents are not re-embedded on every run, batch embedding calls, and consider smaller or self-hosted embedding models for the long tail of low-value documents. Also exclude boilerplate like headers and footers from the chunks before they hit the embedding API.

Can LlamaIndex handle scanned PDFs and tables?

Default loaders cannot, reliably. LlamaParse (the hosted parser) handles complex layouts and tables far better, and for OCR-only PDFs you want a dedicated OCR step upstream that writes back a text layer. Trying to force the open-source loaders through poor-quality scans wastes budget and degrades retrieval.

How do we add citations to LlamaIndex answers?

Preserve metadata through the whole pipeline (file path, page number, section) and use a response synthesizer that emits the node sources alongside the answer. The citations break the moment metadata is lost in a custom parser, so that is the first place to check when they go missing.

How much does LlamaIndex help cost?

The quote is free. Tell us what the retrieval is doing today, show us the kind of documents involved, and we come back with a number before anything is touched.

Free estimate

LlamaIndex giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.