The document that broke your agent (and why it was never the model’s fault)
THE DOCUMENT THAT BROKE YOUR AGENT (AND WHY IT WAS NEVER THE MODEL’S FAULT)
An agent is reading a PDF technical manual. Someone asks it for the installation requirements. The agent answers immediately, with complete confidence. Specific steps. No hedging. No “I don’t know.” Just wrong.
The instinct is to blame the model. Maybe it needs a better prompt. Maybe a larger one would handle the document differently. The team usually spends two days finding that out.
What actually happened
The PDF was read with a standard extraction library, PyPDF2 or pdfplumber, page by page, the way most pipelines start. Then a splitter cut the text into chunks by token count.
Chunk 7 starts mid-sentence on page 4. The installation requirements span two chunks, and neither carries a heading in its metadata, because the heading landed in a different piece. The chunk the retriever returned opens with “ensure all cable connections are secure” and runs for several hundred words about a completely different subsystem.
The model received that chunk and reasoned over it correctly. It did not know what it was looking at, because nothing in the chunk told it.
The real problem
This is not a retrieval problem. The retriever found a passage that matched the question. That’s its job. It did it.
It is not an embedding problem either. A better embedding model cannot recover a section name that was never captured.
The chunk had no structural context. No section name, no heading, no indication of where in the document it sat. The model was handed text without provenance and asked to reason with precision.
The model didn’t fail. The ingestion layer did.
Once you see it, you see it everywhere
This is not a PDF problem.
DOCX files keep heading levels as invisible style names. A naive extraction loop returns everything as flat paragraphs and the hierarchy disappears. Spreadsheets get serialised into strings, column headers either gone or stamped onto every row. HTML pages carry navigation menus and sidebars into every chunk because nothing stripped them first.
The source format changes. The failure mode does not.
In each case, text arrives at the retriever without the structure that made it mean something. The model is left to guess at context the document once had and the pipeline quietly discarded.
What heading-aware chunking actually gives you
Chunks that know where they came from.
Each one carries a section name and the heading path above it, and the schema is the same whether you are ingesting a PDF, a DOCX, or an HTML file. The chunk index tells the agent where it sits in the document. When the retriever returns a passage, the agent knows it came from the Installation section of that specific manual, not from an anonymous block that could be anywhere in ninety pages.
That changes the answer. The agent can cite the section, distinguish an installation requirement from a maintenance note, tell the user where to read more. The token count is roughly the same. The quality of the answer is not.
The cost you do not always see immediately
A wrong answer from an internal support agent erodes trust quickly. Engineers stop relying on the tool and go back to searching the PDF manually. The project becomes another AI experiment that did not stick.
The model gets blamed. Prompts get rewritten, then a different model gets tried. The document pipeline stays exactly as it was, and the same wrong answer comes back in a different form.
The fix was upstream the whole time.
A small honest note
We built EnConvert’s ingestion layer because we kept running into this. One endpoint, any file type, one JSONL output schema. Each chunk carries its section name and heading path so the model gets context it can reason over, not just text.
It is not the only way to solve this. If you have a chunking pipeline that already preserves structure, keep it. We are not here to talk anyone out of working code.
But if you are two days into tuning prompts because the answers keep coming back wrong, check what the model was given before you tune the model. The answer is usually in the chunk.
Free tier: 500 ops per month. enconvert.com.
EnConvert. One unified API for AI agents and RAG pipelines. One API key. One output schema. Whatever the source.
About the Author
Krystin Lim (@theunscriptedk) is the commercial co-founder of EnConvert, a unified ingestion API for AI agents. She writes about developer tools, AI agent infrastructure, and what it looks like to build a technical product as a non-technical founder. 102. figuring it out in public.