How to build a unified ingestion layer for your RAG pipeline from scratch

Most RAG pipelines do not fail at the vector store. They do not fail at retrieval. They fail before any of that: at the point where content moves from a URL, a PDF, or a DOCX file into something a model can actually use.

The failure is usually silent. You get embeddings. They are just embeddings of garbage.

Teams spend weeks tuning chunking strategies and retrieval logic, then feed the model content riddled with navigation noise, broken table ordering, and inconsistent structure. The retrieval step gets blamed. The problem started at ingestion.

The problem in practice

Handling URLs, PDFs, and DOCX independently means a separate toolchain for each one, each with its own failure modes.

For web pages, you start with requests and BeautifulSoup. That works until you hit a page built on React or Next.js where content is injected after the initial load. You add Playwright or Puppeteer. Now you are managing browser instances, handling timeouts, deciding when a page is done rendering, and dealing with sites that detect headless browsers and serve different content. You write extraction logic to identify the main content body and strip navigation, cookie banners, and structural chrome. That logic is mostly heuristic (text density, element position, text-to-link ratio) and it does not generalise cleanly across sites.

For PDFs, pdfplumber handles most cases. Until you hit a two-column academic paper where extraction reads across both columns simultaneously, producing sentences that are half one column and half another. Or a scanned PDF with no text layer, which now needs an OCR step. Or a document where tables are rendered as positioned text elements rather than table structures, and the output is a sequence of cell values with no row or column context. PyMuPDF is faster but has its own layout edge cases. You will likely want both.

For DOCX, python-docx gives you access to the document XML. What it does not give you is clean output. You traverse paragraphs. You traverse table cells separately. You handle tracked changes, revision history, embedded images, and comments that appear in the XML but should not appear in your content. You write your own serialiser to get from the document object model to something resembling readable text.

Each library has a different output format. You write normalisation code to reconcile all three before anything reaches your chunker. Then you add error handling, retries, timeout logic, and logging. Then you find the edge cases you had not seen yet.

This is not a weekend project. It is several weeks of engineering work, and the maintenance burden is ongoing because upstream sources change.

What unified ingestion actually means

Unified ingestion is not about using a single library. It is about producing a consistent output schema regardless of input source.

Clean, LLM-ready output has one form: structured Markdown where headings are preserved as ATX headings, tables render as GFM tables, code blocks carry language identifiers, and the content is the content — not a mixture of content and the structural chrome of whatever format it came from.

The key property is consistency. If a URL produces ## Installation as a heading and a PDF produces Installation as a plain line followed by a blank line, your chunking strategy has to handle both. If it is designed around Markdown heading boundaries, one of those inputs breaks the chunk boundaries silently.

Unified ingestion means your chunking logic, your embedding pipeline, and your retrieval layer all operate against one format. The source type becomes an implementation detail of the ingestion layer, not a branching condition in the rest of your pipeline.

Building it yourself: what it actually takes

Web extraction requires a fetch layer with redirect handling and timeout management, a render layer with a headless browser for JavaScript-heavy pages, a content extraction layer that identifies the main body and strips navigation and chrome, and a conversion layer that produces consistent Markdown. The content extraction step is mostly heuristic and does not generalise perfectly across sites.

PDF extraction requires a text extraction layer for text-layer PDFs, an OCR layer for scanned documents, table detection and reconstruction logic, multi-column layout handling, and heading inference; PDF has no semantic heading structure, only text with larger font sizes.

DOCX extraction requires paragraph traversal, table cell extraction with row and column mapping, heading level inference from style names, and filtering of tracked changes, comments, and embedded object references.

Across all three, you need format normalisation to a common schema, error handling that surfaces failures explicitly rather than returning empty strings, and consistent Markdown output with the same heading syntax, table format, and code block convention regardless of source.

None of this is impossible. It is a real surface area to maintain, and it grows every time a site changes its markup or a library ships a breaking change.

Using an API instead

The EnConvert equivalent is one POST request per source type. The response schema is identical across all three calls. The response schema is identical across all three calls, be it for URL, PDF or a DOCX file.

Your downstream code does not branch by input type. You write the chunking logic, the embedding step, and the retrieval layer once. Error handling is also consistent: non-200 responses follow the same error schema regardless of which endpoint surfaced the problem.

What you give up: control over the parsing internals and the ability to tune extraction for a specific source type. Those are real costs. For most teams they are smaller than the cost of owning three parsers and the edge cases that come with them.

When to build vs buy

If ingestion is a core differentiator for your product, if you are building document intelligence where extraction quality is what you are selling, build it. The edge case handling is where your product value lives and you need control over it.

If ingestion is infrastructure, the calculation is different. You are not building a competitive advantage by writing your own PDF table extractor. You are spending engineering time on a problem that is not specific to what you are building.

The test is simple: would a customer choose your product over a competitor because of how you handle DOCX extraction? If yes, build it. If they would not know or care as long as it works, that is infrastructure. Treat it accordingly.

Early-stage teams in particular should be honest about this. Weeks spent building a robust ingestion layer are weeks not spent on the part of the product that is actually differentiated.

Get your free API key; it's 500 ops/month, no card required.

We launched on Product Hunt yesterday; if you have tried the free tier, an upvote, comment or honest review on the listing would mean a lot to us. producthunt.com/products/enconvert