---
title: "RAG web ingestion API: why the quality gate sits before the chunker"
description: "How EnConvert's RAG web ingestion API crawls a site, renders each page in Chrome, skips pages it couldn't read, and writes heading-aware JSONL chunks."
canonical: "https://www.enconvert.com/blog/rag-web-ingestion-api"
locale: "en"
author: "het Dave"
published: 2026-10-29
---

# RAG web ingestion API: why the quality gate sits before the chunker

- Published: 2026-10-29
- Author: het Dave

How EnConvert's RAG web ingestion API crawls a site, renders each page in Chrome, skips pages it couldn't read, and writes heading-aware JSONL chunks.

In late July we put this line on our trust page, in its list of limits:

> /v2/ingest renders pages but doesn't gate on the score yet, so a blocked page can enter a RAG corpus unflagged; that gate is roadmap, not done.

It was true. Not a great line to have on your trust page when the product is a RAG web ingestion API.

Perceive already returned a `render_quality` score with every page it delivered. Ingest rendered through the same pipeline and threw the verdict away, so a login wall or a soft 404 with some body text was chunked, billed and written into your JSONL next to the real documentation, with nothing in the chunk to say so.

Nothing errors when that happens. The page gets embedded with everything else, and a later query can pull it back as context the model treats as your docs.

The gate shipped on 17 September. A page whose render is blocked, or scores under 0.40, is now dropped before it reaches the chunker, with the reason recorded and no op charged. Chunks that get through carry the page's `render_quality` and `deductions` in their metadata.

EnConvert's RAG web ingestion API is `POST /v2/ingest`. Give it a list of URLs, a sitemap, or a seed URL to crawl, and it renders every page in headless Chrome, scores the render, drops the pages it couldn't read, splits the rest on headings and returns one JSONL file for your vector database. Ingest is generally available. The API it sits in is still labelled beta.

Whether to build ingestion yourself at all is a separate question, answered in [How to build a unified ingestion layer for your RAG pipeline from scratch](https://www.enconvert.com/blog/how-to-build-a-unified-ingestion-layer-for-your-rag-pipeline-from-scratch.md).

## Getting the URL list

A 719-URL sitemap used to come back as `pages_discovered: 50`, with nothing saying the default cap had cut it.

`urls` mode takes an explicit list (up to 1,000). `sitemap` reads the sitemap from a seed `url`, and `crawl` follows links from it to `max_depth` (default 2, up to 5). Both stop at `max_pages` (default 50, up to 1,000), stay on the seed's domain unless you turn off `same_domain_only`, and accept regex `include_patterns` and `exclude_patterns`.

The job now reports `pages_found` (what discovery saw before the cap), sets `discovery_truncated`, and adds a warning with both numbers. On the free plan, set `max_pages` on purpose. 500 ops is 500 pages, and one crawl can spend all of them.

Discovery folds URLs that differ only by `www.` or a trailing slash. It crawls over plain HTTP, with a small, capped browser crawl as the fallback for client-rendered sites whose links only exist after JavaScript runs. Discovery isn't billed. You pay one op per page that's rendered and chunked. Skipped, failed and duplicate pages cost nothing.

## Why ingest always uses the browser

On `/v2/perceive`, an HTML-only read tries a fast fetch with a real Chrome TLS fingerprint first and escalates to a real headless Chrome render when the page looks blocked or scores under 0.40.

Since 16 August ingest doesn't use that fast fetch and always renders in Chrome. Our scorer catches an empty app shell `unhydrated_shell`), but from HTML alone it can't tell a complete server-rendered page from one whose JavaScript hasn't finished filling it in. In our tests on documentation sites with the fast fetch allowed, pre-hydration HTML scored 1.00 and never escalated. The chunks carried React Suspense payloads still parked in `<div hidden id="S:0">`, "Loading" labels from skeleton loaders, cookie banners nobody had dismissed, and, on [docs.pinecone.io](http://docs.pinecone.io), less than half the page's text. For a corpus you'll keep querying, I'd rather pay for the browser render every time.

The same release moved ingest off Crawl4AI's Fit Markdown, whose pruning filter hard-codes `header` in its excluded tags. Any docs site that wraps its title block in an HTML5 `<header>` (Mintlify sites and some Next.js-built docs do) lost its `<h1>` and standfirst on every page. Ingest now uses perceive's main-content pipeline, and [the URL to markdown post](https://www.enconvert.com/blog/url-to-markdown-api.md) covers how that decides what to throw away.

Renders run one page at a time, through one node in one region today (it's on the [trust page](https://www.enconvert.com/trust.md)), so a 1,000-page crawl is a long job. Ingest also takes no credentials, and `auth`, `cookies` or `headers` get a 422. Jobs resume after a restart, so the request is persisted, and we don't want secrets sitting in job state. If a page needs a session cookie, read it with perceive.

## EnConvert trust page

Ingest skips a page if `is_blocked` is true or `render_quality < 0.40`. You need both checks.

Any one of `http_error`, a hard `empty_body`, `unhydrated_shell`, `login_wall` or `soft_404` sinks a page by itself. A page that trips only one of the two bot deductions can land on 0.40 or 0.50 and clear the floor, so `is_blocked` has to catch it.

Skipped pages count toward `pages_failed`, and the reason is recorded in a form like `the page could not be read (render_quality 0.30; http_error)`.

Most jobs won't show you that reason, and it's the gap I'd check before relying on this. A completed job says `pages_failed: 3` and nothing about which three or why.

There's no per-page route yet.

When every page is skipped, the job fails and `error_message` carries the most common skip reason. To find the dropped URLs in a completed job, diff the `metadata.source_url` values in the JSONL against what you sent. In `urls` mode that's exact, because `source_url` is the URL you requested, not where it redirected. (Pages folded as duplicates are missing from the JSONL too. A warning gives the count.)

Here's the gate on a public docs site, with a working page plus a 404 in one `urls` job, then the 404 alone:

Some pages that pass will still be bad. Every web chunk carries its page's deductions, so you can be stricter at load time than we are. Drop anything with `domain_redirect` (the final URL landed on a different registrable domain, a cookie-consent portal for example), or raise your own floor. The scorer is heuristic and misses some block pages.[The web perception post](https://www.enconvert.com/blog/web-perception-api.md) has the misses and the numbers.

## How the chunker splits a page

The chunker is deterministic and uses no embeddings or tokenizer, just the standard library and regex. Boundaries are `#`, `##` and `###`. Deeper headings stay inline as content. Nothing crosses a section, and each chunk carries its full `headings_path`. Code blocks and pipe tables stay whole even past `max_words`, and lists only break between items.

Prose packs greedily up to `max_words` (default 512, range 32 to 4,000). When a paragraph has to break, it breaks on a sentence boundary and repeats the last `sentence_overlap` sentences at the top of the next chunk (default 1, range 0 to 10). Overlap never crosses a heading and stops at a code block, table or list. The sentence splitter is a plain regex on sentence-ending punctuation, so an abbreviation like "Dr." counts as a sentence end.

Size is counted in whitespace-separated words, so the chunker doesn't need your embedding model. If that model has a hard input limit, run a sample of chunks through its tokenizer to get your word-to-token ratio.

Published defaults don't agree on a size. OpenAI's [file search](https://developers.openai.com/api/docs/guides/retrieval) defaults to 800-token chunks with 400 tokens of overlap. Microsoft's [Azure Ai Search guidance](https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents) suggests starting at 512 tokens with 25% overlap. Anthropic's [contextual retrieval write-up](https://www.anthropic.com/news/contextual-retrieval) describes chunks of "usually no more than a few hundred tokens". Our one-sentence overlap is smaller than OpenAI's or Azure's, and a chunk never straddles two sections, the same rule Unstructured's `by_title` strategy follows. Try two settings against your retrieval questions and keep whichever answers better.

On our ingest reference, the signature-checking code sits under "Completion webhooks" and "Verifying the signature". Those headings live in `metadata.headings_path`, not in the chunk's `content`, so a loader that embeds `content` alone never puts them in the vector. Anthropic's post prepends a short context, usually 50 to 100 tokens, to each chunk before embedding. Prepending the heading path is the cheap version of that. 

## The JSONL, and why the ids are deterministic

One JSON object per line, UTF-8. This is the documented shape with placeholder values, not a captured response `render_quality` and `deductions` only appear on web pages, because uploaded files aren't rendered):

```
{"id": "<md5(requested URL)[:12]>-<chunk_index, 4 digits>", "content": "<chunk text>", "metadata": {"source_url": "<the URL you requested>", "title": "<page title>", "headings_path": ["<h1>", "<h2>", "<h3>"], "section": "<last heading>", "word_count": <int>, "chunk_index": <int>, "render_quality": <0.40 to 1.0>, "deductions": {"<name>": <weight>}}}
```

The id is the first 12 hex characters of the MD5 of the requested URL, a hyphen, and the chunk index padded to four digits. Re-ingest the same URL and you get the same ids, so an upsert keyed on `id` replaces rows instead of duplicating them.

We got this wrong on the file side. Until a fix we published on 15 July, we seeded `/v2/ingest/files` chunk ids from the uploaded filename, so two same-named files in one job got identical ids, and an upsert keyed on `id` overwrote one document with the other without an error. File ids now come from the storage key, so they're unique per file and stable within a job, but not across jobs. URL ids didn't change.

The ids depend on the URL and the chunk index, not the content. If a page had nine chunks last month and six today, an upsert writes `-0000` through `-0005` and leaves `-0006` to `-0008` from the old version in your index. So don't upsert blindly. Group the new records by `metadata.source_url` (always the URL you asked for), delete what your index holds for each URL, then write. A page that was skipped this run, or has gone from the site, has no records at all, so also compare the run's `source_url` set against your index. Whatever is missing keeps its old chunks until you retire them.

If two URLs in the same job turn out to be the same page, by final URL after redirects or by content hash, the second produces no chunks and isn't billed. `docs.llamaindex.ai/en/stable/` and `docs.llamaindex.ai/` both redirect to `developers.llamaindex.ai/python/framework/`, and before that check, ingest shipped every chunk twice.

##  Using the RAG web ingestion API from Python

A valid `POST /v2/ingest` answers `202` with an `ing_` job id. You poll `GET /v2/ingest/{job_id}` (no ops) or pass a `webhook_url`. With the Python SDK, plus the grouping from above:

```
python
import json, os, time
from collections import defaultdict

import requests
from enconvert import Enconvert

client = Enconvert(api_key=os.environ["ENCONVERT_API_KEY"])
job = client.v2.ingest(mode="sitemap", url="https://docs.example.com", max_pages=100)

while job.status not in ("completed", "failed", "canceled"):
    time.sleep(10)
    job = client.v2.get_ingest_job(job.job_id)

print(job.status, job.pages_processed, job.pages_failed, job.error_message, job.warnings)

if job.status == "completed":
    by_page = defaultdict(list)
    for line in requests.get(job.output_url, timeout=120).content.splitlines():
        record = json.loads(line)
        by_page[record["metadata"]["source_url"]].append(record)
    # per source_url: delete what your index holds for it, then upsert by record["id"]
```

(The loop splits bytes, not text, because `str.splitlines()` also breaks on U+0085, which JSON leaves unescaped inside a chunk.)

The webhook is HMAC-SHA256 over `<timestamp>.<raw body>`, sent as `X-Enconvert-Signature: sha256=<hex>` with `X-Enconvert-Timestamp`. Reject anything older than 300 seconds. Failed deliveries retry after 1, 4 and 16 seconds, and `POST /v2/ingest/{job_id}/retry-webhook` re-sends later. It fires on `completed` only, so a job where every page was skipped fails without calling you, and the payload has no `pages_failed` or `warnings`. I'd keep a slow poll as a backstop and GET the job once the webhook lands.

`output_url` expires 15 minutes after signing and every GET signs a fresh one, but the JSONL itself is deleted when your plan's retention ends (an hour on the free Founding plan, a week on Indie and Studio, 30 days on Production) unless you have the storage add-on. Download it when the job completes. Agents can run the same flow through our MCP server's `start_ingest` tool [the agent web data post](https://www.enconvert.com/blog/agent-web-data-api.md) covers it).

## What ingest doesn't do

`respect_robots` defaults to false. Set it to true and ingest won't render a disallowed URL for your corpus. In `crawl` mode, though, discovery can still load a disallowed page while following links (over HTTP, or in the small browser fallback for client-rendered sites), and drops it afterwards. What you crawl, and whether you honour robots.txt, is up to you as the operator.

[Files](https://www.enconvert.com/docs/guides/file-ingestion.md) go through `/v2/ingest/files` into the same chunker (PDF, DOCX, PPTX, XLSX, EPUB and more, up to 200 per request), but there's no OCR, so images are rejected.

If you'd rather own the crawl, I'd start with [Crawl4AI](https://github.com/unclecode/crawl4ai). Our renderer is built on it. It's open source (Apache-2.0) and, as of October 2026, does deep crawls with crash recovery and ships its own anti-bot detector. Apify's [Website Content Cawler](https://apify.com/apify/website-content-crawler) is pitched at feeding vector databases and RAG pipelines. Either way, you can apply the same 0.40 floor and `is_blocked` check before your chunker with our MIT-licensed scorer, [render-quality](https://github.com/enconvert/render-quality) `pip install render-quality`). It scores the HTML you already fetched and leaves out the four runtime deductions, which need a browser.

The request parameters, limits and error codes are in the [ingest reference](https://www.enconvert.com/docs/endpoints/ingest.md). Ingest runs on the free plan, and [a key](https://www.enconvert.com/auth?mode=signup) doesn't need a card. A job id is all I need to look at a broken page that got through the gate, [email_me](mailto:het@enconvert.com).
