RAG web ingestion API: why the quality gate sits before the chunker
In late July we put this line on our trust page, in its list of limits:
> /v2/ingest renders pages but doesn't gate on the score yet, so a blocked page can enter a RAG corpus unflagged; that gate is roadmap, not done.
It was true. Not a great line to have on your trust page when the product is a RAG web ingestion API.
Perceive already returned a render_quality score with every page it delivered. Ingest rendered through the same pipeline and threw the verdict away, so a login wall or a soft 404 with some body text was chunked, billed and written into your JSONL next to the real documentation, with nothing in the chunk to say so.
Nothing errors when that happens. The page gets embedded with everything else, and a later query can pull it back as context the model treats as your docs.
The gate shipped on 17 September. A page whose render is blocked, or scores under 0.40, is now dropped before it reaches the chunker, with the reason recorded and no op charged. Chunks that get through carry the page's render_quality and deductions in their metadata.
EnConvert's RAG web ingestion API is POST /v2/ingest. Give it a list of URLs, a sitemap, or a seed URL to crawl, and it renders every page in headless Chrome, scores the render, drops the pages it couldn't read, splits the rest on headings and returns one JSONL file for your vector database. Ingest is generally available. The API it sits in is still labelled beta.
Whether to build ingestion yourself at all is a separate question, answered in How to build a unified ingestion layer for your RAG pipeline from scratch.
Getting the URL list
A 719-URL sitemap used to come back as pages_discovered: 50, with nothing saying the default cap had cut it.
urls mode takes an explicit list (up to 1,000). sitemap reads the sitemap from a seed url, and crawl follows links from it to max_depth (default 2, up to 5). Both stop at max_pages (default 50, up to 1,000), stay on the seed's domain unless you turn off same_domain_only, and accept regex include_patterns and exclude_patterns.
The job now reports pages_found (what discovery saw before the cap), sets discovery_truncated, and adds a warning with both numbers. On the free plan, set max_pages on purpose. 500 ops is 500 pages, and one crawl can spend all of them.
Discovery folds URLs that differ only by www. or a trailing slash. It crawls over plain HTTP, with a small, capped browser crawl as the fallback for client-rendered sites whose links only exist after JavaScript runs. Discovery isn't billed. You pay one op per page that's rendered and chunked. Skipped, failed and duplicate pages cost nothing.
Why ingest always uses the browser
On /v2/perceive, an HTML-only read tries a fast fetch with a real Chrome TLS fingerprint first and escalates to a real headless Chrome render when the page looks blocked or scores under 0.40.
Since 16 August ingest doesn't use that fast fetch and always renders in Chrome. Our scorer catches an empty app shell unhydrated_shell), but from HTML alone it can't tell a complete server-rendered page from one whose JavaScript hasn't finished filling it in. In our tests on documentation sites with the fast fetch allowed, pre-hydration HTML scored 1.00 and never escalated. The chunks carried React Suspense payloads still parked in <div hidden id="S:0">, "Loading" labels from skeleton loaders, cookie banners nobody had dismissed, and, on docs.pinecone.io, less than half the page's text. For a corpus you'll keep querying, I'd rather pay for the browser render every time.
The same release moved ingest off Crawl4AI's Fit Markdown, whose pruning filter hard-codes header in its excluded tags. Any docs site that wraps its title block in an HTML5 <header> (Mintlify sites and some Next.js-built docs do) lost its <h1> and standfirst on every page. Ingest now uses perceive's main-content pipeline, and the URL to markdown post covers how that decides what to throw away.
Renders run one page at a time, through one node in one region today (it's on the trust page), so a 1,000-page crawl is a long job. Ingest also takes no credentials, and auth, cookies or headers get a 422. Jobs resume after a restart, so the request is persisted, and we don't want secrets sitting in job state. If a page needs a session cookie, read it with perceive.
EnConvert trust page
Ingest skips a page if is_blocked is true or render_quality < 0.40. You need both checks.
Any one of http_error, a hard empty_body, unhydrated_shell, login_wall or soft_404 sinks a page by itself. A page that trips only one of the two bot deductions can land on 0.40 or 0.50 and clear the floor, so is_blocked has to catch it.
Skipped pages count toward pages_failed, and the reason is recorded in a form like the page could not be read (render_quality 0.30; http_error).
Most jobs won't show you that reason, and it's the gap I'd check before relying on this. A completed job says pages_failed: 3 and nothing about which three or why.
There's no per-page route yet.
When every page is skipped, the job fails and error_message carries the most common skip reason. To find the dropped URLs in a completed job, diff the metadata.source_url values in the JSONL against what you sent. In urls mode that's exact, because source_url is the URL you requested, not where it redirected. (Pages folded as duplicates are missing from the JSONL too. A warning gives the count.)
Here's the gate on a public docs site, with a working page plus a 404 in one urls job, then the 404 alone:
Some pages that pass will still be bad. Every web chunk carries its page's deductions, so you can be stricter at load time than we are. Drop anything with domain_redirect (the final URL landed on a different registrable domain, a cookie-consent portal for example), or raise your own floor. The scorer is heuristic and misses some block pages.The web perception post has the misses and the numbers.
How the chunker splits a page
The chunker is deterministic and uses no embeddings or tokenizer, just the standard library and regex. Boundaries are #, ## and ###. Deeper headings stay inline as content. Nothing crosses a section, and each chunk carries its full headings_path. Code blocks and pipe tables stay whole even past max_words, and lists only break between items.
Prose packs greedily up to max_words (default 512, range 32 to 4,000). When a paragraph has to break, it breaks on a sentence boundary and repeats the last sentence_overlap sentences at the top of the next chunk (default 1, range 0 to 10). Overlap never crosses a heading and stops at a code block, table or list. The sentence splitter is a plain regex on sentence-ending punctuation, so an abbreviation like "Dr." counts as a sentence end.
Size is counted in whitespace-separated words, so the chunker doesn't need your embedding model. If that model has a hard input limit, run a sample of chunks through its tokenizer to get your word-to-token ratio.
Published defaults don't agree on a size. OpenAI's file search defaults to 800-token chunks with 400 tokens of overlap. Microsoft's Azure Ai Search guidance suggests starting at 512 tokens with 25% overlap. Anthropic's contextual retrieval write-up describes chunks of "usually no more than a few hundred tokens". Our one-sentence overlap is smaller than OpenAI's or Azure's, and a chunk never straddles two sections, the same rule Unstructured's by_title strategy follows. Try two settings against your retrieval questions and keep whichever answers better.
On our ingest reference, the signature-checking code sits under "Completion webhooks" and "Verifying the signature". Those headings live in metadata.headings_path, not in the chunk's content, so a loader that embeds content alone never puts them in the vector. Anthropic's post prepends a short context, usually 50 to 100 tokens, to each chunk before embedding. Prepending the heading path is the cheap version of that.
The JSONL, and why the ids are deterministic
One JSON object per line, UTF-8. This is the documented shape with placeholder values, not a captured response render_quality and deductions only appear on web pages, because uploaded files aren't rendered):
{"id": "<md5(requested URL)[:12]>-<chunk_index, 4 digits>", "content": "<chunk text>", "metadata": {"source_url": "<the URL you requested>", "title": "<page title>", "headings_path": ["<h1>", "<h2>", "<h3>"], "section": "<last heading>", "word_count": <int>, "chunk_index": <int>, "render_quality": <0.40 to 1.0>, "deductions": {"<name>": <weight>}}}The id is the first 12 hex characters of the MD5 of the requested URL, a hyphen, and the chunk index padded to four digits. Re-ingest the same URL and you get the same ids, so an upsert keyed on id replaces rows instead of duplicating them.
We got this wrong on the file side. Until a fix we published on 15 July, we seeded /v2/ingest/files chunk ids from the uploaded filename, so two same-named files in one job got identical ids, and an upsert keyed on id overwrote one document with the other without an error. File ids now come from the storage key, so they're unique per file and stable within a job, but not across jobs. URL ids didn't change.
The ids depend on the URL and the chunk index, not the content. If a page had nine chunks last month and six today, an upsert writes -0000 through -0005 and leaves -0006 to -0008 from the old version in your index. So don't upsert blindly. Group the new records by metadata.source_url (always the URL you asked for), delete what your index holds for each URL, then write. A page that was skipped this run, or has gone from the site, has no records at all, so also compare the run's source_url set against your index. Whatever is missing keeps its old chunks until you retire them.
If two URLs in the same job turn out to be the same page, by final URL after redirects or by content hash, the second produces no chunks and isn't billed. docs.llamaindex.ai/en/stable/ and docs.llamaindex.ai/ both redirect to developers.llamaindex.ai/python/framework/, and before that check, ingest shipped every chunk twice.
Using the RAG web ingestion API from Python
A valid POST /v2/ingest answers 202 with an ing_ job id. You poll GET /v2/ingest/{job_id} (no ops) or pass a webhook_url. With the Python SDK, plus the grouping from above:
python
import json, os, time
from collections import defaultdict
import requests
from enconvert import Enconvert
client = Enconvert(api_key=os.environ["ENCONVERT_API_KEY"])
job = client.v2.ingest(mode="sitemap", url="https://docs.example.com", max_pages=100)
while job.status not in ("completed", "failed", "canceled"):
time.sleep(10)
job = client.v2.get_ingest_job(job.job_id)
print(job.status, job.pages_processed, job.pages_failed, job.error_message, job.warnings)
if job.status == "completed":
by_page = defaultdict(list)
for line in requests.get(job.output_url, timeout=120).content.splitlines():
record = json.loads(line)
by_page[record["metadata"]["source_url"]].append(record)
# per source_url: delete what your index holds for it, then upsert by record["id"](The loop splits bytes, not text, because str.splitlines() also breaks on U+0085, which JSON leaves unescaped inside a chunk.)
The webhook is HMAC-SHA256 over <timestamp>.<raw body>, sent as X-Enconvert-Signature: sha256=<hex> with X-Enconvert-Timestamp. Reject anything older than 300 seconds. Failed deliveries retry after 1, 4 and 16 seconds, and POST /v2/ingest/{job_id}/retry-webhook re-sends later. It fires on completed only, so a job where every page was skipped fails without calling you, and the payload has no pages_failed or warnings. I'd keep a slow poll as a backstop and GET the job once the webhook lands.
output_url expires 15 minutes after signing and every GET signs a fresh one, but the JSONL itself is deleted when your plan's retention ends (an hour on the free Founding plan, a week on Indie and Studio, 30 days on Production) unless you have the storage add-on. Download it when the job completes. Agents can run the same flow through our MCP server's start_ingest tool the agent web data post covers it).
What ingest doesn't do
respect_robots defaults to false. Set it to true and ingest won't render a disallowed URL for your corpus. In crawl mode, though, discovery can still load a disallowed page while following links (over HTTP, or in the small browser fallback for client-rendered sites), and drops it afterwards. What you crawl, and whether you honour robots.txt, is up to you as the operator.
Files go through /v2/ingest/files into the same chunker (PDF, DOCX, PPTX, XLSX, EPUB and more, up to 200 per request), but there's no OCR, so images are rejected.
If you'd rather own the crawl, I'd start with Crawl4AI. Our renderer is built on it. It's open source (Apache-2.0) and, as of October 2026, does deep crawls with crash recovery and ships its own anti-bot detector. Apify's Website Content Cawler is pitched at feeding vector databases and RAG pipelines. Either way, you can apply the same 0.40 floor and is_blocked check before your chunker with our MIT-licensed scorer, render-quality pip install render-quality). It scores the HTML you already fetched and leaves out the four runtime deductions, which need a browser.
The request parameters, limits and error codes are in the ingest reference. Ingest runs on the free plan, and a key doesn't need a card. A job id is all I need to look at a broken page that got through the gate, email_me.