---
title: "LLM web scraping API: why distill rejects keys it doesn't know"
description: "22 of 33 QA tests on /v2/perceive sent a key it didn't know and got a 200. What our LLM web scraping API does with guessed keys and invented fields."
canonical: "https://www.enconvert.com/blog/llm-web-scraping-api"
locale: "en"
author: "Het Dave"
published: 2026-10-29
---

# LLM web scraping API: why distill rejects keys it doesn't know

- Published: 2026-10-29
- Author: Het Dave

22 of 33 QA tests on /v2/perceive sent a key it didn't know and got a 200. What our LLM web scraping API does with guessed keys and invented fields.

Twenty-two of the 33 tests an external testing team ran against EnConvert's `/v2/perceive` endpoint, between 3 and 5 August 2026, sent a top-level key our API didn't have: 18 used `content_only` (a paraphrase of another scraping tool's main-content flag), and 4 sent `markdown_fit`, which was an output name, as a flag. Every one came back 200. The API had dropped the unknown key and run its defaults, and 12 of those 22 came straight from the cache as identical files. So the testers asked whether our cache was keyed by URL alone.

Distill, our LLM web scraping API, was already rejecting unknown keys that week. Three other endpoints weren't.

On 6 August I re-ran all 33 tests live. The cache was fine. The parameter was being deleted before it got there. A fresh URL sent bare, then with `content_only: true`, then with `onlyMainContent: true`, came back as the same stored file every time.

We were strict everywhere else, which is why nobody suspected it. An unknown name inside `outputs` got a 422, and so did `proxy_url`. Our trust page even had a line titled "Reserved paths return 422, never a silent 200". Unknown top-level keys still slipped through on perceive, discover and lookup (the JSON bodies of ingest and watch already rejected them). Our fault for accepting them silently. Since 7 August those three answer an unknown key with a 422, and the message in `errors` names it: `unknown parameter 'content_only'`.

An agent writing client code from memory is in the testers' spot. Parameter names from several scraping APIs blur together, and a 200 gives it no reason to doubt a guess. On the way out, a model can add keys you never asked for, or fill a field with something the page never said. Distill drops the extra keys. It can't catch an invented value.

EnConvert reads web pages for AI agents, and `POST /v2/distill` is its LLM web scraping endpoint. You send up to 50 URLs and a schema, and you get back one row per URL. A completed row's `data` has exactly your schema's keys, and the row carries the `render_quality` score of the page it came from. Distill is generally available, though the API around it is still labelled beta. Our docs list it as a paid-plan endpoint, and the model pass needs AI credits (Indie and up).

## What our LLM web scraping API does with each URL

The schema is either a JSON Schema object or a flat `{field: description}` map, up to 200 top-level properties. Each URL renders once, through the same path as perceive.

If you send a `css_schema` (a `baseSelector` for the repeating element plus `fields`), Crawl4AI's `JsonCssExtractionStrategy` runs over the rendered HTML. That pass costs nothing. It gets 10 seconds per URL, and if it runs out, the URL falls through to the model with a warning. Without a `css_schema`, every field goes to the model.

Whatever is still empty after that goes to the model, and nothing else does. Empty means `null`, an empty string, list or object, or an array whose items lack a declared sub-field. Distill sends just those fields as a reduced schema, in one forced tool call per URL. I chose a forced tool call over a text prompt plus JSON parsing in June, before distill existed, because Crawl4AI's own LLM strategy couldn't force one (it parses a text answer with `json.loads`), and it gave us nowhere to attach prompt caching.

Before the page reaches the model it's converted to Markdown and cut at 26,000 bytes of UTF-8, with the title and meta description put in front. The cut counts bytes, so a page in Chinese or full of emoji can't carry more tokens than the cost cap assumed.

The Markdown step is a fix. The cut used to run on regex-stripped HTML, and on markup-heavy pages (Stripe and GitHub, in our testing) tag soup filled the budget. The visible text never reached the model, so a page that rendered at 1.0 came back with nothing extracted, and we billed the call anyway.

The model pass is capped four ways:

- $0.05 per call, checked against a worst-case estimate before any network request goes out

- $0.50 per distill request

- 50 model calls per request

- your monthly AI-credit balance: $5 on Indie, $15 on Studio, $40 on Production, unused credits roll over (the free Founding plan has none)

A cap never fails the request. The fields it cut off come back `null` with a warning saying why. You pay one op per URL that completes, and a row that fails (a robots block, a crashed render) costs nothing. Model spend comes out of AI credits, not ops, and each row reports its `tokens` and `cost_cents`.

## Pages under the floor never reach the model

Every render gets a `render_quality` score from 0 to 1 (the [web perception post](https://www.enconvert.com/blog/web-perception-api.md) explains it), and below 0.40 we treat the page as failed. Distill skips the model for any page that is blocked or under that floor. Before the August QA fixes it only checked for bot blocks. Now a 404 or an unhydrated single-page app is skipped too. Neither has anything to extract, so a model call there is money spent on nothing. Perceive's schema extraction follows the same rule.

In distill that URL still counts as completed and bills its op, because the render happened. Fields the CSS pass didn't fill come back `null` (or `[]` for an array), with a warning that the model was skipped. Perceive doesn't bill a blocked read (see the [web reading API post](https://www.enconvert.com/blog/web-reading-api.md)). Distill still does, which makes it an exception to the "blocked reads are not billed" line on our pricing page.

Distill rows carry `render_quality` but not `is_blocked` or the named `deductions`, and a blocked page can sit at or above the floor: bot detection alone scores 0.50, an anti-bot challenge alone exactly 0.40. The server still skips the model, but the score won't tell your code why. The gate I'd use checks the score and that at least one field was filled `extraction_tier` other than `none`), as in the code below. To see why a page was skipped, read it with [perceive](https://www.enconvert.com/docs/endpoints/perceive.md).

## What the schema guarantee covers

On a completed row, `data` always has exactly your schema's top-level keys. A missing scalar is `null`, a missing array is `[]`, keys you didn't ask for are dropped, and a stray scalar in an array property gets wrapped in a list.

The flat form makes every field a nullable string, and a CSS `text` field returns the matched text, so a `price` declared as a number can arrive as a string. A model can also misread a page, and normalising keys does nothing about a wrong value. Validate types yourself. `fields_from_css` and `fields_from_llm` count how many fields each pass filled, not which ones.

Warnings explain the nulls. A row tells you when its page was cut to fit the budget, and when the model ran out of room mid-answer or found nothing. You're billed for the call either way.

```
import os
import requests

resp = requests.post(
    "https://api.enconvert.com/v2/distill",
    headers={"X-API-Key": os.environ["ENCONVERT_API_KEY"]},
    json={
        "urls": ["https://example.com/product/widget"],
        "schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "price": {"type": "number"},
                "in_stock": {"type": "boolean"},
            },
        },
    },
    timeout=310,
)
if resp.status_code == 422:
    print(resp.json().get("errors"))  # names the field you got wrong
resp.raise_for_status()

for row in resp.json()["results"]:
    if row["status"] != "completed":
        continue  # render failed, no op billed
    if row["render_quality"] < 0.40 or row["extraction_tier"] == "none":
        continue  # nothing usable, but the op was billed
    data = row["data"]  # name, price, in_stock
    price = data["price"]
    if price is not None and not isinstance(price, (int, float)):
        print("price is not a number:", repr(price))
    print(row["url"], data, row["fields_from_llm"], row["warnings"])
```

##  If you only need one page

If you're reading one page anyway, ask perceive for `structured` from the same render. Its `extract` list takes `metadata`, `structured_data` (the page's JSON-LD), `headings`, `tables`, `main_content` or `all`. None of those use a model or cost anything beyond the read's one op, and a site's JSON-LD may already hold the fields you were about to pay a model for. Add a `schema` and, on Indie and up, the model fills `structured.extracted`, under the same floor rule.

Over MCP, the `extract_structured` tool in `@enconvert/mcp` calls distill, with a required `schema` and, unlike the API, no prompt-only mode (more in the [agent web data post](https://www.enconvert.com/blog/agent-web-data-api.md)). For links, images, screenshots and PDFs with no model involved, see [web content extraction](https://www.enconvert.com/blog/web-content-extraction-api.md).

## Firecrawl, Diffbot and ScrapingBee

I checked all three on their own sites in October 2026.

[Firecrawl](https://docs.firecrawl.dev/features/scrape) returns JSON as one of its scrape formats, and its [pricing page](https://www.firecrawl.dev/pricing) lists JSON mode at 4 credits per page on top of the 1-credit scrape. If you already crawl with Firecrawl, extracting there is one vendor fewer.

[Diffbot Extract](https://www.diffbot.com/docs/extract/) doesn't ask for a schema at all. It classifies the page and returns structured JSON for types such as articles and products with "no rules or per-site configuration required", at one credit per request (two with a proxy).

Distill has no CAPTCHA solving and no proxies behind it. For sites that fight back hard, ([ScrapingBee](https://www.scrapingbee.com/documentation/) has a `stealth_proxy` mode and renders JavaScript by default (5 credits a request). Its docs list `extract_rules`, `ai_query` and `ai_extract_rules` for pulling fields out.

##  Timeouts and the keys we still drop

Distill is one synchronous request, and URLs render one after another, roughly 10 to 30 seconds each by our docs' estimate. The gateway cancels any request that hasn't started responding within 300 seconds, so on that estimate 10 to 30 URLs fit, and the proxy in front of the API can drop it sooner (typically 60 to 120 seconds, the same docs warn). Each URL bills as it finishes, so a timeout doesn't undo the ops for URLs that already completed, and their data never reaches you. I'd send a handful of URLs per call, not 50.

A few places still don't answer an unknown name with a 422. Unknown keys inside perceive's `pdf_options` are dropped silently. The multipart `/v2/ingest/files` route ignores form fields it doesn't know, so a misspelled `max_words` falls back to the default. And `prices`, `contacts` and `technologies` are reserved `extract` names on perceive that aren't live yet. Asking for one gets a warning, not a 422.

With `respect_robots` set to `true` (the default is `false`), a URL the site's robots.txt disallows comes back as a failed row. Which sites you scrape and what you do with the data is on you.

If you try one thing, declare every field in `schema`, add a `css_schema` for the ones a selector can reach, and watch `fields_from_llm` on each row. When it's above zero, I'd spot-check the fields your `css_schema` doesn't cover. If you find another key we accept without a word, tell me which one at ([my_email](mailto:het@enconvert.com)).

The full parameter tables are in the [distill docs](https://www.enconvert.com/docs/endpoints/distill.md).
