Agent web data API: which verbs are GA and which are private beta

I asked Crawl4AI's breadth-first crawler for at most 50 pages in June 2026, while I was building discover (the part of our agent web data API that maps a site's URLs), and it made 152 fetches. In batch mode its max_pages is a loose per-level cap, and a site with many links runs past it. The fix went in on 23 June. Discover drives the crawl in stream mode and breaks out of the loop itself.

The live test that shipped with that fix had the opposite bug. With a 50-page cap, pages_crawled never went above 49. Crawl4AI's stream loop bumps its counter and breaks before it yields the page that hit the limit, so page 50 was fetched and thrown away. On 8 July we gave Crawl4AI one page of headroom and kept the exact bound in our own loop. A test pins that upstream behaviour and fails the day Crawl4AI fixes it.

Both fixed. An agent sets max_urls on purpose and plans from the count it gets back. It can't see 152 fetches behind a cap of 50, or a 49 that should have been 50.

EnConvert is an agent web data API: with one key, an agent can search the web, map a site, read a page, pull fields out of pages, ingest a site into chunks, watch a page for changes and turn an attached file into markdown, through an MCP server, a REST API, ten SDKs or a CLI.

Which parts of the agent web data API are GA

Our docs mark Perceive, Ingest and Distill as generally available, inside an API that's still labelled beta. Discover, Lookup and Watch are private beta: callable on your normal key today, with shapes that can change without notice, so I'd keep them out of anything load-bearing for now.

Every op comes off one monthly counter, file conversions included. Watchers use slots instead, and Distill's LLM pass draws on AI credits.

Search and site maps

For search there's /v2/lookup. It runs on Serper (per the lookup docs) and covers six categories: web, news, images, scholar, patents and maps. Set perceive_top to N, up to 10, and it also reads the top N results through perceive, for at most 1 + N ops (blocked reads aren't billed). Check is_blocked on each perceived hit before reading its outputs.

The MCP tool is thinner. web_search has no enrich option, and its text is the answer box (if any) plus titles, URLs and snippets cut to 200 characters. Perceived pages stay in structuredContent as signed URLs, with only their operation ids in the text. Our own tool description promises page content in one round trip. That oversells it. To read a hit web_search already perceived, have the agent call get_perceive_operation with its id. It inlines the markdown and costs no op, where perceive_url would bill another.

If your agent mostly searches, start with a search-first API. As of October 2026, Tavily puts search, extract, crawl and map behind a hosted a MCP server, and Exa's /contents returns a per-URL statuses array with an error tag when a fetch fails. Lookup is Serper plus our reader, so I'd only choose it if your agent reads most of what it finds and needs to know which reads were blocked.

Discover lists a site's URLs without rendering each page. The default hybrid mode combines sitemaps (declared in robots.txt or probed), RSS and Atom feeds and a breadth-first HTTP crawl, and returns up to max_urls de-duplicated URLs (default 100, max 1,000). My live test on 23 June hit the caps I gave it: 80 URLs from quotes.toscrape.com and 200 from djangoproject.com's sitemap, both flagged truncated. excalidraw.com, crawled over HTTP, came back with 1 URL, the seed itself, because a client-rendered React shell has no links in its HTML.

render_js handles that case. With the default, auto, discover renders the page in the browser when the HTTP crawl finds one URL or fewer, and collects the links the JavaScript added. The MCP tool discover_urls doesn't expose render_js, so MCP calls always get auto.

respect_robots (default false) only filters the returned list. In hybrid and crawl mode, a disallowed page can still be fetched and then dropped, per the discover docs.

Perceive

Perceive has its own posts on what a read returns, the score , the markdown " and the other outputs.

Treat is_blocked or render_quality < 0.40 as no page. A read that comes back is_blocked, or with an http_error or login_wall deduction, returns HTTP 200 with billed: false and outputs: {}. Cache hits bill.

Through MCP, perceive_url puts those signals in the text the model reads Render quality, Source status, the deductions, Blocked: yes (not billed)) and inlines the markdown up to 256 KB. Above that there's no inline copy, only the signed URL every summary lists.

For many URLs there's perceive_batch, with one options block for all of them. The API caps a batch by plan (50, 100 or 400 URLs on Indie, Studio and Production), and the free plan has no batch. Agents didn't find it on their own. MCP 0.3.1 shipped a SKILL.md routing guide on 3 August because they were looping perceive_url one URL at a time, and calling convert_document on web pages.

Distill and ingest

Give /v2/distill a schema and up to 50 URLs. If you also give it CSS selectors css_schema), it runs a free CSS pass first and sends only the fields still missing to an LLM. Without selectors, every field goes to the LLM. Either way the response has the schema's shape. the LLM web scraping API post covers the spend caps.

Ingest turns a whole site or a list of URLs into chunked JSONL. It's always async (a 202 with an ing_ job id, then poll or take a signed webhook) and skips pages that are blocked or score under 0.40 without billing them. the RAG web ingestion API post covers the chunker.

Watch (private beta, paid plans)

Watch re-reads a page on a schedule and reports changes. Checks cost no ops. Each paid plan gets watcher slots (20 on Indie, 100 on Studio, 500 on Production), and the free Founding plan can't create watchers. Before an agent creates one:

- frequency_minutes runs from 60 to 43,200 (hourly to every 30 days). Anything else is a 422.

- Checks always render in Chromium, never the fast fetch, because they're diffed against a Chromium baseline (the browser post has the false alarms behind that).

- A check that scores under 0.40 or comes back blocked is recorded, but never becomes a baseline or fires a change.

- A change emails the owner by default and, if webhook_url is set, sends a POST signed with the project's HMAC-SHA256 secret X-Enconvert-Signature), the same secret ingest webhooks use.

- Three render failures in a row pause the watcher.

No MCP tool reads or rotates that secret, because a secret returned into LLM context ends up in transcripts and logs. Read it in the dashboard or with enconvert ingest webhook-secret show. The CLI has no typed watch commands yet, though its enconvert api passthrough can call /v2/watch.

Files someone attached

anything-to-markdown accepts 22 extensions (PDF, Word, PowerPoint, Excel, CSV, EPUB, HTML, OpenDocument, RTF, plain text) and returns heading-aware markdown with pipe tables. /v2/ingest/files uses the same converter. There's no OCR, so an image gets a 400, and so does a scanned PDF without a text layer.

How the tools reach the agent

Our MCP server, @enconvert/mcp, runs as a local stdio process and registers 24 tools, 18 for the web verbs and 6 for files. ENCONVERT_TOOLSETS web, files, or the default all) chooses which family it registers. The setup wizard configures Claude Code, Claude Desktop, Cursor, Windsurf, VS Code, Zed, Gemini CLI, Codex and OpenCode.

bash
npx @enconvert/mcp setup
# or, by hand:
claude mcp add enconvert -s user -e ENCONVERT_API_KEY=sk_your_key -- npx -y @enconvert/mcp@latest

Restart the assistant afterwards. The wizard keeps the key in ~/.enconvert/config.json (mode 600), never in a client's config, so npx @enconvert/mcp rotate-key switches every wizard-configured client on its next launch. A manual install with -e ENCONVERT_API_KEY keeps its own key, because the environment variable wins. Use a private sk_ key. A pk_ key authenticates and then 403s on all 18 web tools.

Our 0.2.0 MCP Registry entry, published 10 July 2026, read "MCP server for the Enconvert API. Convert URLs and files to PDF, screenshots, and markdown." (The EnConvert name dates from those converter days.) Since 3 August it reads "Unified web ingestion for AI agents: perceive, discover, search, extract, ingest, watch, convert." Version 0.3.0 reached npm on 25 July, but our first new description ran to 168 characters. The registry caps descriptions at 100, rejected 0.3.0 with a 422 and kept the old text up for nine days. npm versions can't be changed, so the publish workflow now checks the registry's limits before it touches npm.

On 31 August, before its first publish, we checked our ClawHub skill for OpenClaw against the live API and found three mistakes in what it told the agent. It said to call POST /v2/search, which 404s (the endpoint is /v2/lookup), it described perceive's markdown as inline text when the REST response gives a 15-minute signed URL, and it got distill's response shape wrong. An agent would have followed all three, word for word.

perceive_url has no headers, cookies or auth, so a read behind a session cookie needs REST or an SDK. The MCP server is a client of the same REST API https://api.enconvert.com, key in X-API-Key), and so are the SDKs for Node/TypeScript, Python, Go, Java, Kotlin, PHP, Ruby, Rust, C# and Swift, and the CLI, which covers conversions, perceive, discover, lookup, distill and ingest.

No hosted MCP server, no browser actions

Firecrawl, Jina, Tavily and Exa all run hosted MCP servers (checked October 2026), so having one is no reason to pick us. Firecrawl's server lists 27 tools in its full profile. Ours has no hosted endpoint, so if your agent runs somewhere that can't spawn npx, use one of theirs.

There are no browser actions. action_chain returns a 422, so a login form or a checkout needs a remote-browser product (the browser post says which).

If you install it today, set ENCONVERT_TOOLSETS=web unless your agent handles attachments (the model then carries 18 tool definitions instead of 24), and put one line in the agent's instructions: Blocked: yes or a render quality under 0.40 means it didn't get the page.

The MCP setup guide has the manual JSON config, and the free plan is 500 ops a month with no card.