Changelog

/v2/ingest previously accepted only URLs — an explicit list, a declared sitemap, or a crawl. It now also accepts uploaded documents, via a new multipart route. File jobs reuse the URL path's job model, chunker, assembled JSONL deliverable, and signed webhook; the mode field is widened to "urls" | "sitemap" | "crawl" | "files", and every existing /v2/ingest route operates on file jobs unchanged.

POST /v2/ingest/files

Takes N uploaded documents, extracts text from each, chunks each with the same semantic chunker applied to crawled pages, and assembles one JSONL containing every chunk from every upload. Returns 202 with the same ing_-prefixed job_id and the same response body as the JSON path. No browser process is spun up — file jobs are CPU-bound extraction, not renders, and are not subject to browser serialization.

Request: multipart/form-data only — no JSON, no base64. Documents go in files (repeated part, named files, not files[]). Remaining fields are form fields, not a body model: max_words (default 512), sentence_overlap (default 1), webhook_url (optional).

Accepted extensions (allowlist on extension — the declared part Content-Type is not consulted): .txt .text .md .markdown .mdown .mkd .html .htm .xhtml .pdf .docx .pptx .xlsx .csv .epub .doc .ppt .xls .odt .ods .odp .rtf. The last seven are not handled by a native extractor — they are converted through an unoserver/LibreOffice subprocess first. Images are rejected at the extension gate with 400; there is no OCR path.

No mode field — the route is hard-wired to mode="files". None of the URL-path fields exist here (max_pages, max_depth, same_domain_only, include_patterns, exclude_patterns, respect_robots, wait_for, wait_timeout_ms). Because the route takes discrete form fields rather than a strict body model, unknown fields are silently ignored instead of returning 422 — a misspelled max_words falls back to the default with no error.

Response (IngestJobResponse, unchanged): job_id, status, mode, pages_discovered, pages_processed, pages_failed, total_chunks, output_url (signed URL to the assembled JSONL, present once completed), error_message, webhook_url, webhook_delivered, created_at, completed_at, warnings. For file jobs pages_discovered is the uploaded file count, and the job moves queued → processing — it never enters discovering, since pages exist at submit. Existing routes apply as-is: GET /v2/ingest/{job_id}, DELETE /v2/ingest/{job_id}, POST /v2/ingest/{job_id}/retry-webhook, GET /v2/ingest (summaries render mode: "files"), GET /v2/ingest/webhook-secret, POST /v2/ingest/webhook-secret/rotate.

Output: chunk records are the same shape as URL jobs — id, content, metadata{source_url, title, headings_path, section, word_count, chunk_index}. For file chunks, metadata.source_url and metadata.title carry the original filename rather than a URL: same key, different semantics per mode. There is no per-chunk marker distinguishing a file chunk from a URL chunk — job mode is the only discriminator. id derives from a hash of metadata.source_url, so two uploads sharing a filename in one request produce colliding chunk ids in the assembled JSONL; both files still process. Filenames are emitted as supplied — unsanitized in metadata; sanitization applies to storage keys only.

Constraints: 200 files per request maximum (400 above it) — the JSON path's 1000-page ceiling does not apply here. Empty file → 400. Per-file byte ceiling is read from the caller's account configuration; over it → 413. No aggregate request-size cap exists — 200 files each at the per-file ceiling is accepted. max_words clamps to 32–4000 and sentence_overlap to 0–10: out-of-range values are silently clamped, not rejected with 422 as on the JSON path (max_words=999999 becomes 4000). max_words remains a soft cap — code blocks and tables stay atomic and may exceed it.

Submit is not instant. Each file is read, magic-byte checked, and staged to storage one at a time inside the request before the 202 returns, so submit wall-clock scales with file count × size and the 300 s request window applies to the upload. Content sniffing runs after the extension gate and rejects only when the bytes resolve to a recognizably different known binary type — indeterminate bytes pass. A single unsupported, empty, oversized, or mismatched file rejects the whole request: no job is created and no file in the batch is processed, but files already staged before the failing one are left orphaned in storage — no job row exists to clean them up. Retrying a rejected batch re-uploads everything.

No conversion timeout except 120 s on the unoserver-routed formats — .pdf, .docx, .pptx, .xlsx, .csv, .epub, text, and HTML extraction is uncapped, and there is no job-level deadline. Uploaded source files are deleted once the JSONL is assembled; that cleanup runs at the tail of assembly only, so a job canceled before it reaches assembly leaves its uploads in storage. webhook_url is scheme-checked at submit (http:// or https://, else 400) and SSRF-screened at delivery; a file job fetches no other URL, so nothing else is screened.

Activity rows for file jobs record zero input bytes and log the same /v2/ingest endpoint as the JSON path, so the two are indistinguishable in activity history.

Summary

Six new endpoint groups (20 routes total) ship under the /v2 prefix: perceive, discover, lookup, distill, ingest, and watch. These cover headless-browser page capture, site URL enumeration, live web search, schema-driven structured extraction, async URL-to-JSONL ingestion, and scheduled page-change monitoring. All routes authenticate via the existing X-API-Key / Authorization: Bearer mechanism used elsewhere in the API. Rate limiting applies to billable POST routes; GET status-polling routes are exempt. There is no shared response envelope — each endpoint defines its own response model; check status/error fields per-endpoint rather than assuming a common shape.

/v2/perceive

• POST /v2/perceive — single-URL headless-browser render producing any combination of requested outputs in one page load: markdown, markdown_fit, html_cleaned, html_raw, screenshot, screenshot_full_page, pdf, links, images, structured (via outputs[]).
• GET /v2/perceive/{operation_id} — poll/fetch a single operation.
• POST /v2/perceive/batch — same flow fanned out over urls[] (max 1000, de-duplicated) under one shared options block.
• GET /v2/perceive/batch/{job_id} — poll aggregate batch status and per-URL results.

Request fields: url (max 2048 chars, http(s):// only), outputs[] (default ["markdown","structured"]), extract[] (tables, metadata, main_content, headings, structured_data implemented; prices, contacts, technologies, all are accepted by the schema but degrade to a warning — not implemented), extraction_schema (aliased schema), wait_for (css:/js: expression), wait_timeout_ms (0–60000, default 30000), js_code (max 20000 chars), viewport, headers/cookies/auth, cache_mode (enabled|bypass|refresh, default enabled), pdf_options, block_resources[], respect_robots (default false), mobile (default false). proxy_url, geolocation, action_chain are accepted by the schema but return 422 — not implemented.

Response (PerceiveResponse): operation_id, status (queued|processing|completed|failed), url/url_final, content_hash, render_quality (0.0–1.0), cache_hit, outputs{} (each a pre-signed URL artifact, default 900s expiry), structured, extraction_tier (heuristic|css|llm), tokens{input,output}, duration_ms, error, warnings[]. Structured extraction always runs a heuristic pass (metadata/JSON-LD/headings/tables/main content); the LLM tier fires only when a schema is supplied, the render isn't flagged bot-blocked, and heuristic fields are unfilled.

Batch specifics: output_mode (manifest|zip, default manifest) — zip bundles all successful artifacts into one archive, exposed via a zip field on PerceiveBatchResponse. Batches process strictly sequentially through a single in-process worker — no concurrency. Batches of ≤10 URLs attempt an inline response, waiting up to 240s before degrading to 202 + job_id; batches of >10 URLs always return 202 immediately. PerceiveBatchResponse fields: job_id, status (queued|processing|completed|failed|partial), output_mode, total/completed/failed/pending, zip, items[] (PerceiveResponse[]), warnings[]. The batch queue is in-memory only — a server restart mid-batch marks remaining rows failed ("interrupted by server restart"); batches must be resubmitted.

Constraints: main_content extraction truncated to 50,000 chars; every URL is SSRF-screened before any row is created; respect_robots=true adds a robots.txt check (403 if disallowed); pdf output always wins hook-chain selection over screenshot when both are requested; cache keyed by a fingerprint of render-affecting fields (URL, outputs, extract list, schema, pdf_options, viewport, mobile, js_code, wait config, block_resources, headers, sorted cookies, auth), 1-hour window.

/v2/discover

• POST /v2/discover — enumerates a site's URLs without rendering (no browser process spun up). Three modes: sitemap (robots.txt-declared sitemaps, probed sitemap.xml/<sitemapindex> recursion, RSS/Atom feeds), crawl (HTTP-only breadth-first crawl harvesting <a href> links from raw HTML — cannot see JS-injected routes on client-rendered SPAs, by design), hybrid (default — union of both, deduplicated).

Request: url (max 2048), mode (default hybrid), max_urls (1–1000, default 100), max_depth (1–5, default 2, crawl mode only), include_patterns[]/exclude_patterns[] (regex, max 50 entries each, compiled at request-validation time — malformed pattern returns 422), same_domain_only (default true), respect_robots (default false; filters the output list post-hoc rather than gating fetches).

Response (DiscoverResponse): url, mode, total, urls[], pages_crawled, truncated, robots_respected, sources{} (raw per-source counts before dedup/filter), warnings[].

Constraints: crawl mode hard-caps actual HTTP fetches at min(max_urls, 50) regardless of max_urls; seed URL and every BFS-followed link are SSRF-screened independently; synchronous request/response — no operation row, no job queue, no persisted artifact. Handler catches all internal errors and returns a generic 500 to avoid leaking library/path details.

/v2/lookup

• POST /v2/lookup — live web search proxy (Serper/Google SERP) across six categories. No caching and no retrieval against previously ingested/watched content — each call is a fresh outbound query. Optionally auto-renders the top-N result URLs through the /v2/perceive flow (markdown-only) in the same round trip via perceive_top.

Request: query (1–512 chars), category (web|news|images|scholar|patents|maps, default web), country/locale/time_filter/location, num_results (1–100, default 10), page (1–10, default 1), autocorrect (default true), perceive_top (0–10, default 0).

Response (LookupResponse): lookup_id, echoed query/category/filters, total, results[] (title, url, snippet, position, source, date, image_url, thumbnail_url, extra{}, optional perceive), perceive_top, perceive_operation_ids[], answer_box, knowledge_graph, warnings[].

Constraints: 15s timeout per attempt, up to 3 attempts with 0.5s/1.0s backoff on 429/500/502/503/504; 401/403 are not retried; a shared circuit breaker gates the provider — an open breaker returns 503 immediately without attempting the call; auto-perceive runs sequentially, not in parallel, and degrades to a per-result warning rather than failing the whole search on a single render failure; result order is passed through from the provider unmodified (no re-ranking); raw provider error bodies are never surfaced to the client (mapped to generic 502/503).

/v2/distill

• POST /v2/distill — schema-driven structured extraction from one or more URLs via a two-pass engine. Pass 1: a free, no-LLM CSS selector extraction against a caller-supplied css_schema. Pass 2: escalates only fields left empty by pass 1 to an LLM extractor (claude-haiku-4-5) with a reduced schema covering just those fields. Results merge and normalize to exactly the caller's schema (missing fields become null/[]).

Request: urls[] (max 50 entries, mutually exclusive with discover_from) or discover_from ({url, mode, max_pages: 1–50}), schema (aliased extraction_schema, required, max 200 top-level properties, JSON-Schema or flat {field: description} form), css_schema (baseSelector, fields[] — name/type(text|attribute|html|regex|nested|list|nested_list)/selector/attribute/pattern/transform, nested recursion depth max 5), wait_for (max 1024 chars), wait_timeout_ms (0–60000, default 30000), headers/cookies, respect_robots (default false). Regex patterns in css_schema are compiled at request time and rejected on a nested-quantifier ReDoS heuristic.

Response (DistillResponse): operation_id, total/completed/failed, results[] (url/url_final, status (completed|failed), data, extraction_tier (css|llm|mixed|none), fields_from_css, fields_from_llm, render_quality, tokens{input,output}, error, warnings[]), warnings[].

Constraints: 50-URL hard ceiling per request; each URL renders sequentially through a shared browser instance (not concurrent); CSS pass is wall-clock bounded at 10s and falls through to the LLM pass with a warning on timeout; LLM pass is skipped (CSS-only fallback, no error) when the render is flagged bot-blocked or a per-request escalation limit is reached (at most one LLM call per URL); LLM request timeout is 60s; SSRF/robots protection is inherited per-URL from the underlying render step — a rejected render fails only that URL (status: "failed") without aborting the rest of the batch.

/v2/ingest

• POST /v2/ingest — creates an ingestion job.
• GET /v2/ingest — paginated job list.
• GET /v2/ingest/{job_id} — job status/progress.
• DELETE /v2/ingest/{job_id} — cancel (idempotent).
• POST /v2/ingest/{job_id}/retry-webhook — manual webhook redelivery.
• GET /v2/ingest/webhook-secret / POST /v2/ingest/webhook-secret/rotate — signing-secret management.

Discovers and/or renders a URL or set of URLs (mode: urls explicit list, or sitemap/crawl — seed URL expanded via the same crawler used by /v2/discover, up to max_pages), renders each page (credential-free — no auth/cookies/custom headers accepted on any URL-mode request), converts HTML to Markdown, and splits it with a heading-aware chunker. On completion, all pages' chunks are concatenated into a single JSONL deliverable (LangChain JSONLoader / LlamaIndex SimpleDirectoryReader / vector-DB bulk-import compatible), returned as a signed download URL. Always asynchronous — POST /v2/ingest returns 202.

Request (IngestRequest, extra="forbid" — unknown fields reject the request): mode (default urls), url (seed, required for sitemap/crawl, max 2048 chars), urls[] (required for urls mode, max 1000 entries, each max 2048 chars), max_pages (1–1000, default 50), max_depth (1–5, default 2), same_domain_only (default true), include_patterns[]/exclude_patterns[] (max 50 each, validated at request time), respect_robots (default false), wait_for (max 1024 chars), wait_timeout_ms (0–60000, default 30000), chunk: {max_words, sentence_overlap} (both bounded), webhook_url (max 2048 chars, http(s):// only at submit).

Response (IngestJobResponse): job_id, status (queued|discovering|processing|completed|failed|canceled), mode, pages_discovered/pages_processed/pages_failed/total_chunks, output_url (present only once status == "completed"), error_message, webhook_url, webhook_delivered, created_at/completed_at, warnings[]. GET /v2/ingest paginates via skip/limit (default 20, max 100) + has_more. DELETE cancellation is observed between page renders — the worker stops without assembling output; terminal jobs are unchanged by a repeat call.

Webhooks: fired automatically on completion if webhook_url is set; best-effort — delivery failure never fails the job. SSRF-screened at delivery time (not at submit time). Signed with a per-project HMAC-SHA256 secret over "<unix_ts>.<raw body>", sent as X-Enconvert-Signature: sha256=<hex hmac> and X-Enconvert-Timestamp: <unix seconds> (timestamp bound into the MAC for replay protection; default consumer-side freshness tolerance 300s). Payload: {job_id, status, output_url, pages_processed, total_chunks}. Retries at 1.0s, 4.0s, 16.0s backoff (4 attempts worst case), each retry re-signed with a fresh timestamp; any 2xx counts as success. POST /v2/ingest/{job_id}/retry-webhook returns 400 if no webhook is configured or the target now resolves to a private address, 409 if the job isn't completed.

/v2/watch

• POST /v2/watch (201) — create a watcher.
• GET /v2/watch — list watchers.
• GET /v2/watch/{watcher_id} — get one.
• GET /v2/watch/{watcher_id}/snapshots — capture history.
• PATCH /v2/watch/{watcher_id} — update.
• DELETE /v2/watch/{watcher_id} — soft delete (idempotent).

Schedules recurring headless-browser captures of a URL and diffs each capture against the prior one, persisting change records and optionally delivering a signed webhook and/or email when a change is detected.

Request (POST, extra="forbid"): url (max 2048, http(s):// only), frequency_minutes (60–43,200, default 60), diff_mode (auto|text|structured|tables|metadata, default auto), track_fields (optional field/selector subset), webhook_url (max 2048, http(s):// only), notify_email (default true). PATCH rejects an all-null body with 422 (no silent no-op).

Response (WatcherResponse): watcher_id (wat_<uuid4hex>), url, status (active|paused|deleted), frequency_minutes, diff_mode, track_fields, webhook_url, notify_email, consecutive_errors, checks_count, last_check_at, next_check_at, last_change_at, created_at/updated_at. GET /v2/watch list returns a leaner WatcherSummary (same minus diff_mode/track_fields/webhook_url/notify_email/updated_at), limit clamped to 1–100.

Diffing: four strategies run under diff_mode=auto — text diffing (similarity-ratio comparison on main-content text, flagged changed below 0.98 similarity, one capped unified diff), structured-list diffing (key-matched add/remove/modify for links by href, JSON-LD structured_data, prices, contacts), table diffing (matched by heading/caption/header signature/position), and metadata diffing (key-by-key). Each Change carries section, kind (added|removed|modified), key, field, before/after — string values >2000 chars truncated; before/after are documented as untrusted page content requiring HTML-escaping by any consumer. Capped at 500 changes per diff (overflow appends a summary record with the true count). A render flagged blocked or scoring below the quality floor is persisted as an audit-only snapshot — no diff, no baseline eligibility, no notification. GET /v2/watch/{watcher_id}/snapshots returns newest-first, limit clamped 1–100 at the handler (200 hard cap in the store).

Webhooks: same signing scheme as /v2/ingest (X-Enconvert-Signature/X-Enconvert-Timestamp, HMAC-SHA256 over "<ts>.<body>") — the signing secret is shared per-project across /v2/ingest and /v2/watch deliveries and is re-screened for SSRF immediately before each send (independent of the scheme-only check performed at create/PATCH time). Retries at 1.0s/4.0s/16.0s, each re-signed with a fresh timestamp; any 2xx is success; a dead endpoint is logged as a non-delivery, never raised. Payload: {event: "change_detected", watcher_id, url, checked_at, similarity, change_count, changes}.

Constraints: frequency_minutes floor of 60 is enforced redundantly at schema validation, flow computation, and poller-claim time; ceiling 43,200 minutes (30 days); 3 consecutive check failures auto-pauses a watcher (status="paused", next_check_at cleared, optional owner email); the poller claims due watchers in batches of ≤50; prior-snapshot reads are scoped to a project-namespaced storage path as a cross-project isolation guard independent of write-time checks.