Web Scraping API for Markdown, Screenshots, and Structured Data#

POST /v2/perceive is EnConvert's web scraping API: it renders a URL once in a real headless browser (JavaScript executed, lazy content loaded) and hands back every output you ask for from that single render: clean Markdown (main content only by default, site chrome stripped), cleaned or raw HTML, a screenshot, a PDF, the link and image inventory, and structured data (page metadata, JSON-LD, headings, tables). File outputs come back as short-lived pre-signed download URLs, the structured block inline, and batches over 10 URLs run asynchronously behind a polled job_id. One request replaces a whole stack of separate calls: url-to-markdown, url-to-screenshot, url-to-pdf, plus your own scraping.

Here is the smallest useful call. Send a URL, get clean Markdown and the page's structured metadata back:

curl -X POST https://api.enconvert.com/v2/perceive \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "outputs": ["markdown", "structured"]
  }'

The response carries a pre-signed download URL for the Markdown file and the structured block inline:

{
    "operation_id": "per_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7",
    "status": "completed",
    "url": "https://example.com/pricing",
    "url_final": "https://example.com/pricing",
    "content_hash": "9f2b8c1a...d4e5",
    "render_quality": 0.93,
    "cache_hit": false,
    "outputs": {
        "markdown": {
            "url": "https://spaces.example.com/...signed...",
            "object_key": "env/files/4127/v2-perceive/per_3f9a..._markdown.md",
            "size_bytes": 8421,
            "content_type": "text/markdown; charset=utf-8",
            "expires_in": 900
        }
    },
    "structured": {
        "metadata": {
            "title": "Pricing",
            "description": "Simple, usage-based pricing."
        },
        "structured_data": [
            {"@type": "Product", "name": "Studio", "offers": {"price": "99"}}
        ]
    },
    "extraction_tier": "heuristic",
    "tokens": {"input": 0, "output": 0},
    "cost_cents": 0.0,
    "duration_ms": 6230,
    "warnings": []
}

Endpoints#

Content-Type: application/json on every POST.

Authentication#

Authenticate with a private key in the X-API-Key header for server-to-server calls. This is the path the examples below use.

X-API-Key: sk_your_private_key

Public keys with a JWT bearer token also work, using the same flow as every other endpoint: generate a token with your pk_ key, then send it as Authorization: Bearer <token>. The full flow, including domain locking and token refresh, is in the authentication guide.

Each API key carries an allowed-endpoints allowlist. If /v2/perceive is not on the key's list, the request is rejected with 403.

How perceive works#

One request triggers one browser render through a shared headless Chrome singleton, then materialises every output from that render. You never pay for the same page twice in a single call.

1. Render. The page is fetched through an automatic multi-engine fallback: a fast real-browser TLS fingerprint first, escalating to headless Chrome when the page is blocked or needs JavaScript, and once more to a stealth-hardened render when a page still looks blocked by anti-bot protection, so more real-world pages come back with usable content. In the browser, cookie banners are dismissed, the page is scrolled to trigger lazy-loaded content, sticky headers are handled, and images are given time to load. This is the same capture pipeline that powers the url-to-pdf endpoint.
2. Materialise. From the rendered DOM, perceive builds whatever you listed in outputs: Markdown, cleaned/raw HTML, links, images, a screenshot, a PDF. The DOM is normalised first so the Markdown reflects what a reader sees, code fences keep their language, card links keep their structure, and interface furniture is removed under only_main_content. See Markdown quality.
3. Extract. If you requested structured output, perceive runs a heuristic pass for page metadata, JSON-LD, headings, and tables. If you also send a schema and your plan carries the LLM tier, an LLM fills the schema when the heuristic pass comes up short.
4. Score. A render-quality score (0.0-1.0) tells a real render from a failed one. Scores below 0.40 mean a failed render: an anti-bot page, a login wall, an HTTP error page, a soft 404, or an empty shell. Consult deductions for the reason and status_code for the upstream status.

Binary and text outputs (Markdown, HTML, screenshots, PDFs, the link and image JSON) are uploaded to storage and returned as pre-signed URLs that expire after 15 minutes. The structured block is returned inline in the JSON. Re-fetch any operation with GET /v2/perceive/{operation_id} to get a fresh set of signed URLs.

Request parameters#

Validation is strict: a request key the schema does not know is rejected with 422 naming the offending field. Unknown keys are never silently ignored. Every 422 body also carries a top-level errors array of human-readable messages alongside the machine-readable detail list.

Core#

Outputs#

outputs accepts any combination of these names:

Markdown quality#

Before the page is converted, the rendered DOM is normalised so that the Markdown reflects what a reader sees rather than how the page was built. This runs on every render, so the result does not depend on which extraction strategy wins for a given page.

Always applied, in both only_main_content modes:

• Code fences keep their language. The language is read from whichever convention the site uses (class="language-python", data-lang, a bare language attribute, or a highlighter wrapper) and normalised, so ```python comes through instead of a bare fence.
• Card links stay readable. A link wrapping a heading and a description becomes a linked title followed by its description, instead of one run-together link such as [DatabaseSupabase provides a full Postgres database...]. The destination URL is preserved.
• Headings stay on one line. A heading whose text sits inside a nested element no longer emits a bare ## with the text stranded below it.
• Adjacent elements no longer concatenate. Layouts that space their items with CSS rather than whitespace produced YesNo and EvaluationDeploymentProduction; those now read as separate words.
• Invisible characters are removed: zero-width spaces used as anchor labels, soft hyphens, and Private Use Area glyphs from icon fonts, which arrive as unprintable tokens.
• Empty elements are dropped: icon-only <i> elements that rendered as stray __, and links whose label is empty.

Additionally, with only_main_content: true:

• Interface controls are removed: buttons, tab strips, keyboard shortcut hints, "Copy page" / "On this page" actions, and "Was this page helpful? Yes/No" rating widgets. A control that carries real content (an FAQ question, a clickable card body) is kept.
• Screen-reader-only text is removed: skip links, and the "Section titled ..." labels many documentation themes attach to every heading.
• Site-declared non-content is honoured: blocks marked data-nosnippet, data-pagefind-ignore, or data-noindex, unless they contain headings or code.
• Duplicate blocks are collapsed: responsive designs that ship a desktop and a mobile copy of the same bar, and carousels that pre-render every frame, appear once.
• Breadcrumbs and eyebrow labels above the page title are dropped.

Deferred content is deliberately kept: an inactive tab panel inside the content region holds a real code sample (the Python example in one tab, the JavaScript one in another), so both reach the Markdown rather than only whichever tab happened to be selected at render time.

Rendering and waiting#

Authenticated and custom requests#

Structured extraction#

When structured is in outputs, the extract list controls which fields perceive pulls. Ask for nothing and it defaults to metadata and structured_data.

To be straight about it: prices, contacts, and technologies are reserved names. Request one today and it does not error. The name lands in the warnings array and is dropped from structured.

Schema-driven extraction#

Send a schema to pull specific fields into structured.extracted:

{
    "url": "https://example.com/product/widget",
    "outputs": ["markdown", "structured"],
    "schema": {
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "price": {"type": "number"},
            "in_stock": {"type": "boolean"}
        }
    }
}

The LLM extraction tier fills the schema, and it fires only when all of these hold: you sent a schema, your plan carries the LLM tier (Indie and up), the page was not scored as blocked, and the heuristic pass left schema fields empty. When it runs, extraction_tier is "llm", and tokens and cost_cents report what that extraction cost; otherwise extraction_tier is "heuristic" and both are zero.

Note. Schema extraction is hard-capped to protect your bill: a single extraction is capped per request, and project spend draws on your monthly AI-credit balance ($5 / $15 / $40 per month on Indie / Studio / Production; unused credits roll over). LLM extraction spends credits, not ops. If a cap is hit or the balance is exhausted, perceive returns the heuristic result with a note in warnings rather than overspending. On a plan without the LLM tier, you get heuristic structured data only.

Response#

Both POST /v2/perceive and GET /v2/perceive/{operation_id} return the same object.

Retrieve an operation#

Signed URLs expire after 15 minutes. To download an output later, re-fetch the operation and perceive re-signs every URL from the stored object keys. No re-render happens, so this does not consume ops.

curl https://api.enconvert.com/v2/perceive/per_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_your_private_key"

An unknown operation ID, or one that belongs to a different project, returns 404. Existence is never leaked across projects.

Direct download#

By default every file output comes back as a pre-signed URL you fetch in a second request. Set direct_download: true on the POST to skip the envelope: the HTTP response body is the artifact bytes, with no JSON, no signed URL, and no second fetch. The request must produce exactly one artifact output (outputs: ["markdown"], outputs: ["pdf"], …), otherwise it is rejected with 400. The metadata that would have been in the JSON rides in response headers instead: Content-Disposition, X-Operation-Id, X-Object-Key, X-Cache-Hit, X-Render-Quality, X-Source-Status-Code, X-Content-Hash, and X-Warnings-Count.

curl -X POST https://api.enconvert.com/v2/perceive \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/post",
    "outputs": ["markdown"],
    "direct_download": true
  }' \
  -o post.md

The GET endpoints stream stored artifacts the same way:

• GET /v2/perceive/{operation_id}?direct_download=true&output=markdown streams one artifact from a past operation. output is required when the operation produced more than one artifact. An artifact past your plan's retention window answers 410.
• GET /v2/perceive/batch/{job_id}?direct_download=true streams the batch ZIP for output_mode: "zip" batches whose archive is ready, and answers 400 otherwise.

direct_download is for single URLs only: POST /v2/perceive/batch rejects it with 422. Set output_mode to "zip" and download the archive instead. See Batch perception.

Batch perception#

POST /v2/perceive/batch perceives a list of URLs that share one options block. Each URL is rendered through the same pipeline as a single call and produces its own operation row.

curl -X POST https://api.enconvert.com/v2/perceive/batch \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/c"
    ],
    "options": {"outputs": ["markdown"]},
    "output_mode": "manifest"
  }'

Batches of 10 URLs or fewer run inline and answer 200 with every result populated. Larger batches answer 202 with a job_id; the URLs are drained one at a time and you poll for results:

curl https://api.enconvert.com/v2/perceive/batch/{job_id} \
  -H "X-API-Key: sk_your_private_key"

The batch response reports aggregate progress and carries one full perceive result per URL once rendered:

{
    "job_id": "bat_8c1a...",
    "status": "partial",
    "output_mode": "manifest",
    "total": 3,
    "completed": 2,
    "failed": 1,
    "pending": 0,
    "items": [
        {"operation_id": "per_...", "status": "completed", "url": "https://example.com/a"},
        {"operation_id": "per_...", "status": "completed", "url": "https://example.com/b"},
        {"operation_id": "per_...", "status": "failed", "url": "https://example.com/c"}
    ]
}

status is queued, processing, completed, failed, partial (some URLs succeeded, some failed), or canceled. Set output_mode to zip to bundle every artifact into a single ZIP, returned under the zip field once the batch finishes.

Durable and resumable#

Batches are restart-safe. If the service restarts while a batch is in flight, the batch resumes automatically and re-renders only the URLs that had not finished, so already-completed URLs keep their artifacts. You never need to resubmit a batch because of a restart.

Cancelling a batch#

DELETE /v2/perceive/batch/{job_id} cancels a running batch. The worker stops between URLs, so URLs already rendered keep their results and the rest are left unstarted. The call is idempotent, so cancelling a batch that has already finished simply returns its current state, and the batch's status becomes canceled.

curl -X DELETE https://api.enconvert.com/v2/perceive/batch/{job_id} \
  -H "X-API-Key: sk_your_private_key"

Caching#

cache_mode controls how perceive treats its 1-hour result cache, keyed by your project, the URL, and the render-affecting request options.

Worth flagging: a cache hit still bills one op against your monthly ops allowance. The allowance meters operations rather than browser renders, so the cache saves you render time, not ops.

Code examples#

curl: Markdown only#

curl -X POST https://api.enconvert.com/v2/perceive \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/post",
    "outputs": ["markdown"]
  }'

curl: Markdown plus structured data#

curl -X POST https://api.enconvert.com/v2/perceive \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "outputs": ["markdown", "structured"],
    "extract": ["metadata", "structured_data", "tables"]
  }'

curl: Full outputs plus PDF#

curl -X POST https://api.enconvert.com/v2/perceive \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/report",
    "outputs": ["markdown", "html_cleaned", "screenshot_full_page", "pdf"],
    "pdf_options": {"format": "A4", "print_background": true}
  }'

Python#

import requests

response = requests.post(
    "https://api.enconvert.com/v2/perceive",
    headers={"X-API-Key": "sk_your_private_key"},
    json={
        "url": "https://example.com/pricing",
        "outputs": ["markdown", "structured"],
        "extract": ["metadata", "tables"],
    },
)
response.raise_for_status()
data = response.json()

# Download the Markdown artifact from its signed URL
markdown_url = data["outputs"]["markdown"]["url"]
markdown_text = requests.get(markdown_url).text

print(data["structured"])
print(markdown_text)

Node.js#

const res = await fetch("https://api.enconvert.com/v2/perceive", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        "X-API-Key": "sk_your_private_key"
    },
    body: JSON.stringify({
        url: "https://example.com/pricing",
        outputs: ["markdown", "structured"],
        extract: ["metadata", "tables"]
    })
});

const data = await res.json();

// Download the Markdown artifact from its signed URL
const markdownText = await fetch(data.outputs.markdown.url).then(r => r.text());

console.log(data.structured);
console.log(markdownText);

If you call EnConvert from Claude, Cursor, or another MCP client, the same capability is exposed as the perceive_url tool. See the MCP server page.

Error responses#

Unknown request keys are rejected with a 422 naming the field, on /v2/perceive, /v2/perceive/batch, /v2/discover, and /v2/lookup alike. They are never silently ignored. Every 422 body carries a top-level errors array of human-readable messages alongside the raw detail list.

The full status-code reference is in the error-codes guide.

Limits#

Frequently asked questions#

How do I convert a web page to Markdown with a REST API?#

Send POST /v2/perceive with {"url": "...", "outputs": ["markdown"]}. The page is rendered in headless Chrome and the response carries a pre-signed download URL for the Markdown file. By default only_main_content strips site chrome so you get the article, not the navigation; set "only_main_content": false for the full page, or add "direct_download": true to receive the Markdown bytes directly in the response body.

Can I get a screenshot and Markdown from the same render?#

Yes. outputs accepts any combination, so ["markdown", "screenshot"] (or screenshot_full_page for the entire scroll height) produces both from a single browser render. You never pay for the same page twice in one call.

Does /v2/perceive render JavaScript pages?#

Yes. Every request runs a real headless-Chrome render: cookie banners are dismissed, the page is scrolled to trigger lazy-loaded content, and you can control the page before capture with wait_for (a CSS selector or JS expression), js_code, and block_resources.

Why did my signed download URL stop working?#

Signed URLs expire after 15 minutes (expires_in: 900). Re-fetch the operation with GET /v2/perceive/{operation_id} to get freshly signed URLs. No re-render happens and no ops are consumed.

Does a cached result still count against my quota?#

Yes. A cache hit bills one op, because the monthly allowance meters operations rather than browser renders. Set cache_mode to bypass to skip the 1-hour cache, or refresh to render fresh and replace the cached entry.