--- title: "Web content extraction API: what each perceive output gives you" description: "Markdown, links, images, structured data, screenshots and PDFs from one URL read and one op: what each output contains and when to use which." canonical: "https://www.enconvert.com/blog/web-content-extraction-api" locale: "en" author: "Het Dave" published: 2026-10-08 --- # Web content extraction API: what each perceive output gives you - Published: 2026-10-08 - Author: Het Dave Markdown, links, images, structured data, screenshots and PDFs from one URL read and one op: what each output contains and when to use which. When we measured the 13 August markdown release, one notebook page dropped 94% once its embedding vectors were truncated. An output cell had printed them as plain text, and every number landed in our markdown. Nothing was broken. The converter kept what was on the page, and no model needs a raw embedding vector in its context, so a web content extraction API has to decide what to leave out. That truncation is the `truncate_data_arrays` option ([changelog](https://www.enconvert.com/changelog/entry/cleaner-perceive-markdown)). A run of 64 or more comma-separated numbers collapses to its first 16 values plus a count `... [truncated of values]`), inside code fences too, since notebook output cells become fenced blocks. Unset, it follows `only_main_content`, which defaults to true. On the default main-content path, if the tidy pass cuts the markdown by more than half, `warnings` says so. `"truncate_data_arrays": false` brings the numbers back. The page was LlamaIndex's Pinecone metadata-filter example, which has since moved to developers.llamaindex.ai and still prints those vectors. I read it twice on 8 October 2026, at about 12:02 UTC. With `truncate_data_arrays` unset, the markdown came back at 10,971 bytes with one warning: "long numeric data arrays were truncated in the markdown (set truncate_data_arrays=false to keep them in full)." With `"truncate_data_arrays": false` it came back at 167,735 bytes with no warnings, so the default file is 93.5% smaller. Both files have 283 lines, because truncation shortens lines and never drops them. Lines 196 to 205 of the default file, unedited: ``` INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" [NodeWithScore(node=TextNode(id_='7fed3d0b-e2d7-432a-9231-3ac02130c00f', embedding=[0.00310940156, -0.0246712118, -0.0222742166, -0.0364649445, -0.00715911388, 0.0117236068, -0.0400604382, -0.0275654588, -0.0157589763, 0.00740136392, 0.0278459582, 0.029962454, 0.0187679715, -0.00440511806, 0.00412780605, 0.00442743069, ... [truncated 1520 of 1536 values]], metadata={'director': 'Christopher Nolan', 'theme': 'Fiction', 'year': 2010}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='7937eb153ccc78a3329560f37d90466ba748874df6b0303b3b8dd3c732aa7688', text='Inception', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.317687511), NodeWithScore(node=TextNode(id_='6b7bdb44-9dc2-48f6-b9cb-05dcaea4889b', embedding=[0.0131883333, -0.0137667693, -0.0396421254, -0.00729471678, -0.0107653309, 0.0171731133, -0.0282276608, -0.0507480912, -0.0210422054, -0.01610622, 0.0254768785, 0.0143452045, 0.0170060098, 0.00128059229, -0.0103797065, -0.000858012936, ... [truncated 1520 of 1536 values]], metadata={'author': 'J.K. Rowling', 'theme': 'Fiction', 'year': 1997}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash='1b24f5e9fb6f18cc893e833af8d5f28ff805a6361fc0838a3015c287510d29a3', text="Harry Potter and the Sorcerer's Stone", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.467440248)] ``` The same ten lines with truncation off. I've cut lines 202 and 203 after their first 12 values, and the marker at each cut gives the full length of the line: ``` INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK" [NodeWithScore(node=TextNode(id_='7fed3d0b-e2d7-432a-9231-3ac02130c00f', embedding=[0.00310940156, -0.0246712118, -0.0222742166, -0.0364649445, -0.00715911388, 0.0117236068, -0.0400604382, -0.0275654588, -0.0157589763, 0.00740136392, 0.0278459582, 0.029962454, [cut here: this line is 23,188 characters long] NodeWithScore(node=TextNode(id_='6b7bdb44-9dc2-48f6-b9cb-05dcaea4889b', embedding=[0.0131883333, -0.0137667693, -0.0396421254, -0.00729471678, -0.0107653309, 0.0171731133, -0.0282276608, -0.0507480912, -0.0210422054, -0.01610622, 0.0254768785, 0.0143452045, [cut here: this line is 23,173 characters long] ``` EnConvert's `/v2/perceive` is a web content extraction API for agents. You send a URL, and from one capture of the page you can take markdown, cleaned or raw HTML, a link map, an image list, structured metadata, a screenshot and a PDF, plus a `render_quality` score that says how far to trust the capture. It's one op per URL however many outputs you ask for. Perceive is generally available, and the beta label sits on the API as a whole. The markdown side has [its own post](https://www.enconvert.com/blog/url-to-markdown-api.md), and Krystin covered raw versus clean text in [How to extract clean web content for RAG pipelines and LLM agents](https://www.enconvert.com/blog/how-to-extract-clean-web-content-for-rag-pipelines-and-llm-agents.md) . I'll stick to the other eight outputs. ## Nine outputs from one web content extraction API call ``` curl -X POST https://api.enconvert.com/v2/perceive \ -H "X-API-Key: $ENCONVERT_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "outputs": ["markdown", "links", "images", "structured", "screenshot_full_page"], "extract": ["metadata", "structured_data", "headings", "tables"] }' ``` File outputs come back as signed URLs under `outputs..url`. `structured` is inline in the JSON. To try it before writing code, use the keyless Perceive tab in the [playground](https://www.enconvert.com/playground.md?mode=perceive) . Here is that request against Wikipedia's Mohs scale article, which has JSON-LD and three real data tables, captured on 8 October 2026 at about 12:05 UTC. I added `"cache_mode": "bypass"` so the read was fresh. I've cut the signed URLs' query strings, the `options_echo` block, and all but the first three rows of each table. Nothing else is changed: ``` { "operation_id": "per_b7adc935f535448fa4c51b890ef00432", "status": "completed", "url": "https://en.wikipedia.org/wiki/Mohs_scale", "url_final": "https://en.wikipedia.org/wiki/Mohs_scale", "content_hash": "376e1f880669317c7f5183636f23f34a8a9aebd9bbf8f5e289c546459b3ad0c9", "status_code": 200, "render_quality": 1.0, "deductions": {}, "is_blocked": false, "billed": true, "options_echo": { ... }, "cache_hit": false, "outputs": { "markdown": { "url": "https://nyc3.digitaloceanspaces.com/econverter/live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_markdown.md?[signed query string cut]", "object_key": "live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_markdown.md", "size_bytes": 28520, "content_type": "text/markdown; charset=utf-8", "expires_in": 900 }, "links": { "url": "https://nyc3.digitaloceanspaces.com/econverter/live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_links.json?[signed query string cut]", "object_key": "live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_links.json", "size_bytes": 49345, "content_type": "application/json", "expires_in": 900 }, "images": { "url": "https://nyc3.digitaloceanspaces.com/econverter/live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_images.json?[signed query string cut]", "object_key": "live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_images.json", "size_bytes": 71784, "content_type": "application/json", "expires_in": 900 }, "screenshot_full_page": { "url": "https://nyc3.digitaloceanspaces.com/econverter/live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_screenshot_full_page.png?[signed query string cut]", "object_key": "live/files/1/v2-perceive/per_b7adc935f535448fa4c51b890ef00432_screenshot_full_page.png", "size_bytes": 1460161, "content_type": "image/png", "expires_in": 900 } }, "structured": { "metadata": { "title": "Mohs scale - Wikipedia", "description": null, "keywords": null, "author": null, "og:image": "https://thumb.wikimedia.org/wikipedia/commons/thumb/f/fd/Mohssche-haerteskala_hg.jpg/960px-Mohssche-haerteskala_hg.jpg?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=thumbnail", "og:image:width": "908", "og:image:height": "1200", "og:title": "Mohs scale - Wikipedia", "og:type": "website" }, "structured_data": [ { "@context": "https://schema.org", "@type": "Article", "name": "Mohs scale", "url": "https://en.wikipedia.org/wiki/Mohs_scale", "sameAs": "http://www.wikidata.org/entity/Q41472", "mainEntity": "http://www.wikidata.org/entity/Q41472", "author": { "@type": "Organization", "name": "Contributors to Wikimedia projects" }, "publisher": { "@type": "Organization", "name": "Wikimedia Foundation, Inc.", "logo": { "@type": "ImageObject", "url": "https://www.wikimedia.org/static/images/wmf-hor-googpub.png" } }, "datePublished": "2001-11-06T03:42:32Z", "dateModified": "2026-08-28T13:41:17Z", "image": "https://upload.wikimedia.org/wikipedia/commons/f/fd/Mohssche-haerteskala_hg.jpg?utm_source=en.wikipedia.org&utm_campaign=index&utm_content=original", "headline": "qualitative ordinal scale characterizing scratch resistance of various minerals" } ], "headings": [ { "level": 2, "text": "Contents" }, { "level": 1, "text": "Mohs scale" }, { "level": 2, "text": "Reference minerals" }, { "level": 2, "text": "Examples" }, { "level": 2, "text": "Use" }, { "level": 2, "text": "Comparison with Vickers scale" }, { "level": 2, "text": "Footnotes" }, { "level": 2, "text": "See also" }, { "level": 2, "text": "References" }, { "level": 2, "text": "Further reading" } ], "tables": [ { "headers": [ "Mohshardness", "Referencemineral", "Chemical formula", "Absolutehardness[14]", "Example image" ], "rows": [ [ "1", "Talc", "Mg3Si4O10(OH)2", "1", "" ], [ "2", "Gypsum", "CaSO4·2H2O", "2", "" ], [ "3", "Calcite", "CaCO3", "14", "" ], ... 7 more rows cut ], "caption": "", "summary": "", "metadata": { "row_count": 10, "column_count": 5, "has_headers": true, "has_caption": false, "has_summary": false, "id": "mwfg", "class": "wikitable sortable jquery-tablesorter" } }, { "headers": [ "Hardness", "Substance" ], "rows": [ [ "0.2–0.4", "Potassium[15]" ], [ "0.5–0.6", "Lithium[15]" ], [ "1", "Talc" ], ... 29 more rows cut ], "caption": "", "summary": "", "metadata": { "row_count": 32, "column_count": 2, "has_headers": true, "has_caption": false, "has_summary": false, "id": "mwARw", "class": "wikitable" } }, { "headers": [ "Mineralname", "Hardness (Mohs)", "Hardness (Vickers)(kg/mm2)" ], "rows": [ [ "Tin", "1.5", "VHN10 = 7–9" ], [ "Bismuth", "2–2.5", "VHN100 = 16–18" ], [ "Gold", "2.5", "VHN10 = 30–34" ], ... 14 more rows cut ], "caption": "", "summary": "", "metadata": { "row_count": 17, "column_count": 3, "has_headers": true, "has_caption": false, "has_summary": false, "id": "mwAeo", "class": "wikitable" } } ] }, "extraction_tier": "heuristic", "tokens": { "input": 0, "output": 0 }, "cost_cents": 0.0, "duration_ms": 36254, "error": null, "warnings": [] } ``` | Output | What You Get | | --- | --- | | `markdown` | Main content by default. Image URLs become alt text, except when the Readability candidate wins | | `html_cleaned` | Crawl4AI's cleaned HTML, with nav, header and footer still in it | | `html_raw` | The HTML as captured (rendered DOM, or the fast fetch's HTML) | | `links` | Object with `internal` and `external` array | | `images` | List of `{src, alt, desc}` entries, plus `width` and `format` when known | | `structured` | Inline JSON: metadata, JSON-LD, headings, tables, main content | | `screenshot` | Viewport PNG, 1920 x 1080 by default | | `screenshot_full_page` | Full-height PNG | | `pdf` | One continuous page, or paginated with `pdf_options` | `links`, `images`, `html_cleaned`, `metadata` and `tables` come from the scraper in [Crawl4AI](https://github.com/unclecode/crawl4ai), the open-source crawler our render runs on, and the filtering in the next two sections is theirs. We add the `render_quality` score and choose which of their fields reach you. ### The links file Each entry in `internal` and `external` has `href`, `text`, `title` and `base_domain`. `base_domain` is the domain with its subdomains dropped, the page's for internal links and the target's for external ones. Read from docs.python.org, a link to blog.python.org counts as internal to python.org. A crawler can walk `internal` and just log `external`, filtering by host first if it should stay on one subdomain. The list covers the whole page whatever `only_main_content` says, nav and footer included. Entries are de-duplicated by resolved URL and the earliest anchor on the page wins, so a link in the nav before the article keeps its nav text. On the Mohs scale page, the first five `internal` entries are all page chrome (a skip link and four sidebar links), and the first two `external` ones are a donation link and a photography-contest banner. Unedited: ``` {"href": "https://en.wikipedia.org/wiki/Mohs_scale", "text": "Jump to content", "title": "", "base_domain": "wikipedia.org"} {"href": "https://en.wikipedia.org/wiki/Main_Page", "text": "Main page", "title": "Visit the main page [alt-z]", "base_domain": "wikipedia.org"} {"href": "https://en.wikipedia.org/wiki/Wikipedia:Contents", "text": "Contents", "title": "Guides to browsing Wikipedia", "base_domain": "wikipedia.org"} {"href": "https://en.wikipedia.org/wiki/Portal:Current_events", "text": "Current events", "title": "Articles related to current events", "base_domain": "wikipedia.org"} {"href": "https://en.wikipedia.org/wiki/Special:Random", "text": "Random article", "title": "Visit a randomly selected article [alt-x]", "base_domain": "wikipedia.org"} {"href": "https://donate.wikimedia.org/?uselang=en&wmf_campaign=en.wikipedia.org&wmf_medium=sidebar&wmf_source=donate", "text": "Donate", "title": "", "base_domain": "wikimedia.org"} {"href": "https://commons.wikimedia.org/wiki/Commons:Wiki_Loves_Monuments_2026_in_the_United_States", "text": "Photograph a historic site, help Wikipedia, and win a prize. Participate in the world's largest photography competition this month!\n Learn more", "title": "", "base_domain": "wikimedia.org"} {"href": "https://www.wikidata.org/wiki/Special:EntityPage/Q41472", "text": "Edit links", "title": "Edit interlanguage links", "base_domain": "wikidata.org"} {"href": "https://commons.wikimedia.org/wiki/Category:Mohs_scale", "text": "Wikimedia Commons", "title": "", "base_domain": "wikimedia.org"} {"href": "https://upload.wikimedia.org/wikipedia/commons/transcoded/1/16/En-us-Mohs.ogg/En-us-Mohs.ogg.mp3", "text": "", "title": "Play audio", "base_domain": "wikimedia.org"} ``` If I only wanted the article's links I'd read them from the markdown, which keeps them inline. And if JavaScript injects links after load, pass a `wait_for` selector that only appears once they exist (that also forces the Chrome render). ### Images and what Crawl4AI drops Crawl4AI filters `images` before you see it. It drops anything that looks like an icon, logo or button (judged by the parent's class, the `src` or the `alt`), images sitting directly in a `