Sitemap Crawler API#

POST /v2/discover is a sitemap crawler API that lists a website's URLs the cheap way: an HTTP-first crawl plus sitemap parsing, with an optional JavaScript render fallback for single-page apps. There is no screenshot, no PDF, and no stored artifact. It returns a flat, de-duplicated list of URLs and a count of where each one came from, synchronously in a single call. It will be the lightweight "map the site" primitive: point it at a domain, get back the pages worth processing, then feed that list into the perceive endpoint or a crawl-mode ingest job.

Here is the smallest useful call. Send a URL, get its map back:

curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

The response is a plain URL list with provenance counters. There are no signed URLs, no operation ID, and no polling:

{
    "url": "https://example.com",
    "mode": "hybrid",
    "total": 47,
    "urls": [
        "https://example.com/",
        "https://example.com/pricing",
        "https://example.com/docs",
        "https://example.com/blog/launch"
    ],
    "pages_crawled": 12,
    "truncated": false,
    "robots_respected": false,
    "sources": {"sitemap": 42, "crawl": 30},
    "warnings": []
}

Endpoints#

/v2/discover exposes a single path. It is stateless: there is no operation row to re-fetch and no job to poll, so there is no companion GET endpoint the way perceive has one.

Content-Type: application/json on the POST.

Authentication#

Authenticate with a private key in the X-API-Key header for server-to-server calls. This is the path the examples below use.

X-API-Key: sk_your_private_key

Public keys with a JWT bearer token also work, using the same flow as every other endpoint: generate a token with your pk_ key, then send it as Authorization: Bearer <token>. The full flow, including domain locking and token refresh, is in the authentication guide.

Each API key carries an allowed-endpoints allowlist. If /v2/discover is not on the key's list, the request is rejected with 403 and a message naming the path that was blocked.

How discover works#

One request runs entirely over HTTP. The singleton headless Chrome that powers perceive is never touched. The crawl path uses Crawl4AI's HTTP crawler strategy (an httpx GET plus an lxml link parse per page), and the sitemap path reuses the same pure-HTTP sitemap and feed helpers as the rest of the platform.

1. Screen the seed. The URL you send is checked for SSRF before any fetch: scheme, embedded credentials, blocked hostnames, and the resolved IP are all validated. A URL that resolves to a private, loopback, link-local, or cloud-metadata address is rejected with 400. The seed is always included as the first entry in its own map.
2. Gather from sitemaps. In sitemap or hybrid mode, discover reads robots.txt and fetches the Sitemap: URLs it declares, probes sitemap.xml (recursing into <sitemapindex> children), and pulls RSS/Atom feed pages. It runs these probes against both the host you sent and the site's registrable (apex) domain, so a sitemap published only on the apex is still found from a subdomain or deep-path seed. A missing or broken sitemap becomes a warning, not an error.
3. Crawl over HTTP. In crawl or hybrid mode, discover runs a breadth-first crawl from the seed up to max_depth, harvesting the <a href> links from each fetched page's raw HTML. Every followed link is SSRF-screened before it is fetched.
4. Normalise and filter. The combined raw list is canonicalised (fragments and tracking params stripped, default ports removed, query keys sorted), then passed through the same-domain check, your include and exclude regex patterns, the robots.txt filter, de-duplication, and finally the max_urls cap.

On the HTTP-only path, a client-rendered single-page app returns only its HTML shell in crawl mode, typically the seed plus zero or one URL. The optional render_js fallback (see JavaScript rendering) closes that gap: in its default "auto" setting, discover renders the page once in the browser and harvests the links it injects at runtime whenever the HTTP crawl comes back with just a shell. sitemap mode remains a fast, browser-free route for SEO-aware SPAs, which usually publish a sitemap.

Request parameters#

Core#

Filtering#

The patterns are real Python regular expressions, compiled at validation time. A malformed pattern is a 422 at the edge, not a 500 mid-crawl. Because the semantics are re.search, a bare substring like "/blog/" matches anywhere in the URL, so anchor with ^/$ if you need a positional match.

Worth flagging up front. respect_robots is enforced at output-filter time, not fetch time. Crawl4AI 0.8.9 has no native robots gate, so in crawl or hybrid mode a disallowed page may still be fetched over HTTP and then discarded before it reaches the response. Unlike perceive, where respect_robots=true rejects a disallowed URL with 403, discover never returns 403 for a robots rule. It silently drops the URL and reports nothing.

Modes#

crawl mode fetches up to your full max_urls (up to 1,000), so a large site is enumerated in one pass. Crawl mode is HTTP-only (no browser), so this stays fast. Operators can lower the ceiling with the DISCOVER_CRAWL_MAX_PAGES environment variable if a very large crawl ever needs bounding. pages_crawled in the response tells you exactly how many GETs ran.

Sitemap fetching handles gzipped sitemaps (sitemap.xml.gz and any sitemap served as application/gzip), and sends realistic browser headers so sites behind a WAF are far more likely to return their sitemap instead of a 403/503.

JavaScript rendering#

crawl and hybrid mode read the raw HTML a server returns, so a client-rendered single-page app that builds its links in the browser is invisible to them. render_js opts into a bounded browser fallback that closes that gap:

The fallback uses the same shared headless Chrome as perceive and is capped at a few pages so the call stays inside the request timeout. Links it finds are counted under the crawl_js key in sources. It applies to crawl and hybrid mode only, because sitemap mode is already browser-free and unaffected.

Response#

POST /v2/discover returns this object directly, with no async job and no second call.

The sources counts are raw: they are the number of URLs each path produced before normalisation, the same-domain check, your filters, and de-duplication ran. They will routinely add up to more than total, because hybrid mode finds the same pages from both the sitemap and the crawl. Use them to see which path is carrying the map, not as a post-filter tally.

Code examples#

curl: default hybrid map#

curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

curl: sitemap only, blog pages, capped at 500#

curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "sitemap",
    "max_urls": 500,
    "include_patterns": ["/blog/"],
    "exclude_patterns": ["/tag/", "/author/"],
    "respect_robots": true
  }'

Python#

import requests

response = requests.post(
    "https://api.enconvert.com/v2/discover",
    headers={"X-API-Key": "sk_your_private_key"},
    json={
        "url": "https://example.com",
        "mode": "hybrid",
        "max_urls": 200,
        "include_patterns": [r"/docs/"],
    },
)
response.raise_for_status()
data = response.json()

print(f"found {data['total']} URLs from {data['sources']}")
for page_url in data["urls"]:
    print(page_url)

Node.js#

const res = await fetch("https://api.enconvert.com/v2/discover", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        "X-API-Key": "sk_your_private_key"
    },
    body: JSON.stringify({
        url: "https://example.com",
        mode: "hybrid",
        max_urls: 200,
        include_patterns: ["/docs/"]
    })
});

const data = await res.json();

console.log(`found ${data.total} URLs from`, data.sources);
data.urls.forEach((pageUrl) => console.log(pageUrl));

A common pattern is to discover first, then render: take data.urls and hand them to the perceive endpoint, using single calls for a handful of pages or its batch path for the whole list.

Error responses#

Note that a failed sitemap fetch or a crawl fault does not produce an error status. Those degrade to entries in the warnings array, and the request still returns 200 with whatever was found. The full status-code reference is in the error-codes guide.

Limits#

Frequently asked questions#

How do I list all URLs on a website with an API?#

Send POST /v2/discover with the site's URL. The default hybrid mode combines sitemap parsing (robots.txt entries, sitemap.xml with index recursion, RSS/Atom feeds) with a breadth-first HTTP crawl, and returns up to max_urls (1–1,000, default 100) de-duplicated URLs in one synchronous response.

Does the discover endpoint use a headless browser or render JavaScript?#

By default it runs over HTTP and only renders when it has to. The render_js option controls this: in the default "auto" mode, discover stays HTTP-only and renders a page in the browser just when the HTTP crawl returns a bare JavaScript shell, harvesting the client-injected links; "always" forces the browser crawl and "never" keeps it strictly HTTP-only. sitemap mode remains a fast, browser-free option for SEO-aware SPAs.

How many pages does crawl mode actually fetch?#

Crawl mode fetches up to your full max_urls (up to 1,000). It stays HTTP-only, so it remains fast; operators can lower the ceiling with the DISCOVER_CRAWL_MAX_PAGES environment variable. Each fetched page also contributes many links. The pages_crawled response field reports exactly how many HTTP GETs ran.

Can I filter which URLs come back?#

Yes. include_patterns and exclude_patterns accept up to 50 Python regexes each (re.search semantics, not glob), and respect_robots: true drops URLs disallowed by the site's robots.txt from the returned list.