Discover

POST /v2/discover enumerates a site's URLs without rendering a single page. There is no browser, no screenshot, no PDF, no stored artifact — just an HTTP-only crawl plus sitemap parsing that returns a flat list of URLs and a count of where each one came from. It is the cheap "map the site" primitive: point it at a domain, get back the pages worth processing, then feed that list into the perceive endpoint or a crawl-mode ingest job.

Here is the smallest useful call. Send a URL, get its map back:

curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_live_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

The response is a plain URL list with provenance counters — no signed URLs, no operation ID, no polling:

{
    "url": "https://example.com",
    "mode": "hybrid",
    "total": 47,
    "urls": [
        "https://example.com/",
        "https://example.com/pricing",
        "https://example.com/docs",
        "https://example.com/blog/launch"
    ],
    "pages_crawled": 12,
    "truncated": false,
    "robots_respected": false,
    "sources": {"sitemap": 42, "crawl": 30},
    "warnings": []
}

Worth flagging up front. /v2/discover is a V2 endpoint, gated by the discover_enabled plan flag. Unlike most V2 endpoints it has no per-operation quota counter — it is cheap (no browser, no storage), so there is no monthly meter to exhaust. If your plan does not include V2 discover, the call returns 402. See the pricing page for which plans carry it.

Endpoints

/v2/discover exposes a single path. It is stateless — there is no operation row to re-fetch and no job to poll, so there is no companion GET endpoint the way perceive has one.

Content-Type: application/json on the POST.

Authentication

Authenticate with a private key in the X-API-Key header for server-to-server calls. This is the path the examples below use.

X-API-Key: sk_live_your_private_key

Public keys with a JWT bearer token also work, using the same flow as every other endpoint — generate a token with your pk_ key, then send it as Authorization: Bearer <token>. The full flow, including domain locking and token refresh, is in the authentication guide.

Each API key carries an allowed-endpoints allowlist. If /v2/discover is not on the key's list, the request is rejected with 403 and a message naming the path that was blocked.

How discover works

One request runs entirely over HTTP. The singleton headless Chrome that powers perceive is never touched — the crawl path uses Crawl4AI's HTTP crawler strategy (an httpx GET plus an lxml link parse per page), and the sitemap path reuses the same pure-HTTP sitemap and feed helpers as the rest of the platform.

1. Screen the seed. The URL you send is checked for SSRF before any fetch: scheme, embedded credentials, blocked hostnames, and the resolved IP are all validated. A URL that resolves to a private, loopback, link-local, or cloud-metadata address is rejected with 400. The seed is always included as the first entry in its own map.
2. Gather from sitemaps. In sitemap or hybrid mode, discover reads robots.txt, probes sitemap.xml (recursing into <sitemapindex> children), and pulls RSS/Atom feed pages. A missing or broken sitemap becomes a warning, not an error.
3. Crawl over HTTP. In crawl or hybrid mode, discover runs a breadth-first crawl from the seed up to max_depth, harvesting the <a href> links from each fetched page's raw HTML. Every followed link is SSRF-screened before it is fetched.
4. Normalise and filter. The combined raw list is canonicalised (fragments and tracking params stripped, default ports removed, query keys sorted), then passed through the same-domain check, your include and exclude regex patterns, the robots.txt filter, de-duplication, and finally the max_urls cap.

Because there is no render, a client-rendered single-page app returns only its HTML shell in crawl mode — typically the seed plus zero or one URL. That is correct behaviour for a no-browser endpoint, not a failure. For SPAs, use sitemap mode (most SEO-aware SPAs publish a sitemap) or fall back to per-URL perceive calls.

Request parameters

Core

Filtering

The patterns are real Python regular expressions, compiled at validation time. A malformed pattern is a 422 at the edge, not a 500 mid-crawl. Because the semantics are re.search, a bare substring like "/blog/" matches anywhere in the URL — anchor with ^/$ if you need a positional match.

Worth flagging up front. respect_robots is enforced at output-filter time, not fetch time. Crawl4AI 0.8.9 has no native robots gate, so in crawl or hybrid mode a disallowed page may still be fetched over HTTP and then discarded before it reaches the response. Unlike perceive, where respect_robots=true rejects a disallowed URL with 403, discover never returns 403 for a robots rule — it silently drops the URL and reports nothing.

Modes

To be straight about it: crawl mode bounds the number of pages it actually fetches to min(max_urls, 50). A max_urls of 1,000 does not issue a thousand HTTP GETs — discover stops fetching at 50 pages, but each page contributes many <a href> links, so the returned list can still reach max_urls. The 50-page fetch ceiling keeps the synchronous request well under the 300-second request timeout. pages_crawled in the response tells you exactly how many GETs ran.

Response

POST /v2/discover returns this object directly — there is no async job and no second call.

The sources counts are raw — they are the number of URLs each path produced before normalisation, the same-domain check, your filters, and de-duplication ran. They will routinely add up to more than total, because hybrid mode finds the same pages from both the sitemap and the crawl. Use them to see which path is carrying the map, not as a post-filter tally.

Code examples

curl — default hybrid map

curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_live_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

curl — sitemap only, blog pages, capped at 500

curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_live_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "sitemap",
    "max_urls": 500,
    "include_patterns": ["/blog/"],
    "exclude_patterns": ["/tag/", "/author/"],
    "respect_robots": true
  }'

Python

import requests

response = requests.post(
    "https://api.enconvert.com/v2/discover",
    headers={"X-API-Key": "sk_live_your_private_key"},
    json={
        "url": "https://example.com",
        "mode": "hybrid",
        "max_urls": 200,
        "include_patterns": [r"/docs/"],
    },
)
response.raise_for_status()
data = response.json()

print(f"found {data['total']} URLs from {data['sources']}")
for page_url in data["urls"]:
    print(page_url)

Node.js

const res = await fetch("https://api.enconvert.com/v2/discover", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        "X-API-Key": "sk_live_your_private_key"
    },
    body: JSON.stringify({
        url: "https://example.com",
        mode: "hybrid",
        max_urls: 200,
        include_patterns: ["/docs/"]
    })
});

const data = await res.json();

console.log(`found ${data.total} URLs from`, data.sources);
data.urls.forEach((pageUrl) => console.log(pageUrl));

A common pattern is to discover first, then render: take data.urls and hand them to the perceive endpoint — single calls for a handful of pages, or its batch path for the whole list.

Plan gating

/v2/discover is gated by the discover_enabled plan flag and behaves differently from the metered V2 endpoints. The metered ones — perceive among them — each burn a monthly counter (perceive_operations, lookup_queries, ingest_pages, watch_checks, distill_operations). Discover has no such counter: it is HTTP-only, it stores nothing, so there is no per-operation meter to deplete.

What that means in practice:

• If your plan carries discover_enabled, you can call it. There is no monthly discover allowance to run out.
• If your plan does not, the call returns 402 with an upgrade prompt.
• Admin plans bypass the gate entirely.

Which plans include discover_enabled is set per tier; current allowances live on the pricing page. V1 conversion plans do not include V2 endpoints — a discover call never consumes V1 conversion quota.

Error responses

Note that a failed sitemap fetch or a crawl fault does not produce an error status — those degrade to entries in the warnings array, and the request still returns 200 with whatever was found. The full status-code reference is in the error-codes guide.

Limits