---
seo_title: Discover: Sitemap Crawler API | EnConvert
meta_desc: Enumerate every URL on a site in one call. Sitemap parsing plus HTTP crawling, with no per-page rendering, returned as a flat list.
keywords: sitemap crawler api, list all urls on a website api, get all pages of a website api, crawl website for urls api, sitemap parser api, website url discovery api, site mapping api, http crawler api
---

# Sitemap Crawler API

<div class="alert alert-info">
<strong>Available on every plan.</strong> Discover is generally available on every plan, Founding included, and it draws on your monthly ops allowance like any other call.
</div>

`POST /v2/discover` is a sitemap crawler API that lists a website's
URLs the cheap way: an HTTP-first crawl plus sitemap parsing, with an
optional JavaScript render fallback for single-page apps. There is no
screenshot, no PDF, and no stored artifact. It returns a flat,
de-duplicated list of URLs and a count of where each one came from,
synchronously in a single call. It is the lightweight "map the site"
primitive: point it at a domain, get back the pages
worth processing, then feed that list into
[the perceive endpoint](/docs/endpoints/perceive.md) or a crawl-mode ingest job.

Here is the smallest useful call. Send a URL, get its map back:

```bash
curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'
```

The response is a plain URL list with provenance counters. There are no
signed URLs, no operation ID, and no polling:

```json
{
    "url": "https://example.com",
    "mode": "hybrid",
    "total": 47,
    "urls": [
        "https://example.com/",
        "https://example.com/pricing",
        "https://example.com/docs",
        "https://example.com/blog/launch"
    ],
    "pages_crawled": 12,
    "truncated": false,
    "robots_respected": false,
    "sources": {"sitemap": 42, "crawl": 30},
    "warnings": []
}
```

---

## Endpoints

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/v2/discover` | Map a site's URLs over HTTP only and return a flat list with source counts. |

`/v2/discover` exposes a single path. It is stateless: there is no
operation row to re-fetch and no job to poll, so there is no companion
`GET` endpoint the way [perceive has one](/docs/endpoints/perceive.md).

**Content-Type:** `application/json` on the `POST`.

---

## Authentication

Authenticate with a private key in the `X-API-Key` header for
server-to-server calls. This is the path the examples below use.

```http
X-API-Key: sk_your_private_key
```

Public keys with a JWT bearer token also work, using the same flow as
every other endpoint: generate a token with your `pk_` key, then send it
as `Authorization: Bearer <token>`. The full flow, including domain
locking and token refresh, is in [the authentication guide](/docs/authentication.md).

Each API key carries an allowed-endpoints allowlist. If `/v2/discover`
is not on the key's list, the request is rejected with `403` and a
message naming the path that was blocked.

---

## How discover works

Unless the `render_js` fallback fires (in its default `auto` mode, only when
the HTTP crawl finds no more than one link, as a bare JavaScript shell does;
see JavaScript rendering below), a request runs entirely over HTTP and the singleton
headless Chrome that powers [perceive](/docs/endpoints/perceive.md) is not
touched. The crawl path
uses Crawl4AI's HTTP crawler strategy (an `httpx` GET plus an `lxml`
link parse per page), and the sitemap path reuses the same pure-HTTP
sitemap and feed helpers as the rest of the platform.

1. **Screen the seed.** The URL you send is checked for SSRF before any
   fetch: scheme, embedded credentials, blocked hostnames, and the
   resolved IP are all validated. A URL that resolves to a private,
   loopback, link-local, carrier-grade NAT, or cloud-metadata address is rejected with
   `400`. The seed is always included as the first entry in its own map.
2. **Gather from sitemaps.** In `sitemap` or `hybrid` mode, discover
   reads `robots.txt` and fetches the `Sitemap:` URLs it declares, probes
   `sitemap.xml` (recursing into `<sitemapindex>` children), and pulls
   RSS/Atom feed pages. It runs these probes against both the host you
   sent and the site's registrable (apex) domain, so a sitemap published
   only on the apex is still found from a subdomain or deep-path seed. A
   missing or broken sitemap becomes a warning, not an error.
3. **Crawl over HTTP.** In `crawl` or `hybrid` mode, discover runs a
   breadth-first crawl from the seed up to `max_depth`, harvesting the
   `<a href>` links from each fetched page's raw HTML. Every followed
   link is SSRF-screened before it is fetched.
4. **Normalise and filter.** The combined raw list is canonicalised
   (fragments and tracking params stripped, default ports removed, query
   keys sorted), then passed through the same-domain check, your include
   and exclude regex patterns, the `robots.txt` filter, de-duplication,
   and finally the `max_urls` cap.

On the HTTP-only path, a client-rendered single-page app returns only its
HTML shell in `crawl` mode, typically the seed plus zero or one URL. The
optional `render_js` fallback (see [JavaScript rendering](#javascript-rendering))
closes that gap: in its default `"auto"` setting, discover renders the
page once in the browser and harvests the links it injects at runtime
whenever the HTTP crawl comes back with just a shell. `sitemap` mode
remains a fast, browser-free route for SEO-aware SPAs, which usually
publish a sitemap.

---

## Request parameters

### Core

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `url` | `string` | -- | The site to map. Must start with `http://` or `https://`. Max 2,048 characters. Required. |
| `mode` | `string` | `"hybrid"` | `sitemap`, `crawl`, or `hybrid`. See [Modes](#modes). |
| `max_urls` | `integer` | `100` | Maximum URLs returned. 1–1,000. The list is capped here and `truncated` flags if more existed. |
| `max_depth` | `integer` | `2` | Crawl depth from the seed, in `crawl`/`hybrid` mode. 1–5. |
| `same_domain_only` | `boolean` | `true` | Keep only URLs on the seed's host. When `false`, off-host links discovered during the crawl are kept too. |
| `render_js` | `string` | `"auto"` | Browser-rendered discovery for JavaScript/SPA sites (`crawl`/`hybrid` only). `auto` renders only when the HTTP crawl returns a shell; `always` forces it; `never` stays HTTP-only. See [JavaScript rendering](#javascript-rendering). |

### Filtering

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `include_patterns` | `string[]` | `[]` | Python regex allowlist (`re.search` semantics, not glob). A URL must match at least one to be kept. Empty means allow all. Max 50 patterns. |
| `exclude_patterns` | `string[]` | `[]` | Python regex denylist. A URL matching any pattern is dropped. Applied after `include_patterns`. Max 50 patterns. |
| `respect_robots` | `boolean` | `false` | When `true`, URLs disallowed by the site's `robots.txt` are dropped from the returned list. |

The patterns are real Python regular expressions, compiled at validation
time. A malformed pattern is a `422` at the edge, not a `500` mid-crawl.
Because the semantics are `re.search`, a bare substring like
`"/blog/"` matches anywhere in the URL, so anchor with `^`/`$` if you
need a positional match.

> **Worth flagging up front.** `respect_robots` is enforced at
> output-filter time, not fetch time. Crawl4AI 0.8.9 has no native robots
> gate, so in `crawl` or `hybrid` mode a disallowed page may still be
> fetched over HTTP and then discarded before it reaches the response.
> Unlike [perceive](/docs/endpoints/perceive.md), where `respect_robots=true`
> rejects a disallowed URL with `403`, discover never returns `403` for a
> robots rule. It silently drops the URL and reports nothing.

### Modes

| `mode` | What it does |
|--------|--------------|
| `sitemap` | `robots.txt` sitemap entries, probed `sitemap.xml` (with index recursion), and RSS/Atom feed pages. Instant, no crawl. |
| `crawl` | Breadth-first HTTP-only crawl from the seed, harvesting `<a href>` links from raw markup. |
| `hybrid` (default) | The de-duplicated union of `sitemap` and `crawl`. |

`crawl` mode fetches up to your full `max_urls` (up to 1,000), so a
large site is enumerated in one pass. Crawl mode is HTTP-only (no
browser), so this stays fast. Operators can lower the ceiling with the
`DISCOVER_CRAWL_MAX_PAGES` environment variable if a very large crawl
ever needs bounding. `pages_crawled` in the response tells you exactly
how many GETs ran.

Sitemap fetching handles **gzipped sitemaps** (`sitemap.xml.gz` and any
sitemap served as `application/gzip`), and sends realistic browser
headers so sites behind a WAF are far more likely to return their
sitemap instead of a `403`/`503`.

### JavaScript rendering

`crawl` and `hybrid` mode read the raw HTML a server returns, so a
client-rendered single-page app that builds its links in the browser is
invisible to them. `render_js` opts into a bounded browser fallback that
closes that gap:

| `render_js` | What it does |
|-------------|--------------|
| `auto` (default) | Run the HTTP crawl first; render in the browser only when it comes back with a bare shell (one URL or none), then harvest the client-injected links. |
| `always` | Always run the browser-rendered crawl. |
| `never` | Stay strictly HTTP-only, the pre-existing behaviour. |

The fallback uses the same shared headless Chrome as
[perceive](/docs/endpoints/perceive.md), with the same browser identity and the
same private-address checks on every connection, and is capped at a few pages so the call
stays inside the request timeout. Links it finds are counted under the
`crawl_js` key in `sources`. It applies to `crawl` and `hybrid` mode only,
because `sitemap` mode is already browser-free and unaffected.

---

## Response

`POST /v2/discover` returns this object directly, with no async job and
no second call.

| Field | Type | Description |
|-------|------|-------------|
| `url` | `string` | The seed URL you sent. |
| `mode` | `string` | The mode that ran: `sitemap`, `crawl`, or `hybrid`. |
| `total` | `integer` | Number of URLs in `urls` (after dedup, filtering, and the cap). |
| `urls` | `string[]` | The de-duplicated, normalised, capped URL list. The seed is always the first candidate. |
| `pages_crawled` | `integer` | HTTP GETs issued by the crawl. `0` in pure `sitemap` mode. |
| `truncated` | `boolean` | `true` when more unique URLs existed than `max_urls` allowed. |
| `robots_respected` | `boolean` | Echoes the `respect_robots` value you sent. |
| `sources` | `object` | Raw URL count per source before dedup/filter, e.g. `{"sitemap": 42, "crawl": 30}`. Keys include `sitemap`, `crawl`, `sitemap_apex` (sitemaps found on the apex domain), and `crawl_js` (links from the JavaScript render fallback). The counts overlap and sum above `total`. |
| `warnings` | `string[]` | Non-fatal notes: a missing sitemap, a crawl that failed, an unreachable `robots.txt`. |

The `sources` counts are raw: they are the number of URLs each path
produced before normalisation, the same-domain check, your filters, and
de-duplication ran. They will routinely add up to more than `total`,
because `hybrid` mode finds the same pages from both the sitemap and the
crawl. Use them to see which path is carrying the map, not as a
post-filter tally.

---

## Code examples

### curl: default hybrid map

```bash
curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'
```

### curl: sitemap only, blog pages, capped at 500

```bash
curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "sitemap",
    "max_urls": 500,
    "include_patterns": ["/blog/"],
    "exclude_patterns": ["/tag/", "/author/"],
    "respect_robots": true
  }'
```

### Python

```python
import requests

response = requests.post(
    "https://api.enconvert.com/v2/discover",
    headers={"X-API-Key": "sk_your_private_key"},
    json={
        "url": "https://example.com",
        "mode": "hybrid",
        "max_urls": 200,
        "include_patterns": [r"/docs/"],
    },
)
response.raise_for_status()
data = response.json()

print(f"found {data['total']} URLs from {data['sources']}")
for page_url in data["urls"]:
    print(page_url)
```

### Node.js

```javascript
const res = await fetch("https://api.enconvert.com/v2/discover", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        "X-API-Key": "sk_your_private_key"
    },
    body: JSON.stringify({
        url: "https://example.com",
        mode: "hybrid",
        max_urls: 200,
        include_patterns: ["/docs/"]
    })
});

const data = await res.json();

console.log(`found ${data.total} URLs from`, data.sources);
data.urls.forEach((pageUrl) => console.log(pageUrl));
```

A common pattern is to discover first, then render: take `data.urls` and
hand them to [the perceive endpoint](/docs/endpoints/perceive.md), using single
calls for a handful of pages or its batch path for the whole list.

---

## Error responses

| Status | Condition |
|--------|-----------|
| `400 Bad Request` | URL is not `http(s)`, carries embedded credentials, has no hostname, or resolves to a private, loopback, link-local, carrier-grade NAT, or cloud-metadata address (SSRF protection). |
| `401 Unauthorized` | Missing or invalid API key / JWT token. |
| `402 Payment Required` | Discover is not enabled on your current plan, or your monthly ops allowance is exhausted. Each `/v2/discover` call bills one op. |
| `403 Forbidden` | `/v2/discover` is not in the API key's allowed endpoints. |
| `422 Unprocessable Entity` | Request validation failed: bad `mode` enum, `max_urls` outside 1–1,000, `max_depth` outside 1–5, over 50 include/exclude patterns, a malformed regex, or a `url` over 2,048 characters. |
| `500 Internal Server Error` | URL discovery failed unexpectedly. The client gets a generic message; full detail goes to server logs only. |

Note that a failed sitemap fetch or a crawl fault does not produce an
error status. Those degrade to entries in the `warnings` array, and the
request still returns `200` with whatever was found. The full status-code
reference is in [the error-codes guide](/docs/reference/errors.md).

---

## Limits

| Limit | Value |
|-------|-------|
| URL length | 2,048 characters |
| `max_urls` | 1–1,000 (default 100) |
| `max_depth` | 1–5 (default 2) |
| `include_patterns` | 50 patterns max |
| `exclude_patterns` | 50 patterns max |
| Pages fetched in crawl mode | up to `max_urls` (max 1,000; lower with `DISCOVER_CRAWL_MAX_PAGES`) |
| Request timeout | 300 seconds |
| Ops per call | 1, billed against the unified monthly allowance |

---

## Frequently asked questions

### How do I list all URLs on a website with an API?

Send `POST /v2/discover` with the site's URL. The default `hybrid` mode combines sitemap parsing (`robots.txt` entries, `sitemap.xml` with index recursion, RSS/Atom feeds) with a breadth-first HTTP crawl, and returns up to `max_urls` (1–1,000, default 100) de-duplicated URLs in one synchronous response.

### Does the discover endpoint use a headless browser or render JavaScript?

By default it runs over HTTP and only renders when it has to. The `render_js` option controls this: in the default `"auto"` mode, discover stays HTTP-only and renders a page in the browser just when the HTTP crawl returns a bare JavaScript shell, harvesting the client-injected links; `"always"` forces the browser crawl and `"never"` keeps it strictly HTTP-only. `sitemap` mode remains a fast, browser-free option for SEO-aware SPAs.

### How many pages does crawl mode actually fetch?

Crawl mode fetches up to your full `max_urls` (up to 1,000). It stays HTTP-only, so it remains fast; operators can lower the ceiling with the `DISCOVER_CRAWL_MAX_PAGES` environment variable. Each fetched page also contributes many links. The `pages_crawled` response field reports exactly how many HTTP GETs ran.

### Can I filter which URLs come back?

Yes. `include_patterns` and `exclude_patterns` accept up to 50 Python regexes each (`re.search` semantics, not glob), and `respect_robots: true` drops URLs disallowed by the site's `robots.txt` from the returned list.