---
seo_title: Discover (Phase 2): Sitemap Crawler API | EnConvert
meta_desc: Private beta, Phase 2: enumerate every URL on a site in one call. Sitemap parsing plus HTTP crawling, with no per-page rendering, returned as a flat list.
keywords: sitemap crawler api, list all urls on a website api, get all pages of a website api, crawl website for urls api, sitemap parser api, website url discovery api, site mapping api, http crawler api
---

# Sitemap Crawler API

<div class="alert alert-warning">
<strong>Private beta.</strong> Discover is callable today with your normal API key, on every plan including Founding, and it draws on your monthly ops allowance like any other call. It is not announced or generally available: the request and response shapes can change without notice, and there is no stability or support commitment, so do not build anything load-bearing on it yet. The roadmap is on <a href="/docs/coming-soon">Coming Soon</a>, and every release is announced in <a href="/changelog">the changelog</a>.
</div>

`POST /v2/discover` is a sitemap crawler API that lists a website's
URLs the cheap way: an HTTP-first crawl plus sitemap parsing, with an
optional JavaScript render fallback for single-page apps. There is no
screenshot, no PDF, and no stored artifact. It returns a flat,
de-duplicated list of URLs and a count of where each one came from,
synchronously in a single call. It will be the lightweight "map the site"
primitive: point it at a domain, get back the pages
worth processing, then feed that list into
[the perceive endpoint](/docs/endpoints/perceive.md) or a crawl-mode ingest job.

Here is the smallest useful call. Send a URL, get its map back:

```bash
curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'
```

The response is a plain URL list with provenance counters. There are no
signed URLs, no operation ID, and no polling:

```json
{
    "url": "https://example.com",
    "mode": "hybrid",
    "total": 47,
    "urls": [
        "https://example.com/",
        "https://example.com/pricing",
        "https://example.com/docs",
        "https://example.com/blog/launch"
    ],
    "pages_crawled": 12,
    "truncated": false,
    "robots_respected": false,
    "sources": {"sitemap": 42, "crawl": 30},
    "warnings": []
}
```

---

## Endpoints

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/v2/discover` | Map a site's URLs over HTTP only and return a flat list with source counts. |

`/v2/discover` exposes a single path. It is stateless: there is no
operation row to re-fetch and no job to poll, so there is no companion
`GET` endpoint the way [perceive has one](/docs/endpoints/perceive.md).

**Content-Type:** `application/json` on the `POST`.

---

## Authentication

Authenticate with a private key in the `X-API-Key` header for
server-to-server calls. This is the path the examples below use.

```http
X-API-Key: sk_your_private_key
```

Public keys with a JWT bearer token also work, using the same flow as
every other endpoint: generate a token with your `pk_` key, then send it
as `Authorization: Bearer <token>`. The full flow, including domain
locking and token refresh, is in [the authentication guide](/docs/authentication.md).

Each API key carries an allowed-endpoints allowlist. If `/v2/discover`
is not on the key's list, the request is rejected with `403` and a
message naming the path that was blocked.

---

## How discover works

One request runs entirely over HTTP. The singleton headless Chrome that
powers [perceive](/docs/endpoints/perceive.md) is never touched. The crawl path
uses Crawl4AI's HTTP crawler strategy (an `httpx` GET plus an `lxml`
link parse per page), and the sitemap path reuses the same pure-HTTP
sitemap and feed helpers as the rest of the platform.

1. **Screen the seed.** The URL you send is checked for SSRF before any
   fetch: scheme, embedded credentials, blocked hostnames, and the
   resolved IP are all validated. A URL that resolves to a private,
   loopback, link-local, or cloud-metadata address is rejected with
   `400`. The seed is always included as the first entry in its own map.
2. **Gather from sitemaps.** In `sitemap` or `hybrid` mode, discover
   reads `robots.txt` and fetches the `Sitemap:` URLs it declares, probes
   `sitemap.xml` (recursing into `<sitemapindex>` children), and pulls
   RSS/Atom feed pages. It runs these probes against both the host you
   sent and the site's registrable (apex) domain, so a sitemap published
   only on the apex is still found from a subdomain or deep-path seed. A
   missing or broken sitemap becomes a warning, not an error.
3. **Crawl over HTTP.** In `crawl` or `hybrid` mode, discover runs a
   breadth-first crawl from the seed up to `max_depth`, harvesting the
   `<a href>` links from each fetched page's raw HTML. Every followed
   link is SSRF-screened before it is fetched.
4. **Normalise and filter.** The combined raw list is canonicalised
   (fragments and tracking params stripped, default ports removed, query
   keys sorted), then passed through the same-domain check, your include
   and exclude regex patterns, the `robots.txt` filter, de-duplication,
   and finally the `max_urls` cap.

On the HTTP-only path, a client-rendered single-page app returns only its
HTML shell in `crawl` mode, typically the seed plus zero or one URL. The
optional `render_js` fallback (see [JavaScript rendering](#javascript-rendering))
closes that gap: in its default `"auto"` setting, discover renders the
page once in the browser and harvests the links it injects at runtime
whenever the HTTP crawl comes back with just a shell. `sitemap` mode
remains a fast, browser-free route for SEO-aware SPAs, which usually
publish a sitemap.

---

## Request parameters

### Core

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `url` | `string` | -- | The site to map. Must start with `http://` or `https://`. Max 2,048 characters. Required. |
| `mode` | `string` | `"hybrid"` | `sitemap`, `crawl`, or `hybrid`. See [Modes](#modes). |
| `max_urls` | `integer` | `100` | Maximum URLs returned. 1–1,000. The list is capped here and `truncated` flags if more existed. |
| `max_depth` | `integer` | `2` | Crawl depth from the seed, in `crawl`/`hybrid` mode. 1–5. |
| `same_domain_only` | `boolean` | `true` | Keep only URLs on the seed's host. When `false`, off-host links discovered during the crawl are kept too. |
| `render_js` | `string` | `"auto"` | Browser-rendered discovery for JavaScript/SPA sites (`crawl`/`hybrid` only). `auto` renders only when the HTTP crawl returns a shell; `always` forces it; `never` stays HTTP-only. See [JavaScript rendering](#javascript-rendering). |

### Filtering

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `include_patterns` | `string[]` | `[]` | Python regex allowlist (`re.search` semantics, not glob). A URL must match at least one to be kept. Empty means allow all. Max 50 patterns. |
| `exclude_patterns` | `string[]` | `[]` | Python regex denylist. A URL matching any pattern is dropped. Applied after `include_patterns`. Max 50 patterns. |
| `respect_robots` | `boolean` | `false` | When `true`, URLs disallowed by the site's `robots.txt` are dropped from the returned list. |

The patterns are real Python regular expressions, compiled at validation
time. A malformed pattern is a `422` at the edge, not a `500` mid-crawl.
Because the semantics are `re.search`, a bare substring like
`"/blog/"` matches anywhere in the URL, so anchor with `^`/`$` if you
need a positional match.

> **Worth flagging up front.** `respect_robots` is enforced at
> output-filter time, not fetch time. Crawl4AI 0.8.9 has no native robots
> gate, so in `crawl` or `hybrid` mode a disallowed page may still be
> fetched over HTTP and then discarded before it reaches the response.
> Unlike [perceive](/docs/endpoints/perceive.md), where `respect_robots=true`
> rejects a disallowed URL with `403`, discover never returns `403` for a
> robots rule. It silently drops the URL and reports nothing.

### Modes

| `mode` | What it does |
|--------|--------------|
| `sitemap` | `robots.txt` sitemap entries, probed `sitemap.xml` (with index recursion), and RSS/Atom feed pages. Instant, no crawl. |
| `crawl` | Breadth-first HTTP-only crawl from the seed, harvesting `<a href>` links from raw markup. |
| `hybrid` (default) | The de-duplicated union of `sitemap` and `crawl`. |

`crawl` mode fetches up to your full `max_urls` (up to 1,000), so a
large site is enumerated in one pass. Crawl mode is HTTP-only (no
browser), so this stays fast. Operators can lower the ceiling with the
`DISCOVER_CRAWL_MAX_PAGES` environment variable if a very large crawl
ever needs bounding. `pages_crawled` in the response tells you exactly
how many GETs ran.

Sitemap fetching handles **gzipped sitemaps** (`sitemap.xml.gz` and any
sitemap served as `application/gzip`), and sends realistic browser
headers so sites behind a WAF are far more likely to return their
sitemap instead of a `403`/`503`.

### JavaScript rendering

`crawl` and `hybrid` mode read the raw HTML a server returns, so a
client-rendered single-page app that builds its links in the browser is
invisible to them. `render_js` opts into a bounded browser fallback that
closes that gap:

| `render_js` | What it does |
|-------------|--------------|
| `auto` (default) | Run the HTTP crawl first; render in the browser only when it comes back with a bare shell (one URL or none), then harvest the client-injected links. |
| `always` | Always run the browser-rendered crawl. |
| `never` | Stay strictly HTTP-only, the pre-existing behaviour. |

The fallback uses the same shared headless Chrome as
[perceive](/docs/endpoints/perceive.md) and is capped at a few pages so the call
stays inside the request timeout. Links it finds are counted under the
`crawl_js` key in `sources`. It applies to `crawl` and `hybrid` mode only,
because `sitemap` mode is already browser-free and unaffected.

---

## Response

`POST /v2/discover` returns this object directly, with no async job and
no second call.

| Field | Type | Description |
|-------|------|-------------|
| `url` | `string` | The seed URL you sent. |
| `mode` | `string` | The mode that ran: `sitemap`, `crawl`, or `hybrid`. |
| `total` | `integer` | Number of URLs in `urls` (after dedup, filtering, and the cap). |
| `urls` | `string[]` | The de-duplicated, normalised, capped URL list. The seed is always the first candidate. |
| `pages_crawled` | `integer` | HTTP GETs issued by the crawl. `0` in pure `sitemap` mode. |
| `truncated` | `boolean` | `true` when more unique URLs existed than `max_urls` allowed. |
| `robots_respected` | `boolean` | Echoes the `respect_robots` value you sent. |
| `sources` | `object` | Raw URL count per source before dedup/filter, e.g. `{"sitemap": 42, "crawl": 30}`. Keys include `sitemap`, `crawl`, `sitemap_apex` (sitemaps found on the apex domain), and `crawl_js` (links from the JavaScript render fallback). The counts overlap and sum above `total`. |
| `warnings` | `string[]` | Non-fatal notes: a missing sitemap, a crawl that failed, an unreachable `robots.txt`. |

The `sources` counts are raw: they are the number of URLs each path
produced before normalisation, the same-domain check, your filters, and
de-duplication ran. They will routinely add up to more than `total`,
because `hybrid` mode finds the same pages from both the sitemap and the
crawl. Use them to see which path is carrying the map, not as a
post-filter tally.

---

## Code examples

### curl: default hybrid map

```bash
curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'
```

### curl: sitemap only, blog pages, capped at 500

```bash
curl -X POST https://api.enconvert.com/v2/discover \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "mode": "sitemap",
    "max_urls": 500,
    "include_patterns": ["/blog/"],
    "exclude_patterns": ["/tag/", "/author/"],
    "respect_robots": true
  }'
```

### Python

```python
import requests

response = requests.post(
    "https://api.enconvert.com/v2/discover",
    headers={"X-API-Key": "sk_your_private_key"},
    json={
        "url": "https://example.com",
        "mode": "hybrid",
        "max_urls": 200,
        "include_patterns": [r"/docs/"],
    },
)
response.raise_for_status()
data = response.json()

print(f"found {data['total']} URLs from {data['sources']}")
for page_url in data["urls"]:
    print(page_url)
```

### Node.js

```javascript
const res = await fetch("https://api.enconvert.com/v2/discover", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        "X-API-Key": "sk_your_private_key"
    },
    body: JSON.stringify({
        url: "https://example.com",
        mode: "hybrid",
        max_urls: 200,
        include_patterns: ["/docs/"]
    })
});

const data = await res.json();

console.log(`found ${data.total} URLs from`, data.sources);
data.urls.forEach((pageUrl) => console.log(pageUrl));
```

A common pattern is to discover first, then render: take `data.urls` and
hand them to [the perceive endpoint](/docs/endpoints/perceive.md), using single
calls for a handful of pages or its batch path for the whole list.

---

## Error responses

| Status | Condition |
|--------|-----------|
| `400 Bad Request` | URL is not `http(s)`, carries embedded credentials, has no hostname, or resolves to a private, loopback, link-local, or cloud-metadata address (SSRF protection). |
| `401 Unauthorized` | Missing or invalid API key / JWT token. |
| `402 Payment Required` | Discover is not enabled on your current plan, or your monthly ops allowance is exhausted. Each `/v2/discover` call bills one op. |
| `403 Forbidden` | `/v2/discover` is not in the API key's allowed endpoints. |
| `422 Unprocessable Entity` | Request validation failed: bad `mode` enum, `max_urls` outside 1–1,000, `max_depth` outside 1–5, over 50 include/exclude patterns, a malformed regex, or a `url` over 2,048 characters. |
| `500 Internal Server Error` | URL discovery failed unexpectedly. The client gets a generic message; full detail goes to server logs only. |

Note that a failed sitemap fetch or a crawl fault does not produce an
error status. Those degrade to entries in the `warnings` array, and the
request still returns `200` with whatever was found. The full status-code
reference is in [the error-codes guide](/docs/reference/errors.md).

---

## Limits

| Limit | Value |
|-------|-------|
| URL length | 2,048 characters |
| `max_urls` | 1–1,000 (default 100) |
| `max_depth` | 1–5 (default 2) |
| `include_patterns` | 50 patterns max |
| `exclude_patterns` | 50 patterns max |
| Pages fetched in crawl mode | up to `max_urls` (max 1,000; lower with `DISCOVER_CRAWL_MAX_PAGES`) |
| Request timeout | 300 seconds |
| Ops per call | 1, billed against the unified monthly allowance |

---

## Frequently asked questions

### How do I list all URLs on a website with an API?

Send `POST /v2/discover` with the site's URL. The default `hybrid` mode combines sitemap parsing (`robots.txt` entries, `sitemap.xml` with index recursion, RSS/Atom feeds) with a breadth-first HTTP crawl, and returns up to `max_urls` (1–1,000, default 100) de-duplicated URLs in one synchronous response.

### Does the discover endpoint use a headless browser or render JavaScript?

By default it runs over HTTP and only renders when it has to. The `render_js` option controls this: in the default `"auto"` mode, discover stays HTTP-only and renders a page in the browser just when the HTTP crawl returns a bare JavaScript shell, harvesting the client-injected links; `"always"` forces the browser crawl and `"never"` keeps it strictly HTTP-only. `sitemap` mode remains a fast, browser-free option for SEO-aware SPAs.

### How many pages does crawl mode actually fetch?

Crawl mode fetches up to your full `max_urls` (up to 1,000). It stays HTTP-only, so it remains fast; operators can lower the ceiling with the `DISCOVER_CRAWL_MAX_PAGES` environment variable. Each fetched page also contributes many links. The `pages_crawled` response field reports exactly how many HTTP GETs ran.

### Can I filter which URLs come back?

Yes. `include_patterns` and `exclude_patterns` accept up to 50 Python regexes each (`re.search` semantics, not glob), and `respect_robots: true` drops URLs disallowed by the site's `robots.txt` from the returned list.
