---
seo_title: RAG Ingestion API: Crawl Sites & Files to JSONL | EnConvert
meta_desc: Crawl a site or upload files for RAG with /v2/ingest. It renders or converts, chunks heading-aware, and exports one JSONL file for LangChain, LlamaIndex, or a vector DB.
keywords: crawl website for rag api, ingest files for rag api, pdf to jsonl for rag, website to jsonl langchain llamaindex, rag data ingestion api, document ingestion api for llm, jsonl for vector database, langchain jsonloader
---

# Crawl Website for RAG API

`POST /v2/ingest` crawls a website for RAG: it turns a site (or an
explicit list of URLs) into RAG-ready chunks and emits **one JSONL file**
that loads straight into LangChain `JSONLoader`, LlamaIndex
`SimpleDirectoryReader`, or a vector-DB bulk import. The endpoint is
always asynchronous: `POST` answers `202` with a `job_id`, you poll
`GET /v2/ingest/{job_id}` or register a `webhook_url`, and a completed job
hands back a pre-signed `output_url` for the JSONL. EnConvert does the
discovery, the headless-Chrome render, the heading-aware chunking, and the
JSONL assembly in one job.

Uploaded **files** ingest through the very same pipeline via
[`POST /v2/ingest/files`](#ingesting-files). PDF, DOCX, PPTX, XLSX, CSV,
HTML, EPUB, and more are converted to Markdown, chunked, and assembled into
the same JSONL. One integration covers both web and file RAG ingestion.

Here is the smallest useful call. Crawl a site and chunk every page it
discovers:

```bash
curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "crawl",
    "url": "https://example.com/docs"
  }'
```

The response is the job record, returned with `202 Accepted`. Note the
status is `queued`, and `output_url` is absent until the job completes:

```json
{
    "job_id": "ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7",
    "status": "queued",
    "mode": "crawl",
    "pages_discovered": 0,
    "pages_processed": 0,
    "pages_failed": 0,
    "total_chunks": 0,
    "webhook_delivered": false,
    "created_at": "2026-06-24T09:14:02.118Z"
}
```

Ingest is **always asynchronous**. Each page renders in a real browser,
which runs 10-30 seconds per URL, far past the 300-second request window
for any non-trivial job. So `POST` answers `202` with a `job_id`, and a
droplet-local worker drains the job out of band. You poll
`GET /v2/ingest/{job_id}` for progress, or register a `webhook_url` to be
told when it finishes.

---

## Endpoints

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/v2/ingest` | Create a **web** ingest job (URL list, sitemap, or crawl). Answers `202` with a `job_id`. |
| `POST` | `/v2/ingest/files` | Create a **file** ingest job from uploaded documents (multipart). Same job + JSONL pipeline. |
| `GET` | `/v2/ingest` | Newest-first list of this project's jobs, with `skip`/`limit` paging. |
| `GET` | `/v2/ingest/{job_id}` | Lifecycle status of one job, with a freshly signed `output_url` once completed. |
| `DELETE` | `/v2/ingest/{job_id}` | Cancel a job. The worker sees the canceled status and stops between pages. |
| `POST` | `/v2/ingest/{job_id}/retry-webhook` | Re-sign and re-POST the completion webhook for a completed job. |
| `GET` | `/v2/ingest/webhook-secret` | Reveal the project's webhook signing secret (dashboard channel). |
| `POST` | `/v2/ingest/webhook-secret/rotate` | Rotate the signing secret. Old signatures stop verifying immediately. |

**Content-Type:** `application/json` on every `POST`.

---

## Authentication

Authenticate with a private key in the `X-API-Key` header for
server-to-server calls. This is the path the examples below use.

```http
X-API-Key: sk_your_private_key
```

Public keys with a JWT bearer token also work, using the same flow as
every other endpoint: generate a token with your `pk_` key, then send it
as `Authorization: Bearer <token>`. The full flow, including domain
locking and token refresh, is in [the authentication guide](/docs/authentication.md).

Each API key carries an allowed-endpoints allowlist. If `/v2/ingest` is
not on the key's list, the request is rejected with `403`. A key scoped
to `/v2/ingest` still reaches the jobs it created: `GET` and `DELETE`
`/v2/ingest/{job_id}` and `POST /v2/ingest/{job_id}/retry-webhook` are
always allowed for a `job_id` (the `ing_…` shape is matched explicitly).
The static list endpoint and the two `webhook-secret` management routes
do **not** inherit that bypass; they require a broader or dashboard-scoped
token.

---

## How ingest works

One job runs through five phases, all durable and restart-safe. If the
worker process restarts mid-job, the job is re-enqueued at boot and
resumes from the page it stopped on. Already-completed pages keep their
staged output and are never re-rendered or re-billed.

1. **Queue.** `POST` validates the request, runs a fast `units=1` ops
   check (plan has ingest enabled and headroom left in the monthly ops
   allowance), inserts the job row, and answers `202`. Nothing is
   persisted if the ops gate fails: a `402` leaves zero rows behind.
2. **Discover.** For `sitemap` and `crawl` mode the worker runs the same
   discovery pass as [the discover endpoint](/docs/coming-soon/discover.md), capped at
   `max_pages`, and SSRF-screens the seed URL. For `urls` mode the
   explicit list is de-duplicated in order; no discovery runs. The
   pre-cap discovery size is reported as `pages_found`; when the site has
   more URLs than `max_pages` allowed, `discovery_truncated` is `true`
   and a `warnings` entry states both numbers, so `pages_discovered`
   (the enqueued count) is never mistaken for the site's size.
3. **Render and chunk.** Each URL renders through the shared headless
   Chrome singleton, the same render pipeline behind
   [the perceive endpoint](/docs/endpoints/perceive.md). The rendered HTML is then
   converted to fit-Markdown and sliced by the heading-aware chunker.
   Renders run sequentially, one page at a time.
4. **Stage.** Each page's chunks are written to a per-page JSONL object in
   storage, keyed deterministically by `(project, job, url)`. This is what
   makes a restart cheap: a resumed job re-uses staged pages instead of
   re-rendering them.
5. **Assemble.** Once every page is done, the per-page objects are
   concatenated into the final `v2-ingest/{job_id}.jsonl`, the staging
   objects are deleted, the job flips to `completed`, and the signed
   completion webhook fires if a `webhook_url` was set.

The ops allowance is re-checked **per page** inside the worker, not just
at submit time; each completed page bills one op. A `crawl` job whose
page count is unknown up front stops cleanly at your monthly cap: the
pages already rendered are billed and kept, and the remaining pages are
marked `skipped` rather than over-spending.

Ingest renders are **credential-free by design**. Unlike `/v2/perceive`,
it does not accept `auth`, `cookies`, or custom `headers`. Nothing secret
is persisted for the durable resume, so the job state on disk never
carries credentials.

---

## Ingesting files

`POST /v2/ingest` crawls the web; `POST /v2/ingest/files` ingests **uploaded
files** through the very same pipeline. Both create the same job, run the
same heading-aware chunker, and produce the same single JSONL deliverable,
so one integration covers web *and* file RAG ingestion.

Send the documents as `multipart/form-data` in the `files` field. Each file
is converted to Markdown by the [anything-to-markdown](/docs/endpoints/convert/documents/anything-to-markdown.md)
converter, then chunked and assembled exactly like a crawled page. Every
[supported input format](/docs/endpoints/convert/documents/anything-to-markdown.md#supported-input-formats)
is accepted: PDF, DOCX, PPTX, XLSX, CSV, HTML, EPUB, OpenDocument, and plain
text/Markdown.

```bash
curl -X POST https://api.enconvert.com/v2/ingest/files \
  -H "X-API-Key: sk_your_private_key" \
  -F "files=@handbook.pdf" \
  -F "files=@pricing.xlsx" \
  -F "files=@faq.docx" \
  -F "max_words=700" \
  -F "webhook_url=https://your-app.example.com/hooks/ingest"
```

The response is the same `IngestJobResponse` as the crawl endpoint, with
`mode` set to `files`:

```json
{
    "job_id": "ing_7c1d8e2f4a5b6c7d8e9f0a1b2c3d4e5f",
    "status": "queued",
    "mode": "files",
    "pages_discovered": 0,
    "pages_processed": 0,
    "pages_failed": 0,
    "total_chunks": 0,
    "webhook_delivered": false,
    "created_at": "2026-07-14T10:15:30.220Z"
}
```

Each uploaded file counts as one "page": it bills one op,
climbs `pages_processed` as it completes, and is labelled by its **filename**
in the JSONL `metadata.source_url`. You poll `GET /v2/ingest/{job_id}`, cancel
with `DELETE`, and receive the signed completion webhook exactly as for a crawl
job. Files are stored only until the JSONL is assembled, then deleted.

### File request parameters

Sent as multipart form fields (not a JSON body):

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `files` | file[] | -- | One or more documents to ingest. 1–200 files per request; each is size-checked against your plan's upload limit. |
| `max_words` | `integer` | `512` | Soft cap on words per chunk. 32–4,000. Code blocks and pipe tables stay atomic. |
| `sentence_overlap` | `integer` | `1` | Sentences repeated between consecutive prose chunks of the same section. 0–10. |
| `webhook_url` | `string` | `null` | HMAC-signed completion callback, with the same signing and retry policy as [completion webhooks](#completion-webhooks) below. |

An unsupported file type, an empty file, or an image (OCR is not performed) is
rejected at submit with a `400`; a file over your plan's per-file size limit is
a `413`.

```python
import requests

BASE = "https://api.enconvert.com"
HEADERS = {"X-API-Key": "sk_your_private_key"}

# Submit several files (always 202).
with open("handbook.pdf", "rb") as a, open("pricing.xlsx", "rb") as b:
    job = requests.post(
        f"{BASE}/v2/ingest/files",
        headers=HEADERS,
        files=[("files", ("handbook.pdf", a)), ("files", ("pricing.xlsx", b))],
        data={"max_words": 700},
    ).json()

# Poll GET /v2/ingest/{job_id} exactly as for a crawl job, then download output_url.
print(job["job_id"], job["status"], job["mode"])  # -> ing_...  queued  files
```

---

## Request parameters

### Source and mode

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `mode` | `string` | `"urls"` | `urls`, `sitemap`, or `crawl`. Selects how the URL set is built. |
| `url` | `string` | `null` | Seed URL for `sitemap`/`crawl` mode. Must start with `http://` or `https://`. Max 2,048 characters. Required for those modes; rejected in `urls` mode. |
| `urls` | `string[]` | `null` | Explicit URLs to ingest in `urls` mode. Non-empty, max 1,000 entries, each `http(s)` and ≤ 2,048 chars. Required for `urls` mode; rejected in `sitemap`/`crawl` mode. |

`url` and `urls` are mutually exclusive: send exactly one source. `urls` mode
requires `urls`; `sitemap` and `crawl` require a seed `url`. Sending the
wrong one for the mode is a `422`.

### Discovery (sitemap / crawl modes)

These are forwarded to the discovery pass and ignored in `urls` mode.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `max_pages` | `integer` | `50` | Cap on URLs discovered **and** ingested. 1–1,000. |
| `max_depth` | `integer` | `2` | Crawl link depth from the seed. 1–5. |
| `same_domain_only` | `boolean` | `true` | Restrict discovery to the seed's domain. |
| `include_patterns` | `string[]` | `[]` | Regex patterns a URL must match to be kept. Max 50. Each is compiled at submit; a bad pattern is a `422`. |
| `exclude_patterns` | `string[]` | `[]` | Regex patterns that drop a matching URL. Max 50. |
| `respect_robots` | `boolean` | `false` | When `true`, a URL disallowed by the site's `robots.txt` is skipped. |

### Rendering

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `wait_for` | `string` | `null` | Wait after navigation for a CSS selector or JS expression before capturing. Max 1,024 characters. |
| `wait_timeout_ms` | `integer` | `30000` | How long `wait_for` may wait, in milliseconds. 0–60,000. |

> **Note.** Ingest does **not** accept `auth`, `cookies`, or `headers`.
> If a page needs credentials to render, ingest is the wrong tool. Use
> [the perceive endpoint](/docs/endpoints/perceive.md), which carries the full
> authenticated-request surface, for that single page.

### Chunking (`chunk` object)

| Parameter | Type | Default | Constraints | Description |
|-----------|------|---------|-------------|-------------|
| `max_words` | `integer` | `512` | 32–4,000 | Soft cap on words per chunk. Heading-aware. Code blocks and pipe tables stay atomic and may exceed this. |
| `sentence_overlap` | `integer` | `1` | 0–10 | Sentences repeated between consecutive prose chunks of the same section. `0` disables overlap. Overlap never crosses a heading boundary. |

The chunker splits on `#`, `##`, and `###` headings, so each chunk belongs
to exactly one section and carries its full heading path. Deeper
headings (`####`–`######`) stay inline as content. Fenced code blocks and
Markdown tables are never split, even when a single block runs over
`max_words`; list items split between items, never mid-item.

### Webhook

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `webhook_url` | `string` | `null` | Endpoint to receive the HMAC-signed completion callback. Max 2,048 characters. Scheme-checked at submit; SSRF-screened at *delivery* time, not submit time. |

---

## Response

`POST`, `GET /v2/ingest/{job_id}`, and `DELETE` all return the same
`IngestJobResponse` object.

| Field | Type | Description |
|-------|------|-------------|
| `job_id` | `string` | Opaque ID (`ing_…`). Use it with the GET/DELETE endpoints and quote it to support. |
| `status` | `string` | `queued`, `discovering`, `processing`, `completed`, `failed`, or `canceled`. |
| `mode` | `string` | The mode you submitted: `urls`, `sitemap`, `crawl`, or `files`. |
| `pages_discovered` | `integer` | Items the job actually enqueued: URLs (the explicit list, or the discovery result capped at `max_pages`), or uploaded files. `pages_processed` + `pages_failed` sum to this once the job is terminal. |
| `pages_found` | `integer` | Unique eligible URLs discovery yielded **before** the `max_pages` cap. For `sitemap` jobs this is the site's true unique count; for `crawl` jobs it is a lower bound (the crawl stops fetching at the cap). Absent for `urls` and `files` jobs. |
| `discovery_truncated` | `boolean` | `true` when discovery found more unique URLs than `max_pages` let the job enqueue. A `warnings` entry spells out the numbers; raise `max_pages` to ingest more of the site. |
| `pages_processed` | `integer` | URLs whose render → chunk → stage completed. |
| `pages_failed` | `integer` | URLs that failed to render or were skipped (e.g. ops allowance exhausted). |
| `total_chunks` | `integer` | Total chunks written across all completed pages. Matches the JSONL line count. |
| `output_url` | `string` | Pre-signed download URL for the final JSONL. Present only once `status` is `completed`; expires after 15 minutes. |
| `error_message` | `string` | Set when `status` is `failed` (e.g. discovery rejected, all pages failed). |
| `webhook_url` | `string` | The completion-webhook target registered for this job, if any. |
| `webhook_delivered` | `boolean` | `true` once the signed completion webhook got a `2xx`. |
| `created_at` | `string` | When the job was created (UTC). |
| `completed_at` | `string` | When the job reached a terminal status (UTC). |
| `warnings` | `string[]` | Non-fatal notes, e.g. discovery truncation: `"discovery found 719 unique URLs; the job was capped at max_pages=50, so 50 pages were enqueued. Raise max_pages to ingest more of the site."` |

> **Note.** `POST` and the per-job `GET`/`DELETE` use
> `response_model_exclude_none`, so fields that are still `null` (like
> `output_url` before completion) are omitted from the JSON rather than
> sent as `null`.

### The JSONL record shape

The final file is newline-delimited JSON. Each line is one chunk:

```json
{"id":"9f2b8c1ad4e5-0000","content":"Pricing is usage-based...","metadata":{"source_url":"https://example.com/pricing","title":"Pricing","headings_path":["Pricing","Plans"],"section":"Plans","word_count":118,"chunk_index":0}}
```

| Field | Type | Description |
|-------|------|-------------|
| `id` | `string` | Deterministic per `(source_url, chunk_index)`: `<md5(url)[:12]>-<index:04d>`. A re-run produces identical ids. |
| `content` | `string` | The retrievable chunk text. Maps to `Document.page_content` in LangChain. |
| `metadata.source_url` | `string` | The page the chunk came from. |
| `metadata.title` | `string` | Page `<title>`, falling back to the first `<h1>`, capped at 512 chars. |
| `metadata.headings_path` | `string[]` | The `h1 → h2 → h3` path the chunk sits under. |
| `metadata.section` | `string` | The innermost heading text (the last entry of `headings_path`). |
| `metadata.word_count` | `integer` | Whitespace-delimited word count of `content`. |
| `metadata.chunk_index` | `integer` | The chunk's index within its page. |

The file is UTF-8, written with `ensure_ascii=false`, so unicode stays
readable. Because `content` is a top-level string and `metadata` is a
sibling object, the same file loads through LangChain
`JSONLoader(content_key="content", json_lines=True)`, LlamaIndex
`SimpleDirectoryReader`, and any line-oriented vector-DB import without
reshaping.

---

## Job lifecycle and polling

A job moves through these states:

```text
queued → discovering → processing → completed | failed | canceled
```

| Status | Meaning |
|--------|---------|
| `queued` | Accepted and waiting for the worker. |
| `discovering` | Running the sitemap/crawl discovery pass (`sitemap`/`crawl` only). |
| `processing` | Rendering and chunking pages. `pages_processed` and `total_chunks` climb live. |
| `completed` | The final JSONL is assembled; `output_url` is signed and ready. |
| `failed` | Discovery was rejected, or every page failed or was skipped. `error_message` explains. |
| `canceled` | A `DELETE` reached the job before it finished. |

Poll the status with the per-job `GET`. This is read-only: it consumes
no ops and re-signs the `output_url` from the stored object key on every
call:

```bash
curl https://api.enconvert.com/v2/ingest/ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_your_private_key"
```

An unknown `job_id`, or one that belongs to a different project, returns
`404`. Existence is never leaked across projects.

### Listing jobs

`GET /v2/ingest` returns this project's jobs newest-first, with `skip` and
`limit` query params. `limit` defaults to `20` and is capped at `100`. The
response carries a `has_more` flag instead of a total count:

```bash
curl "https://api.enconvert.com/v2/ingest?skip=0&limit=20" \
  -H "X-API-Key: sk_your_private_key"
```

```json
{
    "jobs": [
        {
            "job_id": "ing_3f9a...",
            "status": "completed",
            "mode": "crawl",
            "pages_discovered": 42,
            "pages_found": 42,
            "discovery_truncated": false,
            "pages_processed": 41,
            "pages_failed": 1,
            "total_chunks": 1187,
            "output_url": "https://spaces.example.com/...signed...",
            "webhook_configured": true,
            "webhook_delivered": true,
            "created_at": "2026-06-24T09:14:02.118Z",
            "completed_at": "2026-06-24T09:31:55.402Z"
        }
    ],
    "skip": 0,
    "limit": 20,
    "has_more": false
}
```

The list rows reduce `webhook_url` to a `webhook_configured` boolean, so
the list never echoes the raw endpoint back into the table.

### Canceling a job

`DELETE /v2/ingest/{job_id}` sets the job's status to `canceled`. The
worker reads that status between pages and stops without assembling
output. Cancellation is idempotent and race-proof: if assembly already
committed, the `DELETE` matches nothing and the job is returned unchanged
as `completed`. A finished job is never clobbered back to `canceled`.

```bash
curl -X DELETE \
  https://api.enconvert.com/v2/ingest/ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_your_private_key"
```

---

## Completion webhooks

Set `webhook_url` on the `POST` and EnConvert sends one HMAC-signed POST
when the job completes. The payload is compact, key-sorted JSON:

```json
{"job_id":"ing_3f9a...","output_url":"https://spaces.example.com/...signed...","pages_processed":41,"status":"completed","total_chunks":1187}
```

Delivery retries up to three times after the first attempt, with back-off
delays of 1, 4, and 16 seconds, which is four POSTs worst case. Each
attempt is re-signed with a fresh timestamp, so a slow retry chain never
drifts past the consumer's freshness window. A 2xx response is success.
A dead endpoint is recorded as a non-delivery and raises a dashboard
alert, but it never sinks an otherwise-completed job.

The `webhook_url` is **SSRF-screened at delivery time**, not at submit. A
URL that resolves to a private, loopback, or metadata address is stored
inertly and only rejected when EnConvert tries to POST to it.

### Verifying the signature

Each delivery carries two headers:

| Header | Value |
|--------|-------|
| `X-Enconvert-Signature` | `sha256=<hex>`, the HMAC-SHA256 of `<timestamp>.<raw body>`. |
| `X-Enconvert-Timestamp` | The unix-seconds timestamp bound into the signature. |

The signing input is the timestamp, a literal `.`, then the raw request
body. Binding the timestamp into the MAC means a consumer that rejects
stale timestamps gets replay protection for free. The default freshness
window is **300 seconds**. Verify in your handler:

```python
import hashlib
import hmac
import time

SECRET = "whsec_your_signing_secret"   # from GET /v2/ingest/webhook-secret
TOLERANCE_SECONDS = 300


def verify(raw_body: bytes, signature_header: str, timestamp_header: str) -> bool:
    if not signature_header or not timestamp_header:
        return False
    try:
        ts = int(timestamp_header)
    except ValueError:
        return False
    if abs(time.time() - ts) > TOLERANCE_SECONDS:
        return False  # replayed or badly skewed clock

    provided = signature_header.removeprefix("sha256=")
    expected = hmac.new(
        SECRET.encode("utf-8"),
        f"{ts}.".encode("utf-8") + raw_body,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, provided)
```

### Managing the signing secret

`GET /v2/ingest/webhook-secret` reveals the project's secret (creating it
on first call) along with the header names and tolerance your consumer
needs. It is **sensitive** and exposed only over the authenticated
dashboard channel:

```json
{
    "secret": "whsec_...",
    "signature_header": "X-Enconvert-Signature",
    "timestamp_header": "X-Enconvert-Timestamp",
    "signature_scheme": "sha256",
    "replay_tolerance_seconds": 300,
    "rotated": false
}
```

`POST /v2/ingest/webhook-secret/rotate` issues a new secret and sets
`rotated` to `true`. Every signature computed with the previous secret
stops verifying the moment the rotation commits. Rotate after a suspected
leak, then update your consumer.

### Re-delivering a webhook

If your endpoint was down when the job finished,
`POST /v2/ingest/{job_id}/retry-webhook` re-signs and re-POSTs with the
same retry policy:

```bash
curl -X POST \
  https://api.enconvert.com/v2/ingest/ing_3f9a.../retry-webhook \
  -H "X-API-Key: sk_your_private_key"
```

```json
{
    "job_id": "ing_3f9a...",
    "delivered": true,
    "attempts": 1,
    "status_code": 200,
    "detail": "Delivered (HTTP 200)."
}
```

It returns `404` for an unknown or foreign `job_id`, `400` when no
`webhook_url` is configured (or the stored URL now resolves to a
private/internal address), and `409` when the job has not reached
`completed`.

---

## Code examples

### curl: explicit URL list

```bash
curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "urls",
    "urls": [
      "https://example.com/docs/intro",
      "https://example.com/docs/quickstart",
      "https://example.com/docs/api"
    ]
  }'
```

### curl: crawl with chunking and a webhook

```bash
curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "crawl",
    "url": "https://example.com/docs",
    "max_pages": 200,
    "max_depth": 3,
    "include_patterns": ["/docs/"],
    "chunk": {"max_words": 700, "sentence_overlap": 2},
    "webhook_url": "https://your-app.example.com/hooks/ingest"
  }'
```

### Python: submit, poll, download

```python
import time

import requests

BASE = "https://api.enconvert.com"
HEADERS = {"X-API-Key": "sk_your_private_key"}

# 1. Submit (always 202).
job = requests.post(
    f"{BASE}/v2/ingest",
    headers=HEADERS,
    json={"mode": "crawl", "url": "https://example.com/docs", "max_pages": 100},
).json()
job_id = job["job_id"]

# 2. Poll until terminal.
while True:
    job = requests.get(f"{BASE}/v2/ingest/{job_id}", headers=HEADERS).json()
    if job["status"] in ("completed", "failed", "canceled"):
        break
    time.sleep(5)

# 3. Download the JSONL from its signed URL.
if job["status"] == "completed":
    jsonl = requests.get(job["output_url"]).text
    print(f"{job['total_chunks']} chunks across "
          f"{job['pages_processed']} pages")
    print(jsonl.splitlines()[0])
```

### Node.js: submit and poll

```javascript
const BASE = "https://api.enconvert.com";
const HEADERS = {
    "Content-Type": "application/json",
    "X-API-Key": "sk_your_private_key"
};

// 1. Submit.
const submit = await fetch(`${BASE}/v2/ingest`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({
        mode: "crawl",
        url: "https://example.com/docs",
        max_pages: 100
    })
});
let job = await submit.json();

// 2. Poll until terminal.
while (!["completed", "failed", "canceled"].includes(job.status)) {
    await new Promise((r) => setTimeout(r, 5000));
    const poll = await fetch(`${BASE}/v2/ingest/${job.job_id}`, {
        headers: { "X-API-Key": HEADERS["X-API-Key"] }
    });
    job = await poll.json();
}

// 3. Download the JSONL.
if (job.status === "completed") {
    const jsonl = await fetch(job.output_url).then((r) => r.text());
    console.log(`${job.total_chunks} chunks`);
    console.log(jsonl.split("\n")[0]);
}
```

---

## Error responses

| Status | Condition |
|--------|-----------|
| `202 Accepted` | The job was created and enqueued. This is the normal `POST` outcome. |
| `401 Unauthorized` | Missing or invalid API key / JWT token. |
| `402 Payment Required` | Ingest is not on your current plan, or your monthly ops allowance is exhausted. |
| `403 Forbidden` | `/v2/ingest` is not in the API key's allowed endpoints. |
| `404 Not Found` | Unknown `job_id`, or one owned by another project. |
| `409 Conflict` | `retry-webhook` called on a job that has not reached `completed`. |
| `400 Bad Request` | `retry-webhook` called with no `webhook_url` configured, or its stored URL now resolves to a private/internal address. |
| `422 Unprocessable Entity` | Source does not match mode (`urls` without `urls`, or a seed `url` in `urls` mode); a parameter is out of range; or an `include_patterns`/`exclude_patterns` regex does not compile. |
| `500 Internal Server Error` | The job could not be created. The message includes the `job_id` to quote to support. |

A page-level render failure does **not** fail the request or the job. It
increments `pages_failed`, lands the page's error in its own row, and the
job continues. A job only `fails` when discovery is rejected or every page
fails or is skipped. The full status-code reference is in
[the error-codes guide](/docs/reference/errors.md).

---

## Limits

| Limit | Value |
|-------|-------|
| URLs per `urls`-mode request | 1,000 |
| `url` / each `urls` entry length | 2,048 characters |
| `max_pages` (discovery cap) | 1–1,000 |
| `max_depth` | 1–5 |
| `include_patterns` / `exclude_patterns` | 50 each |
| `wait_for` length | 1,024 characters |
| `wait_timeout_ms` | 0–60,000 ms |
| `chunk.max_words` | 32–4,000 (default 512) |
| `chunk.sentence_overlap` | 0–10 (default 1) |
| `webhook_url` length | 2,048 characters |
| Per-job page ceiling (`MAX_PAGES_PER_JOB`) | 1,000 |
| Files per `/v2/ingest/files` request | 1–200 |
| Per-file upload size | Plan-dependent (Founding: 5 MB) |
| `GET /v2/ingest` list `limit` | 1–100 (default 20) |
| Signed `output_url` expiry | 15 minutes |
| Webhook delivery attempts | 4 (initial + 3 retries) |
| Webhook replay tolerance | 300 seconds |
| Monthly ops (shared across every endpoint, 1 per page) | 500 / 3,000 / 15,000 / 50,000 by tier; see [pricing](/pricing.md) |

---

## Frequently asked questions

### How do I crawl a website for RAG with an API?

Send `POST /v2/ingest` with `mode: "crawl"` and a seed `url`. The call answers `202` with a `job_id`; the worker discovers pages, renders each in headless Chrome, chunks the Markdown heading-aware, and assembles one JSONL file you download from the signed `output_url`.

### How do I ingest files (PDFs, Word docs) for RAG?

Send `POST /v2/ingest/files` as `multipart/form-data` with one or more `files`. Each document is converted to Markdown, chunked heading-aware, and assembled into the same single JSONL as a crawl job, so one pipeline covers web and files. PDF, DOCX, PPTX, XLSX, CSV, HTML, EPUB, OpenDocument, and plain-text/Markdown files are supported (up to 200 per request); the full list is on the [anything-to-markdown](/docs/endpoints/convert/documents/anything-to-markdown.md#supported-input-formats) page.

### Does the JSONL output load into LangChain and LlamaIndex directly?

Yes. Each line carries a top-level `content` string with a sibling `metadata` object, so the same file loads through LangChain `JSONLoader(content_key="content", json_lines=True)`, LlamaIndex `SimpleDirectoryReader`, and any line-oriented vector-DB import without reshaping.

### How does the chunker split pages into RAG chunks?

It splits on `#`, `##`, and `###` headings with a soft `max_words` cap (default `512`, range 32–4,000) and optional `sentence_overlap`. Fenced code blocks and Markdown tables are never split, and every chunk carries its full `headings_path`.

### How do I get notified when an ingest job finishes?

Set `webhook_url` on the `POST` and EnConvert sends one HMAC-signed callback (headers `X-Enconvert-Signature` and `X-Enconvert-Timestamp`) with up to three retries after the first attempt. If your endpoint was down, `POST /v2/ingest/{job_id}/retry-webhook` re-signs and re-delivers it.

### Why is output_url missing from my ingest response?

`output_url` is present only once `status` is `completed`. The `POST` answer is a `queued` job with the field omitted. Poll `GET /v2/ingest/{job_id}`, which consumes no ops and re-signs the URL on every call; each signed URL expires after 15 minutes.