Crawl Website for RAG API#

POST /v2/ingest crawls a website for RAG: it turns a site (or an explicit list of URLs) into RAG-ready chunks and emits one JSONL file that loads straight into LangChain JSONLoader, LlamaIndex SimpleDirectoryReader, or a vector-DB bulk import. The endpoint is always asynchronous: POST answers 202 with a job_id, you poll GET /v2/ingest/{job_id} or register a webhook_url, and a completed job hands back a pre-signed output_url for the JSONL. EnConvert does the discovery, the headless-Chrome render, the heading-aware chunking, and the JSONL assembly in one job.

Uploaded files ingest through the very same pipeline via POST /v2/ingest/files. PDF, DOCX, PPTX, XLSX, CSV, HTML, EPUB, and more are converted to Markdown, chunked, and assembled into the same JSONL. One integration covers both web and file RAG ingestion.

Here is the smallest useful call. Crawl a site and chunk every page it discovers:

curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "crawl",
    "url": "https://example.com/docs"
  }'

The response is the job record, returned with 202 Accepted. Note the status is queued, and output_url is absent until the job completes:

{
    "job_id": "ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7",
    "status": "queued",
    "mode": "crawl",
    "pages_discovered": 0,
    "pages_processed": 0,
    "pages_failed": 0,
    "total_chunks": 0,
    "webhook_delivered": false,
    "created_at": "2026-06-24T09:14:02.118Z"
}

Ingest is always asynchronous. Each page renders in a real browser, which runs 10-30 seconds per URL, far past the 300-second request window for any non-trivial job. So POST answers 202 with a job_id, and a droplet-local worker drains the job out of band. You poll GET /v2/ingest/{job_id} for progress, or register a webhook_url to be told when it finishes.

Endpoints#

Content-Type: application/json on every POST.

Authentication#

Authenticate with a private key in the X-API-Key header for server-to-server calls. This is the path the examples below use.

X-API-Key: sk_your_private_key

Public keys with a JWT bearer token also work, using the same flow as every other endpoint: generate a token with your pk_ key, then send it as Authorization: Bearer <token>. The full flow, including domain locking and token refresh, is in the authentication guide.

Each API key carries an allowed-endpoints allowlist. If /v2/ingest is not on the key's list, the request is rejected with 403. A key scoped to /v2/ingest still reaches the jobs it created: GET and DELETE /v2/ingest/{job_id} and POST /v2/ingest/{job_id}/retry-webhook are always allowed for a job_id (the ing_… shape is matched explicitly). The static list endpoint and the two webhook-secret management routes do not inherit that bypass; they require a broader or dashboard-scoped token.

How ingest works#

One job runs through five phases, all durable and restart-safe. If the worker process restarts mid-job, the job is re-enqueued at boot and resumes from the page it stopped on. Already-completed pages keep their staged output and are never re-rendered or re-billed.

1. Queue. POST validates the request, runs a fast units=1 ops check (plan has ingest enabled and headroom left in the monthly ops allowance), inserts the job row, and answers 202. Nothing is persisted if the ops gate fails: a 402 leaves zero rows behind.
2. Discover. For sitemap and crawl mode the worker runs the same discovery pass as the discover endpoint, capped at max_pages, and SSRF-screens the seed URL. For urls mode the explicit list is de-duplicated in order; no discovery runs. The pre-cap discovery size is reported as pages_found; when the site has more URLs than max_pages allowed, discovery_truncated is true and a warnings entry states both numbers, so pages_discovered (the enqueued count) is never mistaken for the site's size.
3. Render and chunk. Each URL renders through the shared headless Chrome singleton, the same render pipeline behind the perceive endpoint. The rendered HTML is then converted to fit-Markdown and sliced by the heading-aware chunker. Renders run sequentially, one page at a time.
4. Stage. Each page's chunks are written to a per-page JSONL object in storage, keyed deterministically by (project, job, url). This is what makes a restart cheap: a resumed job re-uses staged pages instead of re-rendering them.
5. Assemble. Once every page is done, the per-page objects are concatenated into the final v2-ingest/{job_id}.jsonl, the staging objects are deleted, the job flips to completed, and the signed completion webhook fires if a webhook_url was set.

The ops allowance is re-checked per page inside the worker, not just at submit time; each completed page bills one op. A crawl job whose page count is unknown up front stops cleanly at your monthly cap: the pages already rendered are billed and kept, and the remaining pages are marked skipped rather than over-spending.

Ingest renders are credential-free by design. Unlike /v2/perceive, it does not accept auth, cookies, or custom headers. Nothing secret is persisted for the durable resume, so the job state on disk never carries credentials.

Ingesting files#

POST /v2/ingest crawls the web; POST /v2/ingest/files ingests uploaded files through the very same pipeline. Both create the same job, run the same heading-aware chunker, and produce the same single JSONL deliverable, so one integration covers web and file RAG ingestion.

Send the documents as multipart/form-data in the files field. Each file is converted to Markdown by the anything-to-markdown converter, then chunked and assembled exactly like a crawled page. Every supported input format is accepted: PDF, DOCX, PPTX, XLSX, CSV, HTML, EPUB, OpenDocument, and plain text/Markdown.

curl -X POST https://api.enconvert.com/v2/ingest/files \
  -H "X-API-Key: sk_your_private_key" \
  -F "[email protected]" \
  -F "[email protected]" \
  -F "[email protected]" \
  -F "max_words=700" \
  -F "webhook_url=https://your-app.example.com/hooks/ingest"

The response is the same IngestJobResponse as the crawl endpoint, with mode set to files:

{
    "job_id": "ing_7c1d8e2f4a5b6c7d8e9f0a1b2c3d4e5f",
    "status": "queued",
    "mode": "files",
    "pages_discovered": 0,
    "pages_processed": 0,
    "pages_failed": 0,
    "total_chunks": 0,
    "webhook_delivered": false,
    "created_at": "2026-07-14T10:15:30.220Z"
}

Each uploaded file counts as one "page": it bills one op, climbs pages_processed as it completes, and is labelled by its filename in the JSONL metadata.source_url. You poll GET /v2/ingest/{job_id}, cancel with DELETE, and receive the signed completion webhook exactly as for a crawl job. Files are stored only until the JSONL is assembled, then deleted.

File request parameters#

Sent as multipart form fields (not a JSON body):

An unsupported file type, an empty file, or an image (OCR is not performed) is rejected at submit with a 400; a file over your plan's per-file size limit is a 413.

import requests

BASE = "https://api.enconvert.com"
HEADERS = {"X-API-Key": "sk_your_private_key"}

# Submit several files (always 202).
with open("handbook.pdf", "rb") as a, open("pricing.xlsx", "rb") as b:
    job = requests.post(
        f"{BASE}/v2/ingest/files",
        headers=HEADERS,
        files=[("files", ("handbook.pdf", a)), ("files", ("pricing.xlsx", b))],
        data={"max_words": 700},
    ).json()

# Poll GET /v2/ingest/{job_id} exactly as for a crawl job, then download output_url.
print(job["job_id"], job["status"], job["mode"])  # -> ing_...  queued  files

Request parameters#

Source and mode#

url and urls are mutually exclusive: send exactly one source. urls mode requires urls; sitemap and crawl require a seed url. Sending the wrong one for the mode is a 422.

Discovery (sitemap / crawl modes)#

These are forwarded to the discovery pass and ignored in urls mode.

Rendering#

Note. Ingest does not accept auth, cookies, or headers. If a page needs credentials to render, ingest is the wrong tool. Use the perceive endpoint, which carries the full authenticated-request surface, for that single page.

Chunking (chunk object)#

The chunker splits on #, ##, and ### headings, so each chunk belongs to exactly one section and carries its full heading path. Deeper headings (####–######) stay inline as content. Fenced code blocks and Markdown tables are never split, even when a single block runs over max_words; list items split between items, never mid-item.

Webhook#

Response#

POST, GET /v2/ingest/{job_id}, and DELETE all return the same IngestJobResponse object.

Note. POST and the per-job GET/DELETE use response_model_exclude_none, so fields that are still null (like output_url before completion) are omitted from the JSON rather than sent as null.

The JSONL record shape#

The final file is newline-delimited JSON. Each line is one chunk:

{"id":"9f2b8c1ad4e5-0000","content":"Pricing is usage-based...","metadata":{"source_url":"https://example.com/pricing","title":"Pricing","headings_path":["Pricing","Plans"],"section":"Plans","word_count":118,"chunk_index":0}}

The file is UTF-8, written with ensure_ascii=false, so unicode stays readable. Because content is a top-level string and metadata is a sibling object, the same file loads through LangChain JSONLoader(content_key="content", json_lines=True), LlamaIndex SimpleDirectoryReader, and any line-oriented vector-DB import without reshaping.

Job lifecycle and polling#

A job moves through these states:

queued → discovering → processing → completed | failed | canceled

Poll the status with the per-job GET. This is read-only: it consumes no ops and re-signs the output_url from the stored object key on every call:

curl https://api.enconvert.com/v2/ingest/ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_your_private_key"

An unknown job_id, or one that belongs to a different project, returns 404. Existence is never leaked across projects.

Listing jobs#

GET /v2/ingest returns this project's jobs newest-first, with skip and limit query params. limit defaults to 20 and is capped at 100. The response carries a has_more flag instead of a total count:

curl "https://api.enconvert.com/v2/ingest?skip=0&limit=20" \
  -H "X-API-Key: sk_your_private_key"

{
    "jobs": [
        {
            "job_id": "ing_3f9a...",
            "status": "completed",
            "mode": "crawl",
            "pages_discovered": 42,
            "pages_found": 42,
            "discovery_truncated": false,
            "pages_processed": 41,
            "pages_failed": 1,
            "total_chunks": 1187,
            "output_url": "https://spaces.example.com/...signed...",
            "webhook_configured": true,
            "webhook_delivered": true,
            "created_at": "2026-06-24T09:14:02.118Z",
            "completed_at": "2026-06-24T09:31:55.402Z"
        }
    ],
    "skip": 0,
    "limit": 20,
    "has_more": false
}

The list rows reduce webhook_url to a webhook_configured boolean, so the list never echoes the raw endpoint back into the table.

Canceling a job#

DELETE /v2/ingest/{job_id} sets the job's status to canceled. The worker reads that status between pages and stops without assembling output. Cancellation is idempotent and race-proof: if assembly already committed, the DELETE matches nothing and the job is returned unchanged as completed. A finished job is never clobbered back to canceled.

curl -X DELETE \
  https://api.enconvert.com/v2/ingest/ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_your_private_key"

Completion webhooks#

Set webhook_url on the POST and EnConvert sends one HMAC-signed POST when the job completes. The payload is compact, key-sorted JSON:

{"job_id":"ing_3f9a...","output_url":"https://spaces.example.com/...signed...","pages_processed":41,"status":"completed","total_chunks":1187}

Delivery retries up to three times after the first attempt, with back-off delays of 1, 4, and 16 seconds, which is four POSTs worst case. Each attempt is re-signed with a fresh timestamp, so a slow retry chain never drifts past the consumer's freshness window. A 2xx response is success. A dead endpoint is recorded as a non-delivery and raises a dashboard alert, but it never sinks an otherwise-completed job.

The webhook_url is SSRF-screened at delivery time, not at submit. A URL that resolves to a private, loopback, or metadata address is stored inertly and only rejected when EnConvert tries to POST to it.

Verifying the signature#

Each delivery carries two headers:

The signing input is the timestamp, a literal ., then the raw request body. Binding the timestamp into the MAC means a consumer that rejects stale timestamps gets replay protection for free. The default freshness window is 300 seconds. Verify in your handler:

import hashlib
import hmac
import time

SECRET = "whsec_your_signing_secret"   # from GET /v2/ingest/webhook-secret
TOLERANCE_SECONDS = 300


def verify(raw_body: bytes, signature_header: str, timestamp_header: str) -> bool:
    if not signature_header or not timestamp_header:
        return False
    try:
        ts = int(timestamp_header)
    except ValueError:
        return False
    if abs(time.time() - ts) > TOLERANCE_SECONDS:
        return False  # replayed or badly skewed clock

    provided = signature_header.removeprefix("sha256=")
    expected = hmac.new(
        SECRET.encode("utf-8"),
        f"{ts}.".encode("utf-8") + raw_body,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, provided)

Managing the signing secret#

GET /v2/ingest/webhook-secret reveals the project's secret (creating it on first call) along with the header names and tolerance your consumer needs. It is sensitive and exposed only over the authenticated dashboard channel:

{
    "secret": "whsec_...",
    "signature_header": "X-Enconvert-Signature",
    "timestamp_header": "X-Enconvert-Timestamp",
    "signature_scheme": "sha256",
    "replay_tolerance_seconds": 300,
    "rotated": false
}

POST /v2/ingest/webhook-secret/rotate issues a new secret and sets rotated to true. Every signature computed with the previous secret stops verifying the moment the rotation commits. Rotate after a suspected leak, then update your consumer.

Re-delivering a webhook#

If your endpoint was down when the job finished, POST /v2/ingest/{job_id}/retry-webhook re-signs and re-POSTs with the same retry policy:

curl -X POST \
  https://api.enconvert.com/v2/ingest/ing_3f9a.../retry-webhook \
  -H "X-API-Key: sk_your_private_key"

{
    "job_id": "ing_3f9a...",
    "delivered": true,
    "attempts": 1,
    "status_code": 200,
    "detail": "Delivered (HTTP 200)."
}

It returns 404 for an unknown or foreign job_id, 400 when no webhook_url is configured (or the stored URL now resolves to a private/internal address), and 409 when the job has not reached completed.

Code examples#

curl: explicit URL list#

curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "urls",
    "urls": [
      "https://example.com/docs/intro",
      "https://example.com/docs/quickstart",
      "https://example.com/docs/api"
    ]
  }'

curl: crawl with chunking and a webhook#

curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "crawl",
    "url": "https://example.com/docs",
    "max_pages": 200,
    "max_depth": 3,
    "include_patterns": ["/docs/"],
    "chunk": {"max_words": 700, "sentence_overlap": 2},
    "webhook_url": "https://your-app.example.com/hooks/ingest"
  }'

Python: submit, poll, download#

import time

import requests

BASE = "https://api.enconvert.com"
HEADERS = {"X-API-Key": "sk_your_private_key"}

# 1. Submit (always 202).
job = requests.post(
    f"{BASE}/v2/ingest",
    headers=HEADERS,
    json={"mode": "crawl", "url": "https://example.com/docs", "max_pages": 100},
).json()
job_id = job["job_id"]

# 2. Poll until terminal.
while True:
    job = requests.get(f"{BASE}/v2/ingest/{job_id}", headers=HEADERS).json()
    if job["status"] in ("completed", "failed", "canceled"):
        break
    time.sleep(5)

# 3. Download the JSONL from its signed URL.
if job["status"] == "completed":
    jsonl = requests.get(job["output_url"]).text
    print(f"{job['total_chunks']} chunks across "
          f"{job['pages_processed']} pages")
    print(jsonl.splitlines()[0])

Node.js: submit and poll#

const BASE = "https://api.enconvert.com";
const HEADERS = {
    "Content-Type": "application/json",
    "X-API-Key": "sk_your_private_key"
};

// 1. Submit.
const submit = await fetch(`${BASE}/v2/ingest`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({
        mode: "crawl",
        url: "https://example.com/docs",
        max_pages: 100
    })
});
let job = await submit.json();

// 2. Poll until terminal.
while (!["completed", "failed", "canceled"].includes(job.status)) {
    await new Promise((r) => setTimeout(r, 5000));
    const poll = await fetch(`${BASE}/v2/ingest/${job.job_id}`, {
        headers: { "X-API-Key": HEADERS["X-API-Key"] }
    });
    job = await poll.json();
}

// 3. Download the JSONL.
if (job.status === "completed") {
    const jsonl = await fetch(job.output_url).then((r) => r.text());
    console.log(`${job.total_chunks} chunks`);
    console.log(jsonl.split("\n")[0]);
}

Error responses#

A page-level render failure does not fail the request or the job. It increments pages_failed, lands the page's error in its own row, and the job continues. A job only fails when discovery is rejected or every page fails or is skipped. The full status-code reference is in the error-codes guide.

Limits#

Frequently asked questions#

How do I crawl a website for RAG with an API?#

Send POST /v2/ingest with mode: "crawl" and a seed url. The call answers 202 with a job_id; the worker discovers pages, renders each in headless Chrome, chunks the Markdown heading-aware, and assembles one JSONL file you download from the signed output_url.

How do I ingest files (PDFs, Word docs) for RAG?#

Send POST /v2/ingest/files as multipart/form-data with one or more files. Each document is converted to Markdown, chunked heading-aware, and assembled into the same single JSONL as a crawl job, so one pipeline covers web and files. PDF, DOCX, PPTX, XLSX, CSV, HTML, EPUB, OpenDocument, and plain-text/Markdown files are supported (up to 200 per request); the full list is on the anything-to-markdown page.

Does the JSONL output load into LangChain and LlamaIndex directly?#

Yes. Each line carries a top-level content string with a sibling metadata object, so the same file loads through LangChain JSONLoader(content_key="content", json_lines=True), LlamaIndex SimpleDirectoryReader, and any line-oriented vector-DB import without reshaping.

How does the chunker split pages into RAG chunks?#

It splits on #, ##, and ### headings with a soft max_words cap (default 512, range 32–4,000) and optional sentence_overlap. Fenced code blocks and Markdown tables are never split, and every chunk carries its full headings_path.

How do I get notified when an ingest job finishes?#

Set webhook_url on the POST and EnConvert sends one HMAC-signed callback (headers X-Enconvert-Signature and X-Enconvert-Timestamp) with up to three retries after the first attempt. If your endpoint was down, POST /v2/ingest/{job_id}/retry-webhook re-signs and re-delivers it.

Why is output_url missing from my ingest response?#

output_url is present only once status is completed. The POST answer is a queued job with the field omitted. Poll GET /v2/ingest/{job_id}, which consumes no ops and re-signs the URL on every call; each signed URL expires after 15 minutes.