Ingest

POST /v2/ingest turns a site — or an explicit list of URLs — into RAG-ready chunks and emits one JSONL file that loads straight into LangChain JSONLoader, LlamaIndex SimpleDirectoryReader, or a vector-DB bulk import. It replaces the two-step stack you would otherwise build: Firecrawl /crawl to discover and scrape, plus your own chunker to slice the Markdown. EnConvert does the discovery, the headless-Chrome render, the heading-aware chunking, and the JSONL assembly in one job.

Here is the smallest useful call. Crawl a site and chunk every page it discovers:

curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_live_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "crawl",
    "url": "https://example.com/docs"
  }'

The response is the job record, returned with 202 Accepted. Note the status is queued, and output_url is absent until the job completes:

{
    "job_id": "ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7",
    "status": "queued",
    "mode": "crawl",
    "pages_discovered": 0,
    "pages_processed": 0,
    "pages_failed": 0,
    "total_chunks": 0,
    "webhook_delivered": false,
    "created_at": "2026-06-24T09:14:02.118Z"
}

Ingest is always asynchronous. Each page renders in a real browser, which runs 10–30 seconds per URL — far past the 300-second request window for any non-trivial job. So POST answers 202 with a job_id, and a droplet-local worker drains the job out of band. You poll GET /v2/ingest/{job_id} for progress, or register a webhook_url to be told when it finishes.

Worth flagging up front. /v2/ingest is a V2 endpoint with its own quota counter, ingest_pages. V1 conversion plans include no ingest allowance — you need a V2-inclusive plan to call it. One page is billed for every URL whose render, chunk, and JSONL stage completes; pages that fail or get skipped are not billed. See the pricing page for per-tier allowances, and the V2 overview for how the V2 counters differ from V1 conversions.

Endpoints

Content-Type: application/json on every POST.

Authentication

Authenticate with a private key in the X-API-Key header for server-to-server calls. This is the path the examples below use.

X-API-Key: sk_live_your_private_key

Public keys with a JWT bearer token also work, using the same flow as every other endpoint — generate a token with your pk_ key, then send it as Authorization: Bearer <token>. The full flow, including domain locking and token refresh, is in the authentication guide.

Each API key carries an allowed-endpoints allowlist. If /v2/ingest is not on the key's list, the request is rejected with 403. A key scoped to /v2/ingest still reaches the jobs it created — GET and DELETE /v2/ingest/{job_id} and POST /v2/ingest/{job_id}/retry-webhook are always allowed for a job_id (the ing_… shape is matched explicitly). The static list endpoint and the two webhook-secret management routes do not inherit that bypass; they require a broader or dashboard-scoped token.

How ingest works

One job runs through five phases, all durable and restart-safe. If the worker process restarts mid-job, the job is re-enqueued at boot and resumes from the page it stopped on — already-completed pages keep their staged output and are never re-rendered or re-billed.

1. Queue. POST validates the request, runs a fast units=1 quota check (plan has ingest enabled and some monthly headroom), inserts the job row, and answers 202. Nothing is persisted if the quota gate fails — a 402 leaves zero rows behind.
2. Discover. For sitemap and crawl mode the worker runs the same discovery pass as the discover endpoint, capped at max_pages, and SSRF-screens the seed URL. For urls mode the explicit list is de-duplicated in order; no discovery runs.
3. Render and chunk. Each URL renders through the shared headless Chrome singleton — the same render pipeline behind the perceive endpoint — then the rendered HTML is converted to fit-Markdown and sliced by the heading-aware chunker. Renders run sequentially, one page at a time.
4. Stage. Each page's chunks are written to a per-page JSONL object in storage, keyed deterministically by (project, job, url). This is what makes a restart cheap: a resumed job re-uses staged pages instead of re-rendering them.
5. Assemble. Once every page is done, the per-page objects are concatenated into the final v2-ingest/{job_id}.jsonl, the staging objects are deleted, the job flips to completed, and — if a webhook_url was set — the signed completion webhook fires.

The quota counter is re-checked per page inside the worker, not just at submit time. A crawl job whose page count is unknown up front stops cleanly at your monthly cap: the pages already rendered are billed and kept, and the remaining pages are marked skipped rather than over-spending.

Ingest renders are credential-free by design. Unlike /v2/perceive, it does not accept auth, cookies, or custom headers — nothing secret is persisted for the durable resume, so the job state on disk never carries credentials.

Request parameters

Source and mode

url and urls are mutually exclusive — exactly one source. urls mode requires urls; sitemap and crawl require a seed url. Sending the wrong one for the mode is a 422.

Discovery (sitemap / crawl modes)

These are forwarded to the discovery pass and ignored in urls mode.

Rendering

Note. Ingest does not accept auth, cookies, or headers. If a page needs credentials to render, ingest is the wrong tool — use the perceive endpoint, which carries the full authenticated-request surface, for that single page.

Chunking (chunk object)

The chunker splits on #, ##, and ### headings — each chunk belongs to exactly one section and carries its full heading path. Deeper headings (####–######) stay inline as content. Fenced code blocks and Markdown tables are never split, even when a single block runs over max_words; list items split between items, never mid-item.

Webhook

Response

POST, GET /v2/ingest/{job_id}, and DELETE all return the same IngestJobResponse object.

Note. POST and the per-job GET/DELETE use response_model_exclude_none, so fields that are still null (like output_url before completion) are omitted from the JSON rather than sent as null.

The JSONL record shape

The final file is newline-delimited JSON. Each line is one chunk:

{"id":"9f2b8c1ad4e5-0000","content":"Pricing is usage-based...","metadata":{"source_url":"https://example.com/pricing","title":"Pricing","headings_path":["Pricing","Plans"],"section":"Plans","word_count":118,"chunk_index":0}}

The file is UTF-8, written with ensure_ascii=false, so unicode stays readable. Because content is a top-level string and metadata is a sibling object, the same file loads through LangChain JSONLoader(content_key="content", json_lines=True), LlamaIndex SimpleDirectoryReader, and any line-oriented vector-DB import without reshaping.

Job lifecycle and polling

A job moves through these states:

queued → discovering → processing → completed | failed | canceled

Poll the status with the per-job GET. This is read-only — it consumes no quota and re-signs the output_url from the stored object key on every call:

curl https://api.enconvert.com/v2/ingest/ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_live_your_private_key"

An unknown job_id, or one that belongs to a different project, returns 404 — existence is never leaked across projects.

Listing jobs

GET /v2/ingest returns this project's jobs newest-first, with skip and limit query params. limit defaults to 20 and is capped at 100. The response carries a has_more flag instead of a total count:

curl "https://api.enconvert.com/v2/ingest?skip=0&limit=20" \
  -H "X-API-Key: sk_live_your_private_key"

{
    "jobs": [
        {
            "job_id": "ing_3f9a...",
            "status": "completed",
            "mode": "crawl",
            "pages_discovered": 42,
            "pages_processed": 41,
            "pages_failed": 1,
            "total_chunks": 1187,
            "output_url": "https://spaces.example.com/...signed...",
            "webhook_configured": true,
            "webhook_delivered": true,
            "created_at": "2026-06-24T09:14:02.118Z",
            "completed_at": "2026-06-24T09:31:55.402Z"
        }
    ],
    "skip": 0,
    "limit": 20,
    "has_more": false
}

The list rows reduce webhook_url to a webhook_configured boolean — the list never echoes the raw endpoint back into the table.

Canceling a job

DELETE /v2/ingest/{job_id} sets the job's status to canceled. The worker reads that status between pages and stops without assembling output. Cancellation is idempotent and race-proof: if assembly already committed, the DELETE matches nothing and the job is returned unchanged as completed — a finished job is never clobbered back to canceled.

curl -X DELETE \
  https://api.enconvert.com/v2/ingest/ing_3f9a2c1b8e7d4a6f90b1c2d3e4f5a6b7 \
  -H "X-API-Key: sk_live_your_private_key"

Completion webhooks

Set webhook_url on the POST and EnConvert sends one HMAC-signed POST when the job completes. The payload is compact, key-sorted JSON:

{"job_id":"ing_3f9a...","output_url":"https://spaces.example.com/...signed...","pages_processed":41,"status":"completed","total_chunks":1187}

Delivery retries up to three times after the first attempt, with back-off delays of 1, 4, and 16 seconds — four POSTs worst case. Each attempt is re-signed with a fresh timestamp, so a slow retry chain never drifts past the consumer's freshness window. A 2xx response is success; a dead endpoint is recorded as a non-delivery (and raises a dashboard alert), it never sinks an otherwise-completed job.

The webhook_url is SSRF-screened at delivery time, not at submit. A URL that resolves to a private, loopback, or metadata address is stored inertly and only rejected when EnConvert tries to POST to it.

Verifying the signature

Each delivery carries two headers:

The signing input is the timestamp, a literal ., then the raw request body. Binding the timestamp into the MAC means a consumer that rejects stale timestamps gets replay protection for free — the default freshness window is 300 seconds. Verify in your handler:

import hashlib
import hmac
import time

SECRET = "whsec_your_signing_secret"   # from GET /v2/ingest/webhook-secret
TOLERANCE_SECONDS = 300


def verify(raw_body: bytes, signature_header: str, timestamp_header: str) -> bool:
    if not signature_header or not timestamp_header:
        return False
    try:
        ts = int(timestamp_header)
    except ValueError:
        return False
    if abs(time.time() - ts) > TOLERANCE_SECONDS:
        return False  # replayed or badly skewed clock

    provided = signature_header.removeprefix("sha256=")
    expected = hmac.new(
        SECRET.encode("utf-8"),
        f"{ts}.".encode("utf-8") + raw_body,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, provided)

Managing the signing secret

GET /v2/ingest/webhook-secret reveals the project's secret (creating it on first call) along with the header names and tolerance your consumer needs. It is sensitive and exposed only over the authenticated dashboard channel:

{
    "secret": "whsec_...",
    "signature_header": "X-Enconvert-Signature",
    "timestamp_header": "X-Enconvert-Timestamp",
    "signature_scheme": "sha256",
    "replay_tolerance_seconds": 300,
    "rotated": false
}

POST /v2/ingest/webhook-secret/rotate issues a new secret and sets rotated to true. Every signature computed with the previous secret stops verifying the moment the rotation commits — rotate after a suspected leak, then update your consumer.

Re-delivering a webhook

If your endpoint was down when the job finished, POST /v2/ingest/{job_id}/retry-webhook re-signs and re-POSTs with the same retry policy:

curl -X POST \
  https://api.enconvert.com/v2/ingest/ing_3f9a.../retry-webhook \
  -H "X-API-Key: sk_live_your_private_key"

{
    "job_id": "ing_3f9a...",
    "delivered": true,
    "attempts": 1,
    "status_code": 200,
    "detail": "Delivered (HTTP 200)."
}

It returns 404 for an unknown or foreign job_id, 400 when no webhook_url is configured (or the stored URL now resolves to a private/internal address), and 409 when the job has not reached completed.

Code examples

curl — explicit URL list

curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_live_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "urls",
    "urls": [
      "https://example.com/docs/intro",
      "https://example.com/docs/quickstart",
      "https://example.com/docs/api"
    ]
  }'

curl — crawl with chunking and a webhook

curl -X POST https://api.enconvert.com/v2/ingest \
  -H "X-API-Key: sk_live_your_private_key" \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "crawl",
    "url": "https://example.com/docs",
    "max_pages": 200,
    "max_depth": 3,
    "include_patterns": ["/docs/"],
    "chunk": {"max_words": 700, "sentence_overlap": 2},
    "webhook_url": "https://your-app.example.com/hooks/ingest"
  }'

Python — submit, poll, download

import time

import requests

BASE = "https://api.enconvert.com"
HEADERS = {"X-API-Key": "sk_live_your_private_key"}

# 1. Submit (always 202).
job = requests.post(
    f"{BASE}/v2/ingest",
    headers=HEADERS,
    json={"mode": "crawl", "url": "https://example.com/docs", "max_pages": 100},
).json()
job_id = job["job_id"]

# 2. Poll until terminal.
while True:
    job = requests.get(f"{BASE}/v2/ingest/{job_id}", headers=HEADERS).json()
    if job["status"] in ("completed", "failed", "canceled"):
        break
    time.sleep(5)

# 3. Download the JSONL from its signed URL.
if job["status"] == "completed":
    jsonl = requests.get(job["output_url"]).text
    print(f"{job['total_chunks']} chunks across "
          f"{job['pages_processed']} pages")
    print(jsonl.splitlines()[0])

Node.js — submit and poll

const BASE = "https://api.enconvert.com";
const HEADERS = {
    "Content-Type": "application/json",
    "X-API-Key": "sk_live_your_private_key"
};

// 1. Submit.
const submit = await fetch(`${BASE}/v2/ingest`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({
        mode: "crawl",
        url: "https://example.com/docs",
        max_pages: 100
    })
});
let job = await submit.json();

// 2. Poll until terminal.
while (!["completed", "failed", "canceled"].includes(job.status)) {
    await new Promise((r) => setTimeout(r, 5000));
    const poll = await fetch(`${BASE}/v2/ingest/${job.job_id}`, {
        headers: { "X-API-Key": HEADERS["X-API-Key"] }
    });
    job = await poll.json();
}

// 3. Download the JSONL.
if (job.status === "completed") {
    const jsonl = await fetch(job.output_url).then((r) => r.text());
    console.log(`${job.total_chunks} chunks`);
    console.log(jsonl.split("\n")[0]);
}

Plan gating

V2 ingest is metered by its own counter, ingest_pages — separate from V1 conversions and from the other V2 counters. One page is billed for every URL whose render, chunk, and JSONL stage completes; pages that fail or get skipped are not billed.

The submit-time check is a fast units=1 gate: your plan must have ingest enabled, with some monthly headroom. The real spend limiter is the per-page check inside the worker — a crawl job stops cleanly at your monthly cap and marks the remaining pages skipped rather than over-running it.

MAX_PAGES_PER_JOB (1,000) is a per-request ceiling that bounds a single job; it is not your monthly allowance. The monthly ingest_pages quota is the real spend limiter. Current per-tier numbers live on the pricing page; see also how perceive operations are metered for the sibling V2 counter.

Error responses

A page-level render failure does not fail the request or the job — it increments pages_failed, lands the page's error in its own row, and the job continues. A job only fails when discovery is rejected or every page fails or is skipped. The full status-code reference is in the error-codes guide.

Limits