---
title: "Diffbot Alternative for PDF Ingestion | EnConvert"
description: "EnConvert is a Diffbot alternative for PDF ingestion that returns heading-aware JSONL chunks with section metadata and ingests web pages through the same API"
canonical: "https://www.enconvert.com/solutions/diffbot-alternative-for-pdf-ingestion-enconvert"
locale: "en"
---

# Best Diffbot Alternative for PDF Ingestion

A developer building a RAG pipeline needs PDF ingestion that returns heading-aware chunks with section metadata, not a web extraction platform built around structured data from websites. Diffbot is an AI-powered web data extraction platform that uses computer vision and machine learning to classify pages and extract structured data from articles, products, discussions, and other web content. Its Article API can process PDFs through direct POST calls, but the output is article-shaped JSON with a text field, not heading-aware JSONL chunks with section metadata for a vector store, so a chunking step is still needed before the content is useful in a RAG pipeline. For a developer who needs to upload a PDF and receive RAG-ready JSONL chunks, that gap means a second integration. EnConvert is a REST API with a dedicated Ingest endpoint that accepts PDF uploads and returns heading-aware JSONL chunks with section metadata, alongside a Perceive endpoint for web content, so you get RAG-ready chunks from PDFs and clean Markdown from web pages in one integration.

## Example

```python
import os
import json
import time

import requests

DIFFBOT_TOKEN = os.environ["DIFFBOT_TOKEN"]
ENCONVERT_API_KEY = os.environ["ENCONVERT_API_KEY"]
PDF_PATH = "/path/to/your/annual-report.pdf"
PAGE_URL = os.environ["PAGE_URL"]  # a web page to ingest alongside the PDF

# Step 1: Diffbot, PDF POSTed to the Article API, returns article-shaped JSON
with open(PDF_PATH, "rb") as pdf_file:
    diffbot_response = requests.post(
        "https://api.diffbot.com/v3/article",
        params={
            "token": DIFFBOT_TOKEN,
            # Required by Diffbot; for an uploaded PDF it only resolves relative links
            "url": "https://example.com/annual-report.pdf",
        },
        headers={"Content-Type": "application/pdf"},
        data=pdf_file,
        timeout=120,
    )
diffbot_response.raise_for_status()
objects = diffbot_response.json().get("objects")
if not objects:
    raise RuntimeError("Diffbot returned no article object for the PDF")
text = objects[0]["text"]
print(f"Diffbot returned one text field of {len(text.split())} words")

# Output is article-shaped JSON, not heading-aware chunks
# Chunking, headings_path metadata, and JSONL formatting are still needed

# Step 2: EnConvert, upload the PDF to Ingest (multipart field is "files")
with open(PDF_PATH, "rb") as pdf_file:
    upload = requests.post(
        "https://api.enconvert.com/v2/ingest/files",
        headers={"X-API-Key": ENCONVERT_API_KEY},
        files={"files": (os.path.basename(PDF_PATH), pdf_file, "application/pdf")},
        timeout=120,
    )
upload.raise_for_status()
job_id = upload.json()["job_id"]

# Step 3: Poll until the job finishes (queued -> processing -> completed)
for _ in range(150):  # about 5 minutes
    status_response = requests.get(
        f"https://api.enconvert.com/v2/ingest/{job_id}",
        headers={"X-API-Key": ENCONVERT_API_KEY},
        timeout=30,
    )
    status_response.raise_for_status()
    job = status_response.json()
    if job["status"] == "completed":
        break
    if job["status"] in ("failed", "canceled"):
        raise RuntimeError(job.get("error_message") or f"Ingest job {job['status']}")
    time.sleep(2)
else:
    raise TimeoutError(f"Ingest job {job_id} did not finish in time")

# Step 4: Download the JSONL. output_url is signed (valid 15 minutes): plain GET, no API key.
jsonl = requests.get(job["output_url"], timeout=60)
jsonl.raise_for_status()

# One chunk per line; the file is UTF-8
pdf_chunks = [
    json.loads(line)
    for line in jsonl.content.decode("utf-8").splitlines()
    if line.strip()
]
print(f"Loaded {len(pdf_chunks)} chunks ready for embedding")

# Each chunk carries "content" plus metadata including headings_path and section
for chunk in pdf_chunks[:3]:
    print(chunk["metadata"]["headings_path"], "|", chunk["metadata"]["section"])

# Step 5: EnConvert, same API key, web page to clean Markdown via Perceive
perceive_response = requests.post(
    "https://api.enconvert.com/v2/perceive",
    headers={"X-API-Key": ENCONVERT_API_KEY, "Content-Type": "application/json"},
    json={"url": PAGE_URL, "outputs": ["markdown"], "only_main_content": True},
    timeout=60,
)
perceive_response.raise_for_status()
perceived = perceive_response.json()

if perceived.get("error") or "markdown" not in perceived.get("outputs", {}):
    raise RuntimeError(f"Perceive failed for {PAGE_URL}: {perceived.get('error')}")

# The Markdown is a signed artifact URL (valid 15 minutes). Plain GET, no API key.
markdown_response = requests.get(perceived["outputs"]["markdown"]["url"], timeout=60)
markdown_response.raise_for_status()
page_markdown = markdown_response.text
print(f"Perceived {PAGE_URL}: {len(page_markdown)} characters of Markdown")

# No second vendor for web content
```

## Output

| Capability | EnConvert | Diffbot |
| --- | --- | --- |
| Primary use case | Unified ingestion of PDFs and web pages for RAG pipelines and AI agents | AI-powered extraction of structured data from web pages, articles, products, and discussions |
| PDF ingestion | Confirmed: dedicated Ingest endpoint accepts PDF uploads and returns chunks split at H1 to H3 heading boundaries, 512 word default soft cap, tables and code blocks kept whole even when they exceed the cap; each chunk carries headings_path, section, chunk_index, and word_count | Confirmed: PDFs accepted via direct POST to Article API; returns article-shaped JSON, not chunked output |
| RAG-ready chunked output | Confirmed: chunks split at H1 to H3 heading boundaries, 512 word default soft cap, tables and code blocks kept whole even when they exceed the cap; each chunk carries headings_path, section, chunk_index, and word_count | Not documented: Article API returns extracted text as structured JSON, not heading-aware JSONL |
| Web page extraction | Perceive: any URL to clean Markdown, including JavaScript-rendered pages; Ingest also accepts URL lists, sitemaps, and website crawls returning the same JSONL chunk schema as file ingestion | Confirmed: Article, Analyze, and Product APIs return structured JSON by page type |
| Document conversion | Convert: 45+ conversion routes across documents, images, and data formats | Not documented as a dedicated conversion endpoint |
| Summarisation endpoint | Distill: any URL or file to a clean summary in one call | Article API supports optional natural language summary; no dedicated summarisation endpoint documented |
| MCP server | Native, exposes full ingestion toolkit as MCP tools | Confirmed: MCP server for web research, crawling, and Knowledge Graph search |
| Free tier | 500 ops per month, no card required | Confirmed: 10,000 credits per month, no card required |

**WHY ENCONVERT FOR PDF INGESTION**

A dedicated PDF ingestion endpoint means chunks split at heading boundaries so retrieval respects document structure, and each chunk carries a full `headings_path` breadcrumb so the agent knows exactly where in the document a result came from. The same API key covers web page extraction through Perceive, so the pipeline does not need a second vendor for web content. When the pipeline grows beyond PDFs, the same Ingest endpoint handles DOCX and XLSX with the same output schema and no integration changes.

## Get started

[Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Ingest Markdown Files into LlamaIndex with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-files-into-llamaindex-with-enconvert.md)
- [How to Ingest Web Pages into Vertex AI with EnConvert](https://www.enconvert.com/solutions/ingest-web-pages-into-vertex-ai-with-enconvert.md)
- [How to Ingest Markdown into OpenAI File Search with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-openai-file-search-enconvert.md)
- [How to Ingest HTML Files into LlamaIndex with EnConvert](https://www.enconvert.com/solutions/ingest-html-files-into-llamaindex-with-enconvert.md)
- [Best Browserless Alternative for Document Parsing](https://www.enconvert.com/solutions/browserless-alternative-for-document-parsing-enconvert.md)
- [How to Ingest DOCX into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-docx-into-crewai-with-enconvert.md)