---
title: "Ingest PDF into Weaviate — EnConvert API"
description: "Use EnConvert's Ingest endpoint to extract PDF content into Weaviate as clean markdown for RAG pipelines without custom PDF parsing or table extraction"
canonical: "https://www.enconvert.com/solutions/ingest-pdf-into-weaviate"
locale: "en"
---

# How to Ingest PDF into Weaviate

Building a RAG pipeline over PDF documents in Weaviate requires clean, consistently structured text before anything reaches the embedding step. Standard PDF extraction libraries handle simple text-layer PDFs adequately but produce unreliable output on the document types most common in enterprise RAG pipelines: multi-column research papers merge text across columns in reading order, scanned documents return empty strings or OCR noise, and tables lose cell boundaries and become unstructured text blocks. Each failure case requires a custom preprocessing branch that adds fragility to the pipeline without guaranteeing coverage across every document type your pipeline will encounter in production. EnConvert's Ingest endpoint handles extraction across all of these document types and returns clean markdown preserving document structure, ready to chunk and upsert into Weaviate without any additional cleaning.

## Example

```python
import requests
from openai import OpenAI
import weaviate
from weaviate.classes.config import Property, DataType

ENCONVERT_API_KEY = "[•your EnConvert API key]"
OPENAI_API_KEY = "[•your OpenAI API key]"
WEAVIATE_URL = "[•your Weaviate cluster URL]"
WEAVIATE_API_KEY = "[•your Weaviate API key]"
COLLECTION_NAME = "[•Weaviate collection name]"

# Step 1: Extract the PDF as clean markdown
with open("[•path to your PDF file]", "rb") as pdf:
    response = requests.post(
        "https://api.enconvert.com/v1/convert/anything-to-markdown",
        headers={
            "Authorization": f"Bearer {ENCONVERT_API_KEY}",
        },
        files={
            "file": ("[•PDF filename]", pdf, "application/pdf"),
        },
        data={
            "direct_download": "false",
        },
    )

response.raise_for_status()

# The API returns JSON metadata with a pre-signed URL to the .md file
markdown_response = requests.get(response.json()["presigned_url"])
markdown_response.raise_for_status()
markdown = markdown_response.text

# Optional: preview the extracted markdown before chunking
print(markdown)

# Step 2: Chunk the markdown
# Note: character-based chunking may split mid-sentence
# Consider a semantic or sentence-aware splitter for production use
chunk_size = 1000
chunks = [
    markdown[i:i + chunk_size]
    for i in range(0, len(markdown), chunk_size)
]

# Step 3: Generate embeddings using OpenAI
# text-embedding-3-small produces 1536-dimensional vectors by default
# Ensure your Weaviate collection dimension matches before upserting
openai_client = OpenAI(api_key=OPENAI_API_KEY)

embedding_response = openai_client.embeddings.create(
    model="text-embedding-3-small",
    input=chunks,
)

# Validate chunk and embedding counts match before upserting
# Using an explicit check rather than assert which is disabled
# when Python runs in optimised mode
if len(chunks) != len(embedding_response.data):
    raise ValueError(
        f"Chunk count {len(chunks)} does not match "
        f"embedding count {len(embedding_response.data)}"
    )

# Step 4: Connect to Weaviate and get the collection
# Use weaviate.connect_to_local() if running Weaviate locally
client = weaviate.connect_to_weaviate_cloud(
    cluster_url=WEAVIATE_URL,
    auth_credentials=weaviate.auth.AuthApiKey(WEAVIATE_API_KEY),
)

# Note: collection must exist before calling get()
# Create it first if it does not exist in your Weaviate instance
collection = client.collections.get(COLLECTION_NAME)

# Step 5: Upsert chunks and embeddings using batch insert
# batch.dynamic() is a Weaviate v4 feature
# Use batch.fixed_size() or batch.rate_limit() for v4 alternatives
with collection.batch.dynamic() as batch:
    for chunk, embedding in zip(chunks, embedding_response.data):
        batch.add_object(
            properties={
                "text": chunk,
                "source": "[•PDF filename]",
            },
            vector=embedding.embedding,
        )

client.close()
print(f"Upserted {len(chunks)} chunks into {COLLECTION_NAME}")
```

## Output

The Ingest endpoint returns PDF content as structured markdown: body text, headings at correct hierarchy levels, tables rendered as markdown tables with cell boundaries intact, and list content as markdown lists, with page numbers, headers, footers, and formatting artefacts stripped before the response is returned. PyPDFLoader and pdfplumber both split output by page rather than by semantic unit, which means a table or section that spans a page boundary is cut mid-content and produces two incomplete chunks that retrieve poorly. Both also fail silently on scanned pages, returning an empty string or whitespace with no error signal, so gaps in your index are invisible until a retrieval query returns no results for content that should have been indexed.

## Get started

[Get free API key — 500 ops per month, no card required](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Scrape Confluence with OpenAI Agents SDK](https://www.enconvert.com/solutions/scrape-confluence-with-openai-agents-sdk.md)
- [How to Ingest GitHub READMEs into LangChain](https://www.enconvert.com/solutions/ingest-github-readmes-into-langchain.md)
- [How to Scrape arXiv with AutoGen](https://www.enconvert.com/solutions/scrape-arxiv-with-autogen.md)
- [How to Scrape GitHub with CrewAI](https://www.enconvert.com/solutions/scrape-github-with-crewai.md)
- [How to Ingest DOCX into LangChain](https://www.enconvert.com/solutions/ingest-docx-into-langchain.md)