A developer building a RAG pipeline needs to get PDF content into Pinecone as clean, embeddable text. Most PDF extraction libraries handle straightforward text-layer PDFs acceptably but produce inconsistent output as soon as the document type changes: research papers with two-column layouts merge content across columns, scanned documents return empty strings or garbled characters, and formatted financial reports lose table structure entirely. Each edge case requires custom handling code that adds pipeline complexity and maintenance overhead. EnConvert's Ingest endpoint handles the extraction step across all of these document types and returns clean markdown, ready to chunk and embed into Pinecone without any additional preprocessing or cleaning logic.
How to Ingest PDF into Pinecone
Use EnConvert's Ingest endpoint to extract clean markdown from any PDF and upsert into Pinecone for RAG pipelines without custom PDF parsing or extraction logic
Get API keyExample
import requests
from openai import OpenAI
from pinecone import Pinecone
ENCONVERT_API_KEY = "sk_your_enconvert_api_key"
OPENAI_API_KEY = "sk-your_openai_api_key"
PINECONE_API_KEY = "your_pinecone_api_key"
PINECONE_INDEX = "enconvert-docs"
# Step 1: Extract the PDF as clean markdown
with open("[•path to your PDF file]", "rb") as pdf:
response = requests.post(
"https://api.enconvert.com/v2/ingest/files",
headers={
"X-API-Key": ENCONVERT_API_KEY,
},
files={
"file": ("[•PDF filename]", pdf, "application/pdf"),
},
)
response.raise_for_status()
# The API returns JSON metadata with a pre-signed URL to the .md file
markdown_response = requests.get(response.json()["presigned_url"])
markdown_response.raise_for_status()
markdown = markdown_response.text
# Preview the extracted markdown before chunking
print(markdown)
# Step 2: Chunk the markdown for embedding
# Note: character-based chunking may split mid-sentence
# Consider a semantic or sentence-aware splitter for production use
chunk_size = 1000
chunks = [
markdown[i:i + chunk_size]
for i in range(0, len(markdown), chunk_size)
]
# Step 3: Generate embeddings using OpenAI
# text-embedding-3-small produces 1536-dimensional vectors by
# Ensure your Pinecone index dimension matches before upserting
openai_client = OpenAI(api_key=OPENAI_API_KEY)
embedding_response = openai_client.embeddings.create(
input=chunks,
model="text-embedding-3-small",
)
# Step 4: Upsert vectors into Pinecone
# batch_size splits the list into one request per 100 vectors
pc = Pinecone(api_key=PINECONE_API_KEY)
index = pc.Index(PINECONE_INDEX)
vectors = [
{
"id": f"chunk-{i}",
"values": record.embedding,
"metadata": {"text": chunks[i]},
}
for i, record in enumerate(embedding_response.data)
]
index.upsert(vectors=vectors, batch_size=100)
print(f"Upserted {len(vectors)} chunks into {PINECONE_INDEX}")What you get back
The Ingest endpoint returns PDF content as structured markdown: body text, headings at correct hierarchy levels, tables rendered as markdown tables with rows and columns preserved, and list content with proper indentation, with page numbers, headers, footers, and formatting artefacts stripped before the response is returned. PyPDF2 extracts only raw text without structure and fails silently on scanned pages where no text layer exists. pdfplumber requires custom logic for multi-column layouts where text order breaks across columns and tables where cell boundaries are inferred from whitespace rather than structure. Neither produces output that is consistently ready to embed without an additional cleaning step before embedding.