---
title: "Ingest a Research Paper into LlamaIndex | EnConvert"
description: "Use EnConvert to ingest a research paper into LlamaIndex as heading-aware chunks with section metadata, ready to load as nodes and filter by section in one job"
canonical: "https://www.enconvert.com/solutions/ingest-a-research-paper-into-llamaindex-enconvert"
locale: "en"
---

# How to Ingest a Research Paper into LlamaIndex with EnConvert

A developer building a LlamaIndex agent or RAG pipeline that reads research papers needs heading-aware chunks with section metadata, not flat page text that loses the paper’s structure. Loading a research paper with PyPDFLoader or pdfplumber returns text page by page with no heading hierarchy and no section boundaries. Nothing populates metadata a LlamaIndex retriever can filter on. For an agent that must answer questions about methodology, results, or related work, flat page text without section context produces retrieval results the agent cannot attribute to a specific part of the paper. EnConvert’s Ingest endpoint takes the uploaded paper and returns JSONL chunks split at H1 to H3 heading boundaries, each carrying its full headings_path breadcrumb, section name, and chunk index in metadata, so the content is ready to load as LlamaIndex Document objects and retrieve in one call.

## Example

```python
import json
import os
import time

import requests
from llama_index.core import VectorStoreIndex
from llama_index.core.schema import TextNode
from llama_index.core.vector_stores import (
    FilterOperator,
    MetadataFilter,
    MetadataFilters,
)
from llama_index.embeddings.openai import OpenAIEmbedding

# No Content-Type here: requests sets the multipart boundary for the upload
HEADERS = {"X-API-Key": os.environ["ENCONVERT_API_KEY"]}

# Step 1: Upload the research paper
# Text-based PDFs only; scanned or image-only papers are not processed
with open("/path/to/your/attention-based-retrieval-paper.pdf", "rb") as paper:
    upload = requests.post(
        "https://api.enconvert.com/v2/ingest/files",
        headers=HEADERS,
        files={"files": ("attention-based-retrieval-paper.pdf", paper, "application/pdf")},
        timeout=60,
    )
upload.raise_for_status()
job_id = upload.json()["job_id"]

# Step 2: Poll until the job completes
while True:
    status_response = requests.get(
        f"https://api.enconvert.com/v2/ingest/{job_id}",
        headers=HEADERS,
        timeout=60,
    )
    status_response.raise_for_status()
    job = status_response.json()
    if job["status"] == "completed":
        break
    if job["status"] in ("failed", "canceled"):
        raise RuntimeError(job.get("error_message") or f"Ingest job {job['status']}")
    time.sleep(3)

# Step 3: Download the JSONL output
# The output_url is signed (valid 15 minutes). Plain GET, no API key.
jsonl_response = requests.get(job["output_url"], timeout=60)
jsonl_response.raise_for_status()

chunks = [
    json.loads(line)
    for line in jsonl_response.content.decode("utf-8").split("\n")
    if line.strip()
]
print(f"Loaded {len(chunks)} chunks ready for embedding")

# Step 4: Turn each chunk into a TextNode with its metadata preserved
# Nodes go straight into the index, so LlamaIndex does not re-split the chunks.
# headings_path is a list, so it is joined into one string for the node metadata.
nodes = [
    TextNode(
        id_=chunk["id"],
        text=chunk["content"],
        metadata={
            "source_url": chunk["metadata"]["source_url"],
            "title": chunk["metadata"]["title"],
            "section": chunk["metadata"]["section"],
            "headings_path": " > ".join(chunk["metadata"]["headings_path"]),
            "chunk_index": chunk["metadata"]["chunk_index"],
            "word_count": chunk["metadata"]["word_count"],
        },
        excluded_embed_metadata_keys=["chunk_index", "word_count", "source_url", "headings_path"],
    )
    for chunk in chunks
]

# Step 5: Build the vector index
index = VectorStoreIndex(
    nodes=nodes,
    embed_model=OpenAIEmbedding(
        model="text-embedding-3-small",
        api_key=os.environ["OPENAI_API_KEY"],
    ),
)

# Step 6: Retrieve restricted to one section by a metadata filter
# The section values found in this paper (use one of them in the filter below)
print(sorted({node.metadata["section"] for node in nodes}))

filters = MetadataFilters(
    filters=[
        MetadataFilter(
            key="section",
            operator=FilterOperator.EQ,
            value="Experiments",  # replace with a heading from your own paper
        )
    ]
)

retriever = index.as_retriever(
    similarity_top_k=4,
    filters=filters,
)

results = retriever.retrieve("What evaluation datasets and metrics were used?")
for result in results:
    print(result.node.metadata["headings_path"])
    print(result.node.metadata["section"])
    print(result.node.get_content()[:200])
```

## Output

Ingest returns the research paper as JSONL chunks split at H1 to H3 heading boundaries, each carrying its headings_path breadcrumb, section name, and chunk index, so LlamaIndex retrieval can filter by section before vector search runs. Loading a paper with PyPDFLoader or pdfplumber gives flat page text with no heading hierarchy, no section boundaries, and no metadata a LlamaIndex retriever can filter on. The endpoint processes text-based PDFs only; scanned or image-only papers are not processed.

## Get started

[Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Ingest Markdown Files into LlamaIndex with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-files-into-llamaindex-with-enconvert.md)
- [How to Ingest Web Pages into Vertex AI with EnConvert](https://www.enconvert.com/solutions/ingest-web-pages-into-vertex-ai-with-enconvert.md)
- [How to Ingest Markdown into OpenAI File Search with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-openai-file-search-enconvert.md)
- [How to Ingest HTML Files into LlamaIndex with EnConvert](https://www.enconvert.com/solutions/ingest-html-files-into-llamaindex-with-enconvert.md)
- [Best Browserless Alternative for Document Parsing](https://www.enconvert.com/solutions/browserless-alternative-for-document-parsing-enconvert.md)
- [How to Ingest DOCX into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-docx-into-crewai-with-enconvert.md)