LlamaIndex Reader for Web Pages and Site Indexing#

The EnConvert reader for LlamaIndex turns web pages and whole sites into LlamaIndex Documents. Perceive individual URLs into markdown, or crawl an entire site and return one Document per indexing-ready chunk. Every page render carries a render_quality score (0.0–1.0), so a blocked or empty page is flagged rather than silently indexed.

Install#

pip install llama-index-readers-enconvert

Add your API key#

The reader takes one credential: EnConvert API Key.

from llama_index.readers.enconvert import EnConvertReader

reader = EnConvertReader(api_key="sk_...")
# Or set $ENCONVERT_API_KEY and omit the parameter
reader = EnConvertReader()

Generate a private API key in the dashboard. Private keys start with sk_. Public pk_ keys are rejected.

Using EnConvertReader#

The reader supports two modes.

Perceive a few URLs#

Turn individual URLs into markdown Documents:

from llama_index.readers.enconvert import EnConvertReader

reader = EnConvertReader(api_key="sk_...")
docs = reader.load_data(urls=["https://example.com", "https://example.com/pricing"])

Each Document's metadata carries: - url: the source URL - render_quality: a number from 0.0 (blocked/empty) to 1.0 (clean render)

Crawl a site into chunks#

Crawl an entire site and return one Document per content chunk, ready for semantic search:

from llama_index.readers.enconvert import EnConvertReader

reader = EnConvertReader(api_key="sk_...")
docs = reader.load_data(
    ingest_url="https://docs.example.com",
    mode="sitemap",  # or "crawl" or "hybrid"
    max_pages=100
)

Each Document's metadata carries the chunk's own source URL, title, and section context. The reader polls the ingestion job to completion, so load_data() is synchronous.

Building a vector index#

Chain the reader into a vector store and query it:

from llama_index.readers.enconvert import EnConvertReader
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.vector_stores import SimpleVectorStore

# Load documents
reader = EnConvertReader(api_key="sk_...")
docs = reader.load_data(ingest_url="https://docs.example.com", mode="sitemap")

# Build index
index = VectorStoreIndex.from_documents(docs)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("How do I install this?")
print(response)

Troubleshooting#

raise ValueError("Provide either urls or ingest_url, but not both") You passed both urls= and ingest_url= to load_data(). Pick one: perceive individual URLs, or crawl a site. Passing neither raises an error too.

AuthenticationError: 401 Unauthorized The API key is wrong or missing. Verify you passed api_key="sk_..." or set $ENCONVERT_API_KEY to a private key (starts with sk_). Public keys are rejected immediately.

Load time seems very long. Site crawls are asynchronous. The reader polls until the job finishes, which can take several minutes for a large site. For a responsive UX, poll the job status directly via /v2/ingest/{job_id} or use the async variant (if available).

render_quality is very low. A low score (< 0.5) means the page is blocked, empty, or heavy JavaScript that didn't render in time. Check the source URL in a browser to see what the page actually serves. If it's a single-page app, a second render might succeed; if it's blocked, there's no retry.

Source and links#

• Source: enconvert/llama-index-readers-enconvert
• Package: PyPI
• Licence: MIT
• LlamaIndex documentation: docs.llamaindex.ai
• Underlying API: Introduction

Frequently asked questions#

How do I load a single URL into a Document?#

from llama_index.readers.enconvert import EnConvertReader
reader = EnConvertReader(api_key="sk_...")
docs = reader.load_data(urls=["https://example.com"])

Each Document's content is the page's markdown, and metadata carries the url and render_quality score.

How do I crawl and index a whole site?#

from llama_index.readers.enconvert import EnConvertReader
from llama_index.core import VectorStoreIndex

reader = EnConvertReader(api_key="sk_...")
docs = reader.load_data(ingest_url="https://docs.example.com", mode="sitemap", max_pages=100)
index = VectorStoreIndex.from_documents(docs)

# Query
engine = index.as_query_engine()
response = engine.query("What is this?")

How do I check a page's render quality before indexing?#

Every Document's metadata includes render_quality, a score from 0.0 to 1.0. Filter documents before indexing:

high_quality = [doc for doc in docs if doc.metadata.get('render_quality', 1.0) > 0.5]
index = VectorStoreIndex.from_documents(high_quality)

Can I crawl multiple sites?#

Yes. Create one reader per site and call load_data() sequentially for each, or parallelize the calls in your own async code.

Why is my crawl taking so long?#

Site crawls are asynchronous and polled. Large sites (100+ pages) can take several minutes. The reader blocks until done, returning one Document per indexed chunk.