LangChain Loader for Web Pages and Site Crawling#

The EnConvert loader for LangChain turns web pages and whole sites into LangChain Documents. Perceive individual URLs into markdown, or crawl an entire site and return one Document per RAG-ready chunk. Every page render carries a render_quality score (0.0–1.0) in its metadata, so a blocked or empty page is flagged rather than silently trusted.

Install#

pip install langchain-enconvert

Add your API key#

The loader takes one credential: EnConvert API Key.

from langchain_enconvert import EnconvertLoader

loader = EnconvertLoader(api_key="sk_...")
# Or set $ENCONVERT_API_KEY and omit the parameter
loader = EnconvertLoader()

Generate a private API key in the dashboard. Private keys start with sk_. Public pk_ keys are rejected.

Using EnconvertLoader#

The loader supports two modes.

Perceive a few URLs#

Turn individual URLs into markdown Documents:

from langchain_enconvert import EnconvertLoader

loader = EnconvertLoader(urls=["https://example.com", "https://example.com/pricing"])
docs = loader.load()

Each Document's metadata carries: - source: the URL - render_quality: a number from 0.0 (blocked/empty) to 1.0 (clean render)

Crawl a site into chunks#

Crawl an entire site and return one Document per content chunk, ready for RAG:

from langchain_enconvert import EnconvertLoader

loader = EnconvertLoader(
    ingest_url="https://docs.example.com",
    mode="sitemap",  # or "crawl" or "hybrid"
    max_pages=100
)
docs = loader.load()  # Blocks until the crawl completes

Each Document's metadata carries the chunk's own source URL, title, and section context.

The loader polls the ingestion job to completion, so .load() is synchronous. For true async, use .lazy_load() to iterate one Document at a time.

Building a RAG pipeline#

Chain the loader into a text splitter and a vector store:

from langchain_enconvert import EnconvertLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

# Load and split
loader = EnconvertLoader(ingest_url="https://docs.example.com", mode="sitemap")
docs = loader.load()

# Embed and index
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)

# Query
retriever = vectorstore.as_retriever()
relevant = retriever.invoke("How do I install this?")

Troubleshooting#

raise ValueError("Provide either urls or ingest_url, but not both") You passed both urls= and ingest_url= to the loader. Pick one: perceive individual URLs, or crawl a site. Passing neither raises an error too.

AuthenticationError: 401 Unauthorized The API key is wrong or missing. Verify you set api_key= or the $ENCONVERT_API_KEY environment variable to a private key (starts with sk_). Public keys are rejected immediately.

Load time seems very long. Site crawls are asynchronous. The loader polls until the job finishes, which can take several minutes for a large site. For a responsive UX, use the async API (await loader.aload() in an async context) or poll the job status directly via /v2/ingest/{job_id}.

render_quality is very low for pages I expected to read cleanly. A low score (< 0.5) means the page is blocked, empty, or heavy JavaScript that didn't render in time. Check the raw source URL in a browser to see what the page actually serves. If it's a single-page app, a second render might succeed; if it's blocked, there's no retry.

Source and links#

• Source: enconvert/langchain-enconvert
• Package: PyPI
• Licence: MIT
• LangChain documentation: python.langchain.com
• Underlying API: Introduction

Frequently asked questions#

How do I load a single URL into a Document?#

from langchain_enconvert import EnconvertLoader
loader = EnconvertLoader(urls=["https://example.com"])
docs = loader.load()

Each Document's content is the page's markdown, and metadata carries the source URL and render_quality score.

How do I crawl a whole site?#

loader = EnconvertLoader(ingest_url="https://docs.example.com", mode="sitemap", max_pages=100)
docs = loader.load()

The loader blocks until the crawl finishes, then returns one Document per chunk. Use mode="crawl" for HTTP crawling, "sitemap" for parsing the sitemap, or "hybrid" for both.

How do I check a page's render quality?#

Every Document's metadata includes render_quality, a score from 0.0 to 1.0. A low score means the page was blocked or empty. Branch on it before indexing:

for doc in docs:
    if doc.metadata.get('render_quality', 1.0) > 0.5:
        vectorstore.add_documents([doc])

Can I stream Documents one at a time?#

Yes, use .lazy_load():

for doc in loader.lazy_load():
    print(doc.page_content[:100])

What if I need to crawl many sites?#

Create one loader per site and call .load() sequentially, or use .aload() in an async context to fan out crawls in parallel.