Public keys will not work. Keys starting with
pk_ are meant for browser widgets and are rejected by the loader. Read more in
Private Keys.
---
## Using EnconvertLoader
The loader supports two modes.
### Perceive a few URLs
Turn individual URLs into markdown `Document`s:
```python
from langchain_enconvert import EnconvertLoader
loader = EnconvertLoader(urls=["https://example.com", "https://example.com/pricing"])
docs = loader.load()
```
Each Document's metadata carries:
- `source`: the URL
- `render_quality`: a number from 0.0 (blocked/empty) to 1.0 (clean render)
### Crawl a site into chunks
Crawl an entire site and return one Document per content chunk, ready for RAG:
```python
from langchain_enconvert import EnconvertLoader
loader = EnconvertLoader(
ingest_url="https://docs.example.com",
mode="sitemap", # or "crawl" or "hybrid"
max_pages=100
)
docs = loader.load() # Blocks until the crawl completes
```
Each Document's metadata carries the chunk's own `source` URL, title, and section context.
The loader polls the ingestion job to completion, so `.load()` is synchronous. For true async, use `.lazy_load()` to iterate one Document at a time.
---
## Building a RAG pipeline
Chain the loader into a text splitter and a vector store:
```python
from langchain_enconvert import EnconvertLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings
# Load and split
loader = EnconvertLoader(ingest_url="https://docs.example.com", mode="sitemap")
docs = loader.load()
# Embed and index
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = splitter.split_documents(docs)
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(chunks, embeddings)
# Query
retriever = vectorstore.as_retriever()
relevant = retriever.invoke("How do I install this?")
```
---
## Troubleshooting
**`raise ValueError("Provide either urls or ingest_url, but not both")`**
You passed both `urls=` and `ingest_url=` to the loader. Pick one: perceive individual URLs, or crawl a site. Passing neither raises an error too.
**`AuthenticationError: 401 Unauthorized`**
The API key is wrong or missing. Verify you set `api_key=` or the `$ENCONVERT_API_KEY` environment variable to a **private** key (starts with `sk_`). Public keys are rejected immediately.
**`Load time seems very long.`**
Site crawls are asynchronous. The loader polls until the job finishes, which can take several minutes for a large site. For a responsive UX, use the async API (`await loader.aload()` in an async context) or poll the job status directly via `/v2/ingest/{job_id}`.
**`render_quality is very low for pages I expected to read cleanly.`**
A low score (< 0.5) means the page is blocked, empty, or heavy JavaScript that didn't render in time. Check the raw `source` URL in a browser to see what the page actually serves. If it's a single-page app, a second render might succeed; if it's blocked, there's no retry.
---
## Source and links
- **Source**: [enconvert/langchain-enconvert](https://github.com/enconvert/langchain-enconvert)
- **Package**: [PyPI](https://pypi.org/project/langchain-enconvert)
- **Licence**: MIT
- **LangChain documentation**: [python.langchain.com](https://python.langchain.com)
- **Underlying API**: [Introduction](/docs/introduction.md)
---
## Frequently asked questions
### How do I load a single URL into a Document?
```python
from langchain_enconvert import EnconvertLoader
loader = EnconvertLoader(urls=["https://example.com"])
docs = loader.load()
```
Each Document's content is the page's markdown, and metadata carries the `source` URL and `render_quality` score.
### How do I crawl a whole site?
```python
loader = EnconvertLoader(ingest_url="https://docs.example.com", mode="sitemap", max_pages=100)
docs = loader.load()
```
The loader blocks until the crawl finishes, then returns one Document per chunk. Use `mode="crawl"` for HTTP crawling, `"sitemap"` for parsing the sitemap, or `"hybrid"` for both.
### How do I check a page's render quality?
Every Document's metadata includes `render_quality`, a score from 0.0 to 1.0. A low score means the page was blocked or empty. Branch on it before indexing:
```python
for doc in docs:
if doc.metadata.get('render_quality', 1.0) > 0.5:
vectorstore.add_documents([doc])
```
### Can I stream Documents one at a time?
Yes, use `.lazy_load()`:
```python
for doc in loader.lazy_load():
print(doc.page_content[:100])
```
### What if I need to crawl many sites?
Create one loader per site and call `.load()` sequentially, or use `.aload()` in an async context to fan out crawls in parallel.