---
title: "Scrape Substack with LangChain | EnConvert"
description: "Scrape Substack posts into a LangChain pipeline with EnConvert Perceive, returning clean Markdown with navigation and sidebars stripped in a single batch call"
canonical: "https://www.enconvert.com/solutions/scrape-substack-with-langchain-enconvert"
locale: "en"
---

# How to Scrape Substack with LangChain

A developer building a LangChain pipeline that reads Substack posts needs clean page text before ingesting it into a vector store. Substack pages mix post content with subscription CTAs, paywall banners, subscriber counts, comment sections, related post recommendations, and author bio footers, and scraping them by hand requires CSS selectors that break when Substack updates its layout. EnConvert Perceive renders up to 10 Substack pages in a single batch call and returns clean Markdown for each one, with navigation and sidebars stripped by default. The extracted Markdown is ready to load into LangChain Documents and index with no HTML parsing.

## Example

```python
import os

import requests
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings

BASE_URL = "https://api.enconvert.com/v2"
headers = {"X-API-Key": os.environ["ENCONVERT_API_KEY"], "Content-Type": "application/json"}

post_urls = [url.strip() for url in os.environ["SUBSTACK_POST_URLS"].split(",")]  # public Substack post URLs

if len(post_urls) > 10:
    raise ValueError("This example covers inline batches of 10 URLs or fewer")

response = requests.post(f"{BASE_URL}/perceive/batch", headers=headers,
    json={"urls": post_urls, "options": {"outputs": ["markdown"]}}, timeout=300)

if response.status_code == 402:
    raise RuntimeError("Monthly ops allowance exhausted")
response.raise_for_status()

documents = []
for result in response.json()["items"]:
    page_url = result["url"]
    if result.get("status") != "completed":
        print(f"Skipped {page_url}: {result.get('status')}")
        continue
    if result.get("is_blocked"):
        print(f"Skipped {page_url}: blocked page")
        continue
    render_quality = result.get("render_quality", 1.0)
    if render_quality < 0.40:
        print(f"Skipped {page_url}: low render quality ({render_quality:.2f})")
        continue
    markdown_url = result["outputs"]["markdown"]["url"]
    markdown = requests.get(markdown_url, timeout=60)
    markdown.raise_for_status()
    documents.append(Document(
        page_content=markdown.text,
        metadata={"source_url": page_url, "render_quality": render_quality},
    ))

if not documents:
    raise RuntimeError("No Substack posts passed the render checks")

embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small", api_key=os.environ["OPENAI_API_KEY"]
)
vectorstore = FAISS.from_documents(documents, embeddings)
results = vectorstore.similarity_search("What are the main arguments made in these posts?", k=3)
for doc in results:
    print(doc.metadata["source_url"])
    print(doc.page_content[:300])
```

## Output

Perceive returns clean Markdown for each URL in the batch, with navigation and sidebars stripped by default, delivered as pre-signed URLs that you fetch once per post. Scraping Substack pages by hand needs CSS selectors that break when Substack updates its layout, and passing raw HTML into a LangChain Document fills the vector store index with subscription prompts and boilerplate that degrades retrieval quality.

## Get started

[Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Scrape Wikipedia with LlamaIndex](https://www.enconvert.com/solutions/scrape-wikipedia-with-llamaindex-enconvert.md)
- [How to Ingest Markdown into LangChain with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-langchain-with-heading-context.md)
- [Best JigsawStack Alternative for Structured Data Extraction](https://www.enconvert.com/solutions/jigsawstack-alternative-for-structured-data-enconvert.md)
- [How to Ingest Audio Transcripts into Qdrant with EnConvert](https://www.enconvert.com/solutions/ingest-audio-transcripts-into-qdrant-enconvert-api.md)
- [How to Ingest a CSV into pgvector Without Losing Row Context](https://www.enconvert.com/solutions/ingest-csv-into-pgvector-with-row-context-enconvert-api.md)
- [How to Ingest CSV into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-csv-into-crewai-with-enconvert.md)