A developer building a LangChain pipeline that reads HubSpot knowledge base or blog pages needs clean page text before passing it into a retriever or chain. HubSpot pages mix the article with navigation, subscription prompts, related post carousels, and footer links, and scraping them by hand means CSS selectors that break when HubSpot updates its layout. EnConvert Perceive renders up to 10 HubSpot pages in a single batch call and returns clean Markdown for each one, with navigation and sidebars stripped by default. The extracted Markdown is ready to load into LangChain Documents with no HTML parsing.
How to Scrape HubSpot with LangChain
Scrape HubSpot knowledge base and blog pages into a LangChain pipeline with EnConvert Perceive, returning clean Markdown with navigation stripped per batch call
Get API keyExample
python
import os
import requests
from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
BASE_URL = "https://api.enconvert.com/v2"
headers = {
"X-API-Key": os.environ["ENCONVERT_API_KEY"],
"Content-Type": "application/json",
}
page_urls = [url.strip() for url in os.environ["HUBSPOT_PAGE_URLS"].split(",")] # public HubSpot knowledge base article or blog post URLs
if len(page_urls) > 10:
raise ValueError("This example covers inline batches of 10 URLs or fewer")
response = requests.post(
f"{BASE_URL}/perceive/batch",
headers=headers,
json={
"urls": page_urls,
"options": {"outputs": ["markdown"]},
},
timeout=300,
)
if response.status_code == 402:
raise RuntimeError("Monthly ops allowance exhausted")
response.raise_for_status()
documents = []
for result in response.json()["items"]:
page_url = result["url"]
if result.get("status") != "completed":
print(f"Skipped {page_url}: {result.get('status')}")
continue
if result.get("is_blocked"):
print(f"Skipped {page_url}: blocked page")
continue
render_quality = result.get("render_quality", 1.0)
if render_quality < 0.40:
print(f"Skipped {page_url}: low render quality ({render_quality:.2f})")
continue
# Markdown is a pre-signed URL; fetch the content in a second request
markdown_url = result["outputs"]["markdown"]["url"]
markdown = requests.get(markdown_url, timeout=60)
markdown.raise_for_status()
documents.append(
Document(
page_content=markdown.text,
metadata={"source_url": page_url, "render_quality": render_quality},
)
)
if not documents:
raise RuntimeError("No HubSpot pages passed the render checks")
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small", api_key=os.environ["OPENAI_API_KEY"]
)
vectorstore = FAISS.from_documents(documents, embeddings)
results = vectorstore.similarity_search("Summarise what these HubSpot pages cover.", k=3)
for doc in results:
print(doc.metadata["source_url"])
print(doc.page_content[:300])What you get back
Perceive returns clean Markdown for each URL in the batch, with navigation and sidebars stripped by default, delivered as pre-signed URLs that you fetch once per page. Scraping HubSpot pages by hand needs CSS selectors that break when HubSpot updates its layout, and passing raw HTML into a LangChain retriever fills the vector store with navigation and boilerplate that degrades retrieval quality.