A developer building a LlamaIndex pipeline that reads Zendesk support articles needs clean article text before indexing. Zendesk Help Centre pages mix the article with navigation, related article links, and interface elements, and scraping them by hand means CSS selectors that break when Zendesk updates its layout. EnConvert Perceive renders up to 10 Zendesk pages in a single batch call and returns clean Markdown for each one, with navigation and sidebars stripped by default. The extracted Markdown is ready to load into LlamaIndex TextNodes with no HTML parsing.
How to Scrape Zendesk with LlamaIndex
Use EnConvert Perceive to scrape Zendesk support articles into a LlamaIndex pipeline as clean Markdown, with up to 10 pages rendered in a single batch call
Get API keyExample
python
import os
import requests
from llama_index.core import VectorStoreIndex
from llama_index.core.schema import TextNode
# LlamaIndex uses OpenAI for embeddings and answers, and reads OPENAI_API_KEY from the environment
BASE_URL = "https://api.enconvert.com/v2"
headers = {
"X-API-Key": os.environ["ENCONVERT_API_KEY"],
"Content-Type": "application/json",
}
# A batch of 10 URLs or fewer returns inline (HTTP 200).
# More than 10 returns HTTP 202 with a job_id to poll.
article_urls = [url.strip() for url in os.environ["ZENDESK_ARTICLE_URLS"].split(",")] # public Zendesk Help Centre article URLs
if len(article_urls) > 10:
raise ValueError("This example covers inline batches of 10 URLs or fewer")
response = requests.post(
f"{BASE_URL}/perceive/batch",
headers=headers,
json={
"urls": article_urls,
"options": {"outputs": ["markdown"]},
},
timeout=300,
)
if response.status_code == 402:
raise RuntimeError("Monthly ops allowance exhausted")
response.raise_for_status()
if response.status_code == 202:
raise RuntimeError("Batch queued; poll GET /v2/perceive/batch/{job_id} for results")
nodes = []
for result in response.json()["items"]:
page_url = result["url"]
if result.get("status") != "completed":
print(f"Skipped {page_url}: {result.get('status')}")
continue
if result.get("is_blocked"):
print(f"Skipped {page_url}: blocked page")
continue
render_quality = result.get("render_quality", 1.0)
if render_quality < 0.40:
print(f"Skipped {page_url}: low render quality ({render_quality:.2f})")
continue
# The Markdown output is a pre-signed URL; fetch it in a second request
markdown_signed_url = result["outputs"]["markdown"]["url"]
markdown = requests.get(markdown_signed_url, timeout=60)
markdown.raise_for_status()
nodes.append(
TextNode(
text=markdown.text,
id_=page_url,
metadata={"source_url": page_url, "render_quality": render_quality},
)
)
if not nodes:
raise RuntimeError("No Zendesk articles passed the render checks")
index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine(similarity_top_k=3)
print(query_engine.query("Summarise the main troubleshooting steps in these support articles."))What you get back
Perceive returns clean Markdown for each URL in the batch, with navigation and sidebars stripped by default, delivered as pre-signed URLs fetched once per article. Scraping Zendesk pages by hand needs CSS selectors that break when Zendesk updates its layout, and passing raw HTML into LlamaIndex fills the index with navigation and boilerplate that degrades retrieval quality.