---
title: "Scrape Intercom with LlamaIndex | EnConvert"
description: "Scrape Intercom pages into a LlamaIndex pipeline with EnConvert Perceive, returning clean Markdown with navigation and sidebars stripped in a single batch call"
canonical: "https://www.enconvert.com/solutions/scrape-intercom-with-llamaindex-enconvert"
locale: "en"
---

# How to Scrape Intercom with LlamaIndex

A developer building a LlamaIndex pipeline that reads Intercom help centre articles needs clean page text before passing it into a vector index or query engine. Intercom pages mix article content with related article suggestions, chat widget, collection navigation, reaction buttons, breadcrumbs, and “Was this article helpful?” prompts, and scraping them by hand requires CSS selectors that break when Intercom updates its layout. EnConvert Perceive renders up to 10 Intercom pages in a single batch call and returns clean Markdown for each, with navigation and sidebars stripped by default. The extracted Markdown is ready to load into a LlamaIndex VectorStoreIndex with no HTML parsing.

## Example

```python
import os
import requests
from llama_index.core import VectorStoreIndex
from llama_index.core.schema import TextNode

# LlamaIndex uses OpenAI for embeddings and answers, and reads OPENAI_API_KEY from the environment

BASE_URL = "https://api.enconvert.com/v2"
headers = {
    "X-API-Key": os.environ["ENCONVERT_API_KEY"],
    "Content-Type": "application/json",
}

page_urls = [url.strip() for url in os.environ["INTERCOM_ARTICLE_URLS"].split(",")]  # public Intercom help article URLs

if len(page_urls) > 10:
    raise ValueError("This example covers inline batches of 10 URLs or fewer")

response = requests.post(f"{BASE_URL}/perceive/batch", headers=headers,
    json={"urls": page_urls, "options": {"outputs": ["markdown"]}}, timeout=300)

if response.status_code == 402:
    raise RuntimeError("Monthly ops allowance exhausted")
response.raise_for_status()

nodes = []
for result in response.json()["items"]:
    page_url = result["url"]
    if result.get("status") != "completed":
        print(f"Skipped {page_url}: {result.get('status')}")
        continue
    if result.get("is_blocked"):
        print(f"Skipped {page_url}: blocked page")
        continue
    render_quality = result.get("render_quality", 1.0)
    if render_quality < 0.40:
        print(f"Skipped {page_url}: low render quality ({render_quality:.2f})")
        continue
    markdown_url = result["outputs"]["markdown"]["url"]
    markdown = requests.get(markdown_url, timeout=60)
    markdown.raise_for_status()
    nodes.append(TextNode(
        text=markdown.text,
        id_=page_url,
        metadata={"source_url": page_url, "render_quality": render_quality},
    ))

if not nodes:
    raise RuntimeError("No Intercom articles passed the render checks")

index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine(similarity_top_k=3)
print(query_engine.query("Summarise what these Intercom help articles cover."))
```

## Output

Perceive returns clean Markdown for each URL in the batch, with navigation and sidebars stripped by default, delivered as pre-signed URLs that you fetch once per page. Scraping Intercom by hand needs CSS selectors that break when the layout changes, and passing raw Intercom HTML into a LlamaIndex node fills the index with related article links, chat widget markup, and reaction prompts that add noise to every similarity search result.

## Get started

[Get free API key — 500 ops per month, no card required](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Scrape Wikipedia with LlamaIndex](https://www.enconvert.com/solutions/scrape-wikipedia-with-llamaindex-enconvert.md)
- [How to Ingest Markdown into LangChain with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-langchain-with-heading-context.md)
- [Best JigsawStack Alternative for Structured Data Extraction](https://www.enconvert.com/solutions/jigsawstack-alternative-for-structured-data-enconvert.md)
- [How to Ingest Audio Transcripts into Qdrant with EnConvert](https://www.enconvert.com/solutions/ingest-audio-transcripts-into-qdrant-enconvert-api.md)
- [How to Ingest a CSV into pgvector Without Losing Row Context](https://www.enconvert.com/solutions/ingest-csv-into-pgvector-with-row-context-enconvert-api.md)
- [How to Ingest CSV into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-csv-into-crewai-with-enconvert.md)