---
title: "Scrape Confluence with OpenAI Agents SDK | EnConvert"
description: "Scrape Confluence pages into an OpenAI Agents SDK pipeline with EnConvert Perceive, returning clean Markdown with navigation stripped in a single batch call"
canonical: "https://www.enconvert.com/solutions/scrape-confluence-with-openai-agents-sdk-enconvert"
locale: "en"
---

# How to Scrape Confluence with OpenAI Agents SDK

A developer building an OpenAI Agents SDK pipeline that reads Confluence pages needs clean page text before passing it into agent context. Confluence pages mix the article with navigation, breadcrumbs, sidebars, and related page links, and scraping them by hand means CSS selectors that break when Confluence updates its layout. EnConvert Perceive renders up to 10 Confluence pages in a single batch call and returns clean Markdown for each one, with navigation and sidebars stripped by default. The extracted Markdown is ready to pass into an OpenAI Agents SDK tool with no HTML parsing.

## Example

```python
import os
import requests
from agents import Agent, Runner, function_tool

# The OpenAI Agents SDK reads OPENAI_API_KEY from the environment

BASE_URL = "https://api.enconvert.com/v2"
headers = {
    "X-API-Key": os.environ["ENCONVERT_API_KEY"],
    "Content-Type": "application/json",
}

CONFLUENCE_URLS = [url.strip() for url in os.environ["CONFLUENCE_PAGE_URLS"].split(",")]  # public Confluence page URLs

@function_tool
def read_confluence_pages(urls: list[str]) -> str:
    """Read publicly accessible Confluence pages and return the main content
    of each as clean Markdown, labelled with its source URL."""
    if len(urls) > 10:
        raise ValueError("This tool covers inline batches of 10 URLs or fewer")

    response = requests.post(
        f"{BASE_URL}/perceive/batch",
        headers=headers,
        json={
            "urls": urls,
            "options": {"outputs": ["markdown"]},
        },
        timeout=300,
    )

    if response.status_code == 402:
        return "Monthly ops allowance exhausted"
    response.raise_for_status()

    sections = []
    for result in response.json()["items"]:
        page_url = result["url"]

        if result.get("status") != "completed":
            sections.append(f"Source: {page_url}\nRender failed ({result.get('status')}).")
            continue

        if result.get("is_blocked"):
            sections.append(f"Source: {page_url}\nBlocked page. No content returned.")
            continue

        render_quality = result.get("render_quality", 1.0)
        if render_quality < 0.40:
            sections.append(f"Source: {page_url}\nRender failed (quality {render_quality:.2f}).")
            continue

        # Markdown is a pre-signed URL; fetch the content in a second request
        markdown_url = result["outputs"]["markdown"]["url"]
        markdown = requests.get(markdown_url, timeout=60)
        markdown.raise_for_status()

        sections.append(f"Source: {page_url}\n\n{markdown.text}")

    return "\n\n---\n\n".join(sections) or "No pages were returned."

agent = Agent(
    name="Confluence Assistant",
    instructions=(
        "You answer questions about Confluence documentation. "
        "Use read_confluence_pages to read the pages you need, "
        "answer only from what they say, and cite the source URL for every statement."
    ),
    tools=[read_confluence_pages],
    model="gpt-4.1",
)

result = Runner.run_sync(
    agent,
    f"Read these Confluence pages: {', '.join(CONFLUENCE_URLS)} "
    "and summarise the deployment steps.",
)
print(result.final_output)
```

## Output

Perceive returns clean Markdown for each URL in the batch, with navigation and sidebars stripped by default, delivered as pre-signed URLs that you fetch once per page. Scraping Confluence pages by hand needs CSS selectors that break when Confluence updates its layout, and passing raw HTML into an OpenAI agent’s context fills the window with navigation and boilerplate that degrades the quality of the agent’s output.

## Get started

[Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Scrape Wikipedia with LlamaIndex](https://www.enconvert.com/solutions/scrape-wikipedia-with-llamaindex-enconvert.md)
- [How to Ingest Markdown into LangChain with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-langchain-with-heading-context.md)
- [Best JigsawStack Alternative for Structured Data Extraction](https://www.enconvert.com/solutions/jigsawstack-alternative-for-structured-data-enconvert.md)
- [How to Ingest Audio Transcripts into Qdrant with EnConvert](https://www.enconvert.com/solutions/ingest-audio-transcripts-into-qdrant-enconvert-api.md)
- [How to Ingest a CSV into pgvector Without Losing Row Context](https://www.enconvert.com/solutions/ingest-csv-into-pgvector-with-row-context-enconvert-api.md)
- [How to Ingest CSV into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-csv-into-crewai-with-enconvert.md)