A developer building an OpenAI agent that reads Confluence documentation, internal wikis, runbooks, and technical specs needs clean text extraction from pages that are frequently shared via link without requiring a logged-in session. Confluence pages carry significant noise: sidebar navigation, breadcrumb trails, metadata panels, and macro markup that produces noisy and unparseable output when scraped without a preprocessing step. The Confluence REST API requires OAuth or API token authentication with admin-level access scoped to a specific workspace, which is not always available when pages are shared across organisations or accessed outside the originating Atlassian instance. EnConvert's Perceive endpoint takes any publicly accessible or link-shared Confluence URL and returns clean markdown of the page content, ready to register as a tool your OpenAI agent can call on demand.
How to Scrape Confluence with OpenAI Agents SDK
Use EnConvert's Perceive endpoint to scrape Confluence pages into clean markdown for OpenAI Agents SDK tools without Confluence REST API authentication
Get API keyExample
import requests
from agents import Agent, Runner, function_tool
ENCONVERT_API_KEY = "[•your EnConvert API key]"
# Step 1: Fetch and convert a Confluence page to clean markdown
def perceive_confluence_page(url: str) -> str:
response = requests.post(
"https://api.enconvert.com/v2/perceive",
headers={
"X-API-Key": ENCONVERT_API_KEY,
"Content-Type": "application/json",
},
json={
"url": url,
"outputs": ["markdown"],
"direct_download": True,
},
timeout=30,
)
response.raise_for_status()
return response.text
url = "[•Confluence page URL]"
markdown = perceive_confluence_page(url)
print(markdown)
# Example markdown returned by Perceive:
#
# # Deployment Runbook
#
# ## Production deployment
#
# Run the deployment command after checking the release status.
#
# ## Rollback
#
# 1. Stop the current deployment.
# 2. Restore the previous version.
#
# | Environment | Status |
# |-------------|---------|
# | Production | Ready |
# Step 2: Register the Perceive call as an OpenAI Agents SDK function tool
@function_tool
def read_confluence_page(url: str) -> str:
"""Fetch an accessible Confluence page and return its content as markdown."""
return perceive_confluence_page(url)
agent = Agent(
name="Confluence Reader",
instructions=(
"Read accessible Confluence pages when needed. "
"Use the read_confluence_page tool with the page URL, "
"then answer using the returned markdown."
),
tools=[read_confluence_page],
)
result = Runner.run_sync(
agent,
"Read the deployment runbook at "
"[•Confluence page URL] "
"and summarise the rollback steps."
)
print(result.final_output)What you get back
The Perceive endpoint returns the Confluence page as structured markdown: page title, full body content with heading hierarchy preserved, tables with rows and columns intact, and inline code blocks with language markers where specified, with sidebar navigation, breadcrumb trails, contributor metadata panels, and Confluence UI chrome stripped before the response is returned. Raw HTML scraping of the same page produces output embedded with Confluence macro syntax, nested table markup, and layout divs that require a custom Confluence HTML parser to extract readable text. The Confluence REST API avoids the parsing problem but requires OAuth setup with admin-level workspace permissions, which is impractical for ad hoc page access across organisation boundaries.