A developer building an AutoGen research agent that reads arXiv papers needs clean extraction of abstracts, section headings, and body text, not raw HTML with LaTeX markup and rendering artefacts. arXiv abstract pages mix structured metadata with rendered equations and citation markup that no language model can reason over without preprocessing. PDF extraction avoids some of the HTML noise but introduces its own problems: broken section boundaries, mangled equations, and formatting errors that add a cleaning step your pipeline should not need. EnConvert's Perceive endpoint takes any arXiv abstract or HTML paper URL and returns clean markdown of the paper title, author list, abstract, and body text, eliminating the need for PDF parsing or arXiv HTML scraping.
How to Scrape arXiv with AutoGen
Use EnConvert's Perceive endpoint to scrape arXiv papers into clean markdown for AutoGen research agents without PDF parsing or arXiv HTML scraping
Get API keyExample
import requests
API_KEY = "[•your EnConvert API key]"
ARXIV_URL = "[•arXiv abstract page URL]"
# Step 1: Convert the arXiv abstract page to clean markdown
# direct_download returns the markdown bytes as the response body
response = requests.post(
"https://api.enconvert.com/v2/perceive",
headers={
"X-API-Key": API_KEY,
"Content-Type": "application/json",
},
json={
"url": ARXIV_URL,
"outputs": ["markdown"],
"direct_download": True,
},
)
response.raise_for_status()
markdown = response.text
# Preview the clean markdown output before passing downstream
print(markdown)
import asyncio
import requests
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient
API_KEY = "sk_your_enconvert_api_key"
OPENAI_API_KEY = "sk-your_openai_api_key"
# Step 2: Wrap the Perceive call as an AutoGen tool the agent can call on demand
# AutoGen reads the signature and docstring to build the tool schema
def fetch_arxiv_paper(url: str) -> str:
"""Fetch an arXiv paper page and return clean markdown."""
response = requests.post(
"https://api.enconvert.com/v2/perceive",
headers={
"X-API-Key": API_KEY,
"Content-Type": "application/json",
},
json={
"url": url,
"outputs": ["markdown"],
"direct_download": True,
},
)
response.raise_for_status()
return response.text
model_client = OpenAIChatCompletionClient(
model="gpt-4o",
api_key=OPENAI_API_KEY,
)
agent = AssistantAgent(
name="research_agent",
model_client=model_client,
tools=[fetch_arxiv_paper],
)
# Step 3: Run the agent with an arXiv abstract URL
# agent.run is async, so it must be awaited inside an event loop
async def main() -> None:
result = await agent.run(
task="Read https://arxiv.org/abs/2303.08774 and summarise its main contribution."
)
print(result.messages[-1].content)
await model_client.close()
asyncio.run(main())What you get back
The Perceive endpoint returns the arXiv page as structured markdown: paper title, full author list, abstract, and body text with section headings and paragraphs preserved, with arXiv navigation bars, citation sidebars, submission metadata panels, raw math delimiters, and LaTeX rendering artefacts stripped before the response is returned. PDF extraction of the same paper introduces broken equations represented as garbled character sequences, inconsistent section boundary detection, and merged paragraphs where the PDF layout uses columns or narrow margins. Both add a preprocessing step that increases pipeline complexity without improving output quality.