A developer building a CrewAI agent that reads GitHub repositories, READMEs, issues, pull request discussions, or wiki pages needs clean text output, not the raw HTML GitHub's interface produces. GitHub pages are heavily JavaScript-rendered, which means naive scrapers return incomplete content where key sections simply do not appear in the static response. Even when content is present, the raw page carries navigation chrome, sidebar elements, and structural noise that reduces the useful signal for LLM consumption. EnConvert's Perceive endpoint takes any GitHub URL and returns the repository description, README content, file tree structure, and issue or PR text as clean markdown, ready to pass into a CrewAI tool or task without any additional preprocessing.
How to Scrape GitHub with CrewAI
Use EnConvert's Perceive endpoint to scrape GitHub repositories into clean markdown for CrewAI agents without JavaScript rendering or incomplete content
Get API keyExample
import requests
API_KEY = "[•your EnConvert API key]"
GITHUB_URL = "[•GitHub repository, README, issue, PR, or wiki page URL]"
# Step 1: Test the Perceive call directly before wrapping as a CrewAI tool
# direct_download returns the markdown bytes as the response body
response = requests.post(
"https://api.enconvert.com/v2/perceive",
headers={
"X-API-Key": API_KEY,
"Content-Type": "application/json",
},
json={
"url": GITHUB_URL,
"outputs": ["markdown"],
"direct_download": True,
},
)
response.raise_for_status()
markdown = response.text
# Preview the clean markdown output before passing downstream
print(markdown)
import requests
from crewai.tools import BaseTool
from crewai import Agent, Task, Crew
API_KEY = "[•your EnConvert API key]"
# Step 2: Wrap the Perceive call as a CrewAI tool the agent can call on demand
class GitHubPageTool(BaseTool):
name: str = "read_github_page"
description: str = (
"Fetch a GitHub repository, README, issue, pull request, "
"or wiki page and return its contents as clean markdown."
)
def _run(self, url: str) -> str:
response = requests.post(
"https://api.enconvert.com/v2/perceive",
headers={
"X-API-Key": API_KEY,
"Content-Type": "application/json",
},
json={
"url": url,
"outputs": ["markdown"],
"direct_download": True,
},
)
response.raise_for_status()
return response.text
github_tool = GitHubPageTool()
# Step 3: Attach the tool to a CrewAI agent and run
agent = Agent(
role="Open Source Research Analyst",
goal="Understand what a repository does and how to get started with it",
backstory=(
"You review open source projects and explain their purpose, "
"setup steps, and trade-offs to engineers evaluating them."
),
tools=[github_tool],
)
task = Task(
description=(
"Read https://github.com/crewaiinc/crewai and summarise what the "
"project does, how to install it, and what its main components are."
),
expected_output="A short brief covering purpose, installation, and key components.",
agent=agent,
)
crew = Crew(
agents=[agent],
tasks=[task],
)
result = crew.kickoff()
print(result)What you get back
Perceive returns GitHub page content as structured markdown: repository description, full README content with heading hierarchy preserved, file tree structure as a list, and issue or PR discussion text with comments included, with GitHub navigation bars, header banners, sponsor sidebars, and UI chrome stripped before the response is returned. Raw scraping of the same GitHub URL returns incomplete output because JavaScript-rendered sections including rendered markdown, dynamic issue timelines, and PR diff summaries are absent from the static HTML response. A scraper that handles a repository index page reliably will also produce different structured output on an issue thread or wiki page, meaning scraping logic written for one GitHub page type breaks on another.