---
title: "Summarise GitHub with LlamaIndex | EnConvert"
description: "GitHub repo summarisation in LlamaIndex: use EnConvert Distill to return schema-defined fields from up to 50 URLs in one synchronous call, no HTML parsing"
canonical: "https://www.enconvert.com/solutions/summarise-github-with-llamaindex-enconvert"
locale: "en"
---

# How to Summarise GitHub with LlamaIndex

A developer building a LlamaIndex pipeline that indexes GitHub content needs clean, structured fields before those pages can be queried usefully. GitHub repo pages mix README content with contributor sidebars, file trees, star counts, and fork metadata, and scraping them directly requires HTML selectors that break when GitHub updates its layout. EnConvert Distill accepts the GitHub URL, applies a schema the pipeline defines, and returns structured fields in one synchronous call. The pipeline gets structured content ready to index as LlamaIndex nodes with no HTML parsing.

## Example

```python
import asyncio
import os
import requests
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.core import VectorStoreIndex
from llama_index.core.schema import TextNode
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI

# LlamaIndex's OpenAI integration reads OPENAI_API_KEY from the environment

BASE_URL = "https://api.enconvert.com/v2"
headers = {
    "X-API-Key": os.environ["ENCONVERT_API_KEY"],
    "Content-Type": "application/json",
}

# The fields the pipeline receives. The response keys match these names exactly.
REPO_SCHEMA = {
    "description": "The one line repository description shown under the repository name",
    "readme_summary": "A three sentence summary of what the README says the project does",
    "latest_release": "The name and date of the latest release, if the page shows one",
    "primary_language": "The main programming language of the repository",
    "license": "The licence named on the page, if any",
}

def distill_repos(urls: list[str]) -> list[dict]:
    """Calls Distill and returns a list of schema-matched data dicts, one per completed URL."""
    if len(urls) > 50:
        raise ValueError("Distill accepts at most 50 URLs per call.")

    # One synchronous call: no job_id, no polling, no download step
    response = requests.post(
        f"{BASE_URL}/distill",
        headers=headers,
        json={"urls": urls, "schema": REPO_SCHEMA},
        timeout=300,
    )

    # Distill runs on paid plans only, and the free tier returns 402
    if response.status_code == 402:
        raise RuntimeError("Distill requires a paid plan. Free tier returns 402.")

    response.raise_for_status()

    rows = []
    for result in response.json()["results"]:
        # Failed pages come back as rows with a failed status, not as errors
        if result["status"] != "completed":
            print(f"Skipped {result['url']}: {result['status']}")
            continue

        # A render_quality below 0.5 may mean a login wall or challenge page
        render_quality = result.get("render_quality", 1.0)
        if render_quality < 0.5:
            print(f"Low render quality for {result['url']}: {render_quality}")
            continue

        rows.append({"url": result["url"], **result["data"]})
    return rows

def format_repo(row: dict) -> str:
    return (
        f"Source: {row['url']}\n"
        f"Description: {row['description']}\n"
        f"Primary language: {row['primary_language']}\n"
        f"Licence: {row['license']}\n"
        f"Latest release: {row['latest_release']}\n"
        f"README summary: {row['readme_summary']}"
    )

# Option 1: index the fields as nodes and query them
repo_urls = [
    "https://github.com/run-llama/llama_index",
    "https://github.com/langchain-ai/langchain",
    "https://github.com/deepset-ai/haystack",
]

nodes = [
    TextNode(
        text=format_repo(row),
        id_=row["url"],
        metadata={
            "source_url": row["url"],
            "primary_language": row["primary_language"] or "",
            "license": row["license"] or "",
        },
    )
    for row in distill_repos(repo_urls)
]

index = VectorStoreIndex(nodes)
query_engine = index.as_query_engine(similarity_top_k=3)
print(query_engine.query("Which of these projects are written in Python, and under which licence?"))

# Option 2: wrap the call as a tool so an agent can read any repo on demand
def summarise_github_repos(urls: list[str]) -> str:
    """Reads GitHub repository pages and returns structured fields for each page:
    description, README summary, latest release, primary language, and licence."""
    try:
        return "\n\n---\n\n".join(format_repo(row) for row in distill_repos(urls)) or "No pages could be read."
    except (ValueError, RuntimeError) as error:
        return str(error)

repo_tool = FunctionTool.from_defaults(
    fn=summarise_github_repos,
    name="summarise_github_repos",
    description="Reads GitHub repository pages and returns structured fields for each page.",
)

agent = FunctionAgent(
    tools=[repo_tool],
    llm=OpenAI(model="gpt-4.1"),
    system_prompt=(
        "You summarise GitHub repositories. Use summarise_github_repos to read the pages you "
        "need, answer only from what it returns, and cite the page URL for every statement."
    ),
)

async def main():
    response = await agent.run(
        "Read https://github.com/run-llama/llama_index and tell me what the project does, "
        "which language it is written in, and what its latest release is."
    )
    print(str(response))

asyncio.run(main())
```

## Output

Distill returns the schema-defined fields inline as structured JSON in the data object, with no polling, no job_id, and no signed URL to fetch. The render_quality score lets the pipeline detect a login wall or challenge page before indexing incomplete content. Scraping GitHub repo pages directly returns HTML containing contributor lists, file tree entries, and star and fork counts embedded throughout, and a dedicated parser needs CSS selectors that break when GitHub changes its layout.

## Get started

[Get free API key — 500 opGet free API key — 500 ops per month, no card required. Note: Distill runs on paid plans. Free tier returns 402.s per month, no card required](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Summarise GitHub with Pydantic AI](https://www.enconvert.com/solutions/summarise-github-with-pydantic-ai-enconvert.md)
- [How to Summarise Google Docs with LangGraph](https://www.enconvert.com/solutions/summarise-google-docs-with-langgraph-enconvert.md)
- [How to Summarise Google Docs with Pydantic AI](https://www.enconvert.com/solutions/summarise-google-docs-with-pydantic-ai-enconvert.md)
- [How to Summarise YouTube Transcripts with LlamaIndex](https://www.enconvert.com/solutions/summarise-youtube-transcripts-with-llamaindex-enconvert.md)
- [How to Summarise GitHub Issues with CrewAI](https://www.enconvert.com/solutions/summarise-github-issues-with-crewai-enconvert.md)
- [How to Summarise GitHub Issues with LlamaIndex](https://www.enconvert.com/solutions/summarise-github-issues-with-llamaindex-enconvert.md)
