If you are building a LlamaIndex pipeline that reads Wikipedia articles, you need the article content without the structural overhead Wikipedia pages carry. Wikipedia pages include disambiguation headers, tables of contents, reference sections with hundreds of citations, edit history links, sidebar navigation panels, hatnotes, and category footers that inflate token usage and introduce noise into your index before a single query runs. Writing your own extraction and summarisation logic handles common article layouts but breaks when Wikipedia updates its page structure or when articles in different categories use different formatting conventions. EnConvert’s Distill endpoint takes a Wikipedia URL and returns a clean, concise summary of the article content, ready to pass into a LlamaIndex index or query engine without any preprocessing on your end.
How to Summarise Wikipedia Articles with LlamaIndex
Use EnConvert Distill to summarise Wikipedia articles in one API call and pass clean, LLM-ready output directly into a LlamaIndex index or query engine.
Get API keyExample
import os
import requests
from llama_index.core import Document, VectorStoreIndex
from llama_index.embeddings.openai import OpenAIEmbedding
ENCONVERT_API_KEY = os.environ["ENCONVERT_API_KEY"]
WIKIPEDIA_URL = os.environ["WIKIPEDIA_URL"] # e.g. https://en.wikipedia.org/wiki/Retrieval-augmented_generation
# Step 1: Summarise the Wikipedia article with Enconvert Distill
response = requests.post(
"https://api.enconvert.com/v2/distill",
headers={
"X-API-Key": ENCONVERT_API_KEY,
"Content-Type": "application/json",
},
json={
"urls": [WIKIPEDIA_URL],
"schema": {
"summary": "a thorough summary of the article covering its main concepts",
},
},
timeout=60,
)
response.raise_for_status()
item = response.json()["results"][0]
if item.get("error") or not item.get("data"):
raise RuntimeError(f"Distill failed for {WIKIPEDIA_URL}: {item.get('error')}")
summary = item["data"]["summary"]
# Step 2: Pass the summary into a LlamaIndex index as a document
document = Document(
text=summary,
metadata={
"source": WIKIPEDIA_URL,
"type": "wikipedia_summary",
},
)
embed_model = OpenAIEmbedding(
model="text-embedding-3-small",
api_key=os.environ["OPENAI_API_KEY"],
)
index = VectorStoreIndex.from_documents(
[document],
embed_model=embed_model,
)
query_engine = index.as_query_engine()
answer = query_engine.query("What are the main concepts covered in this article?")
print(answer)What you get back
Distill returns the Wikipedia article as a clean summary: the core content condensed into coherent paragraphs or structured points, with disambiguation headers, tables of contents, reference sections, citation lists, edit history links, sidebar navigation, and hatnotes removed before the response is returned. Passing raw Wikipedia page content into a LlamaIndex index instead produces bloated token usage from structural noise, inconsistent content shape across articles from different categories where layout conventions vary, and index vectors that partially represent navigational and citation content rather than the article subject itself. The degradation compounds when indexing multiple articles where consistent chunk quality determines retrieval accuracy.