Building a LangChain knowledge base from open source repositories means getting clean text from GitHub READMEs — installation steps, API references, configuration options, and usage examples — without the navigation chrome and sidebar content that surrounds them on the rendered page. The GitHub REST API raw endpoint returns the source markdown, but raw markdown contains relative links, unrendered badge syntax, and image references that require additional processing before the content is usable in a retrieval context. Scraping the rendered HTML page avoids some of those issues but introduces others: sidebar panels, contributor lists, repository metadata, and navigation elements get mixed into the extracted text. EnConvert's Perceive endpoint fetches the rendered README from any GitHub repository URL and returns clean markdown with heading structure, code blocks, tables, and lists intact, without the rendering logic or HTML parsing your pipeline would otherwise need to handle.
How to Ingest GitHub READMEs into LangChain
Use EnConvert's Perceive endpoint to extract GitHub READMEs as clean markdown for LangChain vector stores preserving headings, code blocks, and tables
Get API keyExample
import requests
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
ENCONVERT_API_KEY = "[•your EnConvert API key]"
OPENAI_API_KEY = "[•your OpenAI API key]"
GITHUB_URL = "[•GitHub repository URL]"
# Step 1: Fetch the rendered README as clean markdown via Perceive
response = requests.post(
"https://api.enconvert.com/v2/perceive",
headers={
"Authorization": f"Bearer {ENCONVERT_API_KEY}",
"Content-Type": "application/json",
},
json={
"url": GITHUB_URL,
"outputs": ["markdown"],
"direct_download": True,
},
)
response.raise_for_status()
markdown = response.text
# Optional: preview the extracted markdown before indexing
print(markdown)
# Step 2: Create a LangChain Document
document = Document(
page_content=markdown,
metadata={"source": GITHUB_URL},
)
# Step 3: Split the README for retrieval
# Separators are tried in order from most to least specific
# so the splitter respects heading boundaries before falling
# back to paragraph and word breaks
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=100,
separators=["\n## ", "\n### ", "\n\n", "\n", " "],
)
chunks = splitter.split_documents([document])
# Step 4: Index the chunks into a vector store
embeddings = OpenAIEmbeddings(
model="text-embedding-3-small",
api_key=OPENAI_API_KEY,
)
# Note: this uses the LangChain Chroma wrapper not the standalone chromadb client
# Chroma.from_documents may also require persist_directory
# or a client argument depending on your version
vector_store = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
collection_name="[•collection name for your GitHub READMEs]",
)
# Step 5: Query the retriever and print matching chunks
# Note: use retriever.invoke() in current LangChain versions
# older versions use retriever.get_relevant_documents()
retriever = vector_store.as_retriever(search_kwargs={"k": 4})
results = retriever.invoke("[•example query against the README content]")
# Print the top matching chunks from the indexed README
for result in results:
print(result.page_content)What you get back
Perceive returns the GitHub README as clean markdown: headings at correct hierarchy levels, code blocks with language identifiers, tables with cell boundaries intact, and list content preserved, with GitHub navigation bars, repository sidebar panels, contributor sections, and page chrome stripped before the response is returned. The GitHub REST API raw endpoint avoids the HTML noise but returns source markdown that contains relative links pointing to repository-relative paths, badge markup that does not resolve outside the GitHub rendering context, and image references that carry no useful text content for a retrieval pipeline. Rendered tables in particular are lost in the raw markdown response, as GitHub applies its own table rendering extensions that are not part of standard markdown and do not round-trip cleanly through the raw API.