---
title: "Ingest Knowledge Bases into AutoGen | EnConvert"
description: "How to ingest a knowledge base into an AutoGen agent pipeline with EnConvert, returning one JSONL chunk schema across documents and web pages for retrieval"
canonical: "https://www.enconvert.com/solutions/ingest-knowledge-bases-into-autogen-enconvert"
locale: "en"
---

# How to Ingest Knowledge Bases into AutoGen

A developer building an AutoGen agent over a knowledge base needs to ingest content from multiple sources, including internal documents, support articles, and product documentation hosted on a website, before passing it into a retrieval layer. Knowledge bases span file types and web sources, each requiring a different parser, a different chunking strategy, and a different output schema, leaving the agent’s retrieval layer to handle inconsistent input. EnConvert’s Ingest endpoint handles PDF, DOCX, PPTX, Markdown, and plain text files as well as URL lists, sitemaps, and website crawls, returning the same JSONL chunk schema regardless of source. Each chunk carries headings_path, section, chunk_index, word_count, and source_url metadata across all sources, ready to pass into an AutoGen agent or vector store.

## Example

```python
import json
import os
import time
import requests
from autogen import AssistantAgent, UserProxyAgent

API_KEY = os.environ["ENCONVERT_API_KEY"]
HEADERS = {"X-API-Key": API_KEY}

def poll_and_download(job_id):
    """Poll the Ingest status endpoint until the job completes, then download the JSONL output."""
    while True:
        status_response = requests.get(
            f"https://api.enconvert.com/v2/ingest/{job_id}",
            headers=HEADERS,
        )
        status_response.raise_for_status()
        job = status_response.json()

        if job["status"] == "completed":
            break
        if job["status"] in {"failed", "canceled"}:
            raise RuntimeError(
                job.get("error_message") or f"Ingest job {job_id} ended with status {job['status']}"
            )
        time.sleep(5)

    # Signed output_url expires after 15 minutes
    jsonl_response = requests.get(job["output_url"])
    jsonl_response.raise_for_status()
    return [json.loads(line) for line in jsonl_response.text.splitlines() if line.strip()]

# Step 1: File ingestion job (DOCX)
with open("internal-handbook.docx", "rb") as f:
    file_job_response = requests.post(
        "https://api.enconvert.com/v2/ingest/files",
        headers=HEADERS,
        files={"files": f},
    )
if file_job_response.status_code == 402:
    raise RuntimeError("Ingest is not on your plan, or the monthly ops allowance is exhausted")
file_job_response.raise_for_status()
file_chunks = poll_and_download(file_job_response.json()["job_id"])

# Step 2: Web ingestion job (URL list)
# Publicly accessible pages only; pages behind a login are skipped
web_job_response = requests.post(
    "https://api.enconvert.com/v2/ingest",
    headers={**HEADERS, "Content-Type": "application/json"},
    json={
        "mode": "urls",
        "urls": [url.strip() for url in os.environ["KB_PAGE_URLS"].split(",")],  # public knowledge base page URLs
    },
)
if web_job_response.status_code == 402:
    raise RuntimeError("Ingest is not on your plan, or the monthly ops allowance is exhausted")
web_job_response.raise_for_status()
web_chunks = poll_and_download(web_job_response.json()["job_id"])

# Step 3: Combine both job outputs — same JSONL schema across file and web sources
all_chunks = file_chunks + web_chunks
print(f"Loaded {len(file_chunks)} file chunks and {len(web_chunks)} web chunks")

context = "\n\n".join(
    f"[section: {c['metadata']['section']} | chunk: {c['metadata']['chunk_index']} | source: {c['metadata']['source_url']}]\n"
    f"{c.get('content')}"
    for c in all_chunks[:15]
)

# Step 4: Pass the unified context into an AutoGen agent
assistant = AssistantAgent(
    name="knowledge_base_analyst",
    llm_config={"config_list": [{"model": "gpt-4.1", "api_key": os.environ["OPENAI_API_KEY"]}]},
    system_message=f"""Answer questions using content from the knowledge base below.
Cite the section name and source URL for each answer.

Context:
{context}
""",
)

user = UserProxyAgent(
    name="user",
    human_input_mode="NEVER",
    code_execution_config=False,
)
user.initiate_chat(assistant, message="What does the handbook say about billing, and does the docs site agree?")
```

## Output

Across both file and web sources, the Ingest endpoint returns JSONL chunks split at H1, H2, and H3 heading boundaries with a 512 word soft cap, tables and code blocks kept whole, and each chunk carrying headings_path, section, chunk_index, word_count, and source_url regardless of whether the source was a file or a web page. Ingesting a knowledge base manually requires a different parser per file type and a separate scraper for web content, each with its own output schema, leaving the retrieval layer to normalise inconsistent input before any agent can use it.

## Get started

[Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Scrape Wikipedia with LlamaIndex](https://www.enconvert.com/solutions/scrape-wikipedia-with-llamaindex-enconvert.md)
- [How to Ingest Markdown into LangChain with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-langchain-with-heading-context.md)
- [Best JigsawStack Alternative for Structured Data Extraction](https://www.enconvert.com/solutions/jigsawstack-alternative-for-structured-data-enconvert.md)
- [How to Ingest Audio Transcripts into Qdrant with EnConvert](https://www.enconvert.com/solutions/ingest-audio-transcripts-into-qdrant-enconvert-api.md)
- [How to Ingest a CSV into pgvector Without Losing Row Context](https://www.enconvert.com/solutions/ingest-csv-into-pgvector-with-row-context-enconvert-api.md)
- [How to Ingest CSV into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-csv-into-crewai-with-enconvert.md)