A developer building an AutoGen agent needs to ingest plain text files, including logs, transcripts, notes, and exported documents, before passing them into an agent’s context or retrieval layer. TXT files have no heading structure, so naive token-based splitting breaks context across chunks and strips provenance entirely. EnConvert’s Ingest endpoint accepts a TXT upload and returns JSONL chunks split at H1, H2, and H3 heading boundaries, with a 512 word soft cap and tables and code blocks kept whole. Each chunk carries headings_path, section, chunk_index, word_count, and source_url metadata, ready to pass into an AutoGen agent or vector store — structured output from a format that has no structure of its own.
How to Ingest TXT with AutoGen
How to ingest TXT files inside an AutoGen pipeline with EnConvert, returning heading-aware JSONL chunks with section metadata for retrieval and agent context
Get API keyExample
python
import json
import os
import time
import requests
from autogen import AssistantAgent, UserProxyAgent
# Step 1: Upload the TXT file to the Ingest endpoint
with open("notes.txt", "rb") as f:
upload_response = requests.post(
"https://api.enconvert.com/v2/ingest/files",
headers={"X-API-Key": os.environ["ENCONVERT_API_KEY"]},
files={"files": f},
)
if upload_response.status_code == 402:
raise RuntimeError("Ingest is not on your plan, or the monthly ops allowance is exhausted")
upload_response.raise_for_status()
job_id = upload_response.json()["job_id"]
# Step 2: Poll until the job completes or fails
while True:
status_response = requests.get(
f"https://api.enconvert.com/v2/ingest/{job_id}",
headers={"X-API-Key": os.environ["ENCONVERT_API_KEY"]},
)
status_response.raise_for_status()
job = status_response.json()
if job["status"] == "completed":
break
if job["status"] in {"failed", "canceled"}:
raise RuntimeError(f"Ingest job {job_id} ended with status {job['status']}")
time.sleep(5)
# Step 3: Download JSONL output from signed output_url (expires after 15 minutes)
jsonl_response = requests.get(job["output_url"])
jsonl_response.raise_for_status()
chunks = [json.loads(line) for line in jsonl_response.text.splitlines() if line.strip()]
# Step 4: Build context block from chunks, preserving section metadata
context = "\n\n".join(
f"[section: {c['metadata']['section']} | chunk: {c['metadata']['chunk_index']} | source: {c['metadata']['source_url']}]\n"
f"{c.get('content')}"
for c in chunks[:10]
)
# Step 5: Pass context into an AutoGen agent
assistant = AssistantAgent(
name="txt_analyst",
llm_config={"config_list": [{"model": "gpt-4.1", "api_key": os.environ["OPENAI_API_KEY"]}]},
system_message=f"""Answer questions using the document context below.
Cite the section and chunk index for each claim.
Context:
{context}
""",
)
user = UserProxyAgent(
name="user",
human_input_mode="NEVER",
code_execution_config=False,
)
user.initiate_chat(assistant, message="Summarise the main points in these notes.")What you get back
The Ingest endpoint returns JSONL chunks split at H1, H2, and H3 heading boundaries with a 512 word soft cap, tables and code blocks kept whole, and each chunk carrying headings_path, section, chunk_index, word_count, and source_url. Splitting a TXT file by token count instead produces fragments with no section context and no metadata, and passing the raw file into an agent context inflates token usage without provenance.