---
title: "Ingest HTML into pgvector with EnConvert"
description: "How to ingest HTML files into pgvector with EnConvert, returning heading-aware JSONL chunks with section metadata ready for embedding and PostgreSQL insertion"
canonical: "https://www.enconvert.com/solutions/ingest-html-into-pgvector-with-enconvert"
locale: "en"
---

# How to Ingest HTML into pgvector with EnConvert

A developer building a RAG pipeline with pgvector needs to ingest HTML files, such as exported documentation, scraped pages, or CMS exports, before loading them into a PostgreSQL vector store. Raw HTML carries navigation menus, script tags, boilerplate, and inline styles that inflate token usage and reduce retrieval precision, and splitting HTML by token count produces chunks with no section context and no metadata. Every new page template then adds another stripping rule to maintain, and the retrieval layer still has no way to filter by section. EnConvert’s Ingest endpoint accepts an HTML upload and returns JSONL chunks split at H1, H2, and H3 heading boundaries with a 512 word soft cap, each carrying headings_path, section, chunk_index, word_count, and source_url metadata, ready to load into pgvector.

## Example

```python
import json
import os
import time
import requests
import psycopg2
from psycopg2.extras import execute_values
from openai import OpenAI

# Step 1: Upload the HTML file to the Ingest endpoint
with open("exported-docs.html", "rb") as f:
    upload_response = requests.post(
        "https://api.enconvert.com/v2/ingest/files",
        headers={"X-API-Key": os.environ["ENCONVERT_API_KEY"]},
        files={"files": f},
    )

if upload_response.status_code == 402:
    raise RuntimeError("Ingest is not on your plan, or the monthly ops allowance is exhausted")
upload_response.raise_for_status()
job_id = upload_response.json()["job_id"]

# Step 2: Poll until the job reaches a terminal state
while True:
    status_response = requests.get(
        f"https://api.enconvert.com/v2/ingest/{job_id}",
        headers={"X-API-Key": os.environ["ENCONVERT_API_KEY"]},
    )
    status_response.raise_for_status()
    job = status_response.json()

    if job["status"] == "completed":
        break
    if job["status"] in {"failed", "canceled"}:
        raise RuntimeError(
            job.get("error_message") or f"Ingest job {job_id} ended with status {job['status']}"
        )
    time.sleep(5)

# Step 3: Download JSONL output from signed output_url (expires after 15 minutes)
jsonl_response = requests.get(job["output_url"])
jsonl_response.raise_for_status()
chunks = [json.loads(line) for line in jsonl_response.text.splitlines() if line.strip()]
print(f"Loaded {len(chunks)} chunks")

# Step 4: Create the pgvector table if it does not exist
conn = psycopg2.connect(os.environ["DATABASE_URL"])
cur = conn.cursor()

cur.execute("CREATE EXTENSION IF NOT EXISTS vector;")
cur.execute("""
    CREATE TABLE IF NOT EXISTS html_chunks (
        id TEXT PRIMARY KEY,
        content TEXT,
        embedding vector(1536),
        headings_path TEXT[],
        section TEXT,
        chunk_index INTEGER,
        word_count INTEGER,
        source_url TEXT
    );
""")
conn.commit()

# Step 5: Generate embeddings and insert chunks with metadata into pgvector
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
rows = []

for chunk in chunks:
    meta = chunk["metadata"]
    embedding = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=chunk.get("content"),
    ).data[0].embedding

    rows.append((
        chunk.get("id"),
        chunk.get("content"),
        embedding,
        meta.get("headings_path"),
        meta.get("section"),
        meta.get("chunk_index"),
        meta.get("word_count"),
        meta.get("source_url"),
    ))

execute_values(
    cur,
    """
    INSERT INTO html_chunks
        (id, content, embedding, headings_path, section, chunk_index, word_count, source_url)
    VALUES %s
    ON CONFLICT (id) DO NOTHING
    """,
    rows,
)
conn.commit()
print(f"Inserted {len(rows)} chunks into pgvector")

cur.close()
conn.close()
```

## Output

The Ingest endpoint returns JSONL chunks split at H1, H2, and H3 heading boundaries with a 512 word soft cap, navigation and boilerplate stripped, tables and code blocks kept whole, and each chunk carrying headings_path, section, chunk_index, word_count, and source_url. Splitting raw HTML by token count instead produces chunks containing navigation markup and boilerplate that inflate pgvector index size and reduce retrieval precision, and stripping HTML manually before chunking is a maintenance burden that grows with every new page template.

## Get started

[Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup)

## Related solutions

- [How to Scrape Wikipedia with LlamaIndex](https://www.enconvert.com/solutions/scrape-wikipedia-with-llamaindex-enconvert.md)
- [How to Ingest Markdown into LangChain with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-langchain-with-heading-context.md)
- [Best JigsawStack Alternative for Structured Data Extraction](https://www.enconvert.com/solutions/jigsawstack-alternative-for-structured-data-enconvert.md)
- [How to Ingest Audio Transcripts into Qdrant with EnConvert](https://www.enconvert.com/solutions/ingest-audio-transcripts-into-qdrant-enconvert-api.md)
- [How to Ingest a CSV into pgvector Without Losing Row Context](https://www.enconvert.com/solutions/ingest-csv-into-pgvector-with-row-context-enconvert-api.md)
- [How to Ingest CSV into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-csv-into-crewai-with-enconvert.md)