--- title: "Captain Alternative for RAG Pipeline Ingestion | EnConvert" description: "EnConvert is a Captain alternative for RAG pipeline ingestion that turns web pages and files into clean Markdown you load into the vector store of your choice" canonical: "https://www.enconvert.com/solutions/captain-alternative-for-rag-pipeline-ingestion-enconvert" locale: "en" --- # Best Captain Alternative for RAG Pipeline Ingestion A developer building a RAG pipeline has to decide how much of the stack to hand to a vendor. Captain is a managed RAG platform for files that handles OCR, chunking, embedding, vector storage, and search behind one query API. That suits teams that want retrieval managed for them, but it ties the index to Captain’s storage layer and does not cover web pages as a general input source. EnConvert sits earlier in the pipeline, with endpoints that turn web pages and files into clean, consistent Markdown or chunked output you load into whichever vector store you choose. The choice is between managed retrieval and an ingestion layer you control. ## Example ```python import json import os import time import requests PDF_PATH = "/path/to/your/contract.pdf" # Captain: index a PDF into a managed collection, then query it CAPTAIN_URL = "https://api.filesearch.dev" CAPTAIN_COLLECTION = "contracts" # The organisation is derived from the API key, so X-Organization-ID is optional captain_headers = {"Authorization": f"Bearer {os.environ['CAPTAIN_API_KEY']}"} # Step 1: Create the collection (idempotent: 201 if created, 200 if it exists) requests.put( f"{CAPTAIN_URL}/v2/collections/{CAPTAIN_COLLECTION}", headers=captain_headers, json={}, timeout=30, ).raise_for_status() # Step 2: Upload the PDF for indexing with open(PDF_PATH, "rb") as f: index_response = requests.post( f"{CAPTAIN_URL}/v2/collections/{CAPTAIN_COLLECTION}/index/file", headers=captain_headers, files={"files": ("contract.pdf", f, "application/pdf")}, data={"processing_type": "advanced"}, timeout=120, ) index_response.raise_for_status() captain_job_id = index_response.json()["job_id"] # Step 3: Poll until indexing completes while True: captain_status = requests.get( f"{CAPTAIN_URL}/v2/jobs/{captain_job_id}", headers=captain_headers, timeout=30, ) captain_status.raise_for_status() captain_job = captain_status.json() if captain_job["status"] == "completed": break if captain_job["status"] in ("completed_with_errors", "failed", "cancelled", "timed_out"): raise RuntimeError( f"Captain indexing {captain_job['status']}: " f"{captain_job.get('error_message') or captain_job.get('result')}" ) time.sleep(5) # Step 4: Query the managed collection query_response = requests.post( f"{CAPTAIN_URL}/v3/collections/{CAPTAIN_COLLECTION}/query", headers=captain_headers, json={"query": "What are the termination terms?", "limit": 10}, timeout=60, ) query_response.raise_for_status() for hit in query_response.json()["results"]: print(f"{hit['score']:.3f} {hit['text'][:120]}") # Chunking, embedding, storage, and ranking are managed by Captain # EnConvert: ingest the PDF, then load chunks into the store you control enconvert_headers = {"X-API-Key": os.environ["ENCONVERT_API_KEY"]} # Step 1: Upload the PDF (multipart field is "files"; always answers 202 with a job_id) with open(PDF_PATH, "rb") as f: submit_response = requests.post( "https://api.enconvert.com/v2/ingest/files", headers=enconvert_headers, files={"files": ("contract.pdf", f, "application/pdf")}, timeout=60, ) submit_response.raise_for_status() job_id = submit_response.json()["job_id"] # Step 2: Poll until the job completes while True: status_response = requests.get( f"https://api.enconvert.com/v2/ingest/{job_id}", headers=enconvert_headers, timeout=30, ) status_response.raise_for_status() job = status_response.json() if job["status"] == "completed": break if job["status"] in ("failed", "canceled"): raise RuntimeError(f"Ingest {job['status']}: {job.get('error_message')}") time.sleep(2) # Step 3: Download the JSONL, one chunk per line. output_url is a signed URL # (valid 15 minutes): plain GET, no API key. jsonl_response = requests.get(job["output_url"], timeout=60) jsonl_response.raise_for_status() # Split on "\n" only: str.splitlines() would also break on U+2028 inside chunk text chunks = [ json.loads(line) for line in jsonl_response.content.decode("utf-8").split("\n") if line.strip() ] # Step 4: Map chunks to the inputs a vector store expects ids = [chunk["id"] for chunk in chunks] texts = [chunk["content"] for chunk in chunks] # headings_path is a list; many stores accept only scalar metadata, so join it metadatas = [ { **chunk["metadata"], "headings_path": " > ".join(chunk["metadata"]["headings_path"]), } for chunk in chunks ] print(f"Loaded {len(chunks)} chunks ready for embedding") # Embed texts with your embedding model, then add ids, texts and metadatas # to the vector store you choose ``` ## Output | Capability | EnConvert | Captain | | --- | --- | --- | | Primary function | Ingestion layer: web pages and files to Markdown or JSONL chunks split at H1 to H3 boundaries, tables and code blocks kept whole, for your own vector store | Managed RAG: OCR, chunking, embedding, vector storage, and search behind one query API | | Web page extraction | Perceive: any URL to clean Markdown, including JavaScript-rendered pages; Ingest also accepts URL lists, sitemaps, and website crawls returning the same JSONL chunk schema as file ingestion | Not documented as a general web page ingestion source | | File ingestion | Ingest: PDF, DOCX, XLSX, CSV, Markdown, and plain text; chunks split at H1 to H3 boundaries, 512 word soft cap, tables and code blocks kept whole | Confirmed: PDF, DOCX, XLSX, CSV, images, and connected cloud storage sources with OCR | | Document conversion | Convert: 45+ conversion routes across documents, images, and data formats | Not documented as a standalone conversion endpoint | | Vector storage and search | Not provided; developer brings their own vector store and embedding model | Managed by Captain; query API returns ranked chunks with scores and citations | | Output format | One consistent schema across all endpoints: Markdown from Perceive and Distill, JSONL chunks from Ingest split at H1 to H3 boundaries with tables and code blocks kept whole | Search results with relevance scores and source metadata | | MCP server | Native, exposes full ingestion toolkit as MCP tools | Confirmed: hosted MCP server for indexing, collection management, and search | | Free tier | 500 ops per month, no card required | Developer plan: 500 indexing credits and 1,000 queries per month | **WHICH TO CHOOSE** Choose Captain if you want retrieval fully managed, your inputs are files and connected storage, and you would rather not operate a vector store. Choose EnConvert if your inputs include web pages, if you want to keep your own vector store and embedding model, or if you need one consistent output schema across every source type. The two sit at different points in the pipeline, and some teams could use EnConvert to prepare content and Captain to search it. ## Get started [Get free API key — 500 ops per month, no card required.](https://www.enconvert.com/auth?mode=signup) ## Related solutions - [How to Ingest Markdown Files into LlamaIndex with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-files-into-llamaindex-with-enconvert.md) - [How to Ingest Web Pages into Vertex AI with EnConvert](https://www.enconvert.com/solutions/ingest-web-pages-into-vertex-ai-with-enconvert.md) - [How to Ingest Markdown into OpenAI File Search with EnConvert](https://www.enconvert.com/solutions/ingest-markdown-into-openai-file-search-enconvert.md) - [How to Ingest HTML Files into LlamaIndex with EnConvert](https://www.enconvert.com/solutions/ingest-html-files-into-llamaindex-with-enconvert.md) - [Best Browserless Alternative for Document Parsing](https://www.enconvert.com/solutions/browserless-alternative-for-document-parsing-enconvert.md) - [How to Ingest DOCX into CrewAI with EnConvert](https://www.enconvert.com/solutions/ingest-docx-into-crewai-with-enconvert.md)