Document Loaders

Enterprise knowledge lives in PDFs, Confluence pages, Slack threads, S3 buckets, and SQL databases. Without loaders, every RAG project reimplements parsing logic. Loaders abstract format differences so your pipeline stays consistent whether the source is a 200-page legal PDF or a live Notion export.

Document loaders are like airport baggage scanners — every suitcase (file format) looks different on the outside, but the scanner normalizes everything into a standard package the rest of the system can route.

Visual Workflows

Start here — study each diagram, then use + / − to zoom if needed.

100%

Loading diagram...

Scroll inside the frame · use + / − to zoom

Key Takeaways

1.Document loaders are the ingestion layer of RAG — they read raw data from files, APIs, databases, and SaaS tools and convert it into a standardized document format (text + metadata) that downstream chunking and embedding pipelines can process.
2.Production loaders must handle: format parsing (PDF, DOCX, HTML, Markdown, CSV), encoding detection, OCR for scanned documents, table extraction, incremental sync (only load changed docs), and metadata preservation (source URL, author, timestamp, ACLs).
3.LangChain's BaseLoader interface returns Document objects with page_content and metadata.
4.For scale, loaders run as async workers in an ingestion queue — fetch → parse → validate → emit to chunking service.
5.Critical production concerns: PII detection before indexing, rate limiting on API sources, idempotent ingestion (dedupe by content hash), and failure isolation (one corrupt PDF shouldn't block the batch).

Real Example

Scenario

A legal firm's RAG system ingests 50,000 PDF contracts from S3. PyMuPDFLoader extracts text per page, attaches metadata {source, page, doc_id, client_id}, and streams documents to the chunker. When a contract is updated, an event triggers re-ingestion of only that document.

What you would do

In RAG Engineering, apply Document Loaders to this scenario: A legal firm's RAG system ingests 50,000 PDF contracts from S3. Identify the inputs, run the technique, validate the output, and note one thing you would monitor in production.

Practice Task

Open the Code Walkthrough below and run it locally. Change one parameter related to Document Loaders (e.g. model, temperature, top_k, or tool name), observe the difference in output, and write 2–3 sentences explaining what changed.

Code Walkthrough

Highlighted lines show where Document Loaders happens in the code.

Document Loaders

1from langchain_community.document_loaders import (  # import dependencies2    PyMuPDFLoader, UnstructuredHTMLLoader, S3DirectoryLoader3)4from langchain_core.documents import Document  # import dependencies5from hashlib import sha256  # import dependencies6
7def load_pdf_with_metadata(path: str, doc_id: str) -> list[Document]:  # define a reusable function8    loader = PyMuPDFLoader(path)9    pages = loader.load()10    for i, doc in enumerate(pages):11        doc.metadata.update({12            "doc_id": doc_id,13            "page": i + 1,14            "source": path,15            "content_hash": sha256(doc.page_content.encode()).hexdigest()[:16],16        })17    return pages  # return the result18
19def load_s3_corpus(bucket: str, prefix: str) -> list[Document]:  # define a reusable function20    loader = S3DirectoryLoader(bucket, prefix=prefix)21    return loader.load()  # return the result

Cheat Sheet

Quick recap

quick ref

•Document = page_content + metadata
•lazy_load() for streaming
•Hash for dedup + idempotency
•OCR for scanned PDFs
•Metadata: source, page, doc_id, tenant_id
•Never index without ACL metadata

Common Mistakes

✕Stripping all metadata — makes filtered retrieval and citations impossible
✕Loading entire corpus into memory instead of streaming
✕Ignoring document permissions — indexing content users shouldn't access
✕No content hashing — duplicate ingestion on every pipeline run
✕Treating OCR output as ground truth without confidence checks