Document Processing & Data Ingestion

Getting real-world files into a form agents can use.

Document parsing & conversion

microsoft/markitdown - Convert Office files, PDFs and more to Markdown.
docling-project/docling - IBM's document parser with layout-aware PDF, tables and OCR; best open option for RAG ingestion.
Unstructured-IO/unstructured - Partition and chunk any document type for LLM pipelines.
opendatalab/MinerU - High-quality PDF-to-Markdown extraction, strong on scientific papers.
datalab-to/marker - Fast PDF to Markdown with GPU acceleration.
firecrawl/anydoc - Rust converter for Word, PowerPoint, Excel, EPUB, CSV and PDF to clean Markdown.
allenai/olmocr - Toolkit for linearizing PDFs for LLM datasets/training.
run-llama/liteparse - Fast open source document parser from LlamaIndex.
oomol-lab/pdf-craft - Converts PDF files into other formats such as Markdown and EPUB.
opendataloader-project/opendataloader-pdf - PDF parser that outputs AI-ready structured data.
breezedeus/Pix2Text - Open-source tool recognizing layouts, tables, formulas, and text as Markdown.
deepdoctection/deepdoctection - Document AI toolkit for layout analysis and information extraction.
apache/tika - Toolkit detecting and extracting metadata and text from 1000+ file types.
mwilliamson/python-mammoth - Convert Word documents (.docx files) to HTML.
jgm/pandoc - Universal markup converter.
firecrawl/pdf-inspector - Rust library for PDF classification and fast text extraction.
KOUISAmine/file-converter-tools - Collection of file conversion utilities.
gotenberg/gotenberg - Docker API for converting documents to PDF.
Kozea/WeasyPrint - Convert HTML and CSS to PDF.
paperless-ngx/paperless-ngx - Self-hosted document management with OCR.
mayan-edms/Mayan-EDMS - Free open-source document management system with OCR and workflow support.
xberg-io/xberg - Rust-core document intelligence extracting text, tables, and metadata.

OCR

PaddlePaddle/PaddleOCR - Multilingual OCR toolkit.
hiroi-sora/Umi-OCR - OCR software, free and offline. 开源、免费的离线OCR软件。支持截屏/批量导入图片，PDF文档识别，排除水印/页眉页脚，扫描/生成二维码。内置多国语言库。.
naptha/tesseract.js - Pure Javascript OCR for more than 100 Languages.
ocrmypdf/OCRmyPDF - OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched.
deepseek-ai/DeepSeek-OCR - Contexts Optical Compression.
datalab-to/surya - High-accuracy open OCR model with layout, reading order, and table recognition in 90+ languages.
tesseract-ocr/tesseract - Tesseract Open Source OCR Engine (main repository).
JaidedAI/EasyOCR - Ready-to-use OCR with 80+ supported languages and writing scripts.
RapidAI/RapidOCR - Multi-language OCR toolkits based on ONNX Runtime, OpenVINO, and PyTorch.
madmaze/pytesseract - A Python wrapper for Google Tesseract.
mindee/doctr - Deep-learning OCR library for document text recognition.
sirfz/tesserocr - A Python wrapper for the tesseract-ocr API.
manisandro/gImageReader - A Gtk/Qt front-end to tesseract-ocr.
mittagessen/kraken - OCR engine for all the languages.
breezedeus/CnSTD - PyTorch-based package for scene text detection and document layout analysis.
Calamari-OCR/calamari - Line based ATR Engine based on OCRopy.
google-research/pix2struct - Google's vision-language model for parsing screenshots and documents into structured text.
datalab-to/chandra - OCR model for complex tables, forms and handwriting with full layout.
bytedance/Dolphin - Document image parsing model using heterogeneous anchor prompting.
zai-org/GLM-OCR - Accurate and fast multimodal OCR model for document understanding.
studio-dots-ai/dots.ocr - Multilingual document layout parsing inside a single vision-language model.
getomni-ai/zerox - OCR and document extraction powered by vision models.
robertknight/ocrs - Rust library and CLI tool for fast OCR from images.
dynobo/normcap - OCR powered screen-capture tool to capture information instead of images.

PDF libraries

Stirling-Tools/Stirling-PDF - #1 PDF Application on GitHub that lets you edit PDFs on any device anywhere.
pymupdf/PyMuPDF - High-performance Python library for PDF data extraction and manipulation.
jsvine/pdfplumber - Plumb PDFs for detailed character, rectangle, and table extraction.
pdfcpu/pdfcpu - PDF tooling for Go and the command line.
py-pdf/pypdf - Pure-Python PDF library for splitting, merging, cropping, and transforming pages.
pdfminer/pdfminer.six - Community maintained fork of pdfminer - we fathom PDF.
qpdf/qpdf - Qpdf: A content-preserving PDF document transformer.
pikepdf/pikepdf - A Python library for reading and writing PDF, powered by QPDF.
camelot-dev/camelot - A Python library to extract tabular data from PDFs.
mozilla/pdf.js - PDF rendering in the browser with JavaScript.
0xabu/pdfannots - Extract annotations from PDF files.
apache/pdfbox - Java library for working with PDF documents.
LibrePDF/OpenPDF - Open-source Java PDF library.
PDFMathTranslate/PDFMathTranslate - AI-powered PDF translation preserving layout; supports Google, DeepL, Ollama, OpenAI.

Structured extraction

enoch3712/ExtractThinker - ExtractThinker is a Document Intelligence library for LLMs, offering ORM-style interaction for flexible and powerful document workflows.
Zipstack/unstract - LLM-driven extraction of structured data from unstructured documents.
ucbepic/docetl - A system for agentic LLM-powered data processing and ETL.
Azure-Samples/azure-ai-content-understanding-python - Azure Content Understanding samples.
MicrosoftLearning/mslearn-ai-information-extraction - Labs on AI-powered information extraction.
run-llama/llama_cloud_services - Knowledge Agents and Management in the Cloud.
bhaskatripathi/pdfGPT - PDF GPT allows you to chat with the contents of your PDF file by using GPT capabilities. The.
instill-ai/instill-core - Instill Core is a full-stack AI infrastructure tool for data, model and pipeline.
grobidOrg/grobid - Machine learning for extracting structured data from scholarly documents.
emcf/thepipe - Extracts clean data from complex documents using vision-language models.

Chunking & data pipelines

feyninc/chonkie - Lightweight ingestion library for fast, efficient, robust RAG pipelines.
isaacus-dev/semchunk - Fast library for splitting text into semantically meaningful chunks.
huggingface/datatrove - Platform-agnostic pipeline blocks for large-scale data processing.
allenai/dolma - Data and tools for generating and inspecting OLMo pre-training data.
ChenghaoMou/text-dedup - All-in-one text de-duplication.
treeverse/dvc - Data Versioning and ML Experiments.
fsspec/filesystem_spec - Unified interface to local and remote filesystems.
piskvorky/smart_open - Stream large files from S3, GCS and HDFS.
google/magika - AI-powered file-type detection.
kurtmckee/feedparser - Universal feed parser for RSS and Atom.
feyninc/chonkiejs - Type-friendly, lightweight, fast text-chunking library for TypeScript.

Media ingestion

yt-dlp/yt-dlp - Download videos from YouTube and 1000+ sites.
python-pillow/Pillow - Imaging library for Python.
FFmpeg/FFmpeg - The standard toolkit for audio and video processing.
Zulko/moviepy - Video editing with Python.

Web content extraction

jina-ai/reader - Converts any URL into LLM-friendly text with a simple URL prefix.
AndyTheFactory/newspaper4k - News article extraction with NLP: titles, authors, dates, summaries.
adbar/htmldate - Fast and robust date extraction from web pages, Python or CLI.
codelucas/newspaper - News article full-text and metadata extraction for Python 3.
goose3/goose3 - A Python 3 compatible version of goose http://goose3.readthedocs.io/en/latest/index.html.
mozilla/readability - A standalone version of the readability lib.
miso-belica/jusText - Heuristic based boilerplate removal tool.
adbar/trafilatura - Python and CLI tool for web crawling, scraping, and text extraction.
matthewwithanm/python-markdownify - Convert HTML to Markdown.
Alir3z4/html2text - Convert HTML to Markdown-formatted text.
fhamborg/news-please - News-please - an integrated web crawler and information extractor for news that just works.
scrapinghub/extruct - Extract embedded metadata from HTML markup.
weblyzard/inscriptis - HTML to text converter preserving layout semantics.

Web archiving

ArchiveBox/ArchiveBox - Self-hosted web archiving: saves pages as HTML, PDFs, media, and WARC.
gildas-lormeau/SingleFile - Saves a complete web page as a single self-contained HTML file.
webrecorder/browsertrix-crawler - High-fidelity browser-based web archiving crawler in Docker.
webrecorder/warcio - Streaming WARC/ARC library for fast web archive IO.