Data, Datasets & Synthetic Data

Data curation, labeling, synthetic generation and datasets.

Datasets

mlabonne/llm-datasets - Curated list of datasets and tools for post-training.
huggingface/datasets - The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools.
tensorflow/datasets - TFDS is a collection of datasets ready to use with TensorFlow, Jax.
mlcommons/croissant - Metadata format making ML datasets discoverable.
huggingface/dataset-viewer - Web app for exploring Hugging Face datasets.
togethercomputer/RedPajama-Data - Open dataset replicating LLaMA training data.
facebookresearch/MetaCLIP - Data curation method behind Meta's CLIP models.

Labeling & annotation

wkentaro/labelme - Image annotation with polygons and AI-assisted labeling.
voxel51/fiftyone - Curate datasets and evaluate visual AI models.
CVHub520/X-AnyLabeling - Desktop data labeling tool with built-in AI-assisted annotation models.
doccano/doccano - Open source annotation tool for machine learning practitioners.
HumanSignal/label-studio - Label Studio is a multi-type data labeling and annotation tool with standardized output format.
cvat-ai/cvat - Computer Vision Annotation Tool (CVAT) is a leading platform for building high-quality visual.
snorkel-team/snorkel - Programmatic labeling for training data without hand labels.
argilla-io/argilla - Collaboration workspace for building high-quality AI datasets.
davidjurgens/potato - Portable annotation tool for collecting LLM and NLP training data.

Synthetic data

sdv-dev/SDV - Synthetic data generation for tabular data.
Data-Centric-AI-Community/fg-data-synthetic - Synthetic data generators for tabular and time-series data.
argilla-io/distilabel - Framework for synthetic data and AI feedback pipelines.
Red-Hat-AI-Innovation-Team/sdg_hub - Building blocks for synthetic data generation workflows.
mostly-ai/mostlyai - Synthetic data generation with privacy guarantees.
joke2k/faker - Library for generating realistic fake data in Python.
ConardLi/easy-dataset - Tool for building datasets for LLM fine-tuning, RAG and evaluation.
NVIDIA-NeMo/DataDesigner - Generate high-quality synthetic datasets from scratch or seed data.
rajapalagummi/SDG-Synthetic-Data-Factory - Python library generating synthetic LLM training and eval data.
Lexsi-Labs/CuratorKIT - Data curation and synthetic data generation for LLM post-training.

Data quality & validation

cleanlab/cleanlab - Cleanlab's open-source library is the standard data-centric AI package for data quality and.
awslabs/deequ - Data quality verification on Spark using declarative checks.
elementary-data/elementary - dbt-native data observability for data and analytics engineers.
unionai-oss/pandera - Lightweight statistical data testing for dataframes.
sodadata/soda-core - Data contracts engine for the modern data stack.
deepchecks/deepchecks - Continuous validation tests for ML models and data.

Data processing

Eventual-Inc/Daft - Distributed data engine for processing images, audio and other multimodal data.
NVIDIA-NeMo/Curator - Data curation toolkit for LLM pretraining pipelines.
datajuicer/data-juicer - Data processing system for LLM pretraining data.
dlt-hub/dlt - Python library for building data pipelines without boilerplate.
tobymao/sqlglot - SQL parser, transpiler and optimizer in pure Python.
ibis-project/ibis - Python dataframe API for analytics across backends.
dask/dask - Parallel computing library scaling Python data workflows.
lance-format/lance - Columnar data format for ML with fast random access.
apache/iceberg - High-performance table format for large analytic datasets.
delta-io/delta - Open-source storage layer bringing ACID to data lakes.
narwhals-dev/narwhals - Compatibility layer between dataframe libraries.
airbytehq/airbyte - Open-source ELT platform with 600+ data connectors.
Lightning-AI/litData - Streaming data loading optimized for model training.
mosaicml/streaming - Streaming datasets for fast, multi-node training.
webdataset/webdataset - Efficient I/O format for large deep learning datasets.
NVIDIA/DALI - GPU-accelerated data loading and augmentation library.
duckdb/duckdb - In-process analytical SQL engine; query Parquet and CSV at speed.
pola-rs/polars - Blazing-fast DataFrame library for Python and Rust.
apache/datafusion - Extensible query engine for building data pipelines in Rust.
datachain-ai/datachain - Typed, versioned datasets over unstructured data in cloud storage.

Data versioning, catalogs & feature stores

treeverse/dvc - Data version control for ML: version datasets and models like code.
datahub-project/datahub - Metadata platform for data discovery, observability, and governance.
quiltdata/quilt - Data versioning and packaging for S3-backed datasets.
feast-dev/feast - The Open Source Feature Store for AI/ML.
quintoandar/butterfree - Build feature stores on Spark with declarative definitions.
treeverse/lakeFS - Git-like data version control for your data lake.
OpenLineage/OpenLineage - Open standard for data and ML pipeline lineage metadata.
MarquezProject/marquez - Collect, aggregate, and visualize data ecosystem metadata.

NLP libraries

explosion/spaCy - Industrial-strength NLP pipelines in Python.
QData/TextAttack - Framework for adversarial attacks on NLP models.
flairNLP/flair - NLP framework with state-of-the-art sequence labeling.
stanfordnlp/stanza - Stanford's multilingual NLP toolkit with neural pipelines.
huggingface/setfit - Few-shot text classification with sentence transformers.
languagetool-org/languagetool - Grammar and style checker for 30+ languages.
MaartenGr/BERTopic - Leveraging BERT and c-TF-IDF to create easily interpretable topics.

BI & analytics

vega/altair - Declarative statistical visualization in Python.
plotly/plotly.py - Interactive charts and dashboards in Python.
metabase/metabase - Open-source BI dashboards anyone can use.
apache/superset - Data exploration and visualization platform at scale.
evidence-dev/evidence - Code-based BI with SQL and Markdown.
lightdash/lightdash - BI on dbt models with version-controlled metrics.
PostHog/posthog - Product analytics and feature flags, self-hostable.

Graph analysis libraries

dmlc/dgl - Deep learning on graphs at scale.
networkx/networkx - Graph analysis and algorithms in Python.
pyg-team/pytorch_geometric - Graph neural networks in PyTorch.
igraph/igraph - Fast graph library for R, Python and C.
networkit/networkit - High-performance network analysis toolkit.
gephi/gephi - Interactive graph exploration and visualization.
rapidsai/cugraph - GPU-accelerated graph analytics.
Qiskit/rustworkx - High-performance graph library in Rust.
petgraph/petgraph - Graph data structures for Rust.
graspologic-org/graspologic - Statistical analysis of network data.