Awesome AI Agent Stack

Structured Output, Guardrails & Safety

Validation, constrained decoding, jailbreak and injection defense.

Structured output

• 567-labs/instructor - Structured outputs from any LLM via Pydantic models.
• dottxt-ai/outlines - Constrained generation: guarantee JSON, regex or grammar-valid output.
• mlc-ai/xgrammar - Fast, portable structured generation engine for LLM outputs.
• guidance-ai/guidance - Microsoft's language for structured, constrained LLM generation.
• noamgat/lm-format-enforcer - Token-level filtering that enforces JSON Schema or regex output.
• microsoft/TypeChat - Natural language interfaces via types with schema-validated output.
• BoundaryML/baml - Structured-output language for building reliable LLM agents.
• google/langextract - Extracts structured information from text with precise source grounding.
• urchade/GLiNER - Lightweight NER model for extracting any entity types from text.
• mangiucugna/json_repair - Repairs malformed JSON from LLMs, APIs, logs, and user input.
• guidance-ai/llguidance - High-performance structured output and grammar-constrained decoding.
• promplate/partial-json-parser - Parses partial and incomplete JSON streamed from LLMs.
• explosion/spacy-llm - Integrates LLMs into structured NLP pipelines with spaCy.
• vamplabAI/sgr-agent-core - Schema-guided reasoning agent framework built on structured outputs.

Guardrails

• guardrails-ai/guardrails - Input/output validators for LLM applications.
• NVIDIA-NeMo/Guardrails - Programmable rails for conversational systems.
• superagent-ai/superagent - Superagent protects your AI applications against prompt injections, data leaks, and harmful.
• microsoft/agent-governance-toolkit - AI Agent Governance Toolkit — Policy enforcement, zero-trust identity, execution sandboxing, and.
• invariantlabs-ai/invariant - Guardrails for secure and robust agent development.
• archestra-ai/OpenAPPA - Deterministic guardrails that constrain agent actions without breaking them.
• ibm-granite/granite-guardian - Detects risks in prompts and responses for LLM safety.
• unitaryai/detoxify - Toxic comment classifier built on HuggingFace Transformers.
• mthamil107/prompt-shield - Prompt injection firewall with PII scanning for LLM inputs.
• data-privacy-stack/presidio - PII detection, redaction and anonymization for text and images.
• mohankrishnaganne/LLM-Evaluation-and-Guardrails-Framework - Production framework for evaluating RAG pipelines and enforcing LLM guardrails.
• royalpinto007/awesome-llm-guardrails - Curated list of open-source guardrails for LLM applications.
• baoguangsheng/fast-detect-gpt - Zero-shot detector for machine-generated text.
• cedar-policy/cedar - Policy language for fine-grained access control in applications.
• nolabs-ai/nono - Zero-trust micro sandboxes for securing agent runtimes.

Prompt injection defense

• agencyenterprise/PromptInject - Framework quantifying prompt robustness to injection.
• praetorian-inc/augustus - LLM vulnerability scanner for prompt injection and jailbreaks.
• liu00222/Open-Prompt-Injection - Toolkit for prompt injection attacks and defenses in LLMs.
• ethz-spylab/agentdojo - Environment evaluating prompt injection attacks on LLM agents.
• AgentPostmortem/Injection-arena - Self-hostable prompt-injection CTF game with stacked AI defenses.
• utkusen/promptmap - Security scanner for prompt injection in custom LLM applications.

Red-teaming & security scanners

• usestrix/strix - Open-source AI penetration testing tool to find and fix your app’s vulnerabilities.
• KeygraphHQ/shannon - Shannon is an AI pentester for web applications and APIs. It analyzes your source code.
• NVIDIA/garak - The LLM vulnerability scanner.
• Giskard-AI/giskard-oss - Open-Source Evaluation & Testing library for LLM Agents.
• confident-ai/deepteam - DeepTeam is a framework to red team LLMs and AI agents.
• microsoft/AI-Red-Teaming-Playground-Labs - AI Red Teaming playground labs to run AI Red Teaming trainings including infrastructure.
• msoedov/agentic_security - Agentic LLM Vulnerability Scanner / AI red teaming kit.
• Pantheon-Security/medusa - AI-first security scanner with 40k+ detection patterns.
• NuGuardAI/nuguard - AI red-teaming toolkit for prompt injections and agents.
• snyk/agent-scan - Security scanner for AI agents, MCP servers and skills.
• cisco-ai-defense/mcp-scanner - Scanner for MCP tools, prompts and supply-chain packages.
• microsoft/PyRIT - Red-teaming framework for identifying genAI risks.
• OWASP/AISVS - Testable security verification requirements for AI applications.
• thinkst/canarytokens - Canary tokens that alert when accessed, for detecting breaches.
• Trusted-AI/adversarial-robustness-toolbox - Library for ML security: evasion, poisoning, extraction, and inference attacks.
• rogue-security/rogue - Red-team platform evaluating AI agents against security policies.
• vxcontrol/pentagi - Autonomous AI agents for penetration testing.
• Autumn-27/ARTEX - Autonomous AI penetration testing system (Chinese).

Model supply-chain security

• safetensors/safetensors - Safe tensor storage format preventing arbitrary code execution.
• trailofbits/fickling - Static analyzer and decompiler for Python pickle model files.
• protectai/modelscan - Scans models for serialization attacks and unsafe code.