Awesome AI Agent Stack

Evaluation, Observability & Code Review

Trace, test and red-team agents, and review the code they write.

Tracing & observability

• langfuse/langfuse - Open-source LLM observability: traces, evals, prompt management; self-hostable.
• Arize-ai/phoenix - OpenTelemetry-based tracing and evaluation for LLM apps.
• traceloop/openllmetry - OpenTelemetry instrumentation for every major LLM library.
• langchain-ai/langsmith-sdk - SDK for LangSmith tracing and datasets.
• raga-ai-hub/RagaAI-Catalyst - Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features.
• evidentlyai/evidently - Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor.
• Helicone/helicone - Open-source LLM observability platform plus AI gateway with caching and cost tracking.
• openlit/openlit - OpenTelemetry-native observability and evaluation for AI and coding agents.
• jaegertracing/jaeger - CNCF distributed tracing platform; the standard trace backend for production systems.
• FailproofAI/failproofai - Observability and enforcement for AI agents in production.
• latitude-dev/latitude-llm - Observability platform for AI agents and LLM apps.
• pydantic/logfire - OpenTelemetry-native observability for AI applications.
• lmnr-ai/lmnr - LLM observability and evaluation platform (Laminar).
• pezzolabs/pezzo - Prompt management and LLM observability toolkit.
• future-agi/future-agi - AI evaluation and observability platform.
• Arize-ai/openinference - OpenTelemetry instrumentation for AI applications.
• Scale3-Labs/langtrace - OpenTelemetry-based LLM observability SDK.
• traceroot-ai/traceroot - Observability for self-improving AI agents.
• databufflabs/databuff - AI-native APM built on OpenTelemetry.
• traceloop/openllmetry-js - Open-source LLM observability for JavaScript.
• evilmartians/agent-prism - Trace visualization for AI agent runs.
• Jwuthri/Tracely-ai - Trace-native CI/CD pipelines for AI agents.
• langfuse/langfuse-js - Langfuse observability SDK for JavaScript.
• langfuse/oss-llmops-stack - Open-source LLMOps stack with observability.
• niklasfrick/spark-dashboard - GPU and LLM inference monitoring dashboard.
• inferock/inferock-bench - LLM cost-tracking proxy with verifiable receipts.
• tma1-ai/tma1 - Local-first observability for AI agents.
• Netis/heron - Agent API performance monitoring via packet probe.
• splunk/token-meter - Dashboard for AI agent token usage and cost.
• theagentplane/tokenops - Token governance for AI agent fleets.
• T-Sunm/rag-ops - LLMOps template for RAG applications.
• llmops-build/llmops - Toolkit for LLM operations.
• dunetrace/dunetrace - Reliability layer for AI agents.
• last9/gpu-telemetry - GPU observability for inference workloads.
• TechNickAI/claude_telemetry - OpenTelemetry wrapper for Claude Code CLI.
• overmind-core/overmind - Platform for continuously improving AI agents.
• alibaba/loongsuite-pilot - Telemetry collector for coding agents.
• slavaZim/episodiq - Economical human-readable logs for agentic trajectories.
• comet-ml/opik - Open-source LLM evaluation and observability platform.
• VTSTech/ACP-Agent-Control-Panel - Lightweight monitoring sidecar and web UI for AI agents.
• SigNoz/signoz - Open-source APM and observability with OpenTelemetry.
• grafana/grafana - Dashboards and alerting for metrics, logs and traces.
• MonishGosar/opentrajectory - Local-first observability for AI agents and coding-agent workflows.
• acailic/agent_debugger - Local-first agent debugger with replay and drift detection.
• KNeegcyao/agent-inspect - DevTools for AI agents: pause, step, and fork counterfactual branches.
• nyktora/noctrace - Network-tab-style waterfall visualizer for Claude Code agent workflows.
• umairinayat/AgentTrace - Open-source observability platform for tracing multi-agent AI systems.
• rajudandigam/agent-inspect - Local-first debugger capturing AI agent execution trees for CI checks.
• raghuece455/AgentMesh - Self-hosted observability, trace replay, and cost analytics for AI agents.
• AgentOps-AI/agentops - Python SDK for AI agent monitoring, cost tracking and benchmarking.

Evaluation

• promptfoo/promptfoo - Test prompts and agents like code; CI-friendly with red-teaming.
• confident-ai/deepeval - Pytest-style unit tests for LLM outputs.
• vibrantlabsai/ragas - Evaluation metrics for RAG pipelines.
• openai/evals - OpenAI's eval framework and registry.
• EleutherAI/lm-evaluation-harness - Standard framework for few-shot LLM evaluation.
• open-compass/opencompass - OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI.
• UKGovernmentBEIS/inspect_ai - LLM evaluation framework from the UK AI Security Institute with 200+ agentic benchmarks.
• microsoft/promptflow - Microsoft's toolkit for building, tracing, and evaluating LLM flows and agents.
• fivetran/great_expectations - Data validation framework for testing eval datasets and pipeline data quality.
• dataelement/bisheng - Open-source LLM application DevOps platform with evaluation.
• coze-dev/coze-loop - Agent development and optimization platform with evaluation loops.
• microsoft/prompty - Prompt asset format for creating, managing, and evaluating prompts.
• plurai-ai/intellagent - Diagnose AI agents through synthetic interactions.
• JudgmentLabs/judgeval - Continuous improvement stack for AI agent evaluation.
• relari-ai/continuous-eval - Data-driven evaluation for LLM applications.
• rhesis-ai/rhesis - Collaboration layer for reviewing agent behavior.
• alphadl/AdaRubrics - Evaluator for AI agent trajectories.
• AgentEvalHQ/AgentEval - Evaluation toolkit for .NET AI agents.
• google/litmus - LLM testing and evaluation tool with a friendly UI.
• HiThink-Research/GAGE - Unified evaluation engine for LLMs and multimodal models.
• skillberry-ai/cap-evolve - Optimize agent skills against your own evals.
• dokimos-dev/dokimos - LLM and agent evaluation for Java and Kotlin.
• TIGER-AI-Lab/RewardHarness - Self-evolving reward framework for agent evaluation.
• meshkovQA/Eval-ai-library - AI model evaluation framework with 15+ metrics.
• cristianodabc/aludel - LLM evaluation and observability for Elixir.
• paradime-io/dbt-llm-evals - Warehouse-native LLM evaluation for dbt.
• maida-ai/maida - Pre-merge behavioral regression gate for AI agents.
• ProofAgent-ai/proofagent-harness - Test harness for AI agents in CI.
• tolitius/cupel - Discover LLMs punching above their weight.
• cvs-health/uqlm - Python package for uncertainty quantification and hallucination detection in LLMs.
• raindrop-ai/workshop - Lets coding agents write and run agent evals.
• truera/trulens - Evaluation and tracking for LLM experiments and AI agents.
• Agenta-AI/agenta - LLMOps: prompt playground, management, evals and observability.
• Kiln-AI/Kiln - Build, evaluate and optimize AI systems with a desktop UI.
• open-compass/VLMEvalKit - Evaluation toolkit covering 220+ multimodal models and 80+ benchmarks.
• perezjoan/UVLM - Unified Python interface for reproducible vision-language model benchmarking.
• google-research/true - Code and data for re-evaluating factual consistency metrics.
• METR/vivaria - Platform for running agent evaluations and elicitation research.
• langwatch/langwatch - Open platform for LLM evaluations and AI agent testing.
• jameswniu/self-hosted-llm-evals-lab - Self-hosted LLM eval lab with systematic prompt ablation.
• sdivyanshu90/build-your-own-llm-evals - Typed monorepo for reproducible offline LLM and RAG evals.
• strands-agents/evals - Agent evaluation with judges, simulations, traces, and red teaming.
• vostride/agent-qa - Open-source self-improving QA agent for software teams.
• CodeEmperor7/rag-eval - Production-style AI assistant with evaluation and monitoring.

Benchmarks

• amazon-science/SWE-PolyBench - Multilingual benchmark for software engineering agents.
• jeinlee1991/chinese-llm-benchmark - Benchmark suite for Chinese large language models.
• EvolvingLMMs-Lab/lmms-eval - Evaluation toolkit for large multimodal models.
• TIGER-AI-Lab/ClawBench - Benchmark for browser-use AI agents.
• OpenBMB/UltraEval-Audio - Benchmark for audio language models.
• solis-team/XRepoTest - Repository-level test generation benchmark.
• alibaba-multimodal-industrial-ai/IndustryBench - Industrial benchmark for multimodal models.
• open-compass/GTA - Benchmark for tool-using AI agents.
• reacher-z/HarnessBench - Benchmark comparing agent harnesses on online tasks.
• PhiloLabs/agentic-vbench - Benchmark of post-production agent tasks.
• InternScience/SciEvalKit - Scientific evaluation toolkit for LLMs.
• HiThink-Research/BizFinBench - Bilingual financial benchmark for LLMs.
• HiThink-Research/MME-Finance - Multimodal finance benchmark for expert reasoning.
• whitecircle/circle-guard-bench - Benchmark for LLM guardrails.
• OpenMOSS/SWE-bench-Science - Coding agent benchmark for scientific tasks.
• OpenMOSS/ABC-Bench - Benchmark for coding agents building backend services.
• LeoYeAI/myclaw-bench - Benchmark for OpenClaw agents.
• Gen-Verse/PAST-Bench - Benchmark for recursive self-improvement in agents.
• thedataquarry/structured-outputs - Benchmark comparing structured output approaches.
• RISE-UNIBAS/humanities_data_benchmark - Benchmark suite for humanities data tasks.
• nickname21kmr/gameops-investigator - Evals for incident investigation agents.
• voratiq/voratiq - Benchmark comparing coding agents head-to-head.
• Vchitect/VBench - Benchmark suite for evaluating video generation models.
• sierra-research/tau-bench - Benchmark for tool-augmented AI agents.
• allenai/reward-bench - Benchmark for evaluating reward models for language models.
• METR/public-tasks - Suite of tasks for evaluating autonomous AI agents.
• sierra-research/tau2-bench - Benchmark for evaluating AI agents on telecom support tasks.
• SWE-bench/SWE-bench - Benchmark harness evaluating models on real-world software issues.
• THUDM/AgentBench - Multi-environment benchmark for evaluating LLMs as agents.
• UKGovernmentBEIS/inspect_evals - Maintained evaluation suites built on the Inspect AI framework.

Security & red-teaming

• meta-llama/PurpleLlama - Llama Guard and CyberSecEval for safe deployments.
• OWASP/www-project-top-10-for-large-language-model-applications - The OWASP Top 10 for LLM apps; read before shipping an agent with tools.
• ifixai-ai/iFixAi - Independent auditing platform for AI agents and models.
• Tencent/AI-Infra-Guard - AI infrastructure security evaluation and red-teaming platform.
• Justin0504/Aegis - Runtime policy enforcement for AI agents.
• getagentseal/agentseal - Security toolkit testing agents against prompt injection.
• aiverify-foundation/moonshot - Toolkit for LLM evaluation and red-teaming.
• AgentSafeLabs/safelabs-eval - Red-teaming evaluation framework for agents.
• techlab-innov/llmtrace - Zero-code LLM security and observability proxy.
• IBM/ares - AI robustness evaluation system.
• guardana/guardana - AI security verification platform.
• mintesnot-teshome/llm-verify - Detect fake AI APIs with behavioral fingerprinting.
• cyberful/cyberful - AI red-teaming for security evaluation.
• AISecurityConsortium/AIGoat - AI security playground for red-teaming practice.
• regaan/basilisk - Open-source AI red-teaming framework with prompt evolution.
• ToseaAI/llm-fingerprint-detector - Verify which LLM an API really serves.
• crucible-security/crucible - Red-teaming and security testing for LLM agents.
• delphisecurity/xaidr - Runtime security for AI agents.
• splx-ai/agentic-radar - Security scanner for agentic workflows.
• ProjectRecon/awesome-ai-agents-security - Living map of AI agent security resources.
• AI45Lab/OpenRT - Open red-teaming framework for LLM safety.
• PurCL/ASTRA - Attacking-team framework for LLM safety evaluation.
• cyberark/FuzzyAI - Automated LLM fuzzing tool for finding jailbreaks and vulnerabilities.

Fairness & governance

• Trusted-AI/AIF360 - Fairness metrics and bias-mitigation algorithms.
• Nexus-Router/nexus - Govern and secure your AI model usage.
• verifywise-ai/verifywise - AI governance platform with LLM evaluations.
• cvs-health/langfair - LLM bias and fairness assessments.
• SantanderAI/mech-gov-framework - Governance regimes for AI systems.
• JehanneDussert/govllm - Continuous LLM governance monitoring for regulated environments.
• debiai/DebiAI - Bias detection and contextual evaluation for AI projects.

Code review

• aryanbrite/openrabbit - AI code review bot posting inline PR feedback.
• antongulin/robin - Free AI code reviews on every pull request via Actions.
• james2doyle/ante-ci-bot - CI action running sub-agent reviewers on pull requests.
• joeblackwaslike/ai-review-bot - Autonomous multi-agent PR reviewer.
• mountainowl/bubo - Self-hosted agentic code review for GitHub and GitLab.
• nhatvu148/kaniscope-action - AI code review action with line-anchored inline comments.
• pgup-ai/jbot-review-action - GitHub Action for AI-powered PR reviews.
• pgup-ai/jbot-review - Agentic PR reviewer as in-repo workflow and GitHub App.
• secondsky/z.ai-code-review-bot - PR review bot with deterministic scanners and inline comments.
• alibaba/open-code-review - Hybrid deterministic + LLM code review, battle-tested at Alibaba scale.
• The-PR-Agent/pr-agent - AI pull-request review, description and improvement suggestions.
• jesseduffield/lazygit - Terminal UI for reviewing what your agent just did to git.
• sourcery-ai/sourcery - AI-powered code review and refactoring for Python.
• kodustech/kodus-ai - AI code review agent for pull requests.
• hyhmrright/brooks-lint - AI code reviews grounded in classic engineering books.
• HexmosTech/git-lrc - AI code reviews delivered on every commit.
• codexstar69/bug-hunter - Adversarial bug hunter for codebases.
• Nayjest/Gito - AI code review assistant for Git workflows.
• miracodeai/mira - Self-hosted AI code reviewer.
• devarshishimpi/codra - Self-hosted AI code review on Cloudflare Workers.
• Juror-AI/juror - AI code review agent for GitHub pull requests.
• simion/reviewd - Local AI pull request reviewer for the terminal.
• openqa-cn/codexqa - Agent skills for code review and defect scanning.
• officefloor/ImpactGate - Code review driven by change impact analysis.
• w1ckedxt/cynical-sally - Brutally honest AI code reviews with refactoring.
• mattzcarey/shippie - Extendable AI code review and QA agent for CI pipelines.
• modem-dev/hunk - Review-first terminal diff viewer for reviewing agent-written code.