Awesome AI Agent Stack

Voice, Speech & Audio

Speech-to-text, text-to-speech and voice agents.

Speech-to-text

• openai/whisper - Robust Speech Recognition via Large-Scale Weak Supervision.
• ggml-org/whisper.cpp - Port of OpenAI's Whisper model in C/C++.
• SYSTRAN/faster-whisper - Faster Whisper transcription with CTranslate2.
• m-bain/whisperX - WhisperX: Automatic Speech Recognition with Word-level Timestamps (& Diarization).
• NVIDIA-NeMo/Speech - NVIDIA's toolkit for building production ASR, TTS, and conversational AI.
• modelscope/FunASR - Alibaba's industrial-grade speech recognition toolkit for production ASR.
• facebookresearch/seamless_communication - Foundational models for state-of-the-art speech and text translation.
• QwenAudio/SenseVoice - Multilingual ASR with language ID, emotion and audio-event detection.
• k2-fsa/sherpa-onnx - Offline ASR, TTS, diarization, VAD and enhancement via ONNX Runtime.
• k2-fsa/sherpa - Speech-to-text server framework built on next-gen Kaldi.
• speechbrain/speechbrain - PyTorch toolkit for ASR, diarization and speech processing.
• alphacep/vosk-api - Offline speech recognition for mobile, Raspberry Pi and servers.
• espnet/espnet - End-to-end speech processing toolkit for ASR, TTS and translation.
• PaddlePaddle/PaddleSpeech - Speech toolkit: streaming ASR/TTS, punctuation, speaker verification.
• argmaxinc/argmax-oss-swift - On-device speech AI for Apple Silicon (WhisperKit).
• ufal/whisper_streaming - Real-time streaming Whisper transcription and translation.
• collabora/WhisperLive - A nearly-live implementation of OpenAI Whisper.
• speaches-ai/speaches - OpenAI-compatible speech-to-text and text-to-speech inference server.
• ahmetoner/whisper-asr-webservice - Whisper ASR exposed as a webservice API.
• KoljaB/RealtimeSTT - Low-latency speech-to-text with VAD and wake-word activation.
• Vaibhavs10/insanely-fast-whisper - Whisper transcription accelerated with transformers and optimum.
• linto-ai/whisper-timestamped - Multilingual ASR with word-level timestamps and confidence scores.
• HaujetZhao/CapsWriter-Offline - Offline hotkey-driven dictation app with high accuracy and low latency.
• moonshine-ai/moonshine - Very low-latency speech-to-text for voice agents and interfaces.
• Picovoice/cheetah - On-device streaming speech-to-text engine.
• Picovoice/leopard - On-device speech-to-text engine for private local transcription.
• kyutai-labs/hibiki - Simultaneous speech translation that adapts its flow as you speak.
• wenet-e2e/wenet - Production-first, production-ready end-to-end speech recognition toolkit.
• k2-fsa/icefall - Recipes and training pipelines for next-gen Kaldi speech models.
• QuentinFuxa/WhisperLiveKit - Real-time local speech-to-text with streaming ASR and speaker diarization.
• Blaizzy/mlx-audio - Text-to-speech, speech-to-text and speech-to-speech library built on Apple MLX.
• thewh1teagle/vibe - Offline desktop app for transcribing audio and video with Whisper.
• MahmoudAshraf97/whisper-diarization - Whisper-based speech recognition with speaker diarization.
• FluidInference/FluidAudio - CoreML speech-to-text, text-to-speech and diarization models for Apple apps.
• jhj0517/Whisper-WebUI - Web UI for generating and translating subtitles with Whisper models.
• cjpais/Handy - Offline, extensible speech-to-text desktop app.
• zachlatta/freeflow - Free, fast dictation app; an open alternative to Wispr Flow.

Text-to-speech & voice cloning

• RVC-Boss/GPT-SoVITS - 1 min voice data can also be used to train a good TTS model! (few shot voice cloning).
• jamiepine/voicebox - The open-source AI voice studio. Clone, dictate, create.
• microsoft/VibeVoice - Open-Source Frontier Voice AI.
• 2noise/ChatTTS - A generative speech model for daily dialogue.
• index-tts/index-tts - An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System.
• fishaudio/fish-speech - State-of-the-art open-source text-to-speech with rapid voice cloning.
• myshell-ai/OpenVoice - Instant multilingual voice cloning from a short reference clip.
• espeak-ng/espeak-ng - Compact open-source speech synthesizer for 100+ languages and accents.
• nari-labs/dia - 1.6B TTS model generating ultra-realistic multi-speaker dialogue in one pass.
• SparkAudio/Spark-TTS - LLM-based bilingual TTS with fine-grained voice control and cloning.
• SWivid/F5-TTS - Flow-matching TTS with faithful zero-shot voice cloning from short samples.
• KittenML/KittenTTS - State-of-the-art TTS under 25MB for CPU and edge devices.
• rany2/edge-tts - Use Microsoft Edge neural TTS voices from Python with no API key.
• yxlllc/DDSP-SVC - Real-time end-to-end singing voice conversion based on DDSP.
• RVC-Project/Retrieval-based-Voice-Conversion-WebUI - Train a voice conversion model on 10 minutes of audio via web UI.
• edwko/OuteTTS - Lightweight interface for the OuteTTS 0.5B/1B voice-cloning models.
• bshall/knn-vc - Any-to-any voice conversion with just nearest neighbors.
• SesameAILabs/csm - Conversational speech generation model for natural dialogue voices.
• QwenLM/Qwen3-TTS - Streaming TTS with voice cloning and natural-language voice design.
• KoljaB/RealtimeTTS - Convert text to speech in realtime with low latency.
• snakers4/silero-models - Pre-trained TTS models with a simple one-line inference API.
• canopyai/Orpheus-TTS - Llama-based TTS with emotion tags and ~200ms streaming latency.
• resemble-ai/chatterbox - SOTA open-source TTS with zero-shot voice cloning and emotion control.
• QwenAudio/CosyVoice - Multilingual voice generation with full inference, training and deployment stack.
• k2-fsa/OmniVoice - High-quality voice-cloning TTS supporting 600+ languages.
• debpalash/VoiceStudio - Fully-local ElevenLabs alternative: cloning, dubbing, dictation, audiobooks.
• DrewThomasson/ebook2audiobook - Convert ebooks to audiobooks with TTS and voice cloning.
• kadirnar/voicehub - Unified inference interface across many open TTS models.
• OpenBMB/VoxCPM - Tokenizer-free multilingual text-to-speech model with voice design and cloning.
• OpenMOSS/MOSS-TTS - Open-source model family for long-form speech and dialogue synthesis.
• OpenMOSS/MOSS-TTS-Nano - 100M-parameter multilingual TTS model for real-time CPU inference.
• WhisperSpeech/WhisperSpeech - Open-source text-to-speech system built by inverting Whisper.
• devnen/Chatterbox-TTS-Server - Self-hosted server and web UI for the Chatterbox TTS model.
• RVC-Project/Retrieval-based-Voice-Conversion - Real-time voice conversion with retrieval-based models.
• shirohata/vc-rs - Real-time RVC voice changer in Rust with GUI and VST3 plugin.
• Turing-Mirror/RVC-Fabric - RVC voice changer studio with a modern GUI.

Voice agents

• gradio-app/fastrtc - Turn Python functions into realtime WebRTC audio/video streams.
• livekit-examples/agent-starter-react - Complete voice AI frontend for LiveKit Agents.
• kyutai-labs/unmute - Make any text LLM listen and speak with low latency.
• microsoft/call-center-ai - AI-driven call center on Azure and OpenAI GPT.
• llmrtc/llmrtc - TypeScript SDK for realtime multimodal voice and vision agents.
• fixie-ai/ultravox - Fast multimodal LLM for real-time voice agents.
• livekit/agents - Realtime multimodal voice-agent framework on LiveKit's WebRTC infrastructure.
• pipecat-ai/pipecat - Pipeline framework for voice and multimodal conversational AI agents.
• kyutai-labs/moshi - Real-time full-duplex speech-text model for conversational voice agents.
• livekit/livekit - End-to-end realtime WebRTC stack connecting humans and AI agents.
• TEN-framework/ten-framework - Open-source framework for building conversational voice AI agents.
• bolna-ai/bolna - Build production conversational voice AI agents quickly.
• roomkit-live/roomkit - Async Python framework for multi-channel conversation agents.
• kyutai-labs/delayed-streams-modeling - Kyutai's streaming speech-to-text and text-to-speech models.
• kyutai-labs/moshi-finetune - Fine-tuning toolkit for the Moshi full-duplex spoken dialogue model.
• agent-next/call-use - Open-source runtime giving AI agents outbound phone-call ability.
• huggingface/speech-to-speech - Pipeline for building voice agents from open-source models.
• GetStream/Vision-Agents - Framework for building real-time voice and vision agents with any model.
• dograh-hq/dograh - Open-source self-hosted voice AI platform, alternative to Vapi and Retell.

Audio processing & analysis

• facebookresearch/audiocraft - Audiocraft is a library for audio processing and generation with deep learning. It features the.
• Picovoice/porcupine - On-device wake word detection engine.
• Picovoice/cobra - On-device voice activity detection engine.
• Picovoice/falcon - On-device speaker diarization engine.
• snakers4/silero-vad - Enterprise-grade pre-trained voice activity detector.
• juanmc2005/diart - Real-time speaker diarization and audio streaming toolkit.
• pyannote/pyannote-audio - Neural building blocks for speaker diarization pipelines.
• deezer/spleeter - Source separation library with pretrained models.
• asteroid-team/asteroid - PyTorch audio source separation toolkit for researchers.
• facebookresearch/audioseal - Localized watermarking for AI-generated speech audio.
• iver56/audiomentations - Audio data augmentation library for robust real-world ML models.
• descriptinc/descript-audio-codec - State-of-the-art neural audio codec with 90x compression.
• lhotse-speech/lhotse - Tools for handling speech data in machine learning projects.
• librosa/librosa - Python library for audio and music analysis.
• MTG/essentia - C++ library for audio analysis, description and synthesis.
• pytorch/audio - Audio signal processing and transforms for PyTorch.
• aubio/aubio - Library for audio and music analysis and onset detection.
• spotify/pedalboard - Python library for audio effects and plugin processing.
• gemelo-ai/vocos - Fast high-quality neural vocoder bridging time and Fourier domains.

Telephony & SIP

• signalwire/freeswitch - Software-defined telecom stack for voice and video.
• asterisk/asterisk - The official Asterisk open-source PBX repository.
• OpenSIPS/opensips - High-performance SIP server for VoIP platforms.
• drachtio/drachtio-server - SIP call processing server controlled via Node.js applications.
• kamailio/kamailio - Open-source SIP server for large real-time communication platforms.
• sipwise/rtpengine - Media proxy for Kamailio SIP deployments.
• baresip/baresip - Modular SIP user agent with audio and video support.
• pjsip/pjproject - Multimedia communication library with a SIP stack.
• jambonz/jambonz-feature-server - Telephony feature server for the Jambonz CPaaS platform.
• OpenVoiceOS/ovos-audio - Audio output daemon for the OpenVoiceOS voice assistant.
• OHF-Voice/wyoming - Peer-to-peer protocol for Home Assistant voice assistants.