Audio & Music

⏭️ Forward Deployed: Voice AI on what works in 2026

Aug 25, 2026 · 36:31

Basil Chatha hosts a fireside chat with voice-AI leaders Basia Sudol (Decagon), Varun Singh (Daily), Steven Diaz (Vapi), Tyler D'Silva (Retell AI), and Sudarshan Kamath (Smallest AI) arguing that production voice agents still run on cascaded STT→LLM→TTS pipelines, not end-to-end speech-to-speech. Varun explains why outbound calls are easier than inbound. Steven and Tyler defend cascaded for guardrails and context optimization; Sudarshan says Smallest's Hydra is multimodal but enterprises still prefer cascaded hybrids. On latency, Tyler says fillers make waits natural, Steven warns giant prompts hurt cost, and Varun suggests specialized sub-agents. They also cover multilingual TTS swapping, model choices, and open-source evals.

Mistral: Voxtral TTS, Forge, Leanstral, & Mistral 4 — w/ Pavan Kumar Reddy & Guillaume Lample

Mar 30, 2026 · 54:02

Mistral releases Voxtral TTS, a 3B-parameter speech generation model using a novel autoregressive flow matching architecture and a 12.5 Hz neural audio codec developed in-house. Pavan Kumar Reddy and Guillaume Lample explain how the model achieves high quality and efficiency by predicting audio latents as a continuous distribution, enabling real-time voice agent applications. The episode details Mistral's stepwise multimodal strategy, starting with transcription and now speech generation, with full duplex voice agents as the next goal. Enterprise deployment via Mistral Forge allows fine-tuning on proprietary data, exemplified by training on rare languages or domain-specific jargon. Guillaume discusses merging capabilities (Mistral Small merging coding, reasoning, vision), commitment to open science through technical reports, and reasoning transfer from formal proofs in Lean. The episode also covers AI for science partnerships and hiring for research and forward-deployed engineering roles.

Personalized AI Language Education — with Andrew Hsu, Speak

Jul 11, 2025 · 1:04:10

Andrew Hsu, CTO of Speak, explains how the company built the "third generation" of language learning by betting on AI speech and language models before they were ready. Speak focused on South Korea early, a counterintuitive move for a San Francisco startup, and now 6% of the Korean population has tried the app. The turning point came with Whisper and GPT in 2022, which enabled Speak to evolve from a listen-and-repeat tool into a full-featured AI tutor that gives real-time feedback and adapts to learners. Speak generates over $50M ARR primarily from consumer subscriptions, and Hsu details the product decisions that made it work: abandoning a free version, building custom ASR for low latency, and emphasizing functional fluency over textbook language. He also reveals that Speak is building a real-time voice platform and a knowledge graph for personalized fluency scores, with an eye toward expanding beyond language into general AI-powered education.

Voice AI Masterclass — Kwindla Hultman Kramer and swyx

May 6, 2025 · 20:51

Shawn Wang and Kwindla Hultman Kramer announce a voice AI masterclass course, diving into the landscape of models like Dia and Parakeet, production deployment challenges, and future trends such as speech-to-speech and real-time video. Kwindla explains that telephony (Twilio) drives 99% of current monetizable voice AI, while open source frameworks like Pipecat enable low-latency multi-modal apps. The course covers turn detection, context management, and evals, with 28 sessions featuring partners like OpenAI, Google, and NVIDIA. The goal is to jumpstart builders from prototype to production, with real-time video expected to hit its inflection point by year-end.

Snipd: The AI Podcast App for Learning — with CEO Kevin Ben-Smith

Mar 14, 2025 · 1:17:46

Snipd CEO Kevin Ben-Smith joins Swyx to explain how the AI-powered podcast app turns passive listening into active learning. Snipd transcribes and diarizes episodes, generates chapters, identifies mentioned books, and summarizes insights on demand — all with a four-person team. Kevin describes moving from self-hosted open-source models (Wave2vec, Pyannote) to a mix of Whisper, OpenAI, Google Gemini, and Perplexity APIs, balancing cost against the need for predictable batch processing vs. real-time user requests. He details technical hurdles like dynamic content resyncing (building a 'Shazam for podcasts') and using LLMs as judges to filter hallucinated book recommendations. The episode also covers Snipd's origin as a hackathon-winning podcast search tool, Swyx's own switch from Overcast, and Kevin's vision for voice-driven post-episode reflection and AI-assisted discovery as the next frontier.

How NotebookLM Was Made

Oct 25, 2024 · 1:13:57

Raiza Martin (NotebookLM lead PM) and Usama Bin Shafqat (AI engineer) explain how Google's NotebookLM built the viral 'Deep Dive' audio overview feature. They reveal the product evolved from Project Tailwind, using Gemini 1.5's long context and DeepMind speech to create a two-persona dialogue format that transforms documents into engaging podcasts. The team learned from 65,000 Discord members, leaned on best-selling author Steven Johnson for a 'tool for thought' workflow, and prioritized a single format over exposed controls to preserve unpredictability and delight. Humor and tension are not explicitly prompted but emerge from giving personas different angles. Evaulation relied on internal taste ('potatoes for chefs') before formal raters, with a Likert scale on dimensions like entertainment and groundedness. Future plans include multilingual support, API access, real-time chat, and codebase podcasting, while managing non-determinism by accepting occasional bad rolls.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.

Making Transformers Sing - with Mikey Shulman of Suno

Mar 14, 2024 · 58:58

In this episode of Latent Space, hosts Alessio and Shawn Wang interview Mikey Shulman, CEO of Suno, about making transformers sing. Suno uses transformers to predict audio tokens end-to-end, avoiding baked-in musical knowledge, with a tokenization secret sauce that also includes non-music audio for better vocal realism. Their models are relatively small (far below 175B parameters) due to latency needs, and they prioritize scaling research over brute-force size. Over half of Suno users employ expert mode, tweaking lyrics and style prompts, rather than easy mode. Shulman argues Suno is not the 'Midjourney of music' because music is inherently social and synchronous, unlike images. He demoed live generation, showing control via tokens like [beat drop] and style modifiers, and revealed future plans for collaborative concerts, continuous DJ modes, and personalized models. He also advocates for hiring economists to avoid Goodhart's law pitfalls in ML benchmarks, especially in audio where aesthetics matter most.

Generating your AI Media Empire - with Youssef Rizk of Wondercraft.ai

Sep 20, 2023 · 1:28:55

Youssef Rizk, co-founder of Wondercraft.ai, explains how his five-month-old startup uses hyperrealistic AI voices to turn blogs, newsletters, and other content into podcasts, arguing that the application layer—not the underlying API—is the real moat. He details Wondercraft's origin from a failed sports-investing startup, the rapid MVP that generated $3K in one day, and the decision to build script generation and audio production tools rather than train its own models. Rizk also announces a new AI dubbing feature for video, discusses the challenges of scaling to 28 languages, and reveals that the company runs on just four people pushing two major features per week. The conversation covers the daily Hacker News Recap podcast—which reached Spotify's top 30 tech podcasts—and Rizk's view that AI-generated content must be clearly disclosed to avoid backlash.