Inference

Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

Aug 3, 2026 · 1:42:54

Philip Kiely and Ali Taha of Baseten join Swyx to explain what actually happens when a 200,000-token request hits a production inference system, arguing that stacking quantization, speculative decoding, and disaggregated prefill/decode can make open models like GLM-5.2 up to 10x faster. They detail cache-aware routing, training traffic-specific speculators, and why quantization errors can cancel out so a more-quantized model beats a less-quantized one. They also reveal how Baseten grafted Kimi's vision encoder onto GLM-5.2, discuss NVIDIA Dynamo as a toolkit rather than a turnkey speedup, and explain why they are bearish on mega kernels. The conversation covers video generation's compute barriers, the trend toward ASIC-like GPUs, and the emerging loop where GLM-5.2 writes the GPU kernels that serve itself.

The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO

Jul 8, 2026 · 59:10

Modal CTO Akshat Bubna explains how the cloud platform evolved from a serverless runtime to an AI cloud built for elastic inference, agent sandboxes, and post-training workloads. He argues that Kubernetes was never designed for bursty GPU-heavy AI workloads, so Modal built a decorator-based infrastructure that co-locates compute requirements with code. The company added GPUs a year before ChatGPT and now powers inference for custom models at Suno, Runway, and robotics firms, using GPU snapshotting to slash cold starts. Modal's new Auto Endpoints incorporate open-source DeFlash speculative decoding for frontier-level performance. For reinforcement learning rollouts, Modal provides up to 100,000 sandboxes simultaneously. The platform spans 17 cloud providers, features private IPv6 networking via eBPF, and supports serverless multi-node training with RDMA. Bubna also reveals Modal's shift from developer experience to agent experience, noting that agents benefit from the same minimalist SDK and observability tools that humans use.

⚡️ Google's Open AI Strategy — Omar Sanseviero, Google DeepMind

May 24, 2026 · 29:59

Omar Sanseviero, Google DeepMind's Head of Developer Experience, explains Gemma 4's novel architecture with per-layer embeddings that enable effective parameter offloading: only 2B of 5B parameters need GPU memory, ideal for on-device inference on phones and Raspberry Pis. The model matches 1.5-year-old state-of-the-art in most areas, with Gemini Nano integrated into Pixel and Samsung phones. Gemma 4 supports multimodal input (audio, images, short video) but not audio output or combined audio-video prompts. Sanseviero notes fine-tuning is declining as out-of-box capabilities improve, but remains relevant for specialized domains like healthcare. He contrasts dense (31B) and MoE (27B) variants, highlighting MoE's inference speed but fine-tuning challenges. The team is expanding globally, with Kaggle joining DeepMind to create community-driven benchmarks for model evaluation.

Agent Inference at the "Speed of Light" — How NVIDIA moves like a $4.3 Trillion Startup

Mar 8, 2026 · 1:26:00

NVIDIA's Nader Khalil and Kyle Kranen join Swyx and Vibhu to explain how the company moves like a $4.3 trillion startup through speed-of-light (SOL) first-principles thinking, agent security boundaries, and the Dynamo inference engine. They argue agents should only do two of three things (files, internet, code) to prevent vulnerabilities, and detail Brev's acquisition to improve developer UX with one-click GPU access and DGX Spark integration. Kyle describes Dynamo as a data center scale inference engine that optimizes serving by scaling out, using prefill/decode disaggregation, Kubernetes-based scheduling, and model-hardware co-design to improve cost, latency, and quality. The episode covers SOL's role in creating urgency, long-context limits and potential 'unhobblers' like multi-head latent attention, and the shift toward CLI-first agent workflows for enterprise tools.

The AI Frontier: from Gemini 3 Deep Think distilling to Flash — Jeff Dean

Feb 12, 2026 · 1:23:31

Jeff Dean, Google's Chief AI Scientist, explains how the company's dual strategy of frontier "Pro" models and distilled "Flash" models, alongside co-designed TPUs and latency-optimization, keeps Gemini at the Pareto frontier. Distillation compresses high-capability large models into smaller, efficient ones, enabling Flash models to exceed earlier Pro generations. He emphasizes energy-based thinking: data movement costs thousands more picojoules than arithmetic, making batching and speculative decoding essential. TPU development requires predicting ML workloads 2–6 years out, often speculatively adding hardware features. Gemini was born after Dean wrote a one-page memo arguing that fragmented model efforts across Google Brain and DeepMind should merge into a single unified multimodal effort. Looking ahead, he predicts personalized models attending to all user data and reasoning speeds of 10,000 tokens/second will transform software engineering.

The Future of Email: Superhuman CTO on Your Inbox As the Real AI Agent (Not ChatGPT) — Loïc Houssier

Dec 11, 2025 · 1:11:02

Loïc Houssier, CTO of Superhuman Mail (recently acquired by Grammarly), joins hosts Alessio and Swyx to detail how Superhuman builds AI into email without adding latency. He explains their agentic framework of small tools vs. a single agent, the fight against “agent laziness,” and their eval process anchored by Rahul’s infamous “what wood was my table?” test. Houssier reveals Superhuman uses local-first caching and Baseten’s box pricing for cost control, stores embeddings in TurboPuffer, and has only three engineers on AI—yet PR throughput rose from four to six per engineer per week. He argues the inbox will power your future AI executive assistant, and that AI will widen the gap between engineers with real fundamentals and those faking it.

⚡️Raising $1.1b to build the fastest LLM Chips on Earth — Andrew Feldman, Cerebras

Oct 1, 2025 · 29:14

Andrew Feldman, CEO of Cerebras, joins Latent Space to discuss their $1.1B fundraise at an $8.1B valuation and their wafer-scale chip that delivers 20x faster inference than NVIDIA's B200 GPUs. He explains how their architecture uses SRAM instead of HBM, providing 2,625x more memory bandwidth by eliminating the narrow straw between compute and memory. Feldman details the decision to accelerate sparse linear algebra rather than specialized convolutions, enabling support for transformers and diffusion models unseen during design. He discusses the explosive growth in AI inference demand, the shift from closed-source to fast open-source models, and the importance of speed—citing Paul Graham's observation that ChatGPT's slowness drives users away. The conversation covers enterprise trends in the 10-30B parameter space, the complexity of building data centers that pull gigawatts of power, and the often-overlooked routing and caching systems that make AI work seamlessly.

⚡️ Beyond Transformers with Power Retention

Sep 23, 2025 · 32:59

Jacob Buckman, CEO of Manifest AI, discusses their solution to AI's expensive computational bottleneck: the ever-growing KV cache for long context inference. Power Retention replaces attention in transformers with a fixed-size memory, achieving 10X training speedup at 64K tokens and 100X inference speedup over FlashAttention. Manifest releases Vidrial, a just-in-time CUDA kernel framework that finds optimal configurations, and Metamorphosis, a process to convert existing transformers like StarCoder-3B into Power Retention models in just two hours of mid-training. Buckman demonstrates PowerCoder, a 3B code model that matches baseline loss after 10K steps and handles contexts up to 32K and beyond. The open-source release includes kernels and tools, aiming to build community trust and encourage adoption by inference providers. He also highlights the need for genuinely long-context datasets beyond internet text, such as human trajectories.

A Technical History of Generative Media

Sep 8, 2025 · 1:04:44

Fal.ai founders Gorkem and Batuhan detail their pivot from dbt pipelines to generative media inference, which now serves 2M developers and 350 models, crossed $100M ARR, and raised a $125M Series C. Key inflection points included Stable Diffusion 1.5 (company pivot), SDXL (first $1M revenue), Flux (jump from $2M to $10M monthly revenue), and Veo 3 (text-to-video with perfect lip-sync). They built a proprietary inference engine with 100+ custom kernels and a serverless GPU stack managing 10,000+ H100 equivalents across 6 cloud providers, typically delivering 1.5-10x speedups over stock PyTorch. Video models now drive 50% of revenue, up from 18% in February, fueled by open-source models like Hunyuan and partnerships with closed labs such as Play.ht and Google DeepMind. They argue advertising is the killer application for generative media and that image/video RL, specialized data pipelines, and cheaper conversational video models are underexplored startup opportunities.

⚡️Accelerators @ 3x NVIDIA H200 perf, Made in the USA - Thomas Sohmers + Mitesh Agrawal, Positron AI

Aug 18, 2025 · 47:56

Thomas Sohmers and Mitesh Agrawal of Positron AI argue memory bandwidth, not compute, is the true bottleneck in AI inference, and their accelerator achieves 93% memory bandwidth utilization—triple NVIDIA's efficiency—enabling 70% faster token generation at 150W. The founders, both Lambda Labs veterans, shipped an FPGA product 15 months after founding, then raised a $51M Series A for an ASIC in late 2026. Their hardware requires no recompilation: it ingests raw binary weights from NVIDIA training and outputs an OpenAI-compatible API. Positron focuses on the decode phase of transformers, where memory-bound matrix-vector multiplication dominates, and already counts Cloudflare and Parasail as customers. The company sells systems directly, prioritizing capital efficiency and ROIC over operating its own cloud.

The AI Agenda: GPT5 leaks and the business of AI News — Steph Palazzolo, The Information

Aug 6, 2025 · 1:14:36

Steph Palazzolo, AI journalist at The Information, explains how she covers the secretive AI industry, from OpenAI's GPT-5 to the inference market where startups like Modal raise billions. She reveals that inference providers are essentially GPU resellers facing margin pressure, and that GPT-5's performance will signal whether pre-training scaling or reinforcement learning drives progress. She critiques Meta's super intelligence ambition as misaligned with its ad-revenue business, and details the talent war with rumored hundred-million-dollar offers from Meta to poach researchers. Palazzolo also discusses the Windsurf acqui-hire backlash, the rise of coding agents like Claude Code, and how journalists protect sources while navigating leaks from CEOs like Sam Altman.

The Shape of Compute (Chris Lattner of Modular)

Jun 13, 2025 · 1:18:18

Chris Lattner of Modular explains how his company is breaking the CUDA monopoly with Mojo and MAX, matching NVIDIA's best inference performance on AMD and NVIDIA GPUs. After three years in R&D, Modular open-sourced its stack and now delivers state-of-the-art Llama 3 serving at over 800 tokens per second. Mojo, a Python-family language, runs kernels faster than Rust and can extend Python without bindings, while MAX provides a full inference framework with automatic kernel fusion and cluster management. Unlike VLLM or SGLang, Modular's container is just a gigabyte, fully open source, and not reliant on proprietary CUDA blobs. DeepSeek's low-level PTX work validated the approach, but Lattner emphasizes that Mojo's portability avoids rewriting kernels for each new GPU architecture. He also details his daily routine, using Cursor for coding, and hiring "elite nerds" to grow the team.

S1: the $6 DeepSeek R1 Competitor (ft. Entropix)

Feb 26, 2025 · 17:43

Tim Kellogg joins hosts Alessio and Swyx to break down his viral blog post on S1, showing how Stanford researchers cloned DeepSeek R1's reasoning for $6 by inserting a 'think token' and fine-tuning on just 1,000 examples. Kellogg argues the real innovation isn't the cost but the technique's simplicity—making advanced inference-time scaling accessible. He explains Entropix, a dynamic sampler that uses model entropy and varentropy to adjust generation on the fly, and notes its creators are starting a company. The conversation contrasts supervised fine-tuning's targeted results with RL's broader reasoning growth, using an analogy: 'RL causes a lot of growth, then SFT trims it.' Kellogg predicts that embedding introspection signals into RL training could solve agent 'doom loops' and unlock higher agency—a key challenge behind products like OpenAI's Deep Research.

Outlasting Noam Shazeer, Crowdsourcing Chai AI w/ 1.4m DAU — with William Beauchamp, Chai Research

Jan 26, 2025 · 1:13:33

William Beauchamp, founder-CEO of Chai AI, explains how he pivoted from algorithmic trading to build a character chatbot platform before Character.ai, growing to 1.4M DAU and $22M+ revenue by crowdsourcing model improvements through Chaiverse. Starting with GPT-J in 2021, he found product-market fit with a therapist bot and shifted to user-generated content, letting users define prompts, images, and names. Competing against well-funded rivals like Character AI and Talkie, Chai ships over 100 LLMs weekly via its developer platform, spending $10M on compute in 2024 and tripling it. Beauchamp argues AI follows an S-curve, not scaling laws, and focuses on inference optimization using rejection sampling and reward models to serve better responses. He prioritizes 'insanely great' products over technology-driven features, noting that audio and image features failed to move metrics, while a data flywheel and aggressive user acquisition drove rapid growth.

DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)

Jan 19, 2025 · 57:18

Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Dec 24, 2024 · 28:08

Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering.

Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI

Nov 25, 2024 · 55:49

Lin Qiao, CEO of Fireworks AI, argues that compound AI systems combining multiple open-source models will outperform closed monolithic models like OpenAI's. She explains Fireworks' evolution from a PyTorch platform to a full-stack inference and customization engine serving 40+ customers including Cursor. Qiao details their distributed inference engine, Fire Optimizer for quality-latency-cost tradeoffs, and upcoming o1-like model built on open-source foundations. She defends their quantization approach after public criticism from rivals, emphasizes specialization over general intelligence, and invites developers to test their free LoRA adapter hosting.

[Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval

Sep 19, 2024 · 53:53

Umar Jamil from Writer presents 'Writing in the Margins,' a paper on improving long-context retrieval in LLMs by leveraging chunked prefill of the KV cache. The method splits a long prompt into chunks, generates query-relevant annotations (margins) after each chunk, and appends them at the end before generating the final answer. This approach solves the 'lost in the middle' problem without fine-tuning, and unlike traditional RAG or separate summarization, it avoids re-prefilling the entire context, reducing cost from double to single prefilling. The technique is compatible with any transformer model and includes overlapping margin generation and classification within the same batch request. Benchmarks show consistent improvements across models, and the implementation is open-source on GitHub.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Feb 28, 2024 · 1:21:09

Ben Firshman, CEO of Replicate, explains how the inference platform grew from a research reproducibility tool into a 2M-user API business by embracing the generative image community and treating open-source AI as a hacker-friendly ecosystem. They accidentally discovered their API when a user reverse-engineered their web form, leading to their first $1k/month customer. Cog, their container standard for ML models, was born from lessons at Docker and the need to make models tinkerable. Ben argues that fine-tuning's low cost makes open-source models sustainable, and that AI engineers (orders of magnitude more than ML engineers) just need to start playing with models. He also discusses GPU scarcity, preferring sustainable pricing over price wars, and reveals that demand is not outpacing supply thanks to aggregating demand.

Building an open AI company - with Ce and Vipul of Together AI

Feb 8, 2024 · 1:15:18

Together AI co-founders Vipul Ved Prakash and Ce Zhang explain why openness is core to their mission, detailing their journey from Apple (Vipul) and Stanford research (Ce) to building an open AI platform. They discuss RedPajama’s evolution into a modular dataset with 40 quality signals, the need for 5,000 tokens/second inference speed, and their investment in state space models like Mamba and Hyena as alternatives to transformers. The company, which runs 7,000-8,000 GPUs (mostly H100s), sees training as a larger workload than inference, with fine-tuning driving top models. They advocate for independent inference benchmarks, publish open research like FlashAttention, and keep some software proprietary. With 38 employees and 45% researchers, they are hiring across the stack, from CUDA to DevOps.

The Four Wars of the AI Stack - Dec 2023 Recap

Jan 26, 2024 · 1:20:58

Swyx and Alessio recap December 2023 by framing the AI landscape as four wars: the Data War (NYT lawsuit demanding destruction of GPTs, OpenAI's data partnerships, synthetic data from DeepMind); the GPU/Inference War (Mixtral price dropping 90% to $0.27/million tokens, benchmark drama between Anyscale and Together, alternative architectures like Mamba); the Multimodality War (Midjourney reaching $200M ARR, ElevenLabs hitting unicorn status, OpenAI and Google building god models); and the RAG/Ops War (LangChain vs LlamaIndex, Pinecone's $750M valuation, Qdrant serving OpenAI and Anthropic internally). They argue agents and open source are not active wars, and predict 2024 will see AI finally reach production, with hardware like Rabbit R1 and Tab capturing unique context.

The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

Dec 5, 2023 · 1:07:55

Dylan Patel of SemiAnalysis argues the 'GPU poor vs GPU rich' divide defines AI, with Google's TPU ramp (millions of units) dwarfing others and Nvidia selling over a million H100s this year. He explains that training costs are irrelevant compared to inference costs, and that memory bandwidth is the key bottleneck for LLM inference (e.g., Llama 70B needs ~2.1 TB/s for human reading speed). Patel advises GPU-poor players to focus on on-device innovations like speculative decoding (Medusa), asynchronous training, or fine-tuning for edge use—not on gaming benchmarks or fine-tuning small models. He details why AI hardware startups (Cerebras, Graphcore) bet wrong on on-chip SRAM, while Google’s TPU/Broadcom partnership and Nvidia’s yearly cadence make alternatives tough. He also warns that rebuilding the semiconductor supply chain in the US is unfeasible due to its fragmentation, and that safety through obscurity doesn't work—open innovation is better.