Baseten

Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

Aug 3, 2026 · 1:42:54

Philip Kiely and Ali Taha of Baseten join Swyx to explain what actually happens when a 200,000-token request hits a production inference system, arguing that stacking quantization, speculative decoding, and disaggregated prefill/decode can make open models like GLM-5.2 up to 10x faster. They detail cache-aware routing, training traffic-specific speculators, and why quantization errors can cancel out so a more-quantized model beats a less-quantized one. They also reveal how Baseten grafted Kimi's vision encoder onto GLM-5.2, discuss NVIDIA Dynamo as a toolkit rather than a turnkey speedup, and explain why they are bearish on mega kernels. The conversation covers video generation's compute barriers, the trend toward ASIC-like GPUs, and the emerging loop where GLM-5.2 writes the GPU kernels that serve itself.

The Future of Email: Superhuman CTO on Your Inbox As the Real AI Agent (Not ChatGPT) — Loïc Houssier

Dec 11, 2025 · 1:11:02

Loïc Houssier, CTO of Superhuman Mail (recently acquired by Grammarly), joins hosts Alessio and Swyx to detail how Superhuman builds AI into email without adding latency. He explains their agentic framework of small tools vs. a single agent, the fight against “agent laziness,” and their eval process anchored by Rahul’s infamous “what wood was my table?” test. Houssier reveals Superhuman uses local-first caching and Baseten’s box pricing for cost control, stores embeddings in TurboPuffer, and has only three engineers on AI—yet PR throughput rose from four to six per engineer per week. He argues the inbox will power your future AI executive assistant, and that AI will widen the gap between engineers with real fundamentals and those faking it.

DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)

Jan 19, 2025 · 57:18

Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.