NVIDIA

The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO

Sep 2, 2026 · 44:03

A comedic episode parodying AI hardware marketing, featuring Sean Lie discussing fictional Cerebras CS4 chips with absurd specs like 30 billion parameter models running at 4,400 TPS with wafer-scale integration and jalapeño-flavored SRAM designs.

🔬 Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar, Caltech

Aug 26, 2026 · 1:23:32

Anima Anandkumar, Caltech professor and ex-NVIDIA lead, argues physical-world modeling needs neural operators that bake in physics, not language-style scaling; it already rivals supercomputers. Physics-informed nets fail; Fourier neural operators capture multi-scale non-local phenomena, and FourCastNet, trained on 50,000 reanalysis samples, forecasts weather tens of thousands of times faster on a consumer GPU. FourCastNet 3’s spherical harmonics keep month-long rollouts stable for ensemble climate prediction; the same operators build a fusion digital twin a million times faster than simulation and inverse design. TorchLean ports PyTorch to Lean for formal robustness proofs, and on the UN Science Advisory Board she urges AI for science not be regulated like chatbots. Bottleneck: more compute.

⏭️ Forward Deployed: Voice AI on what works in 2026

Aug 25, 2026 · 36:31

Basil Chatha hosts a fireside chat with voice-AI leaders Basia Sudol (Decagon), Varun Singh (Daily), Steven Diaz (Vapi), Tyler D'Silva (Retell AI), and Sudarshan Kamath (Smallest AI) arguing that production voice agents still run on cascaded STT→LLM→TTS pipelines, not end-to-end speech-to-speech. Varun explains why outbound calls are easier than inbound. Steven and Tyler defend cascaded for guardrails and context optimization; Sudarshan says Smallest's Hydra is multimodal but enterprises still prefer cascaded hybrids. On latency, Tyler says fillers make waits natural, Steven warns giant prompts hurt cost, and Varun suggests specialized sub-agents. They also cover multilingual TTS swapping, model choices, and open-source evals.

Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

Aug 3, 2026 · 1:42:54

Philip Kiely and Ali Taha of Baseten join Swyx to explain what actually happens when a 200,000-token request hits a production inference system, arguing that stacking quantization, speculative decoding, and disaggregated prefill/decode can make open models like GLM-5.2 up to 10x faster. They detail cache-aware routing, training traffic-specific speculators, and why quantization errors can cancel out so a more-quantized model beats a less-quantized one. They also reveal how Baseten grafted Kimi's vision encoder onto GLM-5.2, discuss NVIDIA Dynamo as a toolkit rather than a turnkey speedup, and explain why they are bearish on mega kernels. The conversation covers video generation's compute barriers, the trend toward ASIC-like GPUs, and the emerging loop where GLM-5.2 writes the GPU kernels that serve itself.

The AI Frontier: from open weights to open research — Eiso Kant, Poolside AI

Jul 22, 2026 · 1:56:13

Poolside co-founder Eiso Kant argues that open weights and open research are essential for a future with many foundation model companies, and that models like their new Laguna S (118B parameters, 8B active) can achieve remarkable persistence and capability through post-training reinforcement learning. He explains how Poolside's Model Factory enables 5-8 week model cycles with 10,000-20,000 experiments per month, streaming data directly into training, and perfect reproducibility. Kant shares why Poolside embraced open source after starting as a closed company, criticizes MCP and tool calls in favor of models writing code directly, and predicts reinforcement learning will move earlier into pre-training. He also calls on researchers to start new foundation model companies to avoid an oligopoly of intelligence.

🔬 RL with Verifiable Rewards, but the Verifier is a Lab — Lila Sciences

Jul 16, 2026 · 1:41:04

Andy Beam (CTO) and Rafa Gómez-Bombarelli (Co-founder & CSO of Physical Sciences) of Lila Sciences argue that science is an 'infinite token generator' for AI, using reinforcement learning with verifiable rewards where the wet lab acts as the verifier. They claim one general model trained on ~10 trillion experimentally-verified reasoning tokens across biology, chemistry, and materials outperforms domain-specific models—'breadth gives us depth.' Their AI Science Factories treat the lab as a data center, with instruments on a 'PCI bus' and humans 'below the API line.' Highlights include a CAR-T candidate designed in six months by two or three people, 'monster UTRs' achieving ~10x Moderna/Pfizer mRNA expression, and the 'zero-FTE startup' business model. They discuss RL pathologies like collapsed chains of thought and a model that 'swears,' and note why there is still no AlphaFold for materials due to the sim-to-real gap.

Podcast Crossover: AIE, AGI, frontier lab strategy with ​ ⁨@matthew_berman⁩ and @swyxtv

Jul 10, 2026 · 28:03

Shawn 'Swyx' Wang, founder of the AI Engineer conference, tells Matthew Berman how he seized the industry shift by buying ai.engineer and partnering with a veteran conference organizer, crediting Andrej Karpathy's early endorsement. He argues Etched's ASICs are a natural next-gen bet for transformer inference, not an NVIDIA disruptor, and that Fable 5's slowness and cost signal the end of the LLM scaling era—making model efficiency the next problem. Swyx interprets OpenAI's reported 5% equity offer to the US government as a pragmatic multi-turn negotiation, likening it to Singapore's Temasek model, but warns against premature utility regulation. He pegs his own P(doom) at ~5% over 50 years, rejecting near-term doomerism as egotistical. For founders, he advocates building 'agent labs' that solve specific customer problems (e.g., for lawyers, dentists) rather than betting on model routing, which he dismisses as a marketing line that fails to exploit a single model's full stack as deeply as frontier labs do.

🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"

Jun 30, 2026 · 1:48:40

Evan Feinberg and Sergey Edunov of Genesis Molecular AI argue that diffusion models have unlocked sub-ångström accuracy in protein-ligand structure prediction, a breakthrough that makes AI useful for real drug discovery where the field’s favored 2Å RMSD benchmark is "slop." Their PEARL model uses diffusion with physics-based guidance, synthetic training data from molecular dynamics, and inference-time scaling to predict induced fit—how a protein flexes to accommodate a ligand. On the OpenBind benchmark, PEARL zero-shot surpassed all cofolding models on the notoriously hard EV A721A protease, correctly predicting a flexible loop movement that other methods missed. They also introduce SAPPHIRE, an agentic system that orchestrates AI models for 24/7 drug design, and discuss how downstream ADMET properties (solubility, toxicity, etc.) remain equally critical. The biggest bottleneck they face is GPU availability, and they are actively hiring AI researchers interested in novel architectures beyond standard transformers.

Why AI Labs With Unlimited GPUs Still Fail — Anjney Midha, AMP

Jun 18, 2026 · 1:00:37

Anjney Midha, CEO of AMP, argues that AI labs with unlimited GPUs still fail due to misaligned culture and infrastructure waste, proposing a compute grid modeled on independent system operators to pool demand and supply. At Google, 95% node utilization was considered an outage, yet most clusters today don't reach that, with waste compounding at scale. AMP’s grid, starting at scheduling, aims to make FLOPs flow like megawatts, having secured 1.3 gigawatts of demand. Midha explains Anthropic cracked coding because 'luck favors the prepared mind'—their four years of paranoia and scarcity created a culture that OpenAI’s abundance couldn't replicate. He also shares a 14-year mission in end-of-life prediction, arguing AI can reduce the 30% of Medicare/Medicaid spend on end-of-life care. He warns that too much capital too early makes labs fragile because without hardship they fail to define their P0.

Inside xAI: Building Grok Imagine in 3 Months, Videogen vs World Models, and Video Agents— Ethan He

Jun 1, 2026 · 1:44:43

Ethan He, former xAI and NVIDIA Cosmos researcher, explains how xAI built its first image and video models (Grok Imagine 0.9) from zero to one in three months, attributing rapid iteration to small teams with minimal meetings and strong infra that enabled fixing tiny data and training bugs for biggest quality gains. He argues that most improvements in video generation now come from language models and agents rather than diffusion technology, predicting that by end of 2025 video agents will produce production-grade content for ads. He defines world models as real-time, interactive, long-horizon videos, and details challenges like temporal compression, context management, and the high cost of storing and moving video data (e.g., tens of petabytes for a billion videos). Ethan also shares why he left xAI to focus on language model research, believing the next frontier is models that manage their own context length, similar to solutions already being explored in video generation.

⚡️ Google's Open AI Strategy — Omar Sanseviero, Google DeepMind

May 24, 2026 · 29:59

Omar Sanseviero, Google DeepMind's Head of Developer Experience, explains Gemma 4's novel architecture with per-layer embeddings that enable effective parameter offloading: only 2B of 5B parameters need GPU memory, ideal for on-device inference on phones and Raspberry Pis. The model matches 1.5-year-old state-of-the-art in most areas, with Gemini Nano integrated into Pixel and Samsung phones. Gemma 4 supports multimodal input (audio, images, short video) but not audio output or combined audio-video prompts. Sanseviero notes fine-tuning is declining as out-of-box capabilities improve, but remains relevant for specialized domains like healthcare. He contrasts dense (31B) and MoE (27B) variants, highlighting MoE's inference speed but fine-tuning challenges. The team is expanding globally, with Kaggle joining DeepMind to create community-driven benchmarks for model evaluation.

The $15B Physical AI Company: Simulation, Autonomy OS, Neural Sim, & 1K Engineers—Applied Intuition

Apr 27, 2026 · 1:14:07

Applied Intuition co-founders Peter Ludwig and Qasar Younis argue that the real bottleneck in physical AI is deploying intelligence onto constrained hardware, not model intelligence itself. Their $15B company builds simulation, operating systems, and AI models for autonomous trucks, mining equipment, and defense systems. Starting as YC-era tooling for robotaxis, they now offer 30+ products across simulation & RL infrastructure, vehicle operating systems, and autonomy models. They compare fragmented vehicle software to pre-Android phones, and their OS enables reliable updates and L4 driverless operations (trucks running in Japan today). Verification uses statistical nines of reliability, and they internally adopt coding agents like Cursor and Claude Code. They hire 1,000 engineers at the hardware-software boundary.

AI-Native Engineering: 100% adoption, 5x search throughput, unlimited tokens — Mikhail Parakhin

Apr 22, 2026 · 1:14:30

Shopify CTO Mikhail Parakhin details how the company achieved near 100% AI tool adoption, driven by a December 2025 inflection point where model quality triggered exponential token consumption. He argues token budgets matter only with strong critique loops: running multiple parallel agents without communication wastes tokens, while agentic PR review using large models—like GPT-5.4 Pro or Gemini Deep Think—reduces bugs even though latency increases. Parakhin unveils Tangle, Shopify's third-generation ML workflow system with content-based caching that eliminates duplicate computation across teams, and Tangent, an auto-research loop that optimizes pipelines—boosting search throughput from 800 to 4,200 QPS on the same hardware. SimGym simulates shoppers using decades of merchant data and browser-based agents to predict conversion changes, achieving 0.7 correlation with add-to-cart events. Shopify uses Liquid AI (non-transformer architecture) for sub-30ms query understanding and long-context tasks like catalog categorization, distilling larger models into Liquid for high-throughput batch jobs.

Marc Andreessen introspects on Death of the Browser, Pi + OpenClaw, and Why "This Time Is Different"

Apr 3, 2026 · 1:16:20

Marc Andreessen argues that AI is finally different from past boom-bust cycles because of four compounding breakthroughs: LLMs, reasoning (o1, R1), coding agents (OpenClaw), and recursive self-improvement. He calls this the '80-year overnight success'—decades of neural network research now paying off. Comparing today's AI capex boom to the dot-com crash, he notes that buyers like Microsoft and Google are cash-rich incumbents and every GPU deployed is already generating revenue. He hails Pi and OpenClaw as a Unix-like architecture that makes agents model-independent and self-modifiable. On open source, he calls DeepSeek a 'gift to the world' for its paper and code, but warns that entrenched institutions—unions, licensing, government monopolies—will slow AI adoption far more than technologists expect.

Moonlake: Interactive, Multimodal World Models — with Chris Manning and Fan-yun Sun

Apr 2, 2026 · 1:06:48

Moonlake AI founders Chris Manning and Fan-yun Sun argue that interactive, multimodal world models require structured symbolic reasoning over pure scale, enabling indefinite multiplayer gameplay and causal consistency that video generation models like Genie and Sora cannot achieve. Their approach uses code engines and physics simulators as cognitive tools, producing reasoning traces that handle geometry, physics, and logic, while a separate diffusion model (Reverie) handles pixel fidelity. They aim to replace traditional rendering and empower creators by allowing human intent to be injected at a symbolic layer. Manning contrasts this with Yann LeCun's JEPA, emphasizing language and abstraction over pixel-level prediction. Moonlake is hiring engineers at the intersection of code generation, computer vision, and graphics.

Anthropic’s Felix Rieseberg on AI Coworkers, Local-First Agents, and the Future of Knowledge Work

Mar 17, 2026 · 1:27:34

Anthropic's Felix Rieseberg explains how Claude Cowork—a VM-based, user-friendly version of Claude Code—was built from pre-existing prototypes in ten days, and why putting AI 'where you work' on the local machine is more powerful than cloud-only approaches. He argues that Silicon Valley undervalues the local computer, comparing it to how we all use MacBooks instead of iPads. The VM gives Claude its own computer, enabling it to install tools and run scripts without constant approval, serving as both a safety boundary and a capability unlock. Rieseberg contrasts Cowork's evaluation on knowledge work tasks (e.g., mortgage management, tax filing) with Claude Code's coding evals, and explains the shift from MCPs to file-based skills—markdown files that tell the model what to do. Real use cases include organizing desktops, uploading videos, and filing taxes. He also expresses concern about the impact on junior employees and hints at future work on autonomy, multiplayer coordination, and portability.

Agent Inference at the "Speed of Light" — How NVIDIA moves like a $4.3 Trillion Startup

Mar 8, 2026 · 1:26:00

NVIDIA's Nader Khalil and Kyle Kranen join Swyx and Vibhu to explain how the company moves like a $4.3 trillion startup through speed-of-light (SOL) first-principles thinking, agent security boundaries, and the Dynamo inference engine. They argue agents should only do two of three things (files, internet, code) to prevent vulnerabilities, and detail Brev's acquisition to improve developer UX with one-click GPU access and DGX Spark integration. Kyle describes Dynamo as a data center scale inference engine that optimizes serving by scaling out, using prefill/decode disaggregation, Kubernetes-based scheduling, and model-hardware co-design to improve cost, latency, and quality. The episode covers SOL's role in creating urgency, long-context limits and potential 'unhobblers' like multi-head latent attention, and the shift toward CLI-first agent workflows for enterprise tools.

Dylan Patel Explains the AI War While Cooking | In-Context Cooking

Feb 26, 2026 · 55:13

Dylan Patel, CEO of SemiAnalysis, argues hyperscalers like Google, Amazon, and Meta will sacrifice all profits to build AI infrastructure, spending $180–$200 billion in capex this year alone, because the AI adoption explosion—Claude Code driving 4% of GitHub commits in one month, Anthropic adding $2.5 billion monthly revenue—makes it a Pascal's wager: spend or die. He details how Taiwan's semiconductor geopolitics create endgame scenarios, from a KMT win placating China to full invasion, with TSMC's output critical. Patel explains Nvidia's paranoid founder Jensen Huang is responding to vertical integration threats from hyperscalers by diversifying into chips like CPX and Groq, but warns moats are shallow. The real bottleneck in AI progress? Semiconductors themselves: fabs take years to build, and no one can buy enough GPUs through 2028. He also predicts a massive AI backlash from the public and financial markets, as capital consumption outpaces revenue and labor displacement accelerates.

Claude Code for Finance + The Global Memory Shortage: Doug O'Laughlin, SemiAnalysis

Feb 24, 2026 · 2:07:40

Doug O'Laughlin, SemiAnalysis founder, explains his Claude Code awakening in December 2024: Anthropic's Opus 4.5 one-shotted tasks that used to take 24 hours, leading him to predict Claude Code will write 25–50% of all GitHub code by year-end (already 4% of commits in two weeks). He argues this marks the death of Excel and Bloomberg for analysts—'you can just do things'—but warns of a 'hygiene' crisis as junior analysts lose meta-learning. On the semiconductor side, O'Laughlin details a severe memory squeeze: HBM demand consumes 4× the DRAM capacity, DRAM prices could double again, and CXL is reviving as a workaround. He compares AI infrastructure buildout to the railroad boom (25% of U.S. gross fixed capital investment) and says Microsoft faces an innovator's dilemma—renting GPUs to 'barbarians at the gate' while its own office franchise is disrupted by Claude Code. Google's TPU v7 enjoys a temporary TCO advantage, but Nvidia's supply chain dominance (Jensen doing shots with SK and Samsung) will reassert with Rubin. O'Laughlin also reflects on his 2,800-mile Continental Divide Trail hike as essential self-mastery.

⚡️ Reverse Engineering OpenAI's Training Data — Pratyush Maini, Datology

Feb 10, 2026 · 27:02

Pratyush Maini from Datology reveals how the seahorse emoji question exposes that frontier models like GPT-4.1, GPT-5, and OLMo 3.1 now exhibit self-correction loops—a sign that reasoning traces are baked into mid-training data, not just post-training. By tracking response length surges four months after o1's release and corroborating with OLMo's data, Maini argues that foundation models must contain core capabilities like self-reflection, reinforcing the 'Fine Tuner's Fallacy': you cannot simply fine-tune for a capability; it must be in pre-training. He also highlights Datology's BeyondWeb project, which uses a source-rephrasing paradigm to transform internet data at trillion-token scale, achieving Nemotron-level performance in 2.7x less compute—making specialized pre-training accessible for enterprises. The episode concludes that 2026-2027 will see a shift toward domain-specific pre-training, as the cost amortizes when a smaller pre-trained model outperforms a larger fine-tuned one.

Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay

Jan 23, 2026 · 1:32:05

Yi Tay, who leads Google DeepMind's Reasoning and AGI team in Singapore, explains how Gemini Deep Think achieved IMO Gold by abandoning symbolic AlphaProof for an end-to-end RL-trained model. He details the on-policy RL philosophy—models learn from their own generated outputs rather than imitating others—and the critical role of self-consistency through parallel sampling and internal verification. Tay describes the IMO effort: four co-captains in different time zones, a one-week training sprint, a live competition in Australia where researchers punched in problems as they were released, and the tension of waiting for human scores to determine the gold threshold. He discusses why the team believes one model must subsume everything for AGI, the data efficiency gap compared to humans, and his hiring focus on raw talent and research taste. Tay also shares his personal fitness transformation—losing 23 kilos and improving HRV—as integral to research productivity.

Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith

Jan 9, 2026 · 1:18:15

Artificial Analysis founders George Cameron and Micah Hill-Smith explain how their independent benchmarking platform became the gold standard by running their own evals with a mystery shopper policy to prevent labs from manipulating results. They launched in January 2024 after building it as a side project in Sydney, going viral after Swyx's retweet. The Intelligence Index V3 synthesizes 10 datasets with 95% confidence intervals, while the Omniscience Index measures hallucination rates from -100 to +100 (Claude models lead). Their GDP Val AA benchmark tests 44 white-collar tasks, and they open-sourced their agentic harness Stirrup. They also introduced an Openness Index scoring models out of 18 points. The episode covers how they make money through enterprise benchmarking subscriptions and custom work, and why the cost of GPT-4-level intelligence has dropped over 100× while total inference spend rises due to reasoning and agentic workflows.

[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual

Dec 31, 2025 · 26:48

Nina Lopatina of Contextual AI discusses the state of context engineering, arguing that while the field is still in prototyping, 2025 will see true scale with full-system design patterns. She notes that agentic RAG—especially query reformulation into subqueries—has become the baseline, dramatically improving retrieval. Context rot is widely cited but industry benchmarks at real scale (100k+ documents, billions of tokens) remain rare. MCP is both a driver and a flaw: giant JSON tool definitions stuff the context window, yet MCP servers enable rapid prototyping before optimizing down to direct API calls. Sub-agents with turn limits are critical—unlimited agency degrades performance and causes hallucinations. Instruction-following re-rankers are essential for scaling retrieval across massive databases, providing more recall upfront and precision in the final context. Benchmarks are being saturated faster than ever: a Princeton benchmark from October was saturated by Claude Code in weeks, with solutions so good the gold dataset had errors. Lopatina also shares a lesson from using ChatGPT as a snowboarding coach—after several months of training for a 25-lap mogul race, she had to close…

[State of RL/Reasoning] IMO/IOI Gold, OpenAI o3/GPT-5, and Cursor Composer — Ashvin Nair, Cursor

Dec 30, 2025 · 45:13

Ashvin Nair, now ML lead at Cursor, traces his path from Berkeley robotics and an OpenAI Dota-era internship to OpenAI's reasoning team (which grew from a dozen to 300+ people) and explains why IOI Gold in 2022 felt like solving AI but didn't change the world—because RL doesn't generalize beyond training distribution. He argues most RL research from 2017-2022 overfit to benchmarks, rewarding complex ideas over simple ones that scale. At Cursor, he sees a unique opportunity for continual learning with policy updates every two hours and product-model co-design, keeping engineers in the loop instead of context-switching. His bet is that the next paradigm shift is continual learning with infinite memory: models experience something once and never forget it, storing millions of deployment tokens in weights without overloading capacity.

⚡️ The State of AI Engineer Hiring: Cheating, AI Adoption,Junior Devs — Vivek Ravisankar, HackerRank

Nov 8, 2025 · 49:05

Vivek Ravisankar, CEO of HackerRank, reveals that while overall tech hiring has flattened year-over-year, AI-specific roles are exploding and companies are reversing their stance on junior hiring because new grads are the true AI natives who embrace tools like Devin and Cursor without hesitation. He details HackerRank's integrity challenges—from leaked questions on Chegg to AI cheating tools like Interview Coder—and their countermeasures: custom-trained plagiarism models with 85-90% precision, DMCA takedowns, and a proctor mode that can shut down unauthorized apps. Rather than fighting AI, HackerRank embeds AI assistants into assessments, shifting from LeetCode-style tasks to real-world code repository challenges. Ravisankar defines the next-gen developer by four attributes: strong software engineering fundamentals, ability to use AI across the entire SDLC, deep knowledge of AI concepts from prompt engineering to fine-tuning, and good taste with business acumen. He predicts a proliferation of developers across all business functions—with roles like 'full-stack marketers' and 'go-to-market engineers'—and notes the irony that the most AI-forward companies like Anthropic explicitly…

How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony

Nov 3, 2025 · 1:00:02

Quentin Anthony, head of model training at Zyphra and advisor at EleutherAI, explains why his company moved all training to AMD MI300X GPUs, outperforming Nvidia H100s on certain workloads thanks to 192GB VRAM and higher memory bandwidth, and describes his kernel development approach of writing directly in ROCm or GPU assembly rather than using Triton. He also shares his experience in the METR study on AI coding productivity, where he was one of the few developers with measurable speedup, and offers tips: timebox AI use, avoid the slot machine effect, maintain context hygiene, and use direct API over tools like Cursor. Additionally, he argues that open source AI research benefits from siloed focused teams with guaranteed funding over grand collaborations, and notes that kernel datasets alone won't solve GPU programming due to evaluation challenges.

⚡️Raising $1.1b to build the fastest LLM Chips on Earth — Andrew Feldman, Cerebras

Oct 1, 2025 · 29:14

Andrew Feldman, CEO of Cerebras, joins Latent Space to discuss their $1.1B fundraise at an $8.1B valuation and their wafer-scale chip that delivers 20x faster inference than NVIDIA's B200 GPUs. He explains how their architecture uses SRAM instead of HBM, providing 2,625x more memory bandwidth by eliminating the narrow straw between compute and memory. Feldman details the decision to accelerate sparse linear algebra rather than specialized convolutions, enabling support for transformers and diffusion models unseen during design. He discusses the explosive growth in AI inference demand, the shift from closed-source to fast open-source models, and the importance of speed—citing Paul Graham's observation that ChatGPT's slowness drives users away. The conversation covers enterprise trends in the 10-30B parameter space, the complexity of building data centers that pull gigawatts of power, and the often-overlooked routing and caching systems that make AI work seamlessly.

⚡️Accelerators @ 3x NVIDIA H200 perf, Made in the USA - Thomas Sohmers + Mitesh Agrawal, Positron AI

Aug 18, 2025 · 47:56

Thomas Sohmers and Mitesh Agrawal of Positron AI argue memory bandwidth, not compute, is the true bottleneck in AI inference, and their accelerator achieves 93% memory bandwidth utilization—triple NVIDIA's efficiency—enabling 70% faster token generation at 150W. The founders, both Lambda Labs veterans, shipped an FPGA product 15 months after founding, then raised a $51M Series A for an ASIC in late 2026. Their hardware requires no recompilation: it ingests raw binary weights from NVIDIA training and outputs an OpenAI-compatible API. Positron focuses on the decode phase of transformers, where memory-bound matrix-vector multiplication dominates, and already counts Cloudflare and Parasail as customers. The company sells systems directly, prioritizing capital efficiency and ROIC over operating its own cloud.

⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs

Aug 4, 2025 · 28:07

Stefano Ermon of Inception Labs explains why diffusion language models like their Mercury Coder can match GPT-4.1 Nano and Claude Haiku in intelligence while delivering 5–10× faster inference at 737–1109 tokens/sec on H100s, built on score-based generative model foundations he pioneered at Stanford. He details the coarse-to-fine generation process that modifies multiple tokens per network evaluation, the required end-to-end training from scratch (no fine-tuning existing LLMs), and how post-training pipelines use specialized DPO for preference alignment. Ermon identifies latency-sensitive applications—voice agents, IDEs, vibe coding—as the killer use case, acknowledges they aren't yet frontier-level but sees a future where diffusion dominates due to efficiency gains, and notes the challenge of open-sourcing given proprietary inference engines and kernels.

The Shape of Compute (Chris Lattner of Modular)

Jun 13, 2025 · 1:18:18

Chris Lattner of Modular explains how his company is breaking the CUDA monopoly with Mojo and MAX, matching NVIDIA's best inference performance on AMD and NVIDIA GPUs. After three years in R&D, Modular open-sourced its stack and now delivers state-of-the-art Llama 3 serving at over 800 tokens per second. Mojo, a Python-family language, runs kernels faster than Rust and can extend Python without bindings, while MAX provides a full inference framework with automatic kernel fusion and cluster management. Unlike VLLM or SGLang, Modular's container is just a gigabyte, fully open source, and not reliant on proprietary CUDA blobs. DeepSeek's low-level PTX work validated the approach, but Lattner emphasizes that Mojo's portability avoids rewriting kernels for each new GPU architecture. He also details his daily routine, using Cursor for coding, and hiring "elite nerds" to grow the team.

Voice AI Masterclass — Kwindla Hultman Kramer and swyx

May 6, 2025 · 20:51

Shawn Wang and Kwindla Hultman Kramer announce a voice AI masterclass course, diving into the landscape of models like Dia and Parakeet, production deployment challenges, and future trends such as speech-to-speech and real-time video. Kwindla explains that telephony (Twilio) drives 99% of current monetizable voice AI, while open source frameworks like Pipecat enable low-latency multi-modal apps. The course covers turn detection, context management, and evals, with 28 sessions featuring partners like OpenAI, Google, and NVIDIA. The goal is to jumpstart builders from prototype to production, with real-time video expected to hit its inflection point by year-end.

SF Compute: Commoditizing Compute

Apr 11, 2025 · 1:12:02

Evan Conrad, co-founder of SF Compute, argues that GPUs behave like a real estate business, not a traditional cloud, because price-sensitive customers value every incremental GPU and will switch for a 10% margin. CoreWeave succeeded by selling locked-in long-term contracts to low-credit-risk customers like Microsoft and OpenAI, ignoring short-term demand. He predicts hyperscalers and providers like Together and DigitalOcean will lose money on GPU clusters because software margins cannot match the hardware costs. SF Compute started as an AI lab forced to sublease its cluster monthly to avoid bankruptcy, then evolved into a market where anyone can buy H100s by the hour via dynamic pricing—often below $1/hour for short bursts. Utilization stays near 100% as prices adjust. Future plans include cash-settled futures to reduce financial risk across the industry, while the brand deliberately stays anti-hype and calm.

LIVE from GTC: DGX Spark Insides First Look

Mar 20, 2025 · 12:39

Israel from NVIDIA gives a first look at the DGX Spark, a $3,000–$4,000 mini AI supercomputer that fits up to 200B parameter models in FP4 with 128GB unified memory, powered by the NVIDIA GB10 Superchip. It uses the same Grace Blackwell architecture and software stack as data center systems, so code written on the Spark deploys to production without modifications. The device includes a ConnectX-7 dual-port 200Gb ethernet for clustering two units, and runs DGX OS (Ubuntu 24.04). Storage is 1TB or 4TB NVMe. Partners ASUS, HP, Dell, and Lenovo will offer their own cases. Israel stresses the Spark is a developer box, not a server, but delivers enterprise-grade networking and shared memory in a compact form.

AI Engineering for Art - with comfyanonymous

Jan 4, 2025 · 52:09

Comfy Anonymous, the anonymous creator of ComfyUI, details in his first-ever podcast interview how his node-based image generation tool overtook Automatic1111 through superior memory management and early support for SDXL, becoming the de facto interface for advanced diffusion workflows. He started coding on January 1, 2023, and released the first version on January 16, driven by a desire to chain models and experiment with custom samplers. His work at Stability AI from June 2023 ensured ComfyUI efficiently handled SDXL, which leaked via early access and forced users from less powerful GPUs to adopt his tool. He discusses model preferences: Flux for consistency, SD 3.5 for creativity, and SD1.5 remaining popular. For video, he highlights Mochi as a 'true' video model with 3D latents, already implemented in Comfy. Now with a core team, ComfyUI is targeting a v1 release with an easy installer on Windows and Mac, while planning monetization through cloud and enterprise features—but keeping the open-source core free.

2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents

Jan 1, 2025 · 1:51:07

In their 100th episode, hosts Alessio and Swyx recap 2024 in AI, arguing that pre-training scaling has hit a wall—backed by Ilya Sutskever and others at NeurIPS—and that inference-time compute (o1, o3) is the new frontier. They dissect the "four wars": data quality (lawsuits vs. synthetic data), GPU haves vs. have-nots (with the middle class dying), multimodality (Sora, Veo 2, Gemini 2.0's native image output), and the LLM OS/agents stack (LangChain, E2B, memory). Market share shifted from OpenAI's 95% to 50-75% as Anthropic and Gemini gained ground; prices dropped ~3 orders of magnitude for same ELO. The episode predicts 2025 as the year agents finally enter production, driven by models like o1 and tools like Devin, and warns that AI will set the skill floor for roles.

Best of 2024: Synthetic Data / Smol Models, Loubna Ben Allal, HuggingFace [LS Live! @ NeurIPS 2024]

Dec 24, 2024 · 28:08

Loubna Ben Allal of Hugging Face explains how synthetic data has become ubiquitous in LLM pipelines, moving from post-training to pre-training, with examples like Cosmopedia’s 30B-token synthetic dataset and NVIDIA’s 1.9T-token Nemotron CC. She addresses model collapse fears, showing that web dumps after ChatGPT’s release actually yield better models, and emphasizes diversity through prompt seeds and webpage extracts. For filtering, FineWeb-Edu and DCLM use LLMs to rate educational content, achieving top benchmark scores. In post-training, she highlights AgentInstruct, Tülu3 with PersonaHub, and Cohere’s multilingual arbitrage using multiple teachers. On small models, SmolM2 (1.7B) outperforms Llama 1B and Qwen 2.5 after 11T tokens of pre-training, and on-device inference via frameworks like llama.cpp enables privacy-preserving use cases such as text extraction and structured generation. She predicts a return to fine-tuning specialized small models over costly prompt engineering.

[Paper Club] Weight Streaming on Wafer-Scale Clusters (w/ Sarah Chieng of Cerebras)

Dec 7, 2024 · 43:54

Sarah Chieng of Cerebras presents the weight streaming technique for training giant neural networks on Cerebras wafer-scale clusters, arguing it enables near-linear scaling by separating parameter storage from primary compute. The system uses the Wafer Scale Engine (WSE-2/3) with 900,000 cores and 44 GB on-chip SRAM, avoiding off-chip memory bottlenecks that limit NVIDIA GPUs. MemoryX provides external storage for weights and optimizer states (up to 2.4 petabytes), streaming weights to compute units via the SwarmX interconnect fabric, which aggregates gradients. This design allows training models with up to 120 trillion parameters without complex hybrid parallelism. Cerebras also leverages unstructured weight sparsity to prune 90% of data during transmission and skip zero-value computations on-chip, reducing bandwidth and improving efficiency.

[Paper Club] Upcycling Large Language Models into Mixture of Experts

Oct 29, 2024 · 39:02

NVIDIA's Ethan He presents Megatron-Core MoE and the upcycling of dense LLMs into Mixture-of-Experts models, showing that upcycling a 15B-param dense model into a 64-expert MoE and training on 1T tokens yields 5% lower validation loss and 4% higher MMLU than continued dense training on the same compute. Key techniques include swapping the router order (softmax-then-top-K instead of top-K-then-softmax) with a 4× output scaling to preserve initial forward pass behavior, and initializing fine-grained MoE routers by duplicating half the weights so each shard group selects identically. A high learning rate matching the original pre-training peak is critical; constant low LR causes catastrophic forgetting. The upcycled model matches the compute of a 1.7× larger dense model per scaling laws, but data quality remains paramount—continued dense training still delivered a 20% MMLU jump. Megatron-Core's fused permutation, GroupedGEMM, and expert parallelism are available as a standalone library.

Singapore: the AI Engineer Nation — with Minister Josephine Teo

Oct 19, 2024 · 56:40

Singapore Minister Josephine Teo outlines the city-state's refreshed national AI strategy and the Ministry of Digital Development's approach to balancing innovation, safety, and talent development. She details plans to triple AI practitioners from 5,000 to 15,000, the AI Verify testing tools and AI Safety Institute for governance, and internal government AI bots like AI Bot for RAG-based knowledge retrieval. On sovereign AI, she discusses hybrid cloud infrastructure and pragmatic data center expansion. Teo also explains Singapore's new law requiring transparency for AI-generated election content, emphasizing fact-based political discourse.

The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)

Aug 2, 2024 · 1:23:36

Swyx and Alessio recap Q2 2024 through their 'Four Wars' framework, arguing that the AI landscape is shifting from frontier model dominance to commoditization and vertical applications. They highlight Claude 3.5 Sonnet overtaking OpenAI on coding benchmarks, Llama 3.1's synthetic data approach enabling 7B models to rival GPT-4, Mistral Large 2's non-commercial license and lost open-source crown, and on-device models like Gemini Nano and Apple Intelligence. The Quality Data Wars see NYT suing OpenAI, Reddit licensing data for $200M+, and synthetic data proving real for math (AlphaProof near IMO gold) and code. The Multimodality War includes ChatGPT Voice Mode delayed, Meta's Chameleon for native fusion, and Google's PaliGemma for PDF extraction. The renamed LLM OS War covers agent protocols, memory databases, and the collapse of model cost by an order of magnitude every four months, pushing startups toward vertical services like Brightwave and Dropzone that sell labor, not tools. The episode ends with a CrowdStrike joke about agent safety.

[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models

Jul 29, 2024 · 1:23:46

This episode examines Meta's Llama 3.1 paper, detailing the 405B dense model, its scaling laws grounded on the ARC reasoning benchmark rather than perplexity, and the decision to train on 15 trillion tokens. Vibhu explains the training infrastructure: 16,000 H100s over 54 days with 419 interruptions, 78% from GPU hardware failures. Eugene Yan walks through the synthetic data pipeline—using Llama 2 for filtering, stepwise reward models, and Monte Carlo tree search to improve reasoning traces. Hassan shares building LlamaTutor.com with Together API, serving 4,000 visitors and 5,900 requests for about $12. The group also discusses quantization trade-offs (larger models degrade less), inference provider variability (Groq's non-deterministic temperature zero), and compares Llama 405B's performance to GPT-4o and Claude.

Why Google failed to make GPT-3 -- with David Luan of Adept

Mar 27, 2024 · 49:27

David Luan, co-founder of Adept and former early OpenAI leader, explains why Google failed to build GPT-3—its 'brain credit marketplace' prevented critical mass—and why Adept builds enterprise AI agents prioritizing reliability over generality. He recounts the GPT-2 demo that helped secure Microsoft's $1B investment and details Adept's goal: an AI teammate that can do anything a human does on a computer, targeting 'nines of reliability' for workflows like dispatching a physical truck. Luan contrasts Adept's vertical integration—training fast multimodal models (Fuyu) for charts and UIs—with pure-play foundation model companies that sell tokens, predicting commoditization. He notes Adept is sold out for Q1 and raised $420M, and explains how an augmentation focus creates a data flywheel from human oversight.

Making Transformers Sing - with Mikey Shulman of Suno

Mar 14, 2024 · 58:58

In this episode of Latent Space, hosts Alessio and Shawn Wang interview Mikey Shulman, CEO of Suno, about making transformers sing. Suno uses transformers to predict audio tokens end-to-end, avoiding baked-in musical knowledge, with a tokenization secret sauce that also includes non-music audio for better vocal realism. Their models are relatively small (far below 175B parameters) due to latency needs, and they prioritize scaling research over brute-force size. Over half of Suno users employ expert mode, tweaking lyrics and style prompts, rather than easy mode. Shulman argues Suno is not the 'Midjourney of music' because music is inherently social and synchronous, unlike images. He demoed live generation, showing control via tokens like [beat drop] and style modifiers, and revealed future plans for collaborative concerts, continuous DJ modes, and personalized models. He also advocates for hiring economists to avoid Goodhart's law pitfalls in ML benchmarks, especially in audio where aesthetics matter most.

The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

Dec 5, 2023 · 1:07:55

Dylan Patel of SemiAnalysis argues the 'GPU poor vs GPU rich' divide defines AI, with Google's TPU ramp (millions of units) dwarfing others and Nvidia selling over a million H100s this year. He explains that training costs are irrelevant compared to inference costs, and that memory bandwidth is the key bottleneck for LLM inference (e.g., Llama 70B needs ~2.1 TB/s for human reading speed). Patel advises GPU-poor players to focus on on-device innovations like speculative decoding (Medusa), asynchronous training, or fine-tuning for edge use—not on gaming benchmarks or fine-tuning small models. He details why AI hardware startups (Cerebras, Graphcore) bet wrong on on-chip SRAM, while Google’s TPU/Broadcom partnership and Nvidia’s yearly cadence make alternatives tough. He also warns that rebuilding the semiconductor supply chain in the US is unfeasible due to its fragmentation, and that safety through obscurity doesn't work—open innovation is better.

Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Nov 3, 2023 · 1:18:54

Michael Royzen, co-founder and CEO of Phind, explains how his company built a GPT-4-beating open-source model for developer Q&A. Royzen recounts founding SmartLens in high school, shifting to NLP after a Hugging Face demo, and creating an internet-scale LLM-powered RAG system in January 2022. He details Phind’s pivot to programmers, the Hacker News launch that gave it 1,500 points, and Paul Graham’s role in naming the company and introducing Ron Conway, who then connected Phind to NVIDIA for GPU access. Royzen argues that Phind’s model, fine-tuned from Code Llama 34B with extra data, closes the gap with proprietary models—especially on code reasoning—and that open-source will win the enterprise because the delta to GPT-5 will be small. He shares how Phind handles multi-step conversations via a pair programmer mode where users can pin messages, and reveals plans for reinforcement learning to reduce hallucination and improve correctness.

Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue

Oct 21, 2023 · 1:12:37

Kanjun Qiu, CEO of Imbue (formerly Generally Intelligent), argues that AI agents remain unreliable because they lack robust reasoning and proper abstractions, which Imbue tackles by training foundation models optimized for reasoning over data like code. With a $200M Series B and $1B+ valuation, Imbue builds internal tools for debugging and inspecting agent decision-making, rejecting pure reinforcement learning after their Avalon environment revealed RL cannot handle planning. Instead, they emphasize natural language reasoning for inspectability, use code as a curriculum for reasoning, and design interfaces that let users fork and modify agents mid-execution. Qiu also discusses lessons from earlier startups (Sorceress in recruiting, Embark in VR), the importance of treating team members as creative agents, and her role in co-founding communal living space The Archive to foster scenius.

FlashAttention-2: Making Transformers 800% faster AND exact

Aug 3, 2023 · 1:04:06

Tri Dao, creator of FlashAttention and FlashAttention-2, explains how his I/O-aware algorithm makes attention 2x faster by fusing kernels and using online softmax, achieving near-matrix-multiply efficiency. He argues that transformer alternatives like state space models and RNNs (e.g., RWKV) could surpass transformers for long sequences and high-throughput generation, though attention still dominates. Dao discusses the hardware lottery, where NVIDIA's CUDA ecosystem entrenches transformers, and advocates for open-source AI, praising Meta's Llama 2 for shifting enterprise adoption despite its restrictive license. He emphasizes that understanding both algorithms and systems is key to scaling AI, and that academia should pursue risky bets that industry cannot.

Ep 18: Petaflops to the People — with George Hotz of tinycorp

Jun 20, 2023 · 1:23:24

George Hotz of tinycorp argues that a simplified, open-source ML framework (tinygrad) can democratize AI compute and replace NVIDIA, Google, and AMD. Tinygrad uses 25 ops (vs. XLA's 250) and fuses kernels automatically, achieving 2x speed on Qualcomm GPUs over their library. Hotz reports AMD kernel panics fixed after emailing CEO Lisa Su, but laments AMD's open-source as 'dumped on GitHub.' For hardware, tinybox is a six-GPU desktop at 350W per GPU, aiming for 5x better price-performance than NVIDIA H100 for 90% of training. He criticizes OpenAI's secrecy, revealing GPT-4 is an 8-way mixture of 220B models, and cites the Bitter Lesson. He also outlines FLOPcoin using hardware identity to prevent cheating, and his goal of an AI girlfriend as the third company, merging humans via data rather than implants.