PyTorch

🔬 Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar, Caltech

Aug 26, 2026 · 1:23:32

Anima Anandkumar, Caltech professor and ex-NVIDIA lead, argues physical-world modeling needs neural operators that bake in physics, not language-style scaling; it already rivals supercomputers. Physics-informed nets fail; Fourier neural operators capture multi-scale non-local phenomena, and FourCastNet, trained on 50,000 reanalysis samples, forecasts weather tens of thousands of times faster on a consumer GPU. FourCastNet 3’s spherical harmonics keep month-long rollouts stable for ensemble climate prediction; the same operators build a fusion digital twin a million times faster than simulation and inverse design. TorchLean ports PyTorch to Lean for formal robustness proofs, and on the UN Science Advisory Board she urges AI for science not be regulated like chatbots. Bottleneck: more compute.

[NeurIPS Best Paper] 1000 Layer Networks for Self-Supervised RL — Kevin Wang et al, Princeton

Dec 31, 2025 · 28:19

Kevin Wang, Ishaan Javali, Michał Bortkiewicz, and Benjamin Eysenbach won the NeurIPS 2025 Best Paper award for RL1000, defying conventional wisdom by scaling reinforcement learning networks to 1,000 layers deep. They show that self-supervised RL—learning representations of states, actions, and future states via contrastive classification—scales where value-based methods collapse, and that critical architectural tricks (residual connections, layer norm, classification instead of regression) unlock performance that multiplies beyond a critical depth. Scaling depth is more parameter-efficient (linear growth) than width (quadratic growth), and Jax GPU-accelerated environments enable collecting hundreds of millions of transitions in hours, providing the data abundance that makes scaling possible. The paper argues that RL is finally ready to scale like language and vision, not by throwing compute at value functions, but by borrowing self-supervised objectives, with implications for robotics via goal-conditioned RL without human supervision or demonstrations, and deployment via deep teacher, shallow student distillation.

⚡️Accelerators @ 3x NVIDIA H200 perf, Made in the USA - Thomas Sohmers + Mitesh Agrawal, Positron AI

Aug 18, 2025 · 47:56

Thomas Sohmers and Mitesh Agrawal of Positron AI argue memory bandwidth, not compute, is the true bottleneck in AI inference, and their accelerator achieves 93% memory bandwidth utilization—triple NVIDIA's efficiency—enabling 70% faster token generation at 150W. The founders, both Lambda Labs veterans, shipped an FPGA product 15 months after founding, then raised a $51M Series A for an ASIC in late 2026. Their hardware requires no recompilation: it ingests raw binary weights from NVIDIA training and outputs an OpenAI-compatible API. Positron focuses on the decode phase of transformers, where memory-bound matrix-vector multiplication dominates, and already counts Cloudflare and Parasail as customers. The company sells systems directly, prioritizing capital efficiency and ROIC over operating its own cloud.

Information Theory for Language Models: Jack Morris

Jul 2, 2025 · 1:18:13

Jack Morris, a Cornell PhD student advised by Sasha Rush, discusses his information-theoretic research on language models, including embedding inversion (recovering text from embeddings with 90% accuracy), the universal geometry of embeddings (aligning different models' latent spaces), and measuring model memorization capacity at 3.6 bits per parameter. He argues that paradigm shifts in AI, from AlexNet to instruction tuning, stem not from new architectures but from new datasets, and that the next breakthrough will likely emerge from a novel data source. The conversation also covers the shift from academia to industry, practical advice for grad students on distributed training, and the implications of embedding inversion for privacy and model alignment.

The Shape of Compute (Chris Lattner of Modular)

Jun 13, 2025 · 1:18:18

Chris Lattner of Modular explains how his company is breaking the CUDA monopoly with Mojo and MAX, matching NVIDIA's best inference performance on AMD and NVIDIA GPUs. After three years in R&D, Modular open-sourced its stack and now delivers state-of-the-art Llama 3 serving at over 800 tokens per second. Mojo, a Python-family language, runs kernels faster than Rust and can extend Python without bindings, while MAX provides a full inference framework with automatic kernel fusion and cluster management. Unlike VLLM or SGLang, Modular's container is just a gigabyte, fully open source, and not reliant on proprietary CUDA blobs. DeepSeek's low-level PTX work validated the approach, but Lattner emphasizes that Mojo's portability avoids rewriting kernels for each new GPU architecture. He also details his daily routine, using Cursor for coding, and hiring "elite nerds" to grow the team.

AI Engineering for Art - with comfyanonymous

Jan 4, 2025 · 52:09

Comfy Anonymous, the anonymous creator of ComfyUI, details in his first-ever podcast interview how his node-based image generation tool overtook Automatic1111 through superior memory management and early support for SDXL, becoming the de facto interface for advanced diffusion workflows. He started coding on January 1, 2023, and released the first version on January 16, driven by a desire to chain models and experiment with custom samplers. His work at Stability AI from June 2023 ensured ComfyUI efficiently handled SDXL, which leaked via early access and forced users from less powerful GPUs to adopt his tool. He discusses model preferences: Flux for consistency, SD 3.5 for creativity, and SD1.5 remaining popular. For video, he highlights Mochi as a 'true' video model with 3D latents, already implemented in Comfy. Now with a core team, ComfyUI is targeting a v1 release with an easy installer on Windows and Mac, while planning monetization through cloud and enterprise features—but keeping the open-source core free.

2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]

Dec 24, 2024 · 42:45

Dan Fu (Together AI/UCSD) and Eugene Cheah (Featherless) survey post-transformer architectures, arguing that state-space models (SSMs) like Mamba and RWKV can match transformer quality while scaling subquadratically in sequence length, enabling efficient long-context and test-time compute. Fu explains how SSMs evolved from S4 to data-dependent selection mechanisms, while Cheah details RWKV's open-source development, including Q-RWKV6—a 32B model converted from Qwen 32B with linear attention that retains 76% MMLU after just days of training. They discuss hybrid models outperforming pure transformers or SSMs, and challenge RAG's necessity: fixed-state models like RWKV may handle up to 128K context for most enterprise workloads, while truly infinite context requires the model to decide what to forget. Hot takes: long context is overhyped for most use cases, but models that can 'watch forever' could excel in time-series and video generation.

Why Compound AI + Open Source will beat Closed AI — with Lin Qiao, CEO of Fireworks AI

Nov 25, 2024 · 55:49

Lin Qiao, CEO of Fireworks AI, argues that compound AI systems combining multiple open-source models will outperform closed monolithic models like OpenAI's. She explains Fireworks' evolution from a PyTorch platform to a full-stack inference and customization engine serving 40+ customers including Cursor. Qiao details their distributed inference engine, Fire Optimizer for quality-latency-cost tradeoffs, and upcoming o1-like model built on open-source foundations. She defends their quantization approach after public criticism from rivals, emphasizes specialization over general intelligence, and invites developers to test their free LoRA adapter hosting.

llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE

Sep 21, 2024 · 23:29

Andrej Karpathy explains llm.c, his from-scratch C implementation of GPT-2 training that outperforms PyTorch by 20% in speed and 30% in memory, arguing that LLMs will eventually act as compilers for custom code. He recounts starting the project out of frustration with PyTorch compile errors, writing all forward and backward passes manually in C, then porting to CUDA with kernel optimizations. A community of contributors (Eric, Arun, Alex R) helped achieve 50% MFU on a single H100 node, training GPT-2 1.6B in 24 hours for $600. Ongoing work adds Llama 3.1 and FP8 support. Karpathy sees llm.c as proof that LLMs can automate such low-level optimization, making Python and PyTorch a temporary crutch.

A Comprehensive Overview of Large Language Models - Latent Space Paper Club

Mar 15, 2024 · 54:30

Brian presents a detailed walkthrough of the survey paper 'A Comprehensive Overview of Large Language Models,' which systematically covers the evolution of LLMs from early attention mechanisms to modern transformer architectures. He explains the shift from sequential RNNs to parallelizable transformers, the three dominant architectures (encoder-only like BERT, encoder-decoder like T5, decoder-only like GPT), and key training objectives (masked language modeling, full language modeling, prefix language modeling). The paper also reviews fine-tuning techniques (instruction tuning, alignment tuning via RLHF), prompting strategies (zero-shot, chain-of-thought), efficient adaptation methods (LoRA, quantization), evaluation benchmarks (GLUE, MMLU), and challenges such as bias, memorization, and privacy leaks. This episode serves as a comprehensive primer for anyone seeking to understand the foundational concepts and latest trends in large language model research.

Open Source AI is AI we can Trust — with Soumith Chintala of Meta AI

Mar 6, 2024 · 1:37:38

Soumith Chintala, creator of PyTorch and engineering lead at Meta AI, argues that open source AI is essential for distributing opportunity and trust. He details PyTorch's complexity—1,000 operators needed for generality—and explains synthetic data as a vehicle for imparting symbolic knowledge where humans already have good symbolic models. He highlights a coordination problem in open source: feedback is lost because frontends like Ooba and Ollama lack feedback buttons, and proposes a centralized sinkhole to collect high-quality feedback. Beyond text, he is excited about robotics, where hardware remains a bottleneck, and Osmo's work to digitize smell, which he compares to images in the 1800s.

The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

Dec 5, 2023 · 1:07:55

Dylan Patel of SemiAnalysis argues the 'GPU poor vs GPU rich' divide defines AI, with Google's TPU ramp (millions of units) dwarfing others and Nvidia selling over a million H100s this year. He explains that training costs are irrelevant compared to inference costs, and that memory bandwidth is the key bottleneck for LLM inference (e.g., Llama 70B needs ~2.1 TB/s for human reading speed). Patel advises GPU-poor players to focus on on-device innovations like speculative decoding (Medusa), asynchronous training, or fine-tuning for edge use—not on gaming benchmarks or fine-tuning small models. He details why AI hardware startups (Cerebras, Graphcore) bet wrong on on-chip SRAM, while Google’s TPU/Broadcom partnership and Nvidia’s yearly cadence make alternatives tough. He also warns that rebuilding the semiconductor supply chain in the US is unfeasible due to its fragmentation, and that safety through obscurity doesn't work—open innovation is better.

RWKV: Reinventing RNNs for the Transformer Era

Aug 31, 2023 · 1:56:51

Eugene, CTO of UILicious, introduces RWKV, a recurrent neural network that matches transformer-level performance while scaling linearly with context length, solving the quadratic cost of attention. He explains how RWKV uses Attention-Free Transformer layers and parallelizable training, achieving competitive results on 7B and 14B models. The community-driven project prioritizes multilingual support, with a tokenizer that handles languages without spaces. Eugene recounts his journey from GPU.js to finding RWKV, and highlights use cases like long HTML analysis where transformers fail. He also discusses the token crisis, diffusion models for text, and how the AI waifu community drives optimization and alignment research. Finally, he advises AI engineers to focus on practical prompting and data curation, noting that deep architecture knowledge is optional.

FlashAttention-2: Making Transformers 800% faster AND exact

Aug 3, 2023 · 1:04:06

Tri Dao, creator of FlashAttention and FlashAttention-2, explains how his I/O-aware algorithm makes attention 2x faster by fusing kernels and using online softmax, achieving near-matrix-multiply efficiency. He argues that transformer alternatives like state space models and RNNs (e.g., RWKV) could surpass transformers for long sequences and high-throughput generation, though attention still dominates. Dao discusses the hardware lottery, where NVIDIA's CUDA ecosystem entrenches transformers, and advocates for open-source AI, praising Meta's Llama 2 for shifting enterprise adoption despite its restrictive license. He emphasizes that understanding both algorithms and systems is key to scaling AI, and that academia should pursue risky bets that industry cannot.

Ep 18: Petaflops to the People — with George Hotz of tinycorp

Jun 20, 2023 · 1:23:24

George Hotz of tinycorp argues that a simplified, open-source ML framework (tinygrad) can democratize AI compute and replace NVIDIA, Google, and AMD. Tinygrad uses 25 ops (vs. XLA's 250) and fuses kernels automatically, achieving 2x speed on Qualcomm GPUs over their library. Hotz reports AMD kernel panics fixed after emailing CEO Lisa Su, but laments AMD's open-source as 'dumped on GitHub.' For hardware, tinybox is a six-GPU desktop at 350W per GPU, aiming for 5x better price-performance than NVIDIA H100 for 90% of training. He criticizes OpenAI's secrecy, revealing GPT-4 is an 8-way mixture of 220B models, and cites the Bitter Lesson. He also outlines FLOPcoin using hardware identity to prevent cheating, and his goal of an AI girlfriend as the third company, merging humans via data rather than implants.