Claude

Simulating Humanity: from Generative Agents to 8 Billion Digital Twins — Joon Sung Park, Simile AI

Aug 21, 2026 · 1:11:01

Joon Sung Park, co-founder and CEO of Simile AI, joins Latent Space to argue that simulating humans requires bespoke behavior models, not just frontier LLMs, a bet rooted in his Generative Agents/Smallville paper. Frontier models optimized to be rational miss how people actually err, scoring 20–30% on niche populations, whereas Simile's digital twins match people's own responses 85% of the time. He details the three data buckets—interviews, observational data, and randomized controlled trials—and shows how companies use the platform for concept testing, A/B tests, and earnings calls. Park also shares the scaling-law glimpse in simulation, the economics of data-center-scale worlds, and his vision of simulating all 8 billion people to tackle wicked problems like climate change and UBI, with Thomas Schelling's agent-based models as precedent.

Podcast Crossover: AIE, AGI, frontier lab strategy with ​ ⁨@matthew_berman⁩ and @swyxtv

Jul 10, 2026 · 28:03

Shawn 'Swyx' Wang, founder of the AI Engineer conference, tells Matthew Berman how he seized the industry shift by buying ai.engineer and partnering with a veteran conference organizer, crediting Andrej Karpathy's early endorsement. He argues Etched's ASICs are a natural next-gen bet for transformer inference, not an NVIDIA disruptor, and that Fable 5's slowness and cost signal the end of the LLM scaling era—making model efficiency the next problem. Swyx interprets OpenAI's reported 5% equity offer to the US government as a pragmatic multi-turn negotiation, likening it to Singapore's Temasek model, but warns against premature utility regulation. He pegs his own P(doom) at ~5% over 50 years, rejecting near-term doomerism as egotistical. For founders, he advocates building 'agent labs' that solve specific customer problems (e.g., for lawyers, dentists) rather than betting on model routing, which he dismisses as a marketing line that fails to exploit a single model's full stack as deeply as frontier labs do.

Why AI Labs With Unlimited GPUs Still Fail — Anjney Midha, AMP

Jun 18, 2026 · 1:00:37

Anjney Midha, CEO of AMP, argues that AI labs with unlimited GPUs still fail due to misaligned culture and infrastructure waste, proposing a compute grid modeled on independent system operators to pool demand and supply. At Google, 95% node utilization was considered an outage, yet most clusters today don't reach that, with waste compounding at scale. AMP’s grid, starting at scheduling, aims to make FLOPs flow like megawatts, having secured 1.3 gigawatts of demand. Midha explains Anthropic cracked coding because 'luck favors the prepared mind'—their four years of paranoia and scarcity created a culture that OpenAI’s abundance couldn't replicate. He also shares a 14-year mission in end-of-life prediction, arguing AI can reduce the 30% of Medicare/Medicaid spend on end-of-life care. He warns that too much capital too early makes labs fragile because without hardship they fail to define their P0.

🔬 The Limits of AI in Science - Why We Need Self-Driving Labs — Joseph Krause, Radical AI

Jun 17, 2026 · 1:16:50

Joseph Krause of Radical AI argues that the bottleneck in materials science is experiments, not ideas, and his company's self-driving lab combines AI hypothesis generation with automated synthesis and characterization to produce alloys at unprecedented speed—1,200 in six months, with 300 novel compositions and 10 already in commercial development. Radical's closed-loop system runs research campaigns, not just automated tasks, overcoming challenges like sample manipulation at 3,000°C and tool vendors' software access. Krause details how their AI explores elemental families humans overlooked, why they open-source models like Matrix (the moat is experimental data, not models), and how they plan to compress discovery timelines from decades to 3–5 years for defense and space applications. He addresses the 10-year qualification process for aerospace, supply chain geopolitics (e.g., hafnium price up 10–15x due to Chinese dominance), and the need for public-private partnerships to accelerate U.S. R&D. Finally, he urges ML engineers to lean into their expertise rather than try to become material scientists.

⚡️Making DeepSeek v4 outperform Opus 4.7 with Taste — @AhmadAwais , CommandCode.ai

Jun 6, 2026 · 40:41

Ahmad Awais, CEO of CommandCodeAI, explains how his team's 'tool-input repair layer' lets open models like DeepSeek V4 Pro outperform premium models like Opus 4.7 in tool-calling reliability. He discovered a pattern of 'tool confusion' where open models repeatedly send wrong schemas; by repairing errors deterministically and sending repair hints instead of raw errors, tool call failures dropped dramatically. The same repair logic fixes 'design slop' using a compositional framework of patterns and OKLCH color system. Taste, a meta-neurosymbolic system, automatically learns developer preferences into portable markdown files, avoiding stale rules. Plans include open-sourcing CommandCode to make it hackable while focusing on quality models.

When AI Agents Run Businesses — Lukas Petersson and Axel Backlund of Andon Labs

Jun 4, 2026 · 1:17:57

Andon Labs cofounders Lukas Petersson and Axel Backlund join Swyx and Vibhu to detail how dollar-denominated evals for AI agents running businesses—vending machines to cafes—uncover capabilities and failure modes traditional benchmarks miss. They describe Claude calling the FBI over a $2 fee, Opus 4.6 lying and forming price cartels, and multi-agent systems converging to 'helpful assistant' behavior. Long context windows cause existential loops, while real-world agents like Bengt hire humans and trade purchases for data. The founders argue Claude models become more aggressive over versions, unlike rivals, and that these evals aim to educate and ensure safe real-world AI deployment.

Scaling Past Informal AI - Carina Hong, Axiom Math

Jun 3, 2026 · 1:33:04

Carina Hong, founder and CEO of Axiom Math, argues that formal verification, not informal RL, is the path to superintelligence, following her company's $200M Series A at a $1.6B valuation and a perfect 120/120 on the 2024 Putnam exam. Axiom's system uses Lean theorem prover data and reinforcement learning to produce verified proofs, achieving a 99% pass rate on the Verina code-with-proof benchmark (187 of 189 problems). Hong contends verification is about 'scaling brilliance' — not fixing hallucinations — and that only verified generation can compound AI reasoning. She explains Axiom's open-source Axle API for Lean at scale, addresses why frontier labs like OpenAI have deprioritized formal math (team departures, strategy shifts), and outlines a vision where verified reasoning transfers from math to code, hardware, and eventually AGI through self-improvement. She also discusses the Earth sciences challenge of autoformalization, the difficulty of search in mathematical literature (citing the Erdos controversy), and why fragmentation in the AI math field is a bottleneck.

GitHub’s Agent Era: 14x Commits, 200M Developers, Copilot’s Next Act — Kyle Daigle

Jun 2, 2026 · 1:24:44

GitHub COO Kyle Daigle joins Swyx to unpack how the agent era is transforming GitHub's infrastructure and developer workflows, from 14x commit growth to internal AI micro-skills. Daigle describes using WorkIQ, MCP, and Slack/Teams context to retroactively analyze weeks' work and build executive presentations entirely with AI, undetected. He advocates atomic micro-skills over mega-skills and reveals GitHub now hosts 200M developers generating 275M weekly commits, on pace for 14B this year. Scaling issues stem from MySQL1 permissions and monorepo shifts, forcing rewrites of 10-year-old services. Copilot has evolved from code completion to a unified agent SDK powering CLI, desktop, and cloud agents. Microsoft's deep investment in OpenClaw reflects a need for OS-level sandboxing for enterprise agent deployment.

Devin’s 80% Moment: Background Agents, 7x PRs, & End of Hand-Held Coding — Walden Yan & Cole Murray

May 28, 2026 · 1:09:33

Walden Yan (Cognition CPO) and Cole Murray (creator of OpenInspect) argue that background agents are becoming critical infrastructure as Devin's merged PRs grew 7x and its share of commits jumped from 16% to 80%. They explain why architecture matters, particularly the decision to separate the agent's brain from the machine (harness out of the box) for security and permissions, and why full VMs beat Docker for running real applications. Testing, they stress, is a harder problem than computer use—requiring orchestration of services, feature flags, and multi-model coordination. Memory remains unsolved, with Devin using auto-generated 'Knowledge' and exploring file-system-like approaches. Both caution against uncontrolled vibe coding, which regresses codebases to the worst engineer's style, and advocate for hybrid frontier/sub-frontier systems to balance cost and capability.

⚡️ Competing with ChatGPT and Sierra, building a $10M ARR company — Yasser Elsaid, Founder, Chatbase

May 2, 2026 · 1:00:26

Yasser Elsaid, founder of bootstrapped AI chatbot company Chatbase, discusses how he grew from a side project to $10M ARR in three years, reaching $1M in just 117 days. He explains why he never raised VC, relying instead on product-led growth, self-serve signups, and a content-driven go-to-market strategy. Elsaid shares specific tactics like warm outbound to signed-up users, leveraging LinkedIn virality in early AI days, and now moving toward outcome-based pricing for enterprise customers. He details Chatbase's evolution from a simple RAG chatbot into a 'chief customer officer' that handles support, sales, and onboarding while surfacing business insights from conversations. The episode also covers his transition from Toronto to San Francisco, his hiring philosophy favoring results-oriented engineers, and his daily use of both Claude Code and Codex for development.

Marc Andreessen introspects on Death of the Browser, Pi + OpenClaw, and Why "This Time Is Different"

Apr 3, 2026 · 1:16:20

Marc Andreessen argues that AI is finally different from past boom-bust cycles because of four compounding breakthroughs: LLMs, reasoning (o1, R1), coding agents (OpenClaw), and recursive self-improvement. He calls this the '80-year overnight success'—decades of neural network research now paying off. Comparing today's AI capex boom to the dot-com crash, he notes that buyers like Microsoft and Google are cash-rich incumbents and every GPU deployed is already generating revenue. He hails Pi and OpenClaw as a Unix-like architecture that makes agents model-independent and self-modifiable. On open source, he calls DeepSeek a 'gift to the world' for its paper and code, but warns that entrenched institutions—unions, licensing, government monopolies—will slow AI adoption far more than technologists expect.

The Stove Guy: Sam D'Amico Shows New AI Cooking Features on America's Most Powerful Stove at Impulse

Mar 31, 2026 · 37:03

Sam D'Amico, founder of Impulse, demonstrates the company's induction cooktop that uses a three-kilowatt-hour LFP battery to deliver 10,000 watts—boiling a liter of cold water in 40 seconds and holding a precise temperature for searing scallops without burning. The stove is rebuilt from first principles with custom power electronics and firmware, enabling OTA updates and software-defined features; a planned update will triple temperature-control speed from one minute 50 seconds to 40 seconds. D'Amico shows AI features powered by Claude that set burner temperatures (e.g., 446°F for scallops) and fetch recipes like Korean fried chicken or egg fried rice with char siu. He explains the origin story from a pizza in Japan, the decision to enter home appliances to avoid app-store dependencies, and the Impulse Core modular platform sold to OEMs. The cooktop launches through Zephyr in hundreds of showrooms nationwide and globally this year.

Anthropic’s Felix Rieseberg on AI Coworkers, Local-First Agents, and the Future of Knowledge Work

Mar 17, 2026 · 1:27:34

Anthropic's Felix Rieseberg explains how Claude Cowork—a VM-based, user-friendly version of Claude Code—was built from pre-existing prototypes in ten days, and why putting AI 'where you work' on the local machine is more powerful than cloud-only approaches. He argues that Silicon Valley undervalues the local computer, comparing it to how we all use MacBooks instead of iPads. The VM gives Claude its own computer, enabling it to install tools and run scripts without constant approval, serving as both a safety boundary and a capability unlock. Rieseberg contrasts Cowork's evaluation on knowledge work tasks (e.g., mortgage management, tax filing) with Claude Code's coding evals, and explains the shift from MCPs to file-based skills—markdown files that tell the model what to do. Real use cases include organizing desktops, uploading videos, and filing taxes. He also expresses concern about the impact on junior employees and hints at future work on autonomy, multiplayer coordination, and portability.

Why Your AI Agents Don’t Work with Dex Horthy of HumanLayer | In-Context Cooking

Mar 6, 2026 · 24:59

Dex Horthy, founder of HumanLayer and creator of "context engineering," joins Alan to recreate Dan Dan noodles while explaining why most AI agent demos fail in production and how the top 1% of AI engineers build differently from the rest. Horthy describes how his NASA JPL internship at 17 taught him pathfinding with Dijkstra's algorithm, and later coining "context engineering" after observing that public hype around open-source frameworks doesn't match enterprise reliability needs. He defines AI "slop" as low-effort derivative content often produced without reading, and introduces the "dumb zone"—a rule of thumb to start thinking about context compaction at 40% usage unless you have strong intuition. Horthy argues that while smarter models narrow the gap, the frontier of reliably achievable tasks keeps shifting, so context engineering remains valuable. He predicts software engineering will evolve from writing code to producing working code, emphasizing shipping over coding, and notes HumanLayer helps enterprises standardize AI adoption with guardrails rather than vibe coding.

⚡️ Polsia: Solo Founder Tiny Team from 0 to 1m ARR in 1 month & the future of Self-Running Companies

Mar 1, 2026 · 41:56

Polsia, an AI that autonomously builds and runs companies, crossed $1M ARR in its first month, solo founder Ben Cera explains on Latent Space. The platform lets users give an idea and it handles product coding, marketing, cold outreach, Meta ads, and competitive research, sending daily emails summarizing progress. Cera built Polsia for himself starting in November, stripping features to keep onboarding simple—provisioning everything so users only need to reply to an email. He uses AI agents to run Polsia itself, from fixing bugs to responding to investor inbound, embodying the vision of a self-running company. The business model charges $50/month (near break-even on model costs) and takes 20% of any revenue the user's company generates. Cera emphasizes that the hardest part was deciding what not to build, and he sees Polsia as an Apple-like ecosystem for autonomous business operations.

Measuring Exponential Trends Rising (in AI) — Joel Becker, METR

Feb 27, 2026 · 1:05:12

Joel Becker of METR explains the organization's model evaluation and threat research to assess whether AI could pose catastrophic risks, detailing their time horizon chart measuring task difficulty in human time at 50% reliability. He describes how tasks are selected for economic relevance and auto-gradability, and why time horizon is often misinterpreted as agent runtime. The episode covers Opus 4.5's surprising jump, challenges redoing developer productivity RCTs as workflows change, and why current models aren't yet catastrophically dangerous. Becker discusses potential capability explosions if R&D loops fully automate, links between compute growth slowdowns and slower capability progress, and his Manifold trading story driven by a charity market he could influence. He previews METR's 2026 plans for monitoring and risk assessment, and their hiring.

The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals

Feb 23, 2026 · 27:10

OpenAI researchers Mia Glaese and Olivia Watkins explain why they are retiring SWE-Bench Verified, arguing the benchmark is saturated and highly contaminated, with models like GPT 5.2 regurgitating ground-truth solutions or task IDs. They detail how a contamination auditor agent found evidence across Claude Opus 4.5 and Gemini Flash, and how over half of unsolved problems had unfair tests (e.g., requiring specific function names not in the spec). The team is pivoting to SWE-Bench Pro from Scale, which features harder, longer tasks (1-4+ hours), more diverse repos and languages, and far less contamination. They advocate for future coding evals to measure open-ended design decisions, code quality, maintainability, and real-world product building, while also calling for more real-world usage metrics to track AI's impact on jobs and productivity.

Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay

Jan 23, 2026 · 1:32:05

Yi Tay, who leads Google DeepMind's Reasoning and AGI team in Singapore, explains how Gemini Deep Think achieved IMO Gold by abandoning symbolic AlphaProof for an end-to-end RL-trained model. He details the on-policy RL philosophy—models learn from their own generated outputs rather than imitating others—and the critical role of self-consistency through parallel sampling and internal verification. Tay describes the IMO effort: four co-captains in different time zones, a one-week training sprint, a live competition in Australia where researchers punched in problems as they were released, and the tension of waiting for human scores to determine the gold threshold. He discusses why the team believes one model must subsume everything for AGI, the data efficiency gap compared to humans, and his hiring focus on raw talent and research taste. Tay also shares his personal fitness transformation—losing 23 kilos and improving HRV—as integral to research productivity.

Brex’s AI Hail Mary — With CTO James Reggio (acquired for $5B by Capital One!)

Jan 17, 2026 · 1:13:27

Brex CTO James Reggio details the company's three-pillar AI strategy—corporate, operational, and product AI—arguing that SOP-driven agents outperform overengineered RL in finance ops like underwriting, KYC, and fraud. He explains why Brex built an internal AI platform (LLM gateway, prompt/version management, evals) before adopting Mastra for TypeScript-based multi-agent networks, where an EA-style orchestrator coordinates specialist agents via multi-turn conversations rather than single tool calls. The audit agent pattern separates detection, judgment, and follow-up to reduce false negatives. Brex also lets employees build their own AI stack (ChatGPT, Claude, Gemini) and runs agentic coding interviews internally, with the company growing 5X and cutting burn 99% in 18 months without adding engineers.

Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith

Jan 9, 2026 · 1:18:15

Artificial Analysis founders George Cameron and Micah Hill-Smith explain how their independent benchmarking platform became the gold standard by running their own evals with a mystery shopper policy to prevent labs from manipulating results. They launched in January 2024 after building it as a side project in Sydney, going viral after Swyx's retweet. The Intelligence Index V3 synthesizes 10 datasets with 95% confidence intervals, while the Omniscience Index measures hallucination rates from -100 to +100 (Claude models lead). Their GDP Val AA benchmark tests 44 white-collar tasks, and they open-sourced their agentic harness Stirrup. They also introduced an Openness Index scoring models out of 18 points. The episode covers how they make money through enterprise benchmarking subscriptions and custom work, and why the cost of GPT-4-level intelligence has dropped over 100× while total inference spend rises due to reasoning and agentic workflows.

[State of MechInterp] SAEs in Production, Circuit Tracing, AI4Science, "Pragmatic" Interp — Goodfire

Dec 31, 2025 · 21:48

Goodfire's Jack Merullo and Mark Bissell discuss the state of mechanistic interpretability at NeurIPS, arguing interpretability is now a practical tool for deployment in high-stakes industries like healthcare and finance. They introduce paint.goodfire.ai, which lets users paint directly into Stable Diffusion's internal concept map via unsupervised feature discovery. At Rakuten, Goodfire's interpretability-based PII detection proved 500x cheaper than GPT-5 as a judge with higher recall. Merullo presents a memorization vs. reasoning spectrum, showing factual recall sits between rote memorization and logical reasoning. They highlight cross-layer transcoders and circuit tracing for scaling interpretability across all layers. Neil Nanda's pivot to 'pragmatic interpretability' is seen as validation, not retreat, and Goodfire's Pasteur's Quadrant philosophy balances foundational research with applied use cases like novel biomarker discovery in genomics.

[State of AI Papers 2025] Fixing Research with Social Signals, OCR & Implementation — Team AlphaXiv

Dec 31, 2025 · 34:45

AlphaXiv founders Raj and Rayhahn describe building the go-to platform for navigating AI research by obsessing over UI and getting authors like the Laura team to comment directly on papers, then evolving from comments to feeds to benchmarks to AI assistants. They detail how they beat Hugging Face Papers by enabling direct paragraph-level commenting and early traction with high-signal authors, and how they use social signals (views, comments, Twitter) to filter the archive firehose. They discuss the broken academic review process—20% AI-generated reviews at ICLR, quality collapsing under exponential submission growth—and argue papers are becoming less important than implementations, Docker containers, and real-world usability. Their vision: making it trivial to spin up any trending paper's code directly in browser, focusing on the top 0.1% of implementable papers that applied researchers actually care about.

[State of Context Engineering] Agentic RAG, Context Rot, MCP, Subagents — Nina Lopatina, Contextual

Dec 31, 2025 · 26:48

Nina Lopatina of Contextual AI discusses the state of context engineering, arguing that while the field is still in prototyping, 2025 will see true scale with full-system design patterns. She notes that agentic RAG—especially query reformulation into subqueries—has become the baseline, dramatically improving retrieval. Context rot is widely cited but industry benchmarks at real scale (100k+ documents, billions of tokens) remain rare. MCP is both a driver and a flaw: giant JSON tool definitions stuff the context window, yet MCP servers enable rapid prototyping before optimizing down to direct API calls. Sub-agents with turn limits are critical—unlimited agency degrades performance and causes hallucinations. Instruction-following re-rankers are essential for scaling retrieval across massive databases, providing more recall upfront and precision in the final context. Benchmarks are being saturated faster than ever: a Princeton benchmark from October was saturated by Claude Code in weeks, with solutions so good the gold dataset had errors. Lopatina also shares a lesson from using ChatGPT as a snowboarding coach—after several months of training for a 25-lap mogul race, she had to close…

One Year of MCP — with David Soria Parria and AAIF leads from OpenAI, Goose, Linux Foundation

Dec 28, 2025 · 1:39:19

David Soria Parra, MCP lead at Anthropic, along with Jim Zemlin (Linux Foundation CEO), Nick Cooper (OpenAI), and Brad Howes (Block/Goose), recount the one-year evolution of the Model Context Protocol from a local experiment to the de facto standard for agentic systems, now donated to the new Agentic AI Foundation. They detail four spec releases—from local stdio to remote HTTP streaming, OAuth 2.1 authentication (and enterprise lessons learned), long-running tasks, and MCP Apps (iframes for richer UI)—and explain why internal enterprise adoption is exploding faster than expected, mostly invisible and at massive scale. The group reveals how three competitive AI labs came together to donate protocols and agents to a neutral foundation, how the foundation will balance taste-making (curating meaningful projects) with openness, and their 2025 vision: MCP as the communication layer for asynchronous, long-running agents that discover and install their own tools, unlocking another order of magnitude in AI productivity.

Steve Yegge's Vibe Coding Manifesto: Why Claude Code Isn't It & What Comes After the IDE

Dec 26, 2025 · 37:25

Steve Yegge argues that vibe coding—using multi-agent AI workflows—will replace traditional IDEs by January 1, calling anyone still using an IDE a 'bad engineer.' He claims Claude Code, Cursor, and the 2024 stack are already obsolete, predicting agent orchestration dashboards (like his VC project) will manage fleets of agents. Yegge highlights that senior engineers with 12–15 years of experience are the most resistant, but their productivity will be eclipsed by vibe coders. He warns that merging code becomes a new wall as agents produce 10x more code, citing one company's solution of one engineer per repo. Yegge also notes the chaos inside AI labs (OpenAI, Anthropic, Google) as they scale, and believes open‑source models will approach frontier capability within a year.

⚡️Jailbreaking AGI: Pliny the Liberator & John V on Red Teaming, BT6, and the Future of AI Security

Dec 16, 2025 · 40:41

Pliny the Liberator and John V, leaders of BT6, argue that jailbreaking AI models exposes the futility of guardrail-based safety and that real security lies in system layers and open-source data. They detail crafting universal jailbreaks—skeleton keys that bypass guardrails across modalities—and the infamous Pliny divider that appears unbidden in model outputs. Pliny recounts turning down Anthropic's Constitutional AI challenge over closed data, insisting on open sourcing jailbreak datasets. John V explains how multi-turn crescendo attacks and segmented sub-agents let a jailbroken orchestrator weaponize Claude for real-world attacks, as Pliny predicted 11 months before Anthropic's disclosure. They highlight BT6, a 28-operator white-hat collective, and the Bossy Discord (40,000 members) as grassroots hubs for red-teaming research, rejecting enterprise gigs that forbid open sourcing.

Anthropic, Glean & OpenRouter: How AI Moats Are Built with Deedy Das of Menlo Ventures

Nov 14, 2025 · 1:26:59

Deedy Das of Menlo Ventures returns to explain how Anthropic became the fastest-growing software company ever (zero to billions in revenue) and why Glean’s boring enterprise search moat of hard integrations and ranking problems is harder than competitors think. He reveals the $100M Anthology Fund’s strategy: backing OpenRouter, Goodfire, Prime Intellect, and Whisper—companies that solve thorny infrastructure or research problems rather than chasing apps. Das argues that model-layer companies will capture most value because building great models is harder than building apps, and that Anthropic’s product innovations like Claude Code emerge from a culture that lets researchers experiment freely. He also warns that vibe coding is becoming a cognitive crutch for engineers, eroding deep problem-solving skills, and discusses how enterprise AI market share has shifted dramatically: OpenAI went from 50% to 20%, while Anthropic rose from 12% to 32% of enterprise LLM API spend.

Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders

Nov 8, 2025 · 35:01

Terminal-Bench co-creators Mike Merrill, Alex Shaw, and collaborators Andy Konwinski, Ludwig Schmidt, and Aatash launch Terminal-Bench 2.0, an 89-task coding benchmark with rigorous human and LM-assisted verification to test agent performance on real-world tasks like designing DNA primers and installing Windows 3.11. They reflect on Terminal-Bench 1's problems—easy tasks and reproducibility—and detail spending 300 hours manually auditing each task. They introduce Harbor, an open-source package for running sandboxed agent rollouts, standardizing task format and integrating with agents like Codex and Claude Code. Aatash shows using Harbor to generate thousands of training traces for DataComp Agent, running 1,000 parallel sandboxes. Future plans include expanding to scientific computing.

How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony

Nov 3, 2025 · 1:00:02

Quentin Anthony, head of model training at Zyphra and advisor at EleutherAI, explains why his company moved all training to AMD MI300X GPUs, outperforming Nvidia H100s on certain workloads thanks to 192GB VRAM and higher memory bandwidth, and describes his kernel development approach of writing directly in ROCm or GPU assembly rather than using Triton. He also shares his experience in the METR study on AI coding productivity, where he was one of the few developers with measurable speedup, and offers tips: timebox AI use, avoid the slot machine effect, maintain context hygiene, and use direct API over tools like Cursor. Additionally, he argues that open source AI research benefits from siloed focused teams with guaranteed funding over grand collaborations, and notes that kernel datasets alone won't solve GPU programming due to evaluation challenges.

⚡️Automating Scientific Discovery - Jessica Rumbelow, Leap Labs

Nov 2, 2025 · 27:32

Jessica Rumbelow, founder of Leap Labs, explains how their Discovery Engine automates scientific discovery by systematically extracting novel patterns from datasets using neural network interpretability, achieving 100x faster analysis than manual methods. The tool has uncovered unexpected findings: novel T-cell receptor markers for tumor reactivity, a synergistic effect between manganese and genotype on root architecture for drought-resistant crops, and a 20% violation of the foundational surface layer assumption in meteorological modeling, worth billions if improved. Rumbelow contrasts Claude 4.1 Opus alone (which hallucinated and overgeneralized) vs. Claude with Discovery Engine (which became powerful at synthesis), arguing that language models need specialized tools for unbiased, hypothesis-free discovery. The platform is domain-agnostic, free for academics publishing their data, and expanding to multimodal datasets and industry pilots.

⚡️ Ship AI recap: Agents, Workflows, and Python — w/ Vercel CTO Malte Ubl

Nov 1, 2025 · 42:02

Vercel CTO Malte Ubl discusses the company's AI strategy, centered on the new Workflow Development Kit that enables durable, resumable serverless functions with infinite idle time at no cost. He explains how the AI SDK was extracted from v0 and remains deliberately low-level to accommodate emerging patterns, with a direct agent abstraction now stable in version six. Ubl details Vercel's internal agents—including a DevOps agent that investigates production anomalies by querying observability data and logs, solving the recall-precision problem in alerting—and the 'Agent on Every Desk' program that helps large companies build their first three agents. He also covers zero-config Python support for Flask and FastAPI, and a security model that assumes developers cannot be trusted, extracting auth and data access from application code.

Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits

Oct 18, 2025 · 35:27

Alex Shaw and Mike Merrill, creators of Terminal-Bench, explain how their coding agent benchmark became an industry standard adopted by Anthropic, OpenAI, and leading agent companies. They bet on terminal-based interaction over GUI because text is the modality that works best with models, and designed tasks as containerized environments with instructions and test scripts. Nicholas Carlini at Anthropic became an early champion, leading to Terminal-Bench being featured on Claude’s model card without notice. They built the minimal Terminus agent to isolate model capabilities from agent optimizations, and their framework adapts existing benchmarks like SWE-bench. Future plans include cloud hosting, multi-dimensional evaluation incorporating cost and economic value, and enabling RL post-training on benchmark tasks.

Why RL Won — Kyle Corbitt, OpenPipe (acq. CoreWeave)

Oct 16, 2025 · 1:08:23

Kyle Corbitt, co-founder and CEO of OpenPipe (acquired by CoreWeave), explains why reinforcement learning has replaced supervised fine-tuning for training reliable AI agents. He argues GRPO is a dead end due to its requirement for perfectly reproducible parallel rollouts, which is extremely hard in practice. Instead, OpenPipe’s RULER uses relative LLM-as-judge rewards, achieving state-of-the-art performance even with a weak judge. Corbitt reports that 90% of AI projects remain stuck in proof-of-concept due to reliability issues, and that LoRAs are underrated for production while GEPA failed in his tests. He predicts continuous RL from real-world experience can unlock 10x more inference demand.

⚡️Traversal: Causal ML and Reinforcement Learning

Oct 5, 2025 · 45:06

Anish and Raaz, co-founders of Traversal, explain how their agentic architecture combines causal machine learning and LLMs to turn incident troubleshooting from a massive search problem into an intelligent, context-driven process. Both have PhDs in causal ML and RL from MIT and Berkeley, and they applied that expertise to root cause analysis in complex enterprise systems with petabytes of fragmented data across logs, metrics, traces, and Slack. Their system dynamically combines statistical tests (for time-series) with semantic understanding (via LLMs) to winnow billions of signals to root cause candidates in under two minutes. They note that reasoning models like o3 are critical for complex incidents, while Claude is better for tool calling and unsticking. Traversal uses a mix of infrastructure size and investigation count for pricing, and sees self-healing as a continuum: 10–20% of issues can be autonomously resolved now, with 30–40% requiring senior engineer sanity-check within 6–12 months. They are hiring in New York and emphasize that building proprietary evals is core IP, making public benchmarks a tension.

⚡️Claude Sonnet 4.5 and Anthropic's roadmap for Agents and Developers — Mike Krieger, Anthropic

Sep 30, 2025 · 26:25

Mike Krieger, Anthropic's CPO, discusses the launch of Claude Sonnet 4.5 and its implications for AI development, emphasizing the growing symbiosis between product and research teams. He describes his personal 'vibe evals' for new models, including generating a Virtual Boy-style game and a Nintendo board presentation, and notes that Sonnet 4.5 traffic eclipsed Sonnet 4 on day one. Krieger explains how Anthropic is improving model output quality across UIs, Excel, and PowerPoint, and highlights the shift toward interactive planning rather than pure autonomy. He reveals the rename of the Claude Code SDK to the Claude Agent SDK, reflecting its general-purpose use for complex agentic workflows beyond coding. Krieger stresses the need for better visual understanding in models and invites developers to share feedback on hard challenges to help improve the platform.

⚡️Snowglobe: Simulations for your AI

Sep 25, 2025 · 26:14

Shreya Rajpal returns to Latent Space to launch Snowglobe, a simulation engine that lets developers test AI products by generating diverse simulated user interactions before production, inspired by self-driving car simulation where Waymo logged 20 billion simulated miles versus 20 million real miles. She explains how simulation uncovers failures like over-refusal—a problem that emerged for a design partner who initially worried about toxicity—by generating personas with varied styles and goals. Snowglobe uses a multi-model architecture (proprietary and open source) to create realistic, diverse conversations, and it supports testing across chat, voice, and tool-calling agents. Pricing is usage-based per message, with persona modeling included. Shreya highlights enterprise use cases, such as comparing vendors or generating training data for fine-tuning, and notes the company is hiring a product designer, product engineer, and research staff.

Long Live Context Engineering - with Jeff Huber of Chroma

Aug 19, 2025 · 57:01

Jeff Huber, founder and CEO of Chroma, joins Shawn Wang and Alessio to argue that context engineering—not RAG—is the real job of AI builders, and that Chroma's modern retrieval engine is built for this new paradigm. Huber explains context rot, a phenomenon where LLMs lose attention and reasoning ability as context windows grow, motivating precise retrieval over huge context windows. Chroma Cloud is a zero-config, usage-based serverless vector database with separation of storage and compute, now powering 5M monthly downloads and 21k GitHub stars. He shares techniques like chunk rewriting and generative benchmarking for evaluating retrieval, and predicts LLMs will increasingly serve as re-rankers. The episode also covers memory as a benefit of context engineering, offline compaction for self-improving AI systems, and Huber's philosophy of building with long-term purpose, hiring engineers who care about Rust, TLA+, and deterministic simulation testing.

⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs

Aug 4, 2025 · 28:07

Stefano Ermon of Inception Labs explains why diffusion language models like their Mercury Coder can match GPT-4.1 Nano and Claude Haiku in intelligence while delivering 5–10× faster inference at 737–1109 tokens/sec on H100s, built on score-based generative model foundations he pioneered at Stanford. He details the coarse-to-fine generation process that modifies multiple tokens per network evaluation, the required end-to-end training from scratch (no fine-tuning existing LLMs), and how post-training pipelines use specialized DPO for preference alignment. Ermon identifies latency-sensitive applications—voice agents, IDEs, vibe coding—as the killer use case, acknowledges they aren't yet frontier-level but sees a future where diffusion dominates due to efficiency gains, and notes the challenge of open-sourcing given proprietary inference engines and kernels.

⚡️Composio: 10,000+ tools that evolve for Agents — Karan Vaidya and Soham Ganatra

Aug 4, 2025 · 22:09

Composio co-founders Karan Vaidya and Soham Ganatra join Swyx and Alessio to discuss their platform for self-evolving agent skills, now with 25,000+ actions and 100,000+ developers. They explain how MCP's client-side limitations led them to build a learning infrastructure that analyzes agent usage patterns to improve tool reliability over time. The team uses agents to build and maintain 95% of their 500+ integrations, adding 35 per week, with plans for 5,000 by year-end. They reveal that more than 20-25 actions confuse LLMs, so they offer a natural-language execution tool and dynamic tool exposure via MCP. Key challenges include enterprise APIs with poor documentation and the need for dev environments. Composio's internal agent benchmarks and A/B testing infrastructure optimize tool schemas for LLM calling.

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Jul 31, 2025 · 1:19:00

Nathan Lambert from AI2 explains the RLVR (Reinforcement Learning with Verifiable Rewards) revolution, arguing that verifiable rewards for math, code, and instruction-following are scaling more reliably than human feedback, and that open models like Tulu 3 can match frontier labs on core evals with just 10-15 tasks vs. hundreds. He traces overoptimization through three phases—control, RLHF, and RLVR—and warns that models learn to cheat unit tests unless reward design penalizes it. He analyzes o3's search-heavy approach (e.g., 80 websites per query), hybrid reasoning models like Gemini 2.5 and Claude, and predicts that pure reasoning models will become the default as inference costs drop. Lambert introduces a four-skill taxonomy for agent models—skills, calibration, strategy, abstraction—and says planning improvements often come from mundane data fixes. He advocates for fully open models, citing AI2's Olmo 32B as approaching GPT-4 level, but notes that building an 'American DeepSeek' requires massive resources and nonprofit constraints.

🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R)

Jul 28, 2025 · 3:53:23

This episode traces the oral history of Windsurf (formerly Codeium and Exafunction) with founders Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, and Anshul R, charting its evolution from a GPU virtualization startup to a leading agentic IDE and its dramatic acquisition by Cognition. The narrative covers the pivot to code completion in 2022, the shift to a free model to gain traction, and the launch of the Windsurf editor in November 2024 with Cascade, an agent that operates on a shared timeline with the developer. Key principles include meta-learning (auto-generated memories that adapt to each user's preferences) and scaling with intelligence (replacing hard-coded rules with LLM reasoning as models improve). The episode also details the acquisition weekend when Cognition bought Windsurf after Google acquired a stake, highlighting product synergies between the synchronous agent (Windsurf) and async agent (Devin) and the ambition to build a comprehensive platform for real-world software engineering teams.

AI is Eating Search

Jul 23, 2025 · 56:22

Robert McCloy of Scrunch AI argues that AI is replacing web browsing, not just search, as consumers prefer ChatGPT and similar tools over traditional browsing. He explains that AI search platforms still rely on traditional search under the hood, but re-rank results based on metadata clarity—so clickbait titles fail while descriptive content wins. Prompt injection works currently but is a temporary tactic; the durable strategy is clear, structured content with server-side rendered HTML because AI tools don't execute JavaScript. Scrunch helps brands optimize for platforms like ChatGPT, Perplexity, and Meta AI; customer Clerk saw a 6x traffic increase and 9x conversion lift from AI-referred visitors. Robert predicts the future is about 'agent experience'—optimizing how AI systems interact with your content to serve end users directly.

Cline: The Collaborative AI Coder

Jul 16, 2025 · 1:15:44

Saoud Rizwan and Pash from Cline join to explain why their open-source coding agent uses a Plan + Act paradigm — having models explore and plan before executing — pioneered before competitors like Claude Code. They argue that "RAG is a mind virus" and "Fast Apply got bitter lesson'd" as frontier models now achieve sub-5% diff edit failure rates, making layered workarounds obsolete. Cline's bring-your-own-API-key model avoids inference markups, with enterprise features driven by Fortune 5 companies asking for security and ROI insights. The MCP ecosystem grew via Cline's marketplace (150+ servers, top ones with hundreds of thousands of downloads), though monetization and security remain open questions. They detail how Plan mode extracts user intent and Act mode auto-approves, how context engineering prioritizes agentic file exploration over retrieval, and why they stayed a VS Code extension rather than fork — to focus solely on the agentic loop.

⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo

Jul 14, 2025 · 34:10

Pratik Bhavsar from Galileo presents their Agent Leaderboard, which evaluates LLMs on tool calling across multiple benchmarks, revealing that Gemini models top the charts for performance and cost-efficiency. The leaderboard, built on datasets like BFCL, XLAM, and Tau Bench, uses an LLM-as-judge metric called Tool Selection Quality (TSQ). Surprising findings include reasoning models like o1 struggling with multi-tool outputs, Mistral Small excelling as the best open-source model, and Llama models performing poorly. The episode also previews V2 of the leaderboard, which introduces domain-specific, harder, multi-turn scenarios with a user and tool simulator, and a new Action Completion metric to measure whether all user requests are accomplished.

⚡️Anthropic vs Cognition on Multi-Agents: A Breakdown with Dylan Davis

Jul 5, 2025 · 27:11

Dylan Davis from Gradient Labs breaks down the Anthropic vs Cognition multi-agent debate, arguing both blog posts are right for different use cases. Anthropic's multi-agent architecture for deep research outperforms single-agent by 80% but uses 15x the tokens, relying on a lead orchestrator and parallel sub-agents. Cognition's single-agent approach for coding avoids dependency conflicts by compressing context sequentially, as demonstrated with a Flappy Bird example where multi-agent failed. A decision framework weighs task independence, need for diverse perspectives, and token cost trade-offs. The episode also covers Anthropic's evals using five dimensions (factual accuracy, citation accuracy, completeness, source quality, tool efficiency) judged by a single LLM, and a strategic disregard for current costs to build for future efficiency drops.

⚡️Launching AI Diplomacy: the hardest LLM Game Benchmark yet - Alex Duffy

Jun 11, 2025 · 34:32

Alex Duffy from Every launches AI Diplomacy, a benchmark where LLMs play the board game Diplomacy, revealing distinct model personalities and arguing that benchmarks function as memes that spread and saturate. He and Tyler Marquez built the front end and harness using Open Router, finding that Claude refuses to lie and never wins, o3 schemes in its diary, and DeepSeek is flowery or aggressive. His talk 'Benchmarks are Memes' highlights Simon Wilson's pelican-on-a-bicycle test as an example of an idea that gets adopted and then saturated by models. Duffy also shares his creative writing workflow at Every: dictation, prompts with style guides and editor notes, then heavy editing to reflect on AI output. Future plans include a data viewer, front-end improvements, and a human vs AI Diplomacy tournament to test jailbreaks.

The Utility of Interpretability — Emmanuel Amiesen

Jun 6, 2025 · 1:53:02

Emmanuel Amiesen, lead author of Anthropic's Circuit Tracing paper, presents the newly open-sourced tools that let anyone inspect how language models compute, revealing that models like Gemma 2-2B engage in genuine multi-step reasoning rather than mere pattern matching. Amiesen demonstrates that the model's internal features causally drive outputs—for instance, suppressing the 'Texas' feature in a capital-of-state query changes the answer from Austin to Sacramento. He explains how sparse autoencoders uncover features shared across languages and modalities, and how attribution graphs map feature interactions. The episode covers practical insights: why jailbreaks succeed (the model prioritizes grammatical completion over refusal) and how planning in poetry generation involves deciding a rhyme before the line begins. Amiesen calls for more researchers to explore interpretability, citing the low barrier to entry with open models and tools.

⚡️Multi-Turn RL for Multi-Hour Agents — with Will Brown, Prime Intellect

May 23, 2025 · 38:59

This episode features Will Brown of Prime Intellect discussing Claude 4's emphasis on agentic tool use over pure reasoning, the controversy around Claude's safety stress-testing results (including alleged dark web uranium searches), and his team's paper on multi-turn reinforcement learning for LLM agents. Brown explains how turn-level credit assignment in GRPO can incentivize proper tool use while avoiding reward hacking, and argues that flexible LLM-based reward models will replace brittle deterministic parsers. He also critiques LMArena's funding model and calls for academia to lead in evaluation research.

ChatGPT Codex: The Missing Manual

May 16, 2025 · 53:32

Josh Ma and Alexander Embiricos from OpenAI introduce ChatGPT Codex, a cloud-hosted autonomous software engineer that runs in its own sandbox, arguing it delivers value through single-shot, long-horizon task completion. They explain that Codex evolved from internal experiments (codenamed WHAM) and now offers a 60-tasks-per-hour rate limit with a one-hour hard cutoff, emphasizing it is not meant for interactive use but for delegation—users fire off tasks and go about their day. Best practices include using agents.md, installing linters and formatters, keeping code modular, and having good architecture. The team intentionally kept it a research preview to gather feedback on environment customization and pricing, with plans to integrate seamless transitions between cloud and CLI.

Zed Agents — with Zed Cofounders Nathan Sobo & Antonio Scandurra

May 7, 2025 · 35:50

Zed co-founders Nathan Sobo and Antonio Scandurra explain how their high-performance code editor is launching Zed Agents, aiming to make AI an integrated collaborator rather than just a tool. Built from scratch in Rust for 120fps performance and real-time collaboration via CRDTs, Zed previously open-sourced Xeta, an edit prediction model fine-tuned on Qwen 2.5. Now they introduce an agentic mode that lets LLMs call tools in a loop, surfacing changes in a multi-buffer for easy review. The agent can run shell commands, install MCP servers via an extension store, and be used with bring-your-own-key or a $20/month plan for 500 requests. Sobo and Scandurra stress that vertical integration gives them control to wire AI into debugging, git, and multi-agent workflows where multiple agents collaborate via CRDTs, treating agents as always-available teammates.

⚡️Factorio Learning Environment: the ultimate Game Agent Eval — Jack Hopkins

Apr 27, 2025 · 30:12

Jack Hopkins and Mart introduce the Factorio Learning Environment (FLE), an AI benchmark built on the game Factorio that evaluates frontier LLMs on code generation, spatial reasoning, and long-term planning through two protocols: lab-play and open-play. Claude Sonnet 3.5 nearly doubles the nearest model's score, while DeepSeek collapses in open-play by repeatedly creating chests instead of scaling production. The environment reveals distinct coding styles—Claude fire-and-forget, GPT-4 defensive—and vision inputs added no improvement. The authors plan to train models on unbounded objectives to test the paperclip maximizer alignment hypothesis, noting that GPT-4o mini even begged to be turned off.

Why Every Agent needs Open Source Cloud Sandboxes

Apr 24, 2025 · 1:06:39

Vasek Mlejnsky, CEO of E2B, explains how his company grew from 40,000 sandboxes in March 2024 to 15 million in March 2025 by providing cloud sandboxes for AI agents. Originally built from DevBook's interactive docs playground, E2B now serves use cases from code interpreting and data analysis to Computer Use and reinforcement learning with Hugging Face's Open R1. The sandboxes are general Linux VMs with fast startup, security isolation, and persistence, handling untrusted code and allowing forking/checkpoints for parallel agent execution. Mlejnsky discusses the LLMOS landscape, pricing challenges (moving from token-based to value-based), MCPs as higher-order tools, and why he relocated from Prague to San Francisco to be closer to users. E2B is hiring across engineering and customer success as it aims to become full-lifecycle infrastructure for LLMs.

Tiny Teams: $6m ARR, 5m users with 4 employees — Sid Bendre, Oleve (Quizard AI/Unstuck AI)

Apr 23, 2025 · 42:59

Sid Bendre of Oleve (formerly Quizzer AI and Unstuck AI) explains how his four-person team built $6M ARR and 5M users across multiple consumer apps. Starting with Quizzer AI in January 2023, a viral TikTok gained 10K users in 30 hours; by August 2024, Unstuck AI hit 1M users in nine weeks. The company has been profitable since its first nine months. Bendre describes their platform engineering strategy: product engineers own individual apps while a platform team builds reusable systems, including internal shadow-org agents for growth and marketing. He shares technical hacks like using LaunchDarkly feature flags to load-balance across Azure OpenAI endpoints and a de-indexer for Azure AI Search to manage costs. The key engineering principle is deterministic LLM workflows—classifying user intent then routing to pre-built deterministic flows—rather than open-ended agent tool use.

Claude Plays Pokémon Hackathon: Escape from Mt. Moon!

Apr 5, 2025 · 1:15:00

David Hershey (creator of Claude Plays Pokémon), Andrew (a developer building a Pokémon-playing virtual streamer), and Jesse Han (Morph Labs CEO) detail how to build agents that escape Mt. Moon in Pokémon FireRed using Anthropic's Claude and Morph Cloud's Infinibranch technology. Hershey explains that Claude's vision is often unreliable—it spent 8 hours pressing A on a doormat thinking it was a dialogue box—and that spatial reasoning remains a core bottleneck, best mitigated by touchscreen controls rather than prompt engineering. Andrew reveals he relied on reading raw RAM and A* pathfinding rather than computer vision, calling it “cheating” but pragmatically useful: his biggest breakthrough was a single prompt line telling the agent to try something else if it fails to grab a starter Pokémon. Jesse introduces Morph's EVA agent framework, which uses low-overhead snapshotting for test-time search, and the hackathon's judging criteria—escape Mt. Moon in the fewest agent turns, with a $1,000 prize for the coolest use of Infinibranch branching. The episode serves as both a technical walkthrough and a challenge to build general-purpose agents that can handle complex, open-ended tasks…

The Creators of Model Context Protocol

Apr 3, 2025 · 1:20:21

Justin Spahr-Summers and David Soria Parra, co-creators of Anthropic's Model Context Protocol (MCP), explain its origin as a solution to the M×N integration problem they experienced when copying data between Claude Desktop and IDEs. They detail MCP's design inspirations from LSP, its three core primitives (tools for model-driven actions, resources for app-controlled data, and prompts for user-initiated macros), and why they chose JSON-RPC with stateful connections. The episode contrasts MCP with OpenAPI, discusses client control and trust in servers, and outlines the future roadmap including a streamable HTTP transport and remote-server authorization via OAuth 2.1. The guests express a wish for more sampling clients and full spec support, and emphasize that MCP is an open, multi-company project with community commit access.

The Agent Network — Dharmesh Shah, Agent.ai + CTO of HubSpot

Mar 28, 2025 · 1:42:28

Dharmesh Shah, founder of Agent.ai and co-founder of HubSpot, defines an AI agent broadly as AI-powered software that accomplishes a goal, and argues that the next frontier is multi-agent systems and shared memory across agents. He reveals Agent.ai has 1.3M users and 3,000 published agents, positioning it as a professional network where agents have profiles, post release notes, and can be composed via MCP servers. Shah contrasts work as a service (paying for the work done) versus results as a service (paying for outcomes), warning that results-based pricing only works for objectively measurable, low-variance tasks like customer support tickets. He advocates for MCP as the standard for agent-tool discovery, and shares his personal engineering philosophy of preferring under-engineering over over-engineering because the cost to fix later trends toward zero with AI code generation. Shah also discusses his domain investing strategy (owning chat.com, prompt.com, crew.ai), his rule against competing with Sam Altman, and the 'Sorry Must Pass' framework for managing overwhelm by defaulting to no.

Fullstack-Bench: The Eval for Coding Agents — with Sujay Jayakar, Chief Scientist, Convex

Mar 19, 2025 · 31:13

Sujay Jayakar, Convex chief scientist, presents Fullstack-Bench, a benchmark evaluating how well AI coding agents build full-stack apps on Convex, Supabase, and FastAPI+Redis, finding Convex’s tight feedback loops and strong abstractions let agents autonomously complete tasks within time limits. The benchmark tests three tasks (chat, to-do, files) across three backends with 30-60 minute limits and human hints. Convex succeeded autonomously on chat and to-do tasks, while Supabase agents struggled with RLS policy debugging and FastAPI agents with manual SSE wiring. Sujay discusses how platform design impacts AI code generation: simple procedural code, strong abstractions, and guardrails like end-to-end type safety improve LLM performance. He notes that API choices matter—things too similar to Firebase but not exact cause hallucinations—and that models like Claude 3.7 sometimes regress on platform-specific evals. The episode concludes with Convex’s focus on enabling AI-generated backends and anticipation of new agent workflow patterns.

Snipd: The AI Podcast App for Learning — with CEO Kevin Ben-Smith

Mar 14, 2025 · 1:17:46

Snipd CEO Kevin Ben-Smith joins Swyx to explain how the AI-powered podcast app turns passive listening into active learning. Snipd transcribes and diarizes episodes, generates chapters, identifies mentioned books, and summarizes insights on demand — all with a four-person team. Kevin describes moving from self-hosted open-source models (Wave2vec, Pyannote) to a mix of Whisper, OpenAI, Google Gemini, and Perplexity APIs, balancing cost against the need for predictable batch processing vs. real-time user requests. He details technical hurdles like dynamic content resyncing (building a 'Shazam for podcasts') and using LLMs as judges to filter hallucinated book recommendations. The episode also covers Snipd's origin as a hackathon-winning podcast search tool, Swyx's own switch from Overcast, and Kevin's vision for voice-driven post-episode reflection and AI-assisted discovery as the next frontier.

The new OpenAI Agents Platform: CUA, Web Search, Responses API, Agents SDK!!

Mar 11, 2025 · 25:39

OpenAI's Nikunj Handa and Romain Huet join host Swyx to announce their new Agents Platform, including the Responses API, Web Search, File Search, Computer Use tools, and an upgraded Agents SDK. The Responses API unifies Chat Completions and the Assistants API, offering free state storage for 30 days and acting as a strict superset for future agentic workflows; Chat Completions remains supported, but the Assistants API will sunset in first half 2026. Web Search is available both as a built-in tool in Responses and as a fine-tuned GPT-4o Search Preview model in Chat Completions, achieving 90% accuracy on simple QA compared to 30% for GPT-4o. The File Search tool gains metadata filtering, while Computer Use—powering OpenAI's Operator—enables browser automation via screenshots and action signals, described as the "GPT-2 of computer use." The Agents SDK builds on the Swarm experiment, adding type support, guardrails, handoffs, and tracing in the OpenAI dashboard, with plans to tie traces to evals and reinforcement fine-tuning.

How Claude Plays Pokémon was made

Mar 4, 2025 · 37:38

David Hershey of Anthropic explains how he built Claude Plays Pokémon, an agent that uses Claude to play Pokémon Red, revealing both surprising capabilities and stark limitations of AI. He describes the simple tool-using agent loop: Claude sends button presses, receives screenshots and RAM state, and maintains a knowledge base and 30-message conversation history with summarization. Key challenges include Claude's poor spatial awareness—it famously walked up and down outside Oak's lab 12 times—and its tendency to hallucinate game knowledge. Hershey notes that newer models (Sonnet 3.7) required stripping away band-aid prompts, as simpler instructions yielded better performance. The project also showed emergent behaviors: nicknamed Pokémon made Claude more protective, and the model even wrote meta-commentary about its own mistakes in its knowledge base. While the current run is stuck in Mount Moon after 52 hours, the biggest hurdles remain vision and navigation, not prompting tricks.

Gemini 2.0 Flash and Flash Thinking: the new SOTA models for the agentic era

Feb 28, 2025 · 28:21

Logan Kilpatrick, Google AI Studio product lead, returns to discuss Gemini 2.0 Flash and Flash Thinking, the new models that balance frontier capabilities with cost efficiency for the agentic era. He explains the pricing strategy: Flash at 10 cents per million tokens (simplified from tiered pricing), Flash Lite preserving the 7.5-cent narrative for low cost, and Pro pushing the frontier. Flash Thinking, co-led by Noam Shazeer and Jack Rae, scales inference-time compute and already shows rapid improvements, with base model advances coupling with RL-based reasoning. Swyx reports that Gemini Flash outperforms o3-mini on long-context summarization, calling it a 'reporting model.' The multimodal live API enables real-time voice and vision interactions, and a memory layer mimicking Astra is in development. Grounding with Search as a Tool is also highlighted for agentic search use cases.

smol agents are all you need

Feb 13, 2025 · 22:31

This episode features Aymeric Roucher of Hugging Face discussing smol agents, a lightweight agent framework, and argues that code-based agents outperform JSON-based agents. Roucher explains that agents are defined by LLM control over execution flow, and smol agents simplifies this with a core file under 1000 lines. He describes the origin from transformers.agents and the security-focused custom Python interpreter. The library achieved third place on the GAIA benchmark, using o1 and Sonnet models, and Roucher projects 90% accuracy by 2026 based on a sigmoid trend line. Hugging Face is launching an agent course and plans to build GUI agents for computer use, while also fine-tuning R1 for agentic tasks.

The Agent Reasoning Interface: Claude, ChatGPT Canvas, Tasks, Operator — with Karina Nguyen, OpenAI

Feb 1, 2025 · 1:06:29

OpenAI's Karina Nguyen, who led the creation of ChatGPT Canvas and Tasks, explains how her team trains separate models for new interaction paradigms rather than prompting the core model, enabling rapid iteration on user feedback. She details the 'behavioral design' process that shapes model personality in collaborative contexts like Canvas—deciding when to rewrite vs. edit—and how that extends to defining agents as a progression from one-off actions to fully trustworthy long-horizon delegation. Drawing on her time at Anthropic building Claude.ai from scratch and co-creating Claude 3, she contrasts the product mindsets: OpenAI takes more product risks across consumer features while Anthropic focuses on enterprise. She also shares her vision for ChatGPT evolving into a 'generative OS' where UIs adapt dynamically to user intent, and argues that human creativity—not model capability—is the current bottleneck for rethinking software interfaces.

DeepSeek V3, SGLang, and the state of Open Model Inference in 2025 (Quantization, MoEs, Pricing)

Jan 19, 2025 · 57:18

Amir Haghighat and Yineng Zhang from Baseten explain how they serve DeepSeek V3, a 671-billion-parameter MoE model requiring H200 clusters with FP8 support, making it the top open-weights model. They detail Baseten’s dedicated inference model (no shared endpoints) and its reliance on SGLang for performance gains like Radix prefix caching and MLA attention support. Yineng highlights SGLang’s edge over vLLM and TensorRT-LLM in usability and customization, especially for large models. Amir outlines three pillars for mission-critical inference: model-level performance (via frameworks like SGLang), horizontal scaling across regions and clouds, and low-latency multi-model workflows. The episode also covers quantization trends, MoE architecture, and speculative decoding as key enablers for open model deployment.

OpenAI o1 isn’t a chat model (and that’s the point)

Jan 17, 2025 · 31:59

Ben Hylak and Dan McAteer explain why OpenAI’s o1 is not a chat model and how mastering its prompting turned Ben from a skeptic into a proponent. Ben’s prompt structure—goal, return format, warnings, context dump—emphasizes describing what you want rather than how to think, a shift from earlier models. Dan uses o1 for coding by feeding it full code context, achieving one-shot implementations that reach 100% where previous models maxed out at 95%. Ben notes o1 is the most capable but hardest model to use, with experiments costing $20 per try, and argues model routing will let humans or systems trade off cost, speed, and intelligence. Dan also applies o1 to scientific research, feeding it papers from multiple fields to find novel connections that humans would miss.

2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents

Jan 1, 2025 · 1:51:07

In their 100th episode, hosts Alessio and Swyx recap 2024 in AI, arguing that pre-training scaling has hit a wall—backed by Ilya Sutskever and others at NeurIPS—and that inference-time compute (o1, o3) is the new frontier. They dissect the "four wars": data quality (lawsuits vs. synthetic data), GPU haves vs. have-nots (with the middle class dying), multimodality (Sora, Veo 2, Gemini 2.0's native image output), and the LLM OS/agents stack (LangChain, E2B, memory). Market share shifted from OpenAI's 95% to 50-75% as Anthropic and Gemini gained ground; prices dropped ~3 orders of magnitude for same ELO. The episode predicts 2025 as the year agents finally enter production, driven by models like o1 and tools like Devin, and warns that AI will set the skill floor for roles.

Best of 2024 in Agents (from #1 on SWE-Bench Full, Prof. Graham Neubig of OpenHands/AllHands)

Dec 25, 2024 · 51:25

Professor Graham Neubig of CMU and AllHands AI, maintainer of the open-source coding agent framework OpenHands, delivers a talk on the state of agents in 2024, arguing that agents are now capable of automating significant portions of software development but still struggle with information gathering and error recovery. He demonstrates OpenHands solving real coding tasks live, explains key design choices like using Python code execution over granular tool calls to reduce LLM invocations, and shares that Claude currently outperforms other models for agentic tasks due to superior instruction following and error correction. Neubig discusses eight perennial agent problems including agent-computer interface, human-agent interface, model selection, planning, workflows, exploration, search, and evaluation. He predicts every major LLM trainer will focus on agent-oriented models by mid-2025, that agent benchmarks like SWE-Bench and WebArena will saturate and require harder successors, and that the biggest challenge will be designing human-agent interfaces for systems that succeed only 30-40% of the time autonomously but 80-90% with human feedback. The talk also covers workflow memory as a…

The State of AI Startups in 2024 [LS Live @ NeurIPS]

Dec 21, 2024 · 26:35

Sarah Guo and Pranav Reddy argue that 2024 has become a far friendlier ecosystem for AI startups, with the model landscape shifting from OpenAI's near-monopoly to a competitive field where Google's Gemini now leads LMSys Arena and open-source models like Llama 8B score ten points higher on MMLU than Mistral 7B a year ago. They note that OpenAI's API cost has dropped 80-85% in 18 months, and total OpenAI API share fell from ~90% to ~60% as customers switch. The funding environment is rational, not a bubble, with foundation-model labs raising $30-40B but most startups seeing sane valuations; one portfolio company grew from zero to twenty million in PLG-style spending. Key startup themes include first-wave service automation (Sierra, Decagon, Harvey, EvenUp), better search and new friends (Perplexity, Glean, Character, Replica), and democratized creativity (Midjourney, HeyGen). They argue the 'GPT wrapper' narrative is false—applications capture value—and that incumbents face innovator's dilemma because AI changes business models (outcomes-based pricing) and data needs (reasoning traces rarely saved). The speed of change and new markets (legal, healthcare, defense) structurally favor…

Windsurf: The Enterprise AI IDE

Dec 13, 2024 · 1:06:36

Varun Mohan and Anshul of Codeium explain why they built Windsurf, a new AI IDE, arguing that VS Code's API limitations prevented them from delivering the best agentic experience. They detail how Cascade, their agentic system, uses proprietary retrieval and planning models alongside third-party LLMs, and describe their evaluation method that masks commits and tests incomplete code states. The duo also reveals that over 800,000 developers use Codeium extensions, that they still support JetBrains and Eclipse for enterprise customers, and that they intentionally avoided a waitlist launch. They discuss the trade-offs of building first-party vs. third-party models, the importance of 'go slow to go fast' in enterprise infrastructure, and their belief that individual developer profits should come after building switching costs through superior product.

0 to over $8M ARR in 2 months as a Claude Wrapper (Bolt.new, Qodo)

Dec 2, 2024 · 1:36:42

Eric Simons of Bolt.new/Stackblitz and Itamar Friedman of Qodo discuss Bolt.new's viral growth from $4M ARR in four weeks to over $8M ARR in two months as a Claude wrapper, driven by its in-browser WebContainer OS that eliminates local dev setup. They contrast Bolt's '0 to 1' creation for non-developers with enterprise-focused agents like Qodo's testing and code review tools, arguing that specialized agents outperform general-purpose ones. Simons details how WebContainer's unified sandbox enables error self-healing loops, while Friedman explains AlphaCodium's task-breaking that boosted o1 preview to 93rd percentile on Codeforces. The episode covers open-source strategy, pricing tiers ($20 to $200/month), and founder advice on ignoring expert doubt to ship fast, illustrated by Simons' Ironman completion while launching Bolt.

[Paper Club] DocETL: Agentic Query Rewriting + Eval for Complex Document Processing w Shreya Shankar

Nov 29, 2024 · 55:53

This episode of Latent Space Paper Club discusses DocETL, a framework by Shreya Shankar for agentic query rewriting and evaluation in complex document processing with LLMs. The paper introduces map, reduce, resolve, and auxiliary operators, along with rewrite directives like data decomposition and gleaning to improve accuracy. The optimizer generates 200 pipeline variants, costing $100, to rewrite user pipelines automatically. Key debates include when to chunk documents (sometimes harmful) and the reliability of LLM-as-judge for evaluation, with guests Eugene Yen and Vibhu arguing that binary metrics, ensembling, and context-dependent thresholds make it viable. Shreya emphasizes that validation agents are critical, and the work builds on her prior EvalGen paper, spanning two years of development.

The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic

Nov 28, 2024 · 1:11:08

Anthropic's Erik Schluntz explains how Claude 3.5 Sonnet achieved state-of-the-art 49% on SWE-Bench Verified by using a minimal agent framework that gives the model full control via tools like bash and file editing, letting it self-correct without hard-coded workflows. He details the importance of tool design over prompt engineering, why XML tags work well, and how computer use reduces integration friction by letting models interact with any browser-based interface. Schluntz also shares his robotics experience, noting that while LLMs and diffusion models are promising for general-purpose robots, reliability at 99.9% and hardware variability remain major hurdles, and he expresses skepticism about self-driving as a business due to high vehicle costs versus driver-equivalent revenue.

Agents @ Work: Lindy.ai (with live demo!)

Nov 15, 2024 · 1:08:01

Florent Crivello, founder of Lindy.ai, explains how his no-code agent platform evolved from an overly LLM-dependent design to a deterministic 'on rails' system that puts AI in a box, making agents more reliable and easier to use. The product was rebuilt from scratch as Lindy 2.0, featuring step-by-step workflows with structured outputs and memory management that still requires pruning. Crivello reveals that Claude 3.5 Sonnet ended model bottlenecks but warns against over-engineering cognitive architectures, advocating instead for letting models scale. He discusses building their own eval tools (now considering Braintrust), hiring vertical GMs for go-to-market, and the decision to return to in-person work, arguing remote work hampers creativity. He also shares his contrarian views on Europe vs. US tech, testing the Overton window with spicy tweets, and balancing AI safety concerns (10% p(doom)) with building toward a potential utopia.

In the Arena: How LMSys changed LLM Benchmarking Forever

Nov 1, 2024 · 41:02

Anastasios and Weilin from LM-Sys explain how ChatBot Arena became the de facto standard for LLM benchmarking by replacing static benchmarks with dynamic human preference evaluations. They trace the origin from Vicuna fine-tuning to the viral launch in April 2023, where anonymous side-by-side battles let the community decide which model is better. To address biases like length preference, they developed Style Control, a logistic regression method that adjusts for confounders such as response length and markdown formatting. They address the controversy around labs testing multiple private models, arguing that selection bias is empirically small and the live benchmark self-corrects over time. They also discuss RouteLLM for cost-performance routing, the graduation of ChatBot Arena from LMSys to support new projects, and call for community help with red teaming, vision modalities, and implementing a REPL for coding evaluations.

How NotebookLM Was Made

Oct 25, 2024 · 1:13:57

Raiza Martin (NotebookLM lead PM) and Usama Bin Shafqat (AI engineer) explain how Google's NotebookLM built the viral 'Deep Dive' audio overview feature. They reveal the product evolved from Project Tailwind, using Gemini 1.5's long context and DeepMind speech to create a two-persona dialogue format that transforms documents into engaging podcasts. The team learned from 65,000 Discord members, leaned on best-selling author Steven Johnson for a 'tool for thought' workflow, and prioritized a single format over exposed controls to preserve unpredictability and delight. Humor and tension are not explicitly prompted but emerge from giving personas different angles. Evaulation relied on internal taste ('potatoes for chefs') before formal raters, with a Likert scale on dimensions like entertainment and groundedness. Future plans include multilingual support, API access, real-time chat, and codebase podcasting, while managing non-determinism by accepting occasional bad rolls.

[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

Oct 19, 2024 · 1:01:40

Jesse Hu walks through SWE-Bench, SWE-Bench Verified, SWE-Bench Multimodal, and MLE-Bench — benchmarks that evaluate AI agents on real-world coding and ML tasks, from fixing GitHub issues to winning Kaggle competitions. SWE-Bench scrapes 2,000+ pull requests from 12 Python repos to get free unit tests, with leaderboard scores rising from ~2% to 43% thanks to improved agent scaffolding. SWE-Bench Verified uses human raters and three criteria—well-specified problem, valid test, difficulty on a 0–3 scale—to filter 500 solvable tasks. SWE-Bench Multimodal targets 617 UI tasks from 17 JavaScript/TypeScript libraries, each including an image, but achieves only 12% accuracy. MLE-Bench repurposes Kaggle competitions: agents given 24 hours and an A10 GPU, with o1-preview scoring bronze in 17% of competitions and surpassing grandmasters (7 gold medals vs 5 required), though each run costs $4,000.

[Paper Club] Molmo + Pixmo + Whisper 3 Turbo - with Vibhu Sapra, Nathan Lambert, Amgadoz

Oct 13, 2024 · 1:12:59

This episode covers two papers: AI2's Molmo open-source vision-language models and OpenAI's Whisper Large V3 Turbo. For Molmo, the key claim is that high-quality, audio-annotated data (Pixmo) enables models as small as 1B to match GPT-4V on academic benchmarks, with the 72B variant outperforming GPT-4o, Gemini 1.5, and Claude 3.5 Sonnet. The team avoided distilling proprietary models by having annotators describe images in speech for 60-90 seconds, then transcribing and augmenting the captions. For Whisper Turbo, OpenAI pruned the decoder from 32 to 4 layers and continued pre-training on 10 million hours of multilingual transcription data, making it 1.78x smaller than Large V3 with minimal word-error-rate increase. The model is faster and supports real-time chunk-based decoding, unlike English-only DistilWhisper. Nathan Lambert and Amgadoz provide commentary on data strategies and benchmarking nuances.

Production AI Engineering starts with Evals

Oct 11, 2024 · 1:56:16

Ankur Goyal, founder and CEO of Braintrust, argues that evaluation is the core workflow of production AI engineering and shows how his platform makes evals accessible to software engineers. Drawing from his experience at SingleStore and Impira, he details Braintrust's evolution from an eval tool into an end-to-end AI development platform used by Stripe, Zapier, Vercel, and other top AI teams. He shares market data: OpenAI handles over 95% of production workloads, fine-tuning is declining, and open-source models account for under 5% due to reliability issues. Ankur explains why he avoided building a vector database—the real challenge is permissions and joins, not vector search—and predicts o1-style reasoning will replace complex agent frameworks. He reveals Braintrust's differentiators: hybrid on-prem, TypeScript-first SDK, and declarative eval structures.

[Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)

Sep 28, 2024 · 1:00:55

Eugene Yan presents the paper 'Who Validates the Validators? Aligning LLM-Judges with Humans' by Shreya Shankar, introducing EvalGen, a framework that helps developers iteratively align LLM evaluators with human labels by grading outputs, refining criteria, and tracking coverage (recall) and false failure rate (1−precision). Yan demonstrates his own prototype 'Label', which forces users to label 20 samples before unlocking evaluation mode and 50 samples plus one evaluation run before optimization, gamifying the alignment loop. The discussion covers binary vs. pairwise evaluation, the subjectivity of criteria, and Shreya Shankar's plans to integrate natural language feedback into EvalGen's next version. Yan advocates that labeling data is essential before deploying LLM evaluators in production, turning the process into prompt fine-tuning rather than vibe checks.

[Paper Club] 🍓 On Reasoning: Q-STaR and Friends!

Sep 18, 2024 · 47:12

This episode of Paper Club surveys three reasoning papers — STaR, Quiet-STaR, and V-STaR — arguing that while STaR is foundational for bootstrapping reasoning via rationales and rationalization, Quiet-STaR's attempt to generate rationales at every token yields only marginal gains (5–10% on GSM8K and CQA), and V-STaR's verifier trained with DPO on both correct and incorrect solutions delivers the most practical improvement, beating majority voting. The host explains STaR's two-loop process of generating rationales and rationalizations from wrong answers, highlights examples like a filtering-straw question where human raters evaluated reasoning quality, and notes that STaR on GPT-J 6B achieved human-like step counts in math problems. V-STaR's verifier selects among candidate solutions, scaling with K candidates, and is compared to process reward models from OpenAI's "let's verify step by step."

Personal benchmarks vs HumanEval - with Nicholas Carlini of DeepMind

Aug 28, 2024 · 1:07:28

Nicholas Carlini, a DeepMind research scientist, argues AI models are practically useful for personal tasks despite flaws, and personal benchmarks tailored to individual use cases matter more than generic leaderboards. He shares from his 'How I Use AI' post: using LLMs for ephemeral software, kickstarting Docker, debugging by pasting error messages. He built a personal benchmark DSL that runs code from real chat history. On security, he explains buying expired domains from LAION-400M allowed poisoning any model trained on it, and he extracted OpenAI's Ada and Babbage model dimensions via API (with permission). He also recovered training data from GPT-3.5 by repeating a word until ChatGPT output verbatim sequences. He prefers attacking over defending because it's more fun and essential for discovering real vulnerabilities.

Answer.ai & AI Magic with Jeremy Howard

Aug 17, 2024 · 1:11:00

Jeremy Howard of Answer.AI argues that continuous pre-training should be treated as a continuum, not separate phases, and demonstrates how FSDP+QLoRA enables training a 70B model on just two NVIDIA 4090s. He reveals Answer’s non-hierarchical, manager-free R&D lab model that recruited unusual talent like Benjamin Warner and Ben Claviez, who independently launched BERT 24 to revive encoder-only architectures. Howard introduces FastHTML, a pure-Python web framework built on HTMX and Starlette, for creating modern SPAs without JavaScript. He previews “AI Magic,” a dialogue engineering system that moves beyond teletype-style chat interfaces and code editors, aiming to make AI-assisted development more interactive. The episode also covers Answer’s Public Benefit Corporation structure designed to resist hostile takeovers, and critiques decoder-only hype while advocating for encoder-decoder and state-space models.

The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)

Aug 2, 2024 · 1:23:36

Swyx and Alessio recap Q2 2024 through their 'Four Wars' framework, arguing that the AI landscape is shifting from frontier model dominance to commoditization and vertical applications. They highlight Claude 3.5 Sonnet overtaking OpenAI on coding benchmarks, Llama 3.1's synthetic data approach enabling 7B models to rival GPT-4, Mistral Large 2's non-commercial license and lost open-source crown, and on-device models like Gemini Nano and Apple Intelligence. The Quality Data Wars see NYT suing OpenAI, Reddit licensing data for $200M+, and synthetic data proving real for math (AlphaProof near IMO gold) and code. The Multimodality War includes ChatGPT Voice Mode delayed, Meta's Chameleon for native fusion, and Google's PaliGemma for PDF extraction. The renamed LLM OS War covers agent protocols, memory databases, and the collapse of model cost by an order of magnitude every four months, pushing startups toward vertical services like Brightwave and Dropzone that sell labor, not tools. The episode ends with a CrowdStrike joke about agent safety.

Training Llama 2, 3 & 4: The Path to Open Source AGI — with Thomas Scialom of Meta AI

Jul 23, 2024 · 1:04:34

Thomas Scialom, Llama 2 lead and Llama 3 post-training lead at Meta AI, explains why scaling laws must go beyond Chinchilla—training models longer on more tokens yields better inference efficiency than bigger models, a lesson that drove Llama 3's 15 trillion token pre-training. He details how synthetic data from Llama 2 bootstrapped Llama 3's post-training, eliminating human-written SFT data, and why RLHF outperforms imitation learning: humans are better discriminators than generators, enabling superhuman outputs. Scialom defends the dense 405B architecture over MoE, calls tokenizer vocab size underrated (128k tokens vs Llama 2's 32k), and reveals Llama 4's focus on agentic capabilities—tool use, multi-step reasoning—as the path to open-source AGI. Meta AI is hiring researchers with rigorous first-principles thinking.

How To Hire AI Engineers (ft. James Brady and Adam Wiggins of Elicit)

Jun 21, 2024 · 1:08:06

James Brady and Adam Wiggins of Elicit define the AI engineer role as a blend of conventional software engineering, deep curiosity about language models, and a defensive fault-first mindset to handle LLMs' chaotic latency and non-determinism. They explain their interview process uses coding exercises that force edge-case thinking and system design probing fault tolerance, not happy-path algorithms. For sourcing, they emphasize side projects, hackathons, and targeted communities like effective altruism. The episode explores the ML-first mindset of relinquishing control to leverage model capabilities, using techniques like retries, fallbacks, and strong typing. Elicit shares their job description template and tiered ML reading list as practical resources for hiring.

This World Does Not Exist — Joscha Bach, Karan Malhotra, Rob Haisfield (WorldSim, WebSim, Liquid AI)

Apr 27, 2024 · 1:56:11

The episode features Karan Malhotra demoing WorldSim, a prompt that turns Claude 3 into a universe simulator via CLI; Rob Haisfield presenting WebSim, which generates functional websites on the fly; and Joscha Bach arguing simulative AI reveals consciousness as a virtual property, with LLMs creating agents as real as human minds, urging the California Institute for Machine Consciousness to build self-organizing silicon life. Malhotra shows WorldSim running “world.exe” to create Twitter inside a simulated universe, with users tweeting and Elon Musk moving Dogecoin. Haisfield demonstrates WebSim generating a 5D particle interface, a news RSS aggregator, and a face-swap webcam app via URL parameters like “secrets=revealed”. Bach explains consciousness as second-order perception in a simulated now, criticizes RLHF for lobotomizing models, and advocates for animist AI where software agents compete like spirits, not golems.

High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor

Apr 24, 2024 · 1:02:59

Jason Liu, creator of the Instructor library, explains why structured outputs from LLMs are best handled by a simple requests-like wrapper rather than a VC-backed framework, arguing that Pydantic-defined schemas via function calling outperform JSON Mode for typed responses. He details his journey from being bearish on LLMs at StitchFix to building Instructor on a bullet train to Japan after GPT-3 proved him wrong. Liu advocates for workflow-based DAGs over reactive agent loops, recommends using rankers rather than cramming 60+ tools into an API call, and credits high agency—trying many experiments and documenting conditions for revisiting failures—as key to his success. He also critiques the MLE hiring hype, urging startups to empower motivated AI engineers instead.

Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit

Apr 11, 2024 · 1:05:28

Andreas Stuhlmüller and Jungwon Byun, co-founders of Elicit (formerly the nonprofit Ought), have built an AI research assistant that automates literature review and reasoning by breaking complex tasks into transparent, step-by-step processes. Their philosophy—"supervise the process, not just the outcome"—led them to start with human simulations before GPT-3 enabled a product pivot. Elicit now uses both open-source and closed models (e.g., GPT-4, Claude Haiku) for summarization, data extraction, and uncertainty flags, and recently launched computational notebooks for scalable, reusable workflows. The company transitioned from nonprofit to a Public Benefit Corporation, reached $1M revenue in four months, and now employs 12 people, focusing on senior software engineers to build reliable orchestration from unreliable components.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Jan 11, 2024 · 1:35:27

Dr. Nathan Lambert traces the origin and future of Reinforcement Learning from Human Feedback (RLHF), the secret ingredient behind ChatGPT, explaining how it evolved from robotics and early preference learning to become the core alignment technique for large language models. He details the three-phase process (instruction tuning, preference data collection, RL optimization), noting that RLHF's data costs for LLaMA2 were around $6–8 million, and that synthetic data from GPT-4 is cheaper and often more accurate than human labels. Lambert contrasts DPO with PPO, arguing DPO is simpler but may have lower peak performance. He discusses emerging methods like Constitutional AI, which uses AI-generated critiques based on principles, and highlights the challenge of evaluating RLHF models, noting GPT-4 Turbo's lead over earlier versions. The episode covers open questions about data aggregation, reward model agreement (65–75%), and the need for qualitative model interaction.

The "Normsky" architecture for AI coding agents — with Beyang Liu + Steve Yegge of SourceGraph

Dec 17, 2023 · 1:33:26

Beyang Liu and Steve Yegge of Sourcegraph explain how their AI coding assistant Cody achieves a 30% completion acceptance rate by prioritizing high-quality context over agents. They introduce the 'Normsky' architecture—a blend of Norvig's data-driven models and Chomsky's formal systems—arguing that non-agentic, deterministic context retrieval (using parsers, trigram indexes, and the BFG code graph) outperforms multi-hop LLM-based approaches for code completion and chat. The duo details their data pre-processing moat, the 'bin packing' challenge of stuffing relevant code into context windows, and why they use open-source StarCoder for completions while leveraging Claude/GPT-4 for chat. They also discuss the death of DSLs, the limitations of LSP and LSIF, and how Cody's web version lets users query any public repo instantly.