Coding Agents

Exo: Harnesses should see their own code and logs — Alex Krentsel, UC Berekeley / Google Research

Aug 15, 2026 · 47:11

Alex Krentsel presents Exo, a fully recursive self-improving agent harness he built with Martín Casado and Anker Goyal, arguing agents can now safely edit their own code and runtime policy. Exo's architecture splits an agent into a stateless executor, a stateful harness with history and secrets, and an isolated sandbox, letting it change every component, roll back automatically, and teleport state—unlike OpenClaw/Pi, which only allow memory, skills, and tool extensions. The episode details Exo cutting its Discord adapter costs by 96% at runtime, plus secret stores and evals to prevent reward hacking. Krentsel argues RSI is newly possible because the harness is code in the same medium as LLM output; Exo runs in production at Brain Trust, with contributors directed to github.com/exoharness/exo.

OpenAI’s Vision for the AI Super App — Akshay Nathan, OpenAI

Jul 28, 2026 · 1:10:57

Akshay Nathan, OpenAI's head of core product engineering, explains why the company merged Codex and ChatGPT into a unified agent harness, arguing that the lines between software engineering and knowledge work are blurring. He reveals that Codex unexpectedly gained traction among non-developers at OpenAI, leading to the launch of ChatGPT Work, which shares the same underlying harness but offers opinionated UX differences like sandboxing and Git visibility. Nathan details how artifacts, Sites, plugins, memory, and sub-agents enable users to replace slide decks and spreadsheets with interactive websites, and shares that ChatGPT Work reached 10 million users shortly after launch. He discusses OpenAI's goal to bring useful agents from developers to all knowledge workers, emphasizing that the default model configuration should work for most users while power users can opt into deeper reasoning, Ultra, or multi-agent setups. Nathan also reflects on how AI transforms product development, noting that ideas and taste become the bottleneck when anyone can build, and warns managers not to conflate motion with progress.

AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan

Jun 22, 2026 · 1:07:31

Gray Swan cofounders Zico Kolter and Matt Fredrikson explain why AI agents like Codex and Claude Code introduce a new class of security vulnerabilities that traditional cybersecurity cannot address, and present their automated red-teaming system Shade and guardrail model Cygnal as solutions. Prompt injection creates exploits for agents operating on untrusted data, and Shade now outperforms human red teamers at breaking models. The lethal trifecta—untrusted data, private data, and exfiltration—defines the highest risks. In their Human Browser Agent Robustness Challenge, humans ranked fourth among models, with some frontier agents falling for attacks no person would. Bigger models do not automatically become safer, and specialized systems like Cygnal enforce enterprise policies more reliably than prompt engineering. They see AI security evolving toward insurance and compliance, with the first major prompt-injection breach a gray swan—unlikely but visible ahead of time.

Devin’s 80% Moment: Background Agents, 7x PRs, & End of Hand-Held Coding — Walden Yan & Cole Murray

May 28, 2026 · 1:09:33

Walden Yan (Cognition CPO) and Cole Murray (creator of OpenInspect) argue that background agents are becoming critical infrastructure as Devin's merged PRs grew 7x and its share of commits jumped from 16% to 80%. They explain why architecture matters, particularly the decision to separate the agent's brain from the machine (harness out of the box) for security and permissions, and why full VMs beat Docker for running real applications. Testing, they stress, is a harder problem than computer use—requiring orchestration of services, feature flags, and multi-model coordination. Memory remains unsolved, with Devin using auto-generated 'Knowledge' and exploring file-system-like approaches. Both caution against uncontrolled vibe coding, which regresses codebases to the worst engineer's style, and advocate for hybrid frontier/sub-frontier systems to balance cost and capability.

Senior Dev: This "Grill Me" Prompt Is Going Viral Among Top Engineers

May 7, 2026 · 22:03

Matt Pocock — the TypeScript educator turned AI engineering teacher who created the viral "Grill Me" prompt and runs AI Hero — argues that classic software engineering fundamentals are more critical than ever in the AI coding era. He explains how Domain-Driven Design's ubiquitous language and deep module patterns let developers contain complexity and delegate implementation to AI. His free "Grill Me" skill (part of his 13,000-star skills repo) forces AI to relentlessly interview humans until reaching shared understanding. He contrasts Claude Code's ease-of-use against Pie's transparent primitives, predicting a shift toward developer observability. The episode also covers his two-month course creation process, why TypeScript is overtaking Python in AI engineering, and why traditional lectures still dominate learning.

AI-Native Engineering: 100% adoption, 5x search throughput, unlimited tokens — Mikhail Parakhin

Apr 22, 2026 · 1:14:30

Shopify CTO Mikhail Parakhin details how the company achieved near 100% AI tool adoption, driven by a December 2025 inflection point where model quality triggered exponential token consumption. He argues token budgets matter only with strong critique loops: running multiple parallel agents without communication wastes tokens, while agentic PR review using large models—like GPT-5.4 Pro or Gemini Deep Think—reduces bugs even though latency increases. Parakhin unveils Tangle, Shopify's third-generation ML workflow system with content-based caching that eliminates duplicate computation across teams, and Tangent, an auto-research loop that optimizes pipelines—boosting search throughput from 800 to 4,200 QPS on the same hardware. SimGym simulates shoppers using decades of merchant data and browser-based agents to predict conversion changes, achieving 0.7 correlation with add-to-cart events. Shopify uses Liquid AI (non-transformer architecture) for sub-30ms query understanding and long-context tasks like catalog categorization, distilling larger models into Liquid for high-throughput batch jobs.

⚡️ The best engineers don't write the most code. They delete the most code. — Stay Sassy

Apr 13, 2026 · 57:30

Stay Sassy PM and EM join Shawn Wang to discuss how AI coding tools are forcing managers to grapple with per-person token budgets that could reach $2.5M annually as consumption-based pricing replaces subsidies, making code review more critical, not less. They argue that managing these budgets is a new bottleneck, requiring companies to decide how much to spend on individual employees—a scale unprecedented outside department-level budgeting. On build vs buy, they caution that many products are more complex than they appear, and that the old frameworks of feature analysis, administration burden, and vendor lock-in still apply. They highlight code review fatigue from running multiple agents as a risk, citing Amazon's six-hour downtime from AI-generated code, and urge teams to maintain a culture where no single person can take down prod. Finally, they suggest automating executives' standard decisions first—since much leadership work is routine—rather than targeting only junior tasks, and predict that the hardest problems in 2026 will remain human.

Extreme Harness Engineering: 1M LOC, 1B toks/day, 0% human code or review — Ryan Lopopolo, OpenAI

Apr 7, 2026 · 1:17:54

Ryan Lopopolo of OpenAI's Frontier team details his team's extreme experiment in harness engineering: building a 1M LOC internal Electron app with zero human-written or reviewed code, relying instead on Codex agents that process 1B tokens daily ($2-3k/day). He argues that humans are the bottleneck and that teams should encode non-functional requirements into specs, skills, and observability tooling rather than prompting agents to 'try harder.' The result is Symphony, a ghost library spec and Elixir reference implementation that automates the entire pull-request lifecycle, including self-review and merge. Lopopolo explains how the team scales by treating software as agent-legible text, using worktrees for multi-agent collaboration, and feeding agent mistakes back into the repository via docs like core-beliefs.md. He also discusses Frontier's enterprise platform for safe agent deployment, noting that success depends on giving agents full context—even company culture and inside jokes—so they can act as full teammates.

Marc Andreessen introspects on Death of the Browser, Pi + OpenClaw, and Why "This Time Is Different"

Apr 3, 2026 · 1:16:20

Marc Andreessen argues that AI is finally different from past boom-bust cycles because of four compounding breakthroughs: LLMs, reasoning (o1, R1), coding agents (OpenClaw), and recursive self-improvement. He calls this the '80-year overnight success'—decades of neural network research now paying off. Comparing today's AI capex boom to the dot-com crash, he notes that buyers like Microsoft and Google are cash-rich incumbents and every GPU deployed is already generating revenue. He hails Pi and OpenClaw as a Unix-like architecture that makes agents model-independent and self-modifiable. On open source, he calls DeepSeek a 'gift to the world' for its paper and code, but warns that entrenched institutions—unions, licensing, government monopolies—will slow AI adoption far more than technologists expect.

Claude Code for Finance + The Global Memory Shortage: Doug O'Laughlin, SemiAnalysis

Feb 24, 2026 · 2:07:40

Doug O'Laughlin, SemiAnalysis founder, explains his Claude Code awakening in December 2024: Anthropic's Opus 4.5 one-shotted tasks that used to take 24 hours, leading him to predict Claude Code will write 25–50% of all GitHub code by year-end (already 4% of commits in two weeks). He argues this marks the death of Excel and Bloomberg for analysts—'you can just do things'—but warns of a 'hygiene' crisis as junior analysts lose meta-learning. On the semiconductor side, O'Laughlin details a severe memory squeeze: HBM demand consumes 4× the DRAM capacity, DRAM prices could double again, and CXL is reviving as a workaround. He compares AI infrastructure buildout to the railroad boom (25% of U.S. gross fixed capital investment) and says Microsoft faces an innovator's dilemma—renting GPUs to 'barbarians at the gate' while its own office franchise is disrupted by Claude Code. Google's TPU v7 enjoys a temporary TCO advantage, but Nvidia's supply chain dominance (Jensen doing shots with SK and Samsung) will reassert with Rubin. O'Laughlin also reflects on his 2,800-mile Continental Divide Trail hike as essential self-mastery.

[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang

Dec 31, 2025 · 17:45

John Yang, creator of SWE-bench, explains the benchmark's evolution from an ignored project to the industry standard after Cognition's Devin launch, and introduces CodeClash as a new paradigm for evaluating long-horizon coding agents via programming tournaments. Yang covers SWE-bench's expansion into nine languages (JavaScript, Rust, Java, C, Ruby) and multimodal tasks, the proliferation of independent variants like SWE-bench Pro and SWE-bench Live, and the shift from unit tests to iterative agent competitions. He addresses the Tau-bench controversy over impossible tasks, arguing they serve as cheat detection, and weighs the tension between long autonomy (5-hour runs) and interactivity (Cognition's fast back-and-forth). Yang highlights the academic data problem—companies like Cognition have rich user interaction data, while academics need user simulators or compelling products like LMArena—and positions CodeClash as a testbed for studying human-AI collaboration by freezing model capability and varying interaction setups.

The Great Evals Debate — Ankur Goyal & Malte Ubl

Dec 7, 2025 · 34:33

Ankur Goyal (Braintrust) and Malte Ubl (Vercel) debate whether offline evals are essential infrastructure or premature optimization for AI coding agents, arguing that the best teams deliberately invest in multiple feedback loops—offline evals, A/B tests, and vibe checks—to build effective AI products. They explain that modern evals are not about manufacturing golden datasets but pulling real user failures from production logs into eval suites to iterate faster. The conversation highlights how evals provide a 'first derivative' that enables aggressive shipping without regression fears, akin to unit tests in traditional software. Coding evals are uniquely verifiable (e.g., 'does it compile?') yet underutilized; Vercel uses them in RL pipelines to fine-tune models that fix trivial errors 100x faster than agentic loops. They also discuss how product managers encode domain expertise through rubrics and LLM-as-judge scoring, and why proprietary evals are competitive moats while public benchmarks serve marketing. The debate concludes that RL environments are a promising frontier for computer-use agents but require specialized expertise to avoid reward hacking.

⚡️ 10x AI Engineers with $1m Salaries — Alex Lieberman & Arman Hezarkhani, Tenex

Nov 19, 2025 · 27:11

Tenex co-founders Alex Lieberman and Arman Hezarkhani explain how they compensate AI engineers by story points instead of hours, enabling some to earn over $1 million annually while delivering 10x productivity gains. The model emerged after Arman downsized his previous company Parthian's engineering team by 90% yet saw a 10x increase in production-ready software output by re-architecting with AI. They avoid gaming by hiring engineers who are long-term selfish or love coding, and pair them with technical strategists incentivized on account retention. Example projects include building a retail camera prototype in two weeks and a trivia app that hit #20 on the App Store in a month. Their default stack is TypeScript with React and Express, and they switch between coding agents like Claude Code and Codex based on daily performance. They identify human capital as their main constraint and context engineering or entropy reduction as the core challenge for autonomous AI engineers.

Anthropic, Glean & OpenRouter: How AI Moats Are Built with Deedy Das of Menlo Ventures

Nov 14, 2025 · 1:26:59

Deedy Das of Menlo Ventures returns to explain how Anthropic became the fastest-growing software company ever (zero to billions in revenue) and why Glean’s boring enterprise search moat of hard integrations and ranking problems is harder than competitors think. He reveals the $100M Anthology Fund’s strategy: backing OpenRouter, Goodfire, Prime Intellect, and Whisper—companies that solve thorny infrastructure or research problems rather than chasing apps. Das argues that model-layer companies will capture most value because building great models is harder than building apps, and that Anthropic’s product innovations like Claude Code emerge from a culture that lets researchers experiment freely. He also warns that vibe coding is becoming a cognitive crutch for engineers, eroding deep problem-solving skills, and discusses how enterprise AI market share has shifted dramatically: OpenAI went from 50% to 20%, while Anthropic rose from 12% to 32% of enterprise LLM API spend.

⚡ Inside Google Labs: Building The Gemini Coding Agent — Jed Borovik, Jules + AIE CODE Preview

Nov 10, 2025 · 43:53

Jed Borovik, Product Lead at Google Labs, explains how Google builds Jules, an autonomous coding agent that runs on its own VM for long-running tasks, challenging the assumption that agents should operate locally. He reveals that as Gemini models improved, Jules' scaffolding simplified, shifting from sub-agent patterns and embedding-based RAG to attention-based search. Borovik discusses context window management for sessions lasting up to 30 days with 2 million tokens, and argues that coding agents will increase demand for software engineers (Jevons paradox), not eliminate jobs. He calls for better specification tools beyond chat, such as multimodal input and interactive planning, to move beyond 'vibe coding' toward verifiable, reliable agentic workflows.

Terminal-Bench 2.0: the most impt coding agent benchmark of 2025 gets a v2! Launch + Q&A w/ founders

Nov 8, 2025 · 35:01

Terminal-Bench co-creators Mike Merrill, Alex Shaw, and collaborators Andy Konwinski, Ludwig Schmidt, and Aatash launch Terminal-Bench 2.0, an 89-task coding benchmark with rigorous human and LM-assisted verification to test agent performance on real-world tasks like designing DNA primers and installing Windows 3.11. They reflect on Terminal-Bench 1's problems—easy tasks and reproducibility—and detail spending 300 hours manually auditing each task. They introduce Harbor, an open-source package for running sandboxed agent rollouts, standardizing task format and integrating with agents like Codex and Claude Code. Aatash shows using Harbor to generate thousands of training traces for DataComp Agent, running 1,000 parallel sandboxes. Future plans include expanding to scientific computing.

Terminal-Bench: Pushing Claude Code, OpenAI Codex, Factory Droid, et al to the limits

Oct 18, 2025 · 35:27

Alex Shaw and Mike Merrill, creators of Terminal-Bench, explain how their coding agent benchmark became an industry standard adopted by Anthropic, OpenAI, and leading agent companies. They bet on terminal-based interaction over GUI because text is the modality that works best with models, and designed tasks as containerized environments with instructions and test scripts. Nicholas Carlini at Anthropic became an early champion, leading to Terminal-Bench being featured on Claude’s model card without notice. They built the minimal Terminus agent to isolate model capabilities from agent optimizations, and their framework adapts existing benchmarks like SWE-bench. Future plans include cloud hosting, multi-dimensional evaluation incorporating cost and economic value, and enabling RL post-training on benchmark tasks.

Amp: The Emperor Has No Clothes

Sep 25, 2025 · 1:21:59

Quinn Slack (CEO) and Thorsten Ball (Amp Dictator) from Sourcegraph discuss Amp Code, a coding agent that ships 15x/day with no code reviews. They explain the strategic pivot from Cody to Amp, emphasizing that building the best coding agent trumps everything else. The team dogfoods Amp internally, using it to build itself, and maintains both a VS Code extension and CLI, with a 50/50 internal split. They argue that model choice is becoming less important than the harness around the model, and they reject features like prompt enhancers, custom subagents, and MCP servers as not adding value. Common failure modes include outsourcing thinking but not typing and using agents without proper context engineering. They predict async agents running 24/7 will dominate, and they target early adopters willing to adapt their codebases and workflows for maximal agent effectiveness.

🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R)

Jul 28, 2025 · 3:53:23

This episode traces the oral history of Windsurf (formerly Codeium and Exafunction) with founders Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, and Anshul R, charting its evolution from a GPU virtualization startup to a leading agentic IDE and its dramatic acquisition by Cognition. The narrative covers the pivot to code completion in 2022, the shift to a free model to gain traction, and the launch of the Windsurf editor in November 2024 with Cascade, an agent that operates on a shared timeline with the developer. Key principles include meta-learning (auto-generated memories that adapt to each user's preferences) and scaling with intelligence (replacing hard-coded rules with LLM reasoning as models improve). The episode also details the acquisition weekend when Cognition bought Windsurf after Google acquired a stake, highlighting product synergies between the synchronous agent (Windsurf) and async agent (Devin) and the ambition to build a comprehensive platform for real-world software engineering teams.

Cline: The Collaborative AI Coder

Jul 16, 2025 · 1:15:44

Saoud Rizwan and Pash from Cline join to explain why their open-source coding agent uses a Plan + Act paradigm — having models explore and plan before executing — pioneered before competitors like Claude Code. They argue that "RAG is a mind virus" and "Fast Apply got bitter lesson'd" as frontier models now achieve sub-5% diff edit failure rates, making layered workarounds obsolete. Cline's bring-your-own-API-key model avoids inference markups, with enterprise features driven by Fortune 5 companies asking for security and ROI insights. The MCP ecosystem grew via Cline's marketplace (150+ servers, top ones with hundreds of thousands of downloads), though monetization and security remain open questions. They detail how Plan mode extracts user intent and Act mode auto-approves, how context engineering prioritizes agentic file exploration over retrieval, and why they stayed a VS Code extension rather than fork — to focus solely on the agentic loop.

⚡️Anthropic vs Cognition on Multi-Agents: A Breakdown with Dylan Davis

Jul 5, 2025 · 27:11

Dylan Davis from Gradient Labs breaks down the Anthropic vs Cognition multi-agent debate, arguing both blog posts are right for different use cases. Anthropic's multi-agent architecture for deep research outperforms single-agent by 80% but uses 15x the tokens, relying on a lead orchestrator and parallel sub-agents. Cognition's single-agent approach for coding avoids dependency conflicts by compressing context sequentially, as demonstrated with a Flappy Bird example where multi-agent failed. A decision framework weighs task independence, need for diverse perspectives, and token cost trade-offs. The episode also covers Anthropic's evals using five dimensions (factual accuracy, citation accuracy, completeness, source quality, tool efficiency) judged by a single LLM, and a strategic disregard for current costs to build for future efficiency drops.

⚡️Warp 2.0: the Agentic Development Environment - Zach Lloyd and Ben Holmes

Jun 25, 2025 · 52:23

Warp CEO Zach Lloyd and engineer Ben Holmes debut Warp 2.0, rebranding from an AI terminal to an Agentic Development Environment (ADE) with a state-of-the-art coding agent that rivals Claude Code and Cursor. The episode demonstrates Warp's new universal input that auto-detects English commands and terminal input, voice-driven agent workflows, multi-threaded agents running in tabs, inline diff editing, MCP support, and REPL translations for SQL and Postgres. Lloyd explains the design philosophy of keeping a terminal-like UX while adding agent management features, and discusses pricing changes (Pro now 2,500 requests, Turbo 10,000, plus usage-based credits) and the challenge of subscription vs. usage-based models. Warp has nearly 600,000 active users with weekly revenue growth of 5–15%, and the team positions itself as a model-agnostic layer above Anthropic, Google, and OpenAI, offering deeper integration than ChatGPT or Claude desktop apps.

ChatGPT Codex: The Missing Manual

May 16, 2025 · 53:32

Josh Ma and Alexander Embiricos from OpenAI introduce ChatGPT Codex, a cloud-hosted autonomous software engineer that runs in its own sandbox, arguing it delivers value through single-shot, long-horizon task completion. They explain that Codex evolved from internal experiments (codenamed WHAM) and now offers a 60-tasks-per-hour rate limit with a one-hour hard cutoff, emphasizing it is not meant for interactive use but for delegation—users fire off tasks and go about their day. Best practices include using agents.md, installing linters and formatters, keeping code modular, and having good architecture. The team intentionally kept it a research preview to gather feedback on environment customization and pricing, with plans to integrate seamless transitions between cloud and CLI.

Claude Code: Anthropic's CLI Agent

May 7, 2025 · 1:16:22

In this episode, Boris Cherny and Cat Wu from Anthropic discuss Claude Code, a CLI-based coding agent they describe as a Unix utility rather than a polished product, emphasizing raw model access and simplicity. The tool started as Boris's personal experiment, then spread internally after engineers used it daily, with Anthropic's product principle of 'do the simple thing first' guiding features like memory via markdown files and context compaction by asking Claude to summarize. They ship updates including web fetch, auto-accept, and thinking via chain of thought, and note that 80% of Claude Code's code is written by Claude Code itself. Average spend is $6/day per user versus $20/month for Cursor, and Boris reports a personal 2x productivity gain, with some engineers seeing 10x. The team is growing and investigating open-sourcing the tool, while continuing to build for the terminal as a raw, composable utility for power users.

The #1 SWE-Bench Verified Agent

Apr 2, 2025 · 31:46

Guy Gur-Ari, co-founder of Augment Code, explains how their coding agent achieved the #1 spot on SWE-Bench Verified without fine-tuning, using off-the-shelf models with techniques like sequential thinking, ensembling, and code base understanding. He shares that orientation and memories help the agent learn codebase conventions, and that MCP support allows extensibility. Gur-Ari contrasts Augment's IDE-first approach with competitors like Devin and Factory, emphasizing meeting developers where they are. He also discusses the future of IDEs, the role of custom models, and recommends the SWE-RL paper for reinforcement learning in coding. The episode features a live demo of the agent implementing a feature from a Linear ticket and creating a PR.

Solve coding, solve AGI [Reflection.ai launch w/ CEO Misha Laskin]

Mar 7, 2025 · 28:12

Misha Laskin of Reflection AI argues that solving autonomous coding is the direct path to AGI, combining reinforcement learning (pioneered by his team on AlphaGo/AlphaZero) with large language models (which they advanced at Google on PaLM, Gemini, ChatGPT). He explains that coding is already ergonomic for LLMs—unlike browser agents that require noisy human mouse data—making it the ideal starting point. Reflection AI is building coding agents that automate backlog tasks (testing, refactoring, migrations, security remediation) for large engineering teams, delivering them via an API that takes a task and codebase and outputs resolved code. Laskin insists superintelligence cannot be built in a vacuum; real-world customer evals are essential, as benchmarks like SWE-bench don't guarantee production reliability. He also stresses the need for open-weight models to prevent a few companies from hoarding superintelligent coding agents.

0 to over $8M ARR in 2 months as a Claude Wrapper (Bolt.new, Qodo)

Dec 2, 2024 · 1:36:42

Eric Simons of Bolt.new/Stackblitz and Itamar Friedman of Qodo discuss Bolt.new's viral growth from $4M ARR in four weeks to over $8M ARR in two months as a Claude wrapper, driven by its in-browser WebContainer OS that eliminates local dev setup. They contrast Bolt's '0 to 1' creation for non-developers with enterprise-focused agents like Qodo's testing and code review tools, arguing that specialized agents outperform general-purpose ones. Simons details how WebContainer's unified sandbox enables error self-healing loops, while Friedman explains AlphaCodium's task-breaking that boosted o1 preview to 93rd percentile on Codeforces. The episode covers open-source strategy, pricing tiers ($20 to $200/month), and founder advice on ignoring expert doubt to ship fast, illustrated by Simons' Ironman completion while launching Bolt.

The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic

Nov 28, 2024 · 1:11:08

Anthropic's Erik Schluntz explains how Claude 3.5 Sonnet achieved state-of-the-art 49% on SWE-Bench Verified by using a minimal agent framework that gives the model full control via tools like bash and file editing, letting it self-correct without hard-coded workflows. He details the importance of tool design over prompt engineering, why XML tags work well, and how computer use reduces integration friction by letting models interact with any browser-based interface. Schluntz also shares his robotics experience, noting that while LLMs and diffusion models are promising for general-purpose robots, reliability at 99.9% and hardware variability remain major hurdles, and he expresses skepticism about self-driving as a business due to high vehicle costs versus driver-equivalent revenue.

[Paper Club] SWE-Bench [OpenAI Verified/Multimodal] + MLE-Bench with Jesse Hu

Oct 19, 2024 · 1:01:40

Jesse Hu walks through SWE-Bench, SWE-Bench Verified, SWE-Bench Multimodal, and MLE-Bench — benchmarks that evaluate AI agents on real-world coding and ML tasks, from fixing GitHub issues to winning Kaggle competitions. SWE-Bench scrapes 2,000+ pull requests from 12 Python repos to get free unit tests, with leaderboard scores rising from ~2% to 43% thanks to improved agent scaffolding. SWE-Bench Verified uses human raters and three criteria—well-specified problem, valid test, difficulty on a 0–3 scale—to filter 500 solvable tasks. SWE-Bench Multimodal targets 617 UI tasks from 17 JavaScript/TypeScript libraries, each including an image, but achieves only 12% accuracy. MLE-Bench repurposes Kaggle competitions: agents given 24 hours and an A10 GPU, with o1-preview scoring bronze in 17% of competitions and surpassing grandmasters (7 gold medals vs 5 required), though each run costs $4,000.

Is finetuning GPT4o worth it?

Aug 22, 2024 · 1:01:16

Ali Pullen, founder of Cosine, discusses Genie, an AI software engineering colleague that achieved 30% on SWE-Bench and 43.8% on SWE-Bench Verified via fine-tuning GPT-4o on billions of tokens of synthetic data. Cosine built Genie by training on the process of software engineering—not just working code—using synthetic runtime errors and a self-improvement loop. The agent follows a four-stage workflow: code retrieval (66% accuracy), planning, writing diffs, and running CI tests. Pullen reveals that model performance degrades linearly beyond 60K tokens in the context window, and that OpenAI dynamically sizes LoRA adapters to handle Cosine's massive dataset. He explains why Genie isn't on the SWE-Bench leaderboard (refusing to publish trajectories to prevent distillation) and outlines plans to scale the dataset and fine-tune on customer codebases for personalized performance.

The "Normsky" architecture for AI coding agents — with Beyang Liu + Steve Yegge of SourceGraph

Dec 17, 2023 · 1:33:26

Beyang Liu and Steve Yegge of Sourcegraph explain how their AI coding assistant Cody achieves a 30% completion acceptance rate by prioritizing high-quality context over agents. They introduce the 'Normsky' architecture—a blend of Norvig's data-driven models and Chomsky's formal systems—arguing that non-agentic, deterministic context retrieval (using parsers, trigram indexes, and the BFG code graph) outperforms multi-hop LLM-based approaches for code completion and chat. The duo details their data pre-processing moat, the 'bin packing' challenge of stuffing relevant code into context windows, and why they use open-source StarCoder for completions while leveraging Claude/GPT-4 for chat. They also discuss the death of DSLs, the limitations of LSP and LSIF, and how Cody's web version lets users query any public repo instantly.

Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Nov 3, 2023 · 1:18:54

Michael Royzen, co-founder and CEO of Phind, explains how his company built a GPT-4-beating open-source model for developer Q&A. Royzen recounts founding SmartLens in high school, shifting to NLP after a Hugging Face demo, and creating an internet-scale LLM-powered RAG system in January 2022. He details Phind’s pivot to programmers, the Hacker News launch that gave it 1,500 points, and Paul Graham’s role in naming the company and introducing Ron Conway, who then connected Phind to NVIDIA for GPU access. Royzen argues that Phind’s model, fine-tuned from Code Llama 34B with extra data, closes the gap with proprietary models—especially on code reasoning—and that open-source will win the enterprise because the delta to GPT-5 will be small. He shares how Phind handles multi-step conversations via a pair programmer mode where users can pin messages, and reveals plans for reinforcement learning to reduce hallucination and improve correctness.