All episodes

How To Hire AI Engineers (ft. James Brady and Adam Wiggins of Elicit)

Jun 21, 2024 · 1:08:06

James Brady and Adam Wiggins of Elicit define the AI engineer role as a blend of conventional software engineering, deep curiosity about language models, and a defensive fault-first mindset to handle LLMs' chaotic latency and non-determinism. They explain their interview process uses coding exercises that force edge-case thinking and system design probing fault tolerance, not happy-path algorithms. For sourcing, they emphasize side projects, hackathons, and targeted communities like effective altruism. The episode explores the ML-first mindset of relinquishing control to leverage model capabilities, using techniques like retries, fallbacks, and strong typing. Elicit shares their job description template and tiered ML reading list as practical resources for hiring.

How AI is Eating Finance - with Mike Conover of Brightwave

Jun 11, 2024 · 1:05:06

Brightwave founder Mike Conover explains how his vertical AI startup builds a 'partner in thought' for finance professionals, using a systems-of-systems approach that decomposes problems into specialized subsystems rather than relying on large context windows or monolithic agents. He argues that the evolution from DALL-E's 1,024-token context to today's million-token models has not solved synthesis; smaller, focused reasoning units still outperform. Conover emphasizes that the real competitive edge lies in custom training data and human annotation, not pre-training, and that models are converging in capability—so value shifts to fine-tuning data that elicits specific behaviors. He details why Brightwave avoids spreadsheets in favor of conversational analysis, and predicts AI will automate idea generation and second-order derivative bets, but humans must remain the final synthesizers and deciders. The episode also covers hiring for vertical AI, the role of knowledge graphs, and the diminishing economic incentive for companies to train their own foundation models.

How to train a Million Context LLM — with Mark Huang of Gradient.ai

May 31, 2024 · 1:12:14

Mark Huang of Gradient.ai explains how his team extended Llama 3 to a 1 million token context window using curriculum learning, RingAttention, and EasyContext, achieving near-perfect GPU utilization. They employed theta scaling from the RoPE paper to interpolate positional encodings, trained on carefully curated datasets including SlimPajamas and synthetic data from GPT-4, and validated with benchmarks like Ruler and needle-in-a-haystack. Huang details the trade-offs between full fine-tuning and LoRA adapters, the challenges of pushing to 4M tokens (degradation from floating-point precision limits), and why long context matters for state management across sessions and grounding multimodal inputs. He calls for community collaboration on long-context evaluations and pairwise multimodal datasets.

LLM Asia Paper Club Survey Round

May 22, 2024 · 55:25

The episode surveys four recent papers on LLM reasoning, uncertainty, interpretability, and efficiency. 'Let's Think Dot by Dot' shows that filler tokens (dots) inserted between input and output enable hidden computation, outperforming no-token baselines on tasks like 3-Sum and 2-Sum Transform. 'Uncertainty Estimation' trains a random forest on hidden-layer activations to predict response confidence, achieving higher AUC than unsupervised methods on Q&A and translation. 'Monosemanticity' uses sparse autoencoders to identify interpretable features (e.g., a DNA-detection feature) in a toy transformer, advancing mechanistic interpretability. 'Medusa' attaches multiple prediction heads to the final hidden state to speculate future tokens, enabling faster decoding without a separate draft model and training in five hours on 60K samples.

This World Does Not Exist — Joscha Bach, Karan Malhotra, Rob Haisfield (WorldSim, WebSim, Liquid AI)

Apr 27, 2024 · 1:56:11

The episode features Karan Malhotra demoing WorldSim, a prompt that turns Claude 3 into a universe simulator via CLI; Rob Haisfield presenting WebSim, which generates functional websites on the fly; and Joscha Bach arguing simulative AI reveals consciousness as a virtual property, with LLMs creating agents as real as human minds, urging the California Institute for Machine Consciousness to build self-organizing silicon life. Malhotra shows WorldSim running “world.exe” to create Twitter inside a simulated universe, with users tweeting and Elon Musk moving Dogecoin. Haisfield demonstrates WebSim generating a 5D particle interface, a news RSS aggregator, and a face-swap webcam app via URL parameters like “secrets=revealed”. Bach explains consciousness as second-order perception in a simulated now, criticizes RLHF for lobotomizing models, and advocates for animist AI where software agents compete like spirits, not golems.

High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor

Apr 24, 2024 · 1:02:59

Jason Liu, creator of the Instructor library, explains why structured outputs from LLMs are best handled by a simple requests-like wrapper rather than a VC-backed framework, arguing that Pydantic-defined schemas via function calling outperform JSON Mode for typed responses. He details his journey from being bearish on LLMs at StitchFix to building Instructor on a bullet train to Japan after GPT-3 proved him wrong. Liu advocates for workflow-based DAGs over reactive agent loops, recommends using rankers rather than cramming 60+ tools into an API call, and credits high agency—trying many experiments and documenting conditions for revisiting failures—as key to his success. He also critiques the MLE hiring hype, urging startups to empower motivated AI engineers instead.

Breaking down the OG GPT Paper by Alec Radford

Apr 23, 2024 · 1:05:03

This episode of the Latent Space Paper Club breaks down Alec Radford's seminal 2018 GPT-1 paper, which introduced generative pre-training of a Transformer decoder on BookCorpus (5 GB, 7,000 books) followed by supervised fine-tuning. Amget explains how the 117M-parameter model achieved state-of-the-art on 9 of 12 NLU tasks, with absolute gains of 8.9% on commonsense reasoning (Stories Cloze) and 5.7% on question answering (RACE). Key innovations include task-agnostic input transformations (e.g., concatenating premise and hypothesis with a delimiter for entailment, using a Siamese architecture for semantic similarity, and scoring answer choices for QA) and an auxiliary language modeling objective during fine-tuning (lambda=0.5). The episode also covers zero-shot heuristics (appending 'very' for sentiment, averaging token log probabilities for QA), ablation studies showing pre-training adds 15% average improvement, and the paper's prescient call to scale up—which became the blueprint for GPT-2, GPT-3, and beyond.

Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit

Apr 11, 2024 · 1:05:28

Andreas Stuhlmüller and Jungwon Byun, co-founders of Elicit (formerly the nonprofit Ought), have built an AI research assistant that automates literature review and reasoning by breaking complex tasks into transparent, step-by-step processes. Their philosophy—"supervise the process, not just the outcome"—led them to start with human simulations before GPT-3 enabled a product pivot. Elicit now uses both open-source and closed models (e.g., GPT-4, Claude Haiku) for summarization, data extraction, and uncertainty flags, and recently launched computational notebooks for scalable, reusable workflows. The company transitioned from nonprofit to a Public Benefit Corporation, reached $1M revenue in four months, and now employs 12 people, focusing on senior software engineers to build reliable orchestration from unreliable components.

Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI

Apr 6, 2024 · 58:54

This episode features Damien Murphy of Deepgram, Ethan of Owl/Bee, and Harrison of LangChain demonstrating how to build personal AI with real-time voice bots, wearable life-recording devices, and memory-enhanced journaling apps. Damien shows building a voice bot with subsecond latency using Deepgram, OpenAI, and open source code, costing about 6.5 cents per five-minute call. Ethan presents his Owl wearable that continuously records audio, triggers actions via hot word 'Scarlett,' and discusses challenges in adding vision and open source adoption. Harrison introduces LangFriend, a journaling app using conversational, semantic, and knowledge graph memory, referencing the Generative Agents paper for recency and importance weighting. The episode also highlights open source projects Whomane, Friend, and ADeus, arguing that hardware, voice, and memory are all necessary components for personal AI.

Why Google failed to make GPT-3 -- with David Luan of Adept

Mar 27, 2024 · 49:27

David Luan, co-founder of Adept and former early OpenAI leader, explains why Google failed to build GPT-3—its 'brain credit marketplace' prevented critical mass—and why Adept builds enterprise AI agents prioritizing reliability over generality. He recounts the GPT-2 demo that helped secure Microsoft's $1B investment and details Adept's goal: an AI teammate that can do anything a human does on a computer, targeting 'nines of reliability' for workflows like dispatching a physical truck. Luan contrasts Adept's vertical integration—training fast multimodal models (Fuyu) for charts and UIs—with pure-play foundation model companies that sell tokens, predicting commoditization. He notes Adept is sold out for Q1 and raised $420M, and explains how an augmentation focus creates a data flywheel from human oversight.

A Comprehensive Overview of Large Language Models - Latent Space Paper Club

Mar 15, 2024 · 54:30

Brian presents a detailed walkthrough of the survey paper 'A Comprehensive Overview of Large Language Models,' which systematically covers the evolution of LLMs from early attention mechanisms to modern transformer architectures. He explains the shift from sequential RNNs to parallelizable transformers, the three dominant architectures (encoder-only like BERT, encoder-decoder like T5, decoder-only like GPT), and key training objectives (masked language modeling, full language modeling, prefix language modeling). The paper also reviews fine-tuning techniques (instruction tuning, alignment tuning via RLHF), prompting strategies (zero-shot, chain-of-thought), efficient adaptation methods (LoRA, quantization), evaluation benchmarks (GLUE, MMLU), and challenges such as bias, memorization, and privacy leaks. This episode serves as a comprehensive primer for anyone seeking to understand the foundational concepts and latest trends in large language model research.

Making Transformers Sing - with Mikey Shulman of Suno

Mar 14, 2024 · 58:58

In this episode of Latent Space, hosts Alessio and Shawn Wang interview Mikey Shulman, CEO of Suno, about making transformers sing. Suno uses transformers to predict audio tokens end-to-end, avoiding baked-in musical knowledge, with a tokenization secret sauce that also includes non-music audio for better vocal realism. Their models are relatively small (far below 175B parameters) due to latency needs, and they prioritize scaling research over brute-force size. Over half of Suno users employ expert mode, tweaking lyrics and style prompts, rather than easy mode. Shulman argues Suno is not the 'Midjourney of music' because music is inherently social and synchronous, unlike images. He demoed live generation, showing control via tokens like [beat drop] and style modifiers, and revealed future plans for collaborative concerts, continuous DJ modes, and personalized models. He also advocates for hiring economists to avoid Goodhart's law pitfalls in ML benchmarks, especially in audio where aesthetics matter most.

Open Source AI is AI we can Trust — with Soumith Chintala of Meta AI

Mar 6, 2024 · 1:37:38

Soumith Chintala, creator of PyTorch and engineering lead at Meta AI, argues that open source AI is essential for distributing opportunity and trust. He details PyTorch's complexity—1,000 operators needed for generality—and explains synthetic data as a vehicle for imparting symbolic knowledge where humans already have good symbolic models. He highlights a coordination problem in open source: feedback is lost because frontends like Ooba and Ollama lack feedback buttons, and proposes a centralized sinkhole to collect high-quality feedback. Beyond text, he is excited about robotics, where hardware remains a bottleneck, and Osmo's work to digitize smell, which he compares to images in the 1800s.

A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate

Feb 28, 2024 · 1:21:09

Ben Firshman, CEO of Replicate, explains how the inference platform grew from a research reproducibility tool into a 2M-user API business by embracing the generative image community and treating open-source AI as a hacker-friendly ecosystem. They accidentally discovered their API when a user reverse-engineered their web form, leading to their first $1k/month customer. Cog, their container standard for ML models, was born from lessons at Docker and the need to make models tinkerable. Ben argues that fine-tuning's low cost makes open-source models sustainable, and that AI engineers (orders of magnitude more than ML engineers) just need to start playing with models. He also discusses GPU scarcity, preferring sustainable pricing over price wars, and reveals that demand is not outpacing supply thanks to aggregating demand.

Truly Serverless Infra for AI Engineers - with Erik Bernhardsson of Modal

Feb 19, 2024 · 1:08:58

Erik Bernhardsson, creator of Annoy and Luigi, founded Modal to build a 'postmodern data stack' with a self-revisioning runtime that eliminates container cold starts and lets developers attach GPUs with a Python decorator. His serverless platform can fan out to thousands of GPUs within seconds, becoming a natural fit for GenAI workloads like Stable Diffusion and fine-tuning (e.g., RAMP fine-tuning 100 models in parallel). Modal differentiates from Replicate by targeting custom models and workflows, and from Modular by offering a managed cloud service rather than licensed software. Erik argues that buying hardware is inefficient for startups and that infrastructure founders must be willing to spend years on load balancing, page faults, and DNS. He shares how his IOI Gold Medal background shapes Modal's talent culture, hiring competitive programmers for complex scheduling and bin-packing problems.

Building an open AI company - with Ce and Vipul of Together AI

Feb 8, 2024 · 1:15:18

Together AI co-founders Vipul Ved Prakash and Ce Zhang explain why openness is core to their mission, detailing their journey from Apple (Vipul) and Stanford research (Ce) to building an open AI platform. They discuss RedPajama’s evolution into a modular dataset with 40 quality signals, the need for 5,000 tokens/second inference speed, and their investment in state space models like Mamba and Hyena as alternatives to transformers. The company, which runs 7,000-8,000 GPUs (mostly H100s), sees training as a larger workload than inference, with fine-tuning driving top models. They advocate for independent inference benchmarks, publish open research like FlashAttention, and keep some software proprietary. With 38 employees and 45% researchers, they are hiring across the stack, from CUDA to DevOps.

The State of AI in production — with David Hsu of Retool

Feb 7, 2024 · 1:06:52

Retool CEO David Hsu shares insights from the company's 2023 State of AI survey, revealing that most AI adoption remains internal and that the hype may be overrated—52% of 1,600 respondents said AI is overrated, and only 27% have AI in production, with 66% of those being internal use cases. He explains Retool's developer-first philosophy, their choice of open-source PG Vector over proprietary solutions, and why they intentionally raised less money at lower valuations to avoid over-dilution. Hsu describes Retool's shift from sales-led to bottom-up growth to reach millions of developers, and highlights the importance of AI workflows over simple chatbots, citing a clothing manufacturer that uses Retool apps with DALL-E to generate patterns. He discusses the competitive landscape, predicting open-source models will eventually catch up to OpenAI, and shares philosophical views on AGI using the plane-vs-bird analogy.

The Four Wars of the AI Stack - Dec 2023 Recap

Jan 26, 2024 · 1:20:58

Swyx and Alessio recap December 2023 by framing the AI landscape as four wars: the Data War (NYT lawsuit demanding destruction of GPTs, OpenAI's data partnerships, synthetic data from DeepMind); the GPU/Inference War (Mixtral price dropping 90% to $0.27/million tokens, benchmark drama between Anyscale and Together, alternative architectures like Mamba); the Multimodality War (Midjourney reaching $200M ARR, ElevenLabs hitting unicorn status, OpenAI and Google building god models); and the RAG/Ops War (LangChain vs LlamaIndex, Pinecone's $750M valuation, Qdrant serving OpenAI and Anthropic internally). They argue agents and open source are not active wars, and predict 2024 will see AI finally reach production, with hardware like Rabbit R1 and Tab capturing unique context.

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Jan 11, 2024 · 1:35:27

Dr. Nathan Lambert traces the origin and future of Reinforcement Learning from Human Feedback (RLHF), the secret ingredient behind ChatGPT, explaining how it evolved from robotics and early preference learning to become the core alignment technique for large language models. He details the three-phase process (instruction tuning, preference data collection, RL optimization), noting that RLHF's data costs for LLaMA2 were around $6–8 million, and that synthetic data from GPT-4 is cheaper and often more accurate than human labels. Lambert contrasts DPO with PPO, arguing DPO is simpler but may have lower peak performance. He discusses emerging methods like Constitutional AI, which uses AI-generated critiques based on principles, and highlights the challenge of evaluating RLHF models, noting GPT-4 Turbo's lead over earlier versions. The episode covers open questions about data aggregation, reward model agreement (65–75%), and the need for qualitative model interaction.

The Accidental AI Canvas - with Steve Ruiz of tldraw

Jan 5, 2024 · 1:27:49

Steve Ruiz, founder of tldraw, recounts how his open-source whiteboard accidentally became a premier multimodal AI canvas, enabling visual prompting with GPT-4V. Originally built as a personal project with obsessive attention to arrows and ink, tldraw gained traction as a developer SDK for infinite canvas apps. The 'Make It Real' feature, which turns wireframes into working HTML using GPT-4V, exploded with 22 million Twitter views, showcasing iterative design through spatial branching and annotation. Ruiz explains how the canvas supports multiple input layers—UI sketches, state charts, screenshots, even API endpoints—and how his fine arts background guided his taste-driven approach. He argues tldraw's moat lies in its web-native, hackable canvas infrastructure, not ephemeral AI integrations, and positions it as the battlefield for comparing multimodal models like OpenAI's and Gemini.

The AI-First Graphics Editor - with Suhail Doshi of Playground AI

Jan 2, 2024 · 1:09:23

Suhail Doshi, co-founder of Mixpanel and founder of Playground AI, argues that image generation is still in a GPT-2 moment and that training open-source foundation models from scratch is necessary to unlock real utility, exemplified by Playground v2's 2.5x preference over Stable Diffusion XL on a 1K prompt benchmark. He explains the pivot from Mighty (cloud-streamed browser) to AI, driven by the belief that shifting compute elsewhere aligns with AI's parallel computation. The episode covers Playground's unique UI—not just a prompt box but a full canvas with preview rendering, seed control, and style filters—and the difficulty of balancing safety with artistic expression, especially around NSFW content. Suhail discusses the under-investment in graphics AI compared to language, the open-source community's role, and his choice to release pre-trained weights for academic research. He also shares lessons from running GPU infrastructure (harder for training than inference) and advises founders to follow curiosity and build projects rather than relying on books.

The "Normsky" architecture for AI coding agents — with Beyang Liu + Steve Yegge of SourceGraph

Dec 17, 2023 · 1:33:26

Beyang Liu and Steve Yegge of Sourcegraph explain how their AI coding assistant Cody achieves a 30% completion acceptance rate by prioritizing high-quality context over agents. They introduce the 'Normsky' architecture—a blend of Norvig's data-driven models and Chomsky's formal systems—arguing that non-agentic, deterministic context retrieval (using parsers, trigram indexes, and the BFG code graph) outperforms multi-hop LLM-based approaches for code completion and chat. The duo details their data pre-processing moat, the 'bin packing' challenge of stuffing relevant code into context windows, and why they use open-source StarCoder for completions while leveraging Claude/GPT-4 for chat. They also discuss the death of DSLs, the limitations of LSP and LSIF, and how Cody's web version lets users query any public repo instantly.

The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

Dec 5, 2023 · 1:07:55

Dylan Patel of SemiAnalysis argues the 'GPU poor vs GPU rich' divide defines AI, with Google's TPU ramp (millions of units) dwarfing others and Nvidia selling over a million H100s this year. He explains that training costs are irrelevant compared to inference costs, and that memory bandwidth is the key bottleneck for LLM inference (e.g., Llama 70B needs ~2.1 TB/s for human reading speed). Patel advises GPU-poor players to focus on on-device innovations like speculative decoding (Medusa), asynchronous training, or fine-tuning for edge use—not on gaming benchmarks or fine-tuning small models. He details why AI hardware startups (Cerebras, Graphcore) bet wrong on on-chip SRAM, while Google’s TPU/Broadcom partnership and Nvidia’s yearly cadence make alternatives tough. He also warns that rebuilding the semiconductor supply chain in the US is unfeasible due to its fragmentation, and that safety through obscurity doesn't work—open innovation is better.

Beating GPT-4 with Open Source Models - with Michael Royzen of Phind

Nov 3, 2023 · 1:18:54

Michael Royzen, co-founder and CEO of Phind, explains how his company built a GPT-4-beating open-source model for developer Q&A. Royzen recounts founding SmartLens in high school, shifting to NLP after a Hugging Face demo, and creating an internet-scale LLM-powered RAG system in January 2022. He details Phind’s pivot to programmers, the Hacker News launch that gave it 1,500 points, and Paul Graham’s role in naming the company and introducing Ron Conway, who then connected Phind to NVIDIA for GPU access. Royzen argues that Phind’s model, fine-tuned from Code Llama 34B with extra data, closes the gap with proprietary models—especially on code reasoning—and that open-source will win the enterprise because the delta to GPT-5 will be small. He shares how Phind handles multi-step conversations via a pair programmer mode where users can pin messages, and reveals plans for reinforcement learning to reduce hallucination and improve correctness.

Powering your Copilot for Data - with Artem Keydunov from Cube.dev

Oct 27, 2023 · 43:01

Artem Keydunov, co-founder of Cube (formerly Statsbot), explains how his early text-to-SQL Slack bot in 2016 led to the creation of Cube as a semantic layer, which he argues is now essential for grounding AI models on structured data. Statsbot failed due to the lack of LLMs, relying on regex and custom models, but the underlying Cube framework—an open-source semantic layer defining metrics and dimensions—solved the context problem. Today, semantic layers index data into text descriptions, enabling agents to generate simple queries against the layer instead of complex SQL, reducing errors. Keydunov highlights that natural language querying is becoming a commodity feature in BI tools, but the real value lies in treating metric definitions as code with version control. He advises AI engineers to use a proper warehouse, a semantic layer like Cube, and tools like LangChain, while cautioning that production systems need extra Python code for math and error handling.

Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue

Oct 21, 2023 · 1:12:37

Kanjun Qiu, CEO of Imbue (formerly Generally Intelligent), argues that AI agents remain unreliable because they lack robust reasoning and proper abstractions, which Imbue tackles by training foundation models optimized for reasoning over data like code. With a $200M Series B and $1B+ valuation, Imbue builds internal tools for debugging and inspecting agent decision-making, rejecting pure reinforcement learning after their Avalon environment revealed RL cannot handle planning. Instead, they emphasize natural language reasoning for inspectability, use code as a curriculum for reasoning, and design interfaces that let users fork and modify agents mid-execution. Qiu also discusses lessons from earlier startups (Sorceress in recruiting, Embark in VR), the importance of treating team members as creative agents, and her role in co-founding communal living space The Archive to foster scenius.

The End of Finetuning — with Jeremy Howard of Fast.ai

Oct 20, 2023 · 1:24:48

Jeremy Howard of Fast.ai argues that fine-tuning language models is essentially continued pre-training, not a separate process, and that the common practice of fine-tuning on a single task causes catastrophic forgetting. He recounts the discovery of single-shot memorization in LLMs, where models memorize entire datasets after one epoch, a phenomenon many practitioners ignore. Howard criticizes the current focus on zero-shot and few-shot learning, advocating for transfer learning and small models. He shares his journey from philosophy to founding Fast.ai, the creation of ULMFit (which inspired GPT), and his ongoing work on making AI accessible. He also discusses his involvement with Modular's Mojo language and the importance of democratizing AI technology.

RAG is a hack - with Jerry Liu of LlamaIndex

Oct 12, 2023 · 1:13:25

In this episode of Latent Space, Jerry Liu of LlamaIndex argues that while RAG is fundamentally a hack, it remains the most practical approach for grounding LLMs with external data, and his open-source framework now serves 600,000 monthly downloads. Liu recounts how LlamaIndex originated from a hackathon at Robust Intelligence, evolving from an experimental tree-index to a modular toolkit spanning data loading, retrieval, synthesis, and agent loops. He explains the trade-offs between RAG and fine-tuning, noting that context window limits force algorithmic rather than learned optimizations, but RAG offers transparency and access control that fine-tuning cannot. Liu details the company’s $8.5M Greylock raise, the launch of LlamaHub for community-contributed data loaders, and the open-source SEC Insights app as a production-grade template. Looking ahead, he emphasizes the need for better retrieval benchmarks and predicts that future personalization will move beyond vector stores into model-internal memory architectures.

Generating your AI Media Empire - with Youssef Rizk of Wondercraft.ai

Sep 20, 2023 · 1:28:55

Youssef Rizk, co-founder of Wondercraft.ai, explains how his five-month-old startup uses hyperrealistic AI voices to turn blogs, newsletters, and other content into podcasts, arguing that the application layer—not the underlying API—is the real moat. He details Wondercraft's origin from a failed sports-investing startup, the rapid MVP that generated $3K in one day, and the decision to build script generation and audio production tools rather than train its own models. Rizk also announces a new AI dubbing feature for video, discusses the challenges of scaling to 28 languages, and reveals that the company runs on just four people pushing two major features per week. The conversation covers the daily Hacker News Recap podcast—which reached Spotify's top 30 tech podcasts—and Rizk's view that AI-generated content must be clearly disclosed to avoid backlash.

RWKV: Reinventing RNNs for the Transformer Era

Aug 31, 2023 · 1:56:51

Eugene, CTO of UILicious, introduces RWKV, a recurrent neural network that matches transformer-level performance while scaling linearly with context length, solving the quadratic cost of attention. He explains how RWKV uses Attention-Free Transformer layers and parallelizable training, achieving competitive results on 7B and 14B models. The community-driven project prioritizes multilingual support, with a tokenizer that handles languages without spaces. Eugene recounts his journey from GPU.js to finding RWKV, and highlights use cases like long HTML analysis where transformers fail. He also discusses the token crisis, diffusion models for text, and how the AI waifu community drives optimization and alignment research. Finally, he advises AI engineers to focus on practical prompting and data curation, noting that deep architecture knowledge is optional.

FlashAttention-2: Making Transformers 800% faster AND exact

Aug 3, 2023 · 1:04:06

Tri Dao, creator of FlashAttention and FlashAttention-2, explains how his I/O-aware algorithm makes attention 2x faster by fusing kernels and using online softmax, achieving near-matrix-multiply efficiency. He argues that transformer alternatives like state space models and RNNs (e.g., RWKV) could surpass transformers for long sequences and high-throughput generation, though attention still dominates. Dao discusses the hardware lottery, where NVIDIA's CUDA ecosystem entrenches transformers, and advocates for open-source AI, praising Meta's Llama 2 for shifting enterprise adoption despite its restrictive license. He emphasizes that understanding both algorithms and systems is key to scaling AI, and that academia should pursue risky bets that industry cannot.

Ep 18: Petaflops to the People — with George Hotz of tinycorp

Jun 20, 2023 · 1:23:24

George Hotz of tinycorp argues that a simplified, open-source ML framework (tinygrad) can democratize AI compute and replace NVIDIA, Google, and AMD. Tinygrad uses 25 ops (vs. XLA's 250) and fuses kernels automatically, achieving 2x speed on Qualcomm GPUs over their library. Hotz reports AMD kernel panics fixed after emailing CEO Lisa Su, but laments AMD's open-source as 'dumped on GitHub.' For hardware, tinybox is a six-GPU desktop at 350W per GPU, aiming for 5x better price-performance than NVIDIA H100 for 90% of training. He criticizes OpenAI's secrecy, revealing GPT-4 is an 8-way mixture of 220B models, and cites the Bitter Lesson. He also outlines FLOPcoin using hardware identity to prevent cheating, and his goal of an AI girlfriend as the third company, merging humans via data rather than implants.