About this episode
<p>Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of <a target="_blank" href="http://www.incompleteideas.net/IncIdeas/BitterLesson.html">The Bitter Lesson.</a> And he thinks LLMs are a dead end.</p><p>After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need <em>some</em> new architecture to enable continual learning.</p><p>And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals.</p><p>This new paradigm will render our current approach with LLMs obsolete.</p><p>In our interview, I did my best to represent the view that LLMs might function as the foundation on which experiential learning can happen… Some sparks flew.</p><p>A big thanks to the <a target="_blank" href="https://www.amii.ca/">Alberta Machine Intelligence Institute</a> for inviting me up to Edmonton and for letting me use their studio and equipment.</p><p>Enjoy!</p><p>Watch on <a target="_blank" href="https://youtu.be/21EYKqUsPfg">YouTube</a>; listen on <a target="_blank" href="https://podcasts.apple.com/us/podcast/richard-sutton-father-of-rl-thinks-llms-are-a-dead-end/id1516093381?i=1000728584744">Apple Podcasts</a> or <a target="_blank" href="https://open.spotify.com/episode/3zAXRCFrHPShU4MuuIx4V5?si=74e5a07b00444b51">Spotify</a>.</p><p>Sponsors</p><p>* <a target="_blank" href="https://labelbox.com/dwarkesh">Labelbox</a> makes it possible to train AI agents in hyperrealistic RL environments. With an experienced team of applied researchers and a massive network of subject-matter experts, Labelbox ensures your training reflects important, real-world nuance. Turn your demo projects into working systems at <a target="_blank" href="https://labelbox.com/dwarkesh">labelbox.com/dwarkesh</a></p><p>* <a target="_blank" href="https://gemini.google.com/">Gemini Deep Research</a> is designed for thorough exploration of hard topics. For this episode, it helped me trace reinforcement learning from early policy gradients up to current-day methods, combining clear explanations with curated examples. Try it out yourself at <a target="_blank" href="https://gemini.google.com/">gemini.google.com</a></p><p>* <a target="_blank" href="https://hudsonrivertrading.com/dwarkesh">Hudson River Trading</a> doesn’t silo their teams. Instead, HRT researchers openly trade ideas and share strategy code in a mono-repo. This means you’re able to learn at incredible speed and your contributions have impact across the entire firm. Find open roles at <a target="_blank" href="https://hudsonrivertrading.com/dwarkesh">hudsonrivertrading.com/dwarkesh</a></p><p>Timestamps</p><p>(00:00:00) – Are LLMs a dead end?</p><p>(00:13:04) – Do humans do imitation learning?</p><p>(00:23:10) – The Era of Experience</p><p>(00:33:39) – Current architectures generalize poorly out of distribution</p><p>(00:41:29) – Surprises in the AI field</p><p>(00:46:41) – Will The Bitter Lesson still apply post AGI?</p><p>(00:53:48) – Succession to AIs</p> <br/><br/>Get full access to Dwarkesh Podcast at <a href="https://www.dwarkesh.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4">www.dwarkesh.com/subscribe</a>