Marvin's Memos

The Scaling Hypothesis - Gwern

The provided source is an article titled "The Scaling Hypothesis" by Gwern, which explores the idea that the key to achieving artificial general intelligence (AGI) lies i

The Bitter Lesson - Rich Sutton

The article, "The Bitter Lesson," argues that the most effective approach to artificial intelligence (AI) research is to focus on general methods that leverage computatio

Larger and more instructable language models become less reliable

This study examines the reliability of large language models (LLMs) as they grow larger and are trained to be more "instructable". The authors investigate three key aspec

AlphaChip + A PRELIMINARY EVALUATION OF OPENAI’S O1 ON PLANBENCH

The first source, a research paper from Arizona State University, explores the abilities of large language models (LLMs) to plan, using a benchmark called PlanBench. Whil

Llama 3.2 + Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models

The sources describe the latest advancements in the field of large language models (LLMs) with a focus on multi-modality, meaning the models are able to process and under

Sparse Attention with Linear Units - Rectified Linear Attention (ReLA)

This research paper proposes a new method for achieving sparsity in attention models, called Rectified Linear Attention (ReLA). ReLA replaces the softmax function with a

Sparse and Continuous Attention Mechanisms

This research paper proposes a novel approach to attention mechanisms in neural networks, extending them from discrete to continuous domains. This extension is based on t

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

FlashAttention-2 is a new algorithm that improves upon FlashAttention, a method for speeding up and reducing memory usage of the attention layer in Transformers, which is

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

This episode looks at 'FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness', a novel attention algorithm that significantly improves the speed and

The Intelligence Age - Sam Altman

This episode looks at "The Intelligence Age", by Sam Altman, who argues that we are on the cusp of a new era driven by artificial intelligence. The author posits that dee

A Path Towards Autonomous Machine Intelligence - Yann LeCun

This episode breaks down the 'A Path Towards Autonomous Machine Intelligence' research paper, written by Yann LeCun, which proposes a novel architecture for autonomous ma

Machines Of Loving Grace - Dario Amodei

This episode looks at Dario Amodei's essay, "Machines of Loving Grace," which explores the potential for powerful artificial intelligence (AI) to revolutionise society fo

Situational Awareness, The Decade Ahead - Leopold Aschenbrenner

This episode breaks down the paper titled "Situational Awareness: The Decade Ahead" by Leopold Aschenbrenner, written in June 2024. Aschenbrenner, formerly of OpenAI, arg

Round Up : Top 30 Essential AI Papers

Rounding up of Top 30 Essential AI Papers. The sources cover a wide range of topics including the effectiveness of recurrent neural networks, the use of attention mechani

Lost in the Middle: How Language Models Use Long Contexts

This episode breaks down the 'Lost in the Middle: How Language Models Use Long Contexts' research paper, which investigates how language models use long contexts, specifi

Zephyr: Direct Distillation of LM Alignment

This episode breaks down the 'Zephyr: Direct Distillation of LM Alignment' research paper, which describes ZEPHYR-7B, a smaller language model (LLM) aligned with user int

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

This episode breaks down the 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' paper, which introduces Retrieval-Augmented Generation (RAG), a new approa

Dense Passage Retrieval for Open-Domain Question Answering

This episode breaks down the 'Dense Passage Retrieval for Open-Domain Question Answering' research paper from Facebook AI and other institutions which examines dense repr

Better & Faster Large Language Models via Multi-token Prediction

This episode breaks down the 'Multi-token Prediction' research paper, which proposes a novel approach to training large language models (LLMs) called multi-token predicti

Kolmogorov Complexity and Algorithmic Randomness

This episode breaks down the 'Kolmogorov Complexity' paper, which discusses the fascinating topic of algorithmic information theory, which explores the inherent complexit