How my views on AI changed every year 2017-2024

created: ; modified:

Introduction

I was originally going to write a piece called “why i no longer believe in AGI” but then I remembered that I changed my mind on this like 5 times and I have no idea what I’ll believe in another few years. So I figured it would be more honest to just write up the entire timeline and then append future changes to it.

Very briefly, I went from feeling that AGI will take decades in 2017-2019 to feeling that it’s imminent within the next few years in 2022-2023, and now I’m so confused by what people mean by “AGI” that I can’t even say if it’s a coherent concept worth discussing.

I mean, clearly, AI is an important technology. LLMs are already used by millions of people and are very helpful in some areas of work.

But are LLMs or their immediate successors about to replace humans and put us all on UBI-subsistence existence? I don’t think so. Is there such a thing as “general intelligence”? I don’t think so (Alvaro De Menard suggests otherwise in a comment). Is the eschaton about to be immanentized & are we due for an imminent rapture? Very tempting to believe! But I really don’t think so.

If you think my takes are dumb and should be dismissed, you’re welcome to check out the appendix where I give my own reasons for why you might want to dismiss my current takes.

Anyway, here’s the entire timeline, 2017-2024, in reverse order, since my more recent thoughts are better-developed and more interesting, I think:

2024

  1. I spent very little time explicitly thinking about AI in 2024, mostly studying physics, reading a lot of history, and writing.
    1. Alvaro De Menard’s Reading Notes: Civilization & Capitalism, 15th-18th Century, Vol. I: The Structures of Everyday Life is required reading.
    2. I kept thinking about potentially working interfaces and AI products intermittently from late 2023 to early 2024, but eventually figured that my original desire to build them was based on feeling that LLMs are much smarter than they seem to me now.
  2. This for me is the single biggest datapoint: when I coded every day, I used GPT-4 religiously. Since I stopped coding, I haven’t been using it at all. I feel like LLMs are super useful for in-distribution work (e.g. coding, bs writing of the kind that shouldn’t even be written in the first place) but not at all useful for out-of-distribution work (e.g. research, learning deeply, real writing (e.g. this piece)).
  3. Max Shirokawa noted to me that “LLMs have very high crystallized intelligence but very low fluid intelligence” which seems exactly right to me.
    1. I’ve taken up to saying that LLMs are basically verbally fluent databases.
      1. See The Many Ways that Digital Minds Can Know
    2. Also a good explainer on yapping being confused for something deep: The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con
  4. Seems that DeepMind’s original thesis about “solving intelligence and then using it to solve everything else” fundamentally misunderstands what intelligence is. Intelligence is the ability to solve specific problems, therefore it necessarily exists in the context of all in which it lives and what came before it and the goal of “solving intelligence” is meaningless.
  5. Lots of people working on AGI tell me that they’re trying to “solve reasoning”. I still have no idea what this means.
  6. Relatedly, by mid-2024 I stopped feeling any AI fomo, even when visiting SF, and lost all extra interest in AI. It just feels like any other kind of big technology now.
  7. I feel like my feeling of LLMs being dumb is supported by examples like asking them “9.11 and 9.9 - which is bigger?" (2) or “How many ‘Rs’ in the word ‘Strawberry’?" or “You open a door and win a Ferrari! Should you switch door?" or “what is the second to last word in your response?".
    1. People say “oh it’s the tokenizer” “just use chain-of-thought” “we’ll fix the system 2 reasoning soon” but what I’m getting from this is that whatever neural networks are doing is literally just not at all what we consider to be thinking or reasoning; and the entire system 1/2 distinction is nonsensical anyway I have no idea how people can be talking about it seriously.
    2. This means that working on neural networks is NOT getting us closer to AGI, except indirectly.
    3. And this means that making claims about AGI based on neural network progress is not that different from making AGI claims based on the success of “classic ML” algorithms or things like Monte Carlo Tree Search techniques.
  8. Tyler Cowen’s take that we’re going to see strong AI but not AGI for decades seems right.
  9. I feel like most computation in the brain is chemical, not electrical. Electricity produces too much heat. I think the implications of this are underappreciated. e.g.
  10. I would not be surprised at this point if AI is actually going to slow down science in the same way that I think unlimited data+compute killed the deeper and more imaginative economics research, reducing everything to math and regressions, filling the literature with endless garbage research one needs to weed through to get to something real. Hasn’t been much conceptual economics progress since the advent of computers & p-values.
    1. Also, since many of the most talented people who’d be doing basic science are now doing ML (hi Trenton..).
    2. My general take is: if you need to do statistics, your evidence isn’t strong enough.
    3. “AI slop”.

2023

  1. In January, I got fully mentally committed to alignment and decided that OpenAI is going to be the last company and I must join it to work on alignment to make sure the world doesn’t get destroyed by AGI/ASI.
    1. I wrote down in my document called why alexey decided against working on ai products that “everything that could happen will happen in 10 years”.
  2. In February, I tweeted that “non-ironically shoggoth memes might be the greatest alignment advance of the last year — finally made it easy to picture what an llm really is, instead of allowing us to keep pretending that it’s kind of just a dude on a computer or something”.
    1. I stopped thinking about LLMs as shoggoths about 6 months later and started to think of them as “persona imitators” with personas supplied entirely by the training data; and since the training data is human data (or synthetic data we came up with), all the inner scary “shoggoth” parts of LLMs are just human shadows.
    2. As of mid-2024 I feel kind of blackpilled on language as the medium where most of the thinking happens – it just seems to be much more about communication (I think Milan Cvitkovic was making this point to me in early 2023 and I dismissed it then).
    3. One implication being that LLMs might actually not understand our preferences as well as they seem to because our preferences are not well-encoded in words.
    4. The “it” in AI models is the dataset.
  3. In March, reading Tyler Cowen’s Existential risk, AI, and the inevitable turn in human history, it seemed incredibly sensible, but by then I was so into trying to get into OpenAI to do alignment that it couldn’t change the course of my actions. It’s still the best general piece of writing about AI risk I’ve read.
  4. Joel Einbinder was telling me that the whole alignment thing I got really into seems kind of meaningless and extremely poorly defined.
  5. In May-June, while hanging out at OpenAI, I kept trying to make ChatGPT follow my simple instructions in the style of “don’t give up if you fail, keep trying to complete a task” and was utterly unable to do this. From this, in the summer of 2023, I got a very deep feeling that LLMs actually have absolutely no idea what’s being asked of them and really are nothing more than yapping Chinese rooms.
    1. I’m still confused by people taking yapping as a sign of consciousness.
  6. By August, I lost the ability to see fast takeoff actually happening at all & started to feel like the entire AGI/ASI thing is just an incredibly good meme, very much in line with the original polytheism->monotheism transition, Yudkowsky being a classic doom cult founder, and that every single doom cult in the last 2,000 years was wrong.
    1. Yudkowsky in 1996: “Let’s say that we want this meme [Singularity] to infect a majority of all receptive individuals. If necessary, we can retarget another version for a majority of all influential individuals, technologically sophisticated or otherwise.”
    2. I think most people keep believing in AGI/ASI because it’s just really difficult to get the question “but what if rapture is coming within our lifetimes after all?” out of your head once it got in.
    3. Yud’s meme engineering worked!
    4. I found Arjun Ramani and Zhengdong Wang’s Why transformative AI is really, really hard to achieve to be excellent. I remember having a conversation with him in January 2023 where he was making a lot of the points from the piece and decided to never speak to him ever again (hi Zhengdong!).
  7. By September, I totally lost interest in working on alignment/safety and started writing Is AI alignment on track? Is it progressing… too fast?, which I published in October.

2022

  1. Throughout the first half of 2022, I was starting to develop a feeling that maybe my 2017-2019 takes were unjustified, given how little ML I knew and that maybe I should be thinking about AI much more.
  2. In August, I went to a futurist conference. There, an AI researcher showed me that most of the existing alignment research is really bad. I started to suspect that we might fail at AI alignment which would be extremely bad for the world because alignment researchers aren’t doing good alignment research.
    1. Some of the people at the conference had AGI timelines of 3-5 years, which I was starting to find not entirely crazy.
    2. I also noticed at this conference that I was most interested in talking to AI people compared to anyone else.
  3. Also in August, I tried out OpenAI’s DALL-E and was blown away by it. I felt like human artists/designers were done and the rest of us were going to be done very soon. This was the moment where I think I was set on the AI path, even though it took several more months to really commit to it.
  4. In September-October I tried doing AI research. Briefly: I thought that neural networks meta-learn and generalize much better than they actually did and was proven wrong by Jacob Buckman. In more detail:
    1. I had a strong feeling that neural networks can generalize what it means to be e.g. a good chess player.
    2. One implication of this is that if we train an NN to play as a player with ELOs of 1000, 1100 and 1200, we can ask it to play as a player with ELO 1300 and it’ll be able to do this somewhat well.
    3. This idea was developed with Jacob Buckman (I think he came up with it, actually), so we decided to try to test it experimentally.
    4. I ended up dropping out of the project after a few weeks because I got convinced by my alignment friends that it’s morally bad to be doing this kind of capabilities research.
    5. Jacob eventually finished and published the project himself.
    6. My original intuition was totally wrong and his original intuition (opposite of mine) was totally right: neural networks didn’t generalize what it meant to play as a player with a particular ELO at all.
    7. I feel like I should’ve taken this result more seriously because it suggested that my original 2017-2020 intuitions about how poorly NNs understand what’s going on were correct. Instead I just dismissed this result.
    8. Jacob kept telling me throughout that I believe in neural networks too much and that it would require fundamental new theoretical breakthroughs in our understanding of them and of reinforcement learning to get to ASI.
  5. In October, Nate Sesti and I built a GPT-3 writing interface and it ended up writing a significant chunk of my Planes are still decades away from displacing most bird jobs, If the moon doesn’t need gravity, why do we? The necessity of understanding for general intelligence, and What is the alternative to utilitarianism?. In retrospect, I notice that GPT-3 only helped to write the funny nonsense parts of all of these posts, and whenever I tried to get it to help me write something serious later, it failed.
  6. In November, I got access to an early version of GPT-4. I cried when I first used it. I felt like I was about to be replaced by it. Seeing GPT-4 in late 2022 made me convinced that GPT-5 = AGI.
  7. By this point, I started to dismiss anyone who wasn’t thinking that AGI is coming imminently.
    1. I remember hanging out with Leopold Aschenbrenner around this time and him saying something like “let’s say 20% chance that AGI is not even a coherent concept and all of our thinking about it is hopelessly confused” when discussing whether to work on AI. I thought this was the most ridiculous thing I’ve ever heard someone say.
  8. I think the final thing that convinced me that AI is an existential danger and alignment is the thing to do was the video We Were Right! Real Inner Misalignment by Robert Miles because of how it showed AI mislearning the value function in a setting that seemed pretty realistic, with seemingly extremely direct apocalyptic consequences.
  9. I think my favorite ML paper in 2022 was Transformers are Sample-Efficient World Models by Vincent Micheli, Eloi Alonso, and François Fleuret.
  10. In December, I wrote down insane, in retrospect, qualitative and quantitative predictions for AI progress, the key point being that by 2031 the world will be unrecognizable.
    1. For me personally, the most important prediction (as of mid-2024) is about me using GPT-4 at least for 30 minutes a day upon its release and using a bunch of AI-first apps for e.g. writing.
    2. Well, GPT-4 has been out for almost 1.5 years and the median amount of time I’m using it for is 0 minutes/day; and whenever someone integrates AI features into existing products (e.g. Google Docs), I only get annoyed.
    3. In a remarkable feat of prescience, in that document I noted:

      I hope to unseal this document in December 2023 and think “how could I possibly write this” [about things going so fast].

  11. By December, I was feeling like I should work on alignment because it’s clearly the most important thing, but still couldn’t quite accept it mentally.

2021

  1. A friend visited me for several days and kept asking me why I bother with the institutions of science instead of doing AI/alignment. I kept dismissing him but I think it was the first thing that really lodged the idea that maybe I was wrong about us being decades away from AGI.
  2. I asked Boyden if he thinks his neuro research makes any sense at this point & if it seems to him like neural networks are going to AGI. He got mad at me, which at the time I thought was just him being an old person unwilling to abandon his decades-long research program.

2020

  1. Basically wasn’t thinking about AI in 2020.
  2. I watched this criticism of the GPT-3 paper by Yannic Kilcher and it strengthened my conviction that NNs aren’t real intelligence.
    1. His take is that LLMs are simply storing the training data in their weights and then regirgitating+interpolating it back fluently (video at 23:10). He has a lot of concrete examples where it looks like GPT-3 is doing “thinking” when it’s actually regurgitation that’s not being caught by the authors of the GPT-3 paper.
    2. This is especially funny to me because when I became convinced that AGI is coming very soon in 2022 I remembered how persuasive this video was for me in 2020 and I kept thinking about watching it again and checking out its arguments given my new beliefs but somehow felt anxious that I might be wrong again and ended up not going back to this video and rewatching it until late 2023 when I’d already stopped being worried about AGI again. Something very deep in this.

2019

  1. Took a deep learning class offered to computer science majors at my university. Was thoroughly bored throughout and never used anything I learned.
    1. At some point I ended up reproducing a paper from one of the top conferences and discovered that a key algorithm implemented in the supplied code was different from the algorithm published in the paper; this did not inspire any optimism about the state of ML.
  2. Got somewhat more optimistic about AGI coming soon after all, with OpenAI beating the best DOTA team and DeepMind beating the best Starcraft 2 players.
  3. Got super disappointed by neural networks in a seeming confirmation of my original 2017 intuitions after I watched Alex007’s deep dive into how exactly DeepMind’s AlphaStar plays StarCraft.
    1. It seems that AlphaStar was essentially winning in StarCraft by superior micro and tactical superiority enabled by superhuman dexterity+reaction time rather than any kind of deep strategic understanding of the game.
    2. Upon deeper investigation, it seemed to me that the way OpenAI’s neural network beat DOTA was very similar. Tricks and superhuman micro w/ 0 strategic depth.
      1. Sherjil Ozair: “To me, this confirms the hypothesis that model-free RL doesn’t scale to complex problems. Even after restricting the game to 17 heroes, and providing 45k years worth of data, model-free RL learns fragile one-trick policies which beat humans only due to mechanical superiority.”
  4. James Ough strongly thought that neural networks are not on the path to AGI (I don’t remember his arguments), but I thought he had good taste and it convinced me even further.

2018

  1. In January 2018, I applied to master’s programs in Neuroscience at LMU Munich and ETH Zurich because (according to my motivation letter) I wanted to “understand the way humans learn, make decisions, and how intelligence arises”.
  2. Marginal Revolution’s Will truckers be automated? (from the comments) supported my intuition that AGI is way harder than people think by pointing out that the jobs are much more complicated than it seems from the example of a truck driver job.
    1. Reminds me of Reality has a surprising amount of detail.

2017

  1. 2017 was the first year in which I thought about AI seriously.
    1. It seemed that VR and AI were going to be the two most important technologies in the coming decade.
    2. Started to wonder if I can solve StarCraft 2 and then build AGI.
  2. Tried to take an ML class in the computer science department of my university. Dropped it almost immediately – it had too much math and I felt weird proving theorems about the behavior of cross-entropy when I was interested in how intelligence works.
  3. Played around with OpenAI Gym and deep reinforcement learning agents. Didn’t feel like real learning or something.
  4. Watched all ML lectures from Caltech’s CS 156 taught by Yaser Abu-Mostafa. Every idea and technique taught in class made me feel like ML is not at all concerned with how learning or intelligence works.
  5. By mid-2017, I had a strong feeling that none of current ML is how AGI is going to be achieved; and that yet unknown neuroscience/bio-inspired approaches are the way to go.
    1. One implication of this being that it’ll take decades to get to AGI.
    2. This was one of the biggest reasons I got to work on bio and institutions of basic science in the coming years.

Subscribe to get updates.

Acknowledgements

Thanks to Amanda Geiser, Siméon Campos, Zhengdong Wang, and Filip for feedback.

Appendix: Reasons to dismiss my current AI takes

  1. I’ve done very little technical research/engineering in either ML or neuro or any other field of science or engineering for that matter.
  2. I spent a few months hanging out at OpenAI in mid-2023 and got kicked out in October. Perhaps, I back-justified my current AI skepticism simply to deal with the fact that I couldn’t figure out how to join one of the AGI companies.
  3. I really dislike working on stuff that I feel like someone else could do. I got interested in AI when both AI products and AI safety were yet to become fashionable (before ChatGPT came out). Since then, both fields have exploded. Perhaps, by mid-2024 I simply could no longer find an area of AI where I could really make a major unique contribution, but ended up justifying it to myself by convincing myself that the entire field of AI isn’t as promising as it seems.

Appendix: more on AlphaStar

More on the deep dive into AlphaStar video from 2019:

  1. The video makes the point that the neural network has absolutely no understanding of the game at all, constantly doing really stupid strategic moves. That is, neural networks don’t “get” StarCraft even though they can play incredibly well.
  2. A particularly dumb example of AlphaStar’s behavior is at 34:40 in the linked video: AlphaStar attacks rocks instead of enemy units. Why? Because it learned by imitation learning that humans sometimes attack rocks. Except that humans only ever do it by accident. AlphaStar just copied this meaningless accidental human behavior and never unlearned it, even after it transitioned to playing only against itself.
    1. YouTube comment: “The neural network calculated everything correctly. 100% of winners make misclicks, which means they’re important and useful for winning.”
    2. One of the most remarkable things about AlphaStar playing SC2 is the fact that it sees the entire map all at once, instead of having a limited POV + minimap like humans. I don’t see any good reason for why they would do this, given that DeepMind cared so much about presenting this result as AI playing with human-level restrictions. The only conclusion I can arrive at from this is that they simply couldn’t make a NN learn to use the limited POV + minimap. This is remarkable, given how much training data for imitation learning on human players + compute they had. Points to some severe NN limitations.
    3. The especially sad part about this is that DeepMind literally lied to everyone about how AlphaStar works in a pretty fundamental way. They claimed that it uses human-style POV while in reality it has an entirely different way of selecting units arbitrarily from the entire map without the need to look at any particular piece of the map, all while moving its camera around pretending to use it to select units. Kind of insane if you think about it. Video mark at 59:00.
    4. I’m not aware of anyone in the English-speaking internet picking this up — only this one Ukrainian mathematician/SC2 broadcaster.
  3. A friend notes: “if synthetic data is as easy as people say, why didn’t it work for AlphaStar? For a much easier domain than all human intelligence (‘just’ StarCraft), how many self play games did it have to play just to get that good?”
  1. How I Use “AI” by Nicholas Carlini:

    But the reason I think that the recent advances we’ve made aren’t just hype is that, over the past year, I have spent at least a few hours every week interacting with various large language models, and have been consistently impressed by their ability to solve increasingly difficult tasks I give them. And as a result of this, I would say I’m at least 50% faster at writing code for both my research projects and my side projects as a result of these models.

    Most of the people online I find who talk about LLM utility are either wildly optimistic, and claim all jobs will be automated within three years, or wildly pessimistic, and say they have contributed nothing and never will.

    So in this post, I just want to try and ground the conversation. I’m not going to make any arguments about what the future holds. I just want to provide a list of 50 conversations that I (a programmer and research scientist studying machine learning) have had with different large language models to meaningfully improve my ability to perform research and help me work on random coding side projects.

  2. Ninety-five theses on AI by Sam Hammond
  3. Fifty AI theses by Stephen Malina
  4. Jagged Intelligence by Andrej Karpathy
  5. Why LLMs are Much Smarter than You and Dumber than Your Cat by Daniel Jeffries
  6. Superintelligence: The Idea That Eats Smart People by Maciej Cegłowski
  7. On The Impossibility of AI Alignment by Kevin Lacker:

    I think it is possible for humans to survive and thrive in a world with unaligned superintelligence.

    The best metaphor I have here is that in some sense, weak superintelligences already exist. Corporations and governments are generally smarter than individual humans. Nike (to pick some mundane corporation) is orders of magnitude more powerful than the average human. More money, more ability to affect the world. More able to write software.

    These superintelligences are not aligned. They are kind of aligned… but not really aligned. Nike wants to maximize its profits far more than it wants any particular human value. The French government wants all sorts of things that vaguely correspond to what humans want, but it’s not like there’s a provable mathematical relationship there. It’s kind of aligned.

    … In particular, they [rules] don’t require alignment. Corporations don’t have the same goals as humans or other corporations or governments. We accept that. There is a plurality of goals. Different entities have different goals, both corporations and humans. We may feel a little bit bad that Nike has these inhuman goals of optimizing shoe sales, but it’s generally acceptable.

    This also gives us an alternative strategy to AI alignment. Instead of demonstrating that the system is “aligned”, demonstrate that you can give it some rules, and it will always follow those rules.

    … A technical aside: in general, you cannot take in an arbitrary function and prove anything about it, due to the halting problem. However, if your programming language is constrained in some way, to make it not Turing-complete, or if you permit the prover to sometimes say “I don’t know”, then the problem is possible again. I am not trying to propose we solve the halting problem here.

    So. It is theoretically possible to have a function, 1000 lines of Python code, and prove that no matter what, it will not print out the word “paperclip”. I think it should be much easier than proving something is “aligned” in a fuzzy way. But right now, this is more or less beyond our abilities. The idea of provably secure systems has been around for decades. But it has always been too hard to get working for more than toy problems.

    Perhaps AI itself could help us build these restricted systems. AI could get really good at scanning code for vulnerabilities, good enough so that it was able to sign off on most code bases, and say “We can prove there are no security flaws here.”

  8. Language is primarily a tool for communication rather than thought (via hardmaru)
  9. “Language models learn from the cross entropy objective a Talmudic reasoning style, because cross entropy teaches you that if the words “hellish” and “hello” get tokenized as hell-ish and hell-o then predicting “hell” in both cases is half credit, so they’re closely related." by John David Pressman.
  10. On intelligence as conversion ratio by François Chollet:

    A consequence of intelligence being a conversion ratio is that it is bounded. You cannot do better than optimality – perfect conversion of the information you have available into the ability to deal with future situations.

    You often hear people talk about how future AI will be omnipotent since it will have “an IQ of 10,000” or something like that – or about how machine intelligence could increase exponentially. This makes no sense. If you are very intelligent, then your bottleneck quickly becomes the speed at which you can collect new information, rather than your intelligence.

    In fact, most scientific fields today are not bounded by the limits of human intelligence but by experimentation.

Appendix: a comment from Cody Breene

In a lot of ways this traced the schizophrenic path I’ve had with AI since DALLE. Initially, I thought I must accept Yud because the syllogism is so compelling and the critiques of him were so, so bad (I tweeted a request for counters to his arguments and they were all garbage). But it was all too “prove or disprove the existence of God in 3 sentences” for me. It was like he stopped at Descartes' description of the potentiality of an ever present demon deceiving him at all times, without moving past the thought experiment (but I digress).

This is what struck me the most from your piece: “When I coded every day, I used GPT-4 religiously. Since I stopped coding, I haven’t used it at all. I feel like LLMs are super useful for in-distribution work (e.g. coding, bs writing) but not at all useful for out-of-distribution work (e.g. research, real writing)."

I read a lot of Taleb in 2017, and I think there’s a strong Talebian critique against the p(doom)ers that relates to the point you made above. Essentially, LLMs operate in “mediocristan”, in other words they’re operating within a bell curve, there are no insane “fat-tailed” responses. But to become “super intelligent” (whatever the fuck that means), it requires some mastery of “extremistan” worlds, e.g. geo-politics. But these sort of “game-theoretic” worlds are by definition in “extremistan”, they’re fat-tailed, they’re defined as such because they RESIST prediction.

Appendix: a comment from Alvaro De Menard

If there is no general intelligence how do you explain the fact that humans can perform tasks that are hilariously far outside their evolutionary “training set”?

How human do rocket science when no rockets or math or paper or even writing or alphabets in cave

Like not only is there general intelligence

It’s un fucking believably general

Outrageously general

Like how the fuck does this even work or make the slightest bit of sense that writing computer code feels like the most natural thing in the world for me when computers didn’t even exist until 5 minutes ago

Appendix: Young Eliezer

From Jon Evans' Extropia’s Children, Chapter 1:

It is clear from even a casual perusal of the Extropians archive (maintained by Wei Dai) that within a few months, teenage Eliezer Yudkowsky became one of this extraordinary cacophony’s pre-eminent voices. Extropian Eliezer is impressively sympathetic. He describes himself as “an Algernon,” after the famous story about a man made so intelligent he can no longer befriend other humans. He writes: “The only reason I’m here is to save humanity,” and “if I could find a way to spend every waking minute working, I would do so.” He speaks highly of “the masses,” and is generally remarkably what-would-today-be-called-woke for the nineties in general, much less an extremely online teenage boy.

He is also absolutely convinced the end of the world as we know it is nigh, in the form of the Singularity. He says “the Singularity may well be before 2000,” and calls 1997 “the leading edge of the Singularity.” He cannot wait. The man who will become the world’s leading doomsayer of catastrophic artificial intelligence is convinced at this age of the exact opposite: that superintelligences will necessarily be far more ethical than us.

His fatal intellectual flaw is already apparent. His self-proclaimed ‘landmark work’ is, allegedly, “A Practical Guide to Neurosurgical Intelligence Enhancement Using Current Technology.” Unfortunately there is nothing practical about it. This is unsurprising; however precocious, he’s only seventeen! The problem is that he doesn’t seem to realize it’s vague, theoretical, and handwavey, the polar opposite of a practical guide. Similarly, his next major work, “Coding a Transhuman AI,” is a 50,000-word essay which suggests many notions but, despite its title, contains … no actual software code. He will completely rewrite this piece a few years later. The rewritten version? Still no code.

This “lots of concepts, zero implementation” pattern keeps recurring: “After halting work on the ill-fated business concept, I spent the next few months writing 500K of design notes for a programming language called Flare.” Note: design notes, rather than the actual language, or, say, a compiler. Flare never launched. Eliezer can write code; he just never ships anything.

Appendix: Yann LeCunn on the “Doomer’s Delusion”

  1. AI is likely to kill us all
  2. Hence AI must be monopolized by a small number of companies under tight regulatory control.
  3. Hence AI systems must have a remote kill switch.
  4. Hence foundation model builders must be eternally liable for bad uses of their models and derived versions of it.
  5. Hence open source AI must be banned.
  6. But open source is popular, so we’re just going to say that we are pro open source but also say we need some sort of regulatory agency to oversee it.
  7. We’re going to scare the hell out of the public and their representatives with prophecies of doom.
  8. But we’ll make sure to appear much more respectable than the most extreme doomers.
  9. To look even more respectable, we’ll create a one-person institute to promote AI safety.
  10. We’ll get insane levels of funding from well-intentioned but clueless billionaires who are scared sh*tless of catastrophe scenarios, got too rich too quickly, have too much time on their hands, but should know better.
  11. We’ll claim that the majority of prominent scientists agree with us, even though said scientists are an infinitesimal minority in the AI community.

Appendix: Sherjil Ozair & Teortaxes on interpolation

Sherjil Ozair:

Claude does amazing things that humans have already done and posted the work on the internet.

I’m far from an LLM skeptic, but now I understand why @fchollet is doing what he’s doing.

The magic fades when you see through the implicit retrieval that is happening here.

Teortaxes:

If you don’t get what Sherjil is talking about, maybe this will make it clear.

LLMs are carpets of memoized functions, repositories of vector programs – but not novel, alien minds.
People in frontier companies are high on their own supply… distilled from our shared legacy.

Teortaxes:

LLMs can generalize. But “it was in the training data, verbatim” hypothesis keeps being unreasonably powerful for explaining Emergent Abilities of LLMs – especially for SoTA Anthropic models, hyped up by developers like they’re their precocious children.