Best of Holden Karnofsky and Sam Altman

created: 2022-12-03; modified: 2022-12-03

Holden Karnofsky

The Most Important Century (in a nutshell)

The long-run future is radically unfamiliar. Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.

The long-run future could come much faster than we think, due to a possible AI-driven productivity explosion.

The relevant kind of AI looks like it will be developed this century - making this century the one that will initiate, and have the opportunity to shape, a future galaxy-wide civilization.

These claims seem too “wild” to take seriously. But there are a lot of reasons to think that we live in a wild time, and should be ready for anything.

We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, we aren’t ready for this.

Learning By Writing

The “traditionally” hard parts of this process are steps 4 and 6: spotting weaknesses in arguments, trying to resist the temptation to “stick to my guns” when my original hypothesis isn’t looking so good, etc.

But step 3 is a different kind of challenge: trying to “always have a hypothesis” and re-articulating it whenever it changes. By doing this, I try to continually focus my reading on the goal of forming a bottom-line view, rather than just “gathering information.” I think this makes my investigations more focused and directed, and the results easier to retain. I consider this approach to be probably the single biggest difference-maker between “reading a ton about lots of things, but retaining little” and “efficiently developing a set of views on key topics and retaining the reasoning behind them."

Writing is an incredible amplifier of thought, allowing to do long-term deep thinking and synthesis. Therefore, learning by writing is necessary for figuring out what’s really going with wicked problems.

Note, however, that just writing is not sufficient: to fully learn you also have to also poke, tinker, build, and try falsify your predictions and conclusions against ground truth as much as possible.

If you don’t do this and stick with only thinking and writing, your conclusions will not be grounded and you will develop very complex & articulate arguments that nonetheless might have little correspondence with reality.

The Wicked Problem Experience

“This will never end.” Did I just spend two weeks reading terrible papers about wells, iron supplementation and community health workers? Ugh and I’ve only gotten through 10 more charities, so I’m only about ⅓ of the way through the list as a whole. I was supposed to be just writing up what we found, I can’t take a 6-week detour!

The over-ambitious deadline. All right, I’ll sprint and get it done in a week. [1 week later] Well, now I’m 60% way through the whole list. !@#$

“This is garbage.” What am I even doing anyway? I’m reading all this literature on wells and unilaterally deciding that it doesn’t count as “proof of impact” the way that Population Services International’s surveys count as “proof of impact.” I’m the zillionth person to read these papers; why are we creating a website out of these amateur judgments? Who will, or SHOULD, care what I think? …

This is just way out of whack. Every time I try to add enough meat to what we’re doing that it’s worth publishing at all, the timeline expands another 2 months, AND we still aren’t close to having a path to a quality product that will mean something to someone.

For example, when dealing with a wicked problem like described by Holden above, do not make a mistake of simply reading and analyzing the literature.

At the very least, talk with people who study the subject professionally and check your thinking against them.

Better, talk with people who involved with the things you study. In this case, they would be non-profit workers, clinical trial designers & organizers, medical personnel on the ground, and people who are given iron supplementation or who live in villages where the wells are operating.

Ideally, actually go to the countries where these charaties operate and see with your own eyes how the iron supplementation is administered and touch the wells set up in villages without fresh water with your hands, see how people use them, and drink from them.

Three Key Issues I’ve Changed My Mind About

(1) the importance of potential risks from advanced artificial intelligence, particularly the value alignment problem; (2) the potential of many of the ideas and people associated with the effective altruism community; (3) the properties to look for when assessing an idea or intervention, and in particular how much weight to put on metrics and “feedback loops” compared to other properties. My views on these subjects have changed fairly dramatically over the past several years, contributing to a significant shift in how we approach them as an organization.

Why we can’t take expected value estimates literally (even when they’re unbiased)

The approach we oppose: “explicit expected-value” (EEV) decisionmaking

We term the approach this post argues against the “explicit expected-value” (EEV) approach to decisionmaking. It generally involves an argument of the form:

I estimate that each dollar spent on Program P has a value of V [in terms of lives saved, disability-adjusted life-years, social return on investment, or some other metric]. Granted, my estimate is extremely rough and unreliable, and involves geometrically combining multiple unreliable figures – but it’s unbiased, i.e., it seems as likely to be too pessimistic as it is to be too optimistic. Therefore, my estimate V represents the per-dollar expected value of Program P.

I don’t know how good Charity C is at implementing Program P, but even if it wastes 75% of its money or has a 75% chance of failure, its per-dollar expected value is still 25%*V, which is still excellent.

Sam Altman

Startup Advice

Make something people want.

In general, don’t start a startup you’re not willing to work on for ten years.

Growth solves (nearly) all problems.

Hire smart and effective people that are committed to what you’re doing. The last five words there are important.

Hire people that you could describe as animals.

You make what you measure.

A lot of the best ideas seem silly or bad initially—you want an idea at the intersection of “seems like bad idea” and “is good idea”. (It’s important to note you need to be contrarian and right, not simply contrarian.)

Find a mentor that will teach you how to manage.

How to hire

At Stripe, I believe they call this the Sunday test—would you be likely to come into the office on a Sunday because you want to hang out with this person? Liking the people you work with is pretty important to the right kind of company culture. Only a few times have I ever seen a scenario where I didn’t like an otherwise very good candidate. I only made the hire once, and it was a mistake.

Idea Generation

YC once tried an experiment of funding seemingly good founders with no ideas. I think every company in this no-idea track failed. It turns out that good founders have lots of ideas about everything, so if you want to be a founder and can’t get an idea for a company, you should probably work on getting good at idea generation first. …

It’s important to be in the right kind of environment, and around the right kind of people. You want to be around people who have a good feel for the future, will entertain improbable plans, are optimistic, are smart in a creative way, and have a very high idea flux. These sorts of people tend to think without the constraints most people have, not have a lot of filters, and not care too much what other people think. …

Stay away from people who are world-weary and belittle your ambitions. Unfortunately, this is most of the world. But they hold on to the past, and you want to live in the future.

You want to be able to project yourself 20 years into the future, and then think backwards from there. Trust yourself—20 years is a long time; it’s ok if your ideas about it seem pretty radical.

Another way to do this is to think about the most important tectonic shifts happening right now. How is the world changing in fundamental ways? Can you identify a leading edge of change and an opportunity that it unlocks? The mobile phone explosion from 2008-2012 is the most recent significant example of this—we are overdue for another!

Startup advice, briefly

Startups, Role Models, Risk, and Y Combinator

Here’s the secret: everyone starting a startup for the first time is scared, and everyone feels like a bit of an imposter. Even the most successful founders doubt themselves and their startups many times in the early days. But founders improve very quickly.

Founder Depression

Stupid Apps and Changing the World

Facebook, Twitter, reddit, the Internet itself, the iPhone, and on and on and on—most people dismissed these things as incremental or trivial when they first came out.

Successful people

Productivity

My system has three key pillars: “Make sure to get the important shit done”, “Don’t waste time on stupid shit”, and “make a lot of lists”.

I highly recommend using lists. I make lists of what I want to accomplish each year, each month, and each day.

How To Be Successful

The most successful people I know are primarily internally driven; they do what they do to impress themselves and because they feel compelled to make something happen in the world. After you’ve made enough money to buy whatever you want and gotten enough social status that it stops being fun to get more, this is the only force I know of that will continue to drive you to higher levels of performance.

The days are long but the decades are short

Go out of your way to be around smart, interesting, ambitious people. Work for them and hire them (in fact, one of the most satisfying parts of work is forging deep relationships with really good people). Try to spend time with people who are either among the best in the world at what they do or extremely promising but totally unknown. It really is true that you become an average of the people you spend the most time with.

Machine intelligence, part 1

Most machine intelligence development involves a “fitness function”—something the program tries to optimize. At some point, someone will probably try to give a program the fitness function of “survive and reproduce”. Even if not, it will likely be a useful subgoal of many other fitness functions. It worked well for biological life. Unfortunately for us, one thing I learned when I was a student in the Stanford AI lab is that programs often achieve their fitness function in unpredicted ways.

Evolution will continue forward, and if humans are no longer the most-fit species, we may go away. In some sense, this is the system working as designed. But as a human programmed to survive and reproduce, I feel we should fight it.

Machine intelligence, part 2

The US government, and all other governments, should regulate the development of SMI. In an ideal world, regulation would slow down the bad guys and speed up the good guys—it seems like what happens with the first SMI to be developed will be very important.

How to build the future (interview, transcript)

early on, like a lot of other people, I think I was like, “Well, I wanna make money. I wanna, you know, be in the press,” or whatever else and I wouldn’t say that doesn’t motivate me at all, but it’s honestly not a big factor at this point. I just sort of like, there’s a set of things that I think are important and I wanna work on those. I think another point about motivation that people don’t talk about enough is related to burnout. So I was told and I thought for a long time that you get burned out from working too hard. And at least in my own experience, what I found is that burnout actually comesfrom failing and things not working. Momentum is really energizing. The lack of momentum is super draining. And I find that I have infinite energy to work on things that I find interesting and that are working and almost none to work on things that I either don’t find interesting or aren’t working.