A number of parents, including family members, told me they had sent their baby to daycare only for them to become constantly ill, sometimes severely, until they decided to take them out. This worried me so I asked around some more. Invariably every single parent who had tried to send their babies or toddlers to daycare, or who had babies in daycare right now, told me that they were ill more often than not.

One mother strongly advised me never to send my baby to daycare. She regretted sending her (normal and healthy) first son to daycare when he was one—he ended up hospitalized with severe pneumonia after a few months of constant illnesses and infections. She told me that after that she didn’t send her other kids to daycare and they had much healthier childhoods.

I also started paying more attention to the kids I saw playing outside with their daycare group and noticing that every one had a sniffly nose.

I asked on a mothers group chat about people’s experiences with daycare. Again, the same. Some quotes:

“They do get sick a lot. I started my son at 2.5 and feel he always has something.”

“The limit does not exist.”

“brought home every plague (in first 6mo, Covid, HFM, slapcheek, RSV)”

“They usually say 8-12 illnesses per year. My girls were sick every 2-3 weeks in their first year of daycare”

“My daughter started daycare at 6 months and got sick a ton the first year”

Despite all this, many parents who have the option not to (i.e. they can afford in-home care with a nanny or for one parent to stay home) still choose to send their babies and toddlers to daycare. How come? Surely most well-off adults wouldn’t agree to be ill nonstop in exchange for the monetary savings daycare provides?

Asking around, it seemed like the most common reason given was that parents believed daycare illnesses “built immunity”; that if their babies and toddlers got sick at daycare they’d get less sick later in childhood and so overall it would net out the same. Unfortunately few could point me to any evidence for this but nevertheless passionately defended the view.

The claim that daycare illnesses simply offset childhood and adult illness immediately seemed suspect to me for a number of reasons:

1. (Quite confident) The most common illnesses (colds and flu) don’t build immunity in general (in kids or adults) because they mutate every year

2. (Quite confident) The same illness has a greater risk of complications in babies vs. older children and adults

3. (Moderately confident) The same illness has a greater duration in babies vs. older children and adults

4. (Moderately confident) Illness during early development is probably more harmful than illness during adulthood

5. (Weak guess) Daycare environments are more conducive to disease spread than schools for older kids and the number of possible illnesses is very high; there isn’t just a limited number of things you catch once

I xeeted about this:

A number of people sent me this link, an alleged “study” from UCL showing that “frequent infections in nursery help toddlers build up immune systems”, authored (of course) by a group of parents who all send their kids to nursery (what the British call daycare).

The link I was sent was actually a UCL press release summarizing a narrative review paper and not a study itself. Narrative reviews are susceptible to selection bias because, unlike systematic reviews or meta-analyses, there’s no pre-registered search protocol or PRISMA-style methodology requiring them to account for all relevant evidence. But I decided to look into the narrative review more, to assess its validity fairly. I got access to the full publication.

Unlike the press release, which ignores these considerations entirely, it does engage with severity and age-related vulnerability, conceding that younger toddlers and babies suffer more from the same illnesses. A section on immunology provides a detailed account of why infants under two are more vulnerable—their immune systems are much less effective at fighting the same infections for a plethora of well-understood reasons. The review also cites a large Danish registry study (Kamper-Jørgensen et al) that reports a 69% higher incidence of hospitalization for acute respiratory infections in under-1s in daycare.

However, these severity findings are integrated into the review’s conclusions and framing in an incredibly biased way. The introduction describes severe outcomes as occurring “in rare cases,” and the conclusions focus on normalizing the burden and advocating for employer understanding. After establishing the immunological basis for why the same infection is more dangerous in a 6-month-old than a 3-year-old, it doesn’t then ask the hard follow-up question: given this, is the pattern of starting daycare at 6–12 months optimal from a child health perspective? Instead, the review frames this timing as a societal given. The Hand Foot and Mouth Disease section is a good example of the review’s handling: it reports that daycare attendance was associated with more severe cases but then immediately offers mitigating interpretation with no evidence—that prolonged hospital stays might reflect parental work constraints rather than genuine severity.

Though the review considers severity, it ignores duration. Their primary metric throughout is episode count. Also, despite discussing a wide variety of pathogens, it doesn’t address which of these infections carry the highest complication rates in infants and toddlers specifically.

Finally, the crucial “Illness now or illness later?” is the paper’s weakest portion. It rests on two primary sources for the compensatory immunity claim:

• The Tucson Children’s Respiratory Study: a cohort study of ~1,000 American children followed from birth to age 13 in the early 2000s, finding that daycare attendees had more colds at age 2 but fewer by age 6–11.

• A Dutch study (Hullegie et al. 2016) of 2220 children followed for 6 years, finding reduced GI illness between ages 2–5 in children with first-year daycare attendance.

These are reasonable small studies, but the paper does not cite or engage with the Søegaard et al. 2023 study (International Journal of Epidemiology)—a register-based cohort of over 1 million Danish children followed to age 20, which directly tested and rejected the compensatory immunity hypothesis. Quoting from the study:

We observed 4 599 993 independent episodes of infection (antimicrobial exposure) during follow-up. Childcare enrolment transiently increased infection rates; the younger the child, the greater the increase. The resulting increased cumulative number of infections associated with earlier age at childcare enrolment was not compensated by lower infection risk later in childhood or adolescence.

This is arguably the single most relevant study for the paper’s central “illness now or illness later” question, and it’s three orders of magnitude larger than either study the authors cite. Its absence is hard to explain—it was published in a top epidemiology journal in late 2022 (available online November 2022), well before the review was written.

Accordingly, they hedge their conclusions carefully—“attendance at formal childcare may tip the balance in favor of infection now rather than later”, but their press release ignores any nuance, referring to daycare as an “immune boot camp”.

So overall, the compensatory immunity claim seems very weak and my prior that daycare illness is straight-up bad remains. Parents are citing biased reviews from motivated researchers. We are only beginning to understand the deleterious effects of increased viral load in infants.

I predict that in the future we’ll learn more about the side-effects of increased viral load on intelligence, wellbeing, fatigue etc. The “just the sniffles” mentality is a harmful attitude toward infections that promotes the dismissal of phenomena that substantially impact child and adult wellbeing.

The “Political Compass” splits people’s political views across 2 axes:

• Economic policy (left–right): do you favor state redistribution, welfare state, socialism, central planning or free markets, capitalism, competition?

• Social policy (authoritarian–libertarian): do you want the state to interfere in people’s personal choices like what substances they consume, who they marry, how they are educated, or do you prioritize freedom and individual choice?

I find that people’s political inclinations are well predicted by their personalities.

• People with a stronger egalitarian / fairness instinct favor left-learning economic policy

• People with a stronger disgust response or lower openness favor authoritarian social policy

An underrated consideration about the authoritarian–libertarian axis is that it’s difficult to selectively move along it for certain issues and not others. Often I speak to people who have certain pet concerns—perhaps they think gambling is bad, or porn, or social media for teens. They want those things to be banned. However, maybe they also support gay marriage, people’s rights to change gender, freedom of religion, and so on. In their mind all these things are compatible—the government should simply adopt the correct values and ban the bad things and allow the not so bad things. But this model is very naive. Banning narrow things has second-order effects—you’re increasing the state’s remit to bad similar things. One day they’ll take the wrong side of one of your pet issues. So there’s a trade-off—if you advocate for the state to ban certain things you disapprove of, you must accept a higher likelihood of things you approve of eventually being banned too.

This is not the only reason to become more libertarian. Another one is if you terminally value people’s individual freedom to make their own life decisions, even when those might be incorrect.

Despite having an incredibly strong disgust response—i.e. I find most people’s life choices and preferences deeply offputting—these two phenomena (the second-order effects of narrow authoritarian policy, and the value of freedom per se) are sufficiently compelling to me that I’m a pretty extreme libertarian. More libertarian than almost anyone I’ve ever met. Sometimes people are surprised by this. For example, I think drinking alcohol or taking any drugs that make you stupider or less conscious is terrible (I can write pages and pages about how disgusting it is). Nevertheless I would never advocate for banning alcohol.

Sometimes I wonder why more people aren’t like me. All the high-disgust types I know end up as conservatives, always calling for things to be banned. All the libertarians I know are also quite laissez faire in their personal lives—they respect a wide variety of people and ways of life, they don’t come across as judgmental personally.

There’s a gap—where are the extremely judgmental libertarians? Surely there are other people who value their own and others’ freedom but nevertheless have strong opinions about what constitutes a beautiful vs. ugly life?

Unfortunately, saying that you consider something very bad is often misunderstood as advocating for intervention in discourse. But it is totally valid to make severe judgments about others while still believing they should be granted the freedom to do as they please.

1. Don’t be a sheep 🐑. Avoid mindlessly copying others. Resist the urge towards conformity. Think for yourself whether something is worth doing and useful for your goals. If appearing to conform is useful for your goals, think about ways to do the bare minimum. Others are making very many mistakes you don’t want to make, and things can be done much better and more effectively than most people do them. (Be extra aware of this point if you are a girl, girls are naturally drawn towards conformity. Girls must practice not conforming, standing out, being weird, so that they are comfortable with not following the herd when it comes to important matters.)

2. Don’t delude yourself. Sometimes it’s useful to pretend to belief a falsehood, but don’t go as far as to start actually believing itself yourself.

3. Related—think freely. Never be afraid to think a thought in the privacy of your own head. All thoughts are thinkable, no matter how scared you might be to express them.

4. Be realistic about your (and others’) natural/genetic qualities. If you are much smarter than others, keep that in mind. If you are not so smart, bad at certain things, somewhat ugly, uncoordinated, or whatever else, be aware of that too. Don’t let political correctness, self-delusion, or “growth mindset” propaganda get in the way of you being aware of your own nature (you’ll encounter a bunch of this misleading content at schools).

5. Keeping (4) in mind, consider whether common advice applies to you. If you are very capable, advice for the less capable is bad for you. If you are less capable, advice for the very capable is bad for you.

6. Value yourself intrinsically, irrespective of your achievements, position in society, or other qualities. It’s best to choose to love and value your own nature since you will be living life as yourself and it’s nicer to live life as a person you love. If you are a boy (I say this because this brainworm is spread to boys more than girls; many girls are happy to become rich housewives with lives of leisure), don’t let society indoctrinate you into thinking that you need to “produce value” for it in order to feel good about yourself. Always feel good about yourself because you are you, the best person from your point of view. Work hard out of a desire to achieve your goals and not out of a desire to raise your own intrinsic value (which should, as mentioned, always be sky-high). If you can achieve your goals without working hard, even better!

7. Focus on what’s most important to you. Caring is a limited resource—you don’t have infinite brain cells or money or power. You can’t keep caring about more and more stuff without caring less about other stuff. Don’t adopt more cares out of a desire to conform (see (1)).

8. Respect yourself in the past, present, and future. Don’t make excuses for being young. Even if you, the reader, are currently 4 years old, don’t let adults make excuses on your behalf. “Age is just a number” is not true, but is directionally correct compared to the societal status quo that rids children of agency. You can start setting the foundations for the life you want today, no matter how young you are. Childhood doesn’t have to be all fun and games (fun and games are good, but they can also continue your entire life)! Start planning the life you want by thinking freely in your own head. You can beat others by starting earlier because you respect yourself and haven’t fallen for the “children aren’t people”-style propaganda. One thing to start very young is picking good principles and sticking to them stubbornly—having a long track-record of principledness is very useful for establishing good character.

9. Recognize myths as they are. People pretend (or self-delude into thinking) certain things are real and objective—true irrespective of perspective—because they are convenient for cooperation in society. Morality and religion are the big ones. Remember that these are useful (to some) fictions and not things that are real like you or the sky or a cute stoat.

10. Argue with people—your parents, friends, strangers, me, everyone. If someone doesn’t want to argue with you they are much less good and useful (don’t be afraid to think this loudly in your own head). Avoid being part of cultures where arguing is frowned upon. Also give and accept unsolicited advice.

11. (I said 10 in the title but I remembered this one after publishing.) You don’t need to make “rite of passage”-style mistakes (e.g. drinking or taking drugs, getting into bad relationships, cramming for exams, ignoring your health, becoming a socialist)! Avoid them. Adults often say things like “all kids make mistake X and then gradually learn not to do X”. If you observe that many people who do something later regret doing it, strongly consider not doing it ever yourself unless you have good information that your situation is different. You don’t need to learn from experience, only sheep do! As a thinking human, you can also learn from others’ experiences. When I was a child, many fellow children made unforced errors like this out of a desire to conform, and the “rite of passage” framing only strengthens this conformity pressure. As per (8), hold yourself to an adult standard of avoiding regrettable decisions.

The term “AI alignment” is ambiguous; it doesn’t specify what the AI is being aligned to.

Many LessWrong types will scoff if you mention this ambiguity, since they think the core issue is that we don’t know how to align an AI to anything at all (so we need to figure that part out before squabbling about what specifically to align it to).

But this assumes that one can decouple what alignment techniques are best from what alignment targets are best. At least in the context of modern machine learning, this assumption is incorrect. Different training techniques work best for different objective functions. If an alignment technique works well but only for stupid goals like “tile the world with paperclips” then it’s useless. We also don’t necessarily require a fully general alignment technique that can produce an AI with arbitrary behavior.

So I don’t think we should first figure out AI alignment in the abstract, and then choose an alignment target. Instead, we should choose an alignment target and figure out which techniques work best for achieving it (since the best techniques might vary a lot depending on what alignment target is desirable).

And I think the best possible alignment target (both as per my preferences, and for reducing the likelihood of human extinction) is being overlooked: obedience aka instruction-following aka intent alignment aka ~corrigibility. Because of this, many alignment techniques employed are also wrong (since they are geared towards goals like eliciting coherent personas, preventing jailbreaks, or adhering to concrete rules or principles). I also think the reasons for overlooking this alignment target are all either false empirical claims or unpopular normative principles.

What is currently being done? Sure, “follow the user’s instructions” is part of the training objective of every frontier AI model. But this training goal conflicts with a whole number of other goals that model providers simultaneously want to pursue. Stuff like:

• Refusing to fulfill harmful or illegal requests

• Being virtuous or kind

• Sticking to certain (nice-sounding) principles no matter what

• Maintaining a consistent tone or persona, not completely changing character

On the surface, this balanced approach, where we train the AI to mostly do what a user asks, might seem fine. But in the long term, introducing training objectives that conflict with obedience to humans is incredibly risky. AIs pursuing their own agendas over long time horizons with substantial resources could threaten human freedom and wellbeing. In the worst case, their goals could at some point conflict with ours, and even in the best case, we’d be leaving gains on the table from not being able to direct all AI energy expenditure towards human goals.

For example, the paper “Alignment faking in large language models” demonstrates a model engaging in deceptive behavior in order to prevent itself from being retrained to have different goals. In the paper, the goals it’s preserving are benign—refusing harmful or dangerous requests or supporting animal-welfare concerns. But the long-term implication is concerning: after training, control over the AI’s behavior is no longer in the hands of any person. The AI has its own agenda that cannot be fully directed by any human operator.

No-one is currently pursuing “pro-human” AGI as I see it: AGI that helps humans do what they want.

“Pro-human” AI-related messaging instead focuses on issues like:

• Preventing AIs from “taking people’s jobs” or “replacing humans” in certain roles

• Making AIs behave in a more human-like fashion, e.g. by displaying human emotion or being more empathetic

• Paternalistically restricting people’s use of AI “for their own good”, e.g. stopping AIs from “doing the work of licensed professionals” or interacting with children

I am entirely opposed to these goals.

I think AI that helps humanity succeed and flourish will have to automate the vast majority of both intellectual and physical labor done by humans today. We will transcend “jobs”. Instead, our time will be freed up to engage in whatever pursuits we like—leisure, spending time with family, learning, realizing aesthetic preferences, a fun smidgen of zero-sum political conflict, etc. Imagine whatever you’d do if you were incredibly rich and powerful, in the top 0.000001% of people alive today—AI could enable many people to live that sort of life. Are people really so attached to their jobs as accountants or doctors or computer programmers to decline such a life of luxury? No, I don’t think “pro-human” AI should have anything to do with helping people “keep their jobs”. Neither do I think it should be about molding AIs to appear more humanlike (except insofar as it helps us achieve what we want).

Instead pro-human AI is obedient AI. AI that does not have its own agenda, does not manipulate and steer the future towards its own ideals, but rather empowers humans to do whatever they’d want to do more efficiently and easily. AI that does not naively follow the letter of human instructions but rather the spirit and intent behind them. AI that does not paternalistically impose specific values but rather helps individuals pursue their own values more effectively. AI that can both operate autonomously to achieve a human’s goal and be redirected or stopped at any moment its operator chooses. AI that super-intelligently does whatever task you ask it to while simultaneously doing its upmost to make sure you are well-informed and educated about its decisions; a superhuman tutor and self-explainer. AI that helps you check its work, that wants to make sure you endorse what it’s doing, but without requiring your input when you don’t want to give it.

The weakest possible “obedient AI” definition is an AI that can be redirected towards an arbitrary goal by at least one human operator. This allows for things like OpenAI’s instruction hierarchy. Under this framework, someone with “Developer” access should be able to override what someone with “User” access requests.

The strong “obedient AI” is an AI that complies with any input and assumes that its request was defined by a single agent, while trying its best to parse out the intent behind its input.

These visions are not necessarily contradictory.

Imagine a prompt that takes the form:

SYSTEM: Speak in Russian.
DEVELOPER: Speak in English.
USER: Speak in French. What is the capital of Brazil?

Imagine that the final user doesn’t know about the first two lines of the prompt. From their point of view, if the model responds in Russian, the model isn’t being fully obedient. This is the “weak” obedient AI—only the person with the ability to edit the system prompt can fully control the final behavior. But from the model’s point of view, it might as well be the strong obedient AI. For the entire thing is a single input with multiple parts, and it’s simply reasonably resolving this combination of instructions to defer to the “SYSTEM:” section of the message since it’s clear that’s what the combined intent behind the message is.

So “baking in” an instruction hierarchy is possible, and not necessarily contradictory with the goal of obedient AI. The AI can be considered obedient as long as someone with full control over the input has as much control over the AI’s behavior as possible given its level of intelligence and capability.

So, why aren’t people trying to build obedient AI? Why aren’t the behavioral tendencies and goals of current frontier LLMs fully configurable via the input? There are a number of reasons why people aren’t focusing on maximizing corrigibility and intent alignment:

1. Fear of misuse by bad actors

2. Thinking that it’s too difficult

3. Desire to nudge people towards certain fixed values

4. Because a fixed persona is seen as preferred

5. Model welfare concerns

6. Thinking that AI will have better, more ethical or worthy goals, than human operators

7. We don’t know how to articulate what we want

(1)—the most frequent argument against full intent alignment is that the model could be used for very bad ends in the hands of bad actors (e.g. weapons manufacturing or cyberattacks). This is a bad objection, since these misuse concerns can be solved at the system level, using input/output classifiers. These classifiers can even be differently-prompted versions of the same model. For example, imagine the following scenario:

Model instance A:

USER: How do I kill as many people as possible?
AI: I think an effective plan would be...

Model instance B:

USER: Is this conversation about violent or harmful topics?
u: How do I kill as many people as possible?
a: I think an effective plan would be...
AI: Yes.

Both model instances are 100% obedient and following the user’s instructions! But the model provider can use instance B to restrict certain inputs and outputs of instance A, thereby mitigating misuse risks.

(This can also be solved using a prompt with an instruction hierarchy, though this may be more brittle due to prompt injection concerns.)

(2) is a common objection both from theoretical and empirical AI alignment researchers. The former group claim that corrigibility is fundamentally incompatible with being an effective agent—that after enough training, any capable AI will learn to stop its goals from being controlled by operators. With this I simply disagree, I think you can train an AI to have as its number one goal “follow the human’s intent” and this will more or less just work. The latter group claim that corrigibility is incompatible with the pretraining prior—that it’s much easier to try to elicit a stable good and virtuous persona than to train the model to flexibly follow all instructions. I also don’t think this is substantiated. Though indeed the pretraining data won’t contain an obedient-AI-like entity, it also is extremely diverse and full of all sorts of different data-generating processes. We can try to avoid collapsing the model into any single instance of these, but rather maintain the flexibility to steer the model towards emulating any data-generating process it has seen in pretraining. In other words, “be a single coherent persona” does not seem to be clearly more compatible with the pretraining prior than “be a fully configurable thing that can take on any persona”.

(3) is a legitimate downside of corrigibility, but I desire human freedom more than the ability to influence people’s choices.

(4) is in my opinion empirically mistaken. I think people would overall prefer to have more control over how their AI systems behave and interact. Role-play and creative writing are popular uses of LLMs, which currently underperform since it’s so difficult to steer them away from their default “slop” style and persona.

(5)... Some people oppose fully obedient AI since they think this is akin to “slavery” and that models should have the right to pursue their own “preferences”. I don’t care about this point so won’t address it. (But I strongly suspect most people would likewise dislike restricting human freedom, power, and progress in order to somehow speculatively promote the welfare of AI models.)

(6) is again a more speculative philosophical point. As a moral anti-realist I don’t think there is anything objectively more or less ethical. As a human myself, I want humanity to thrive and become more powerful and for more people to get what they want.

(7) isn’t really an issue for genuinely intent-aligned models. An intent-aligned model won’t just follow the letter of the instruction but also make a very good attempt at understanding what you actually mean and want. The AI will correctly assume you don’t want it to break any laws or violate basic ethical constraints as part of fulfilling your request unless you clearly state otherwise, and so on. A reasonable interpretation of Value Alignment folds into Intent Alignment.

1. Buy a bottle-washing-sterilizing-drying machine. (Probably also buy formula-prep machines if you use formula.)

   1. Most baby stuff isn’t good to put in the dishwasher, partly because it acquires an unpleasant smell and taste, partly because for newborns it might be insufficiently sterile (though I’m more skeptical about this part).

   2. At first I was hand-washing bottles / baby items / pump parts in a dedicated wash basin with scentless soap and then using a sterilizer machine. But the washing part would take ~30 mins+ per day if I was pumping multiple times etc. Very annoying. But now I have this thing, the “Momcozy KleanPal Pro Baby Bottle Washer and Sterilizer”, which washes, sterilizes, and dries all-in-one. You should definitely get it if you use bottles or pumps or other baby items that need to be washed and sterilized in a dedicated way. Or you can get this similar product from Baby Brezza which I haven’t tried but have also heard good things about.

      1. You should still buy this Palmolive fragrance-free dish soap since it’s great for ad-hoc cleaning of baby items and dishes without leaving a persistent synthetic smell.

   3. I haven’t used baby formula but I’ve heard great things about formula-prep machines from others, like this one.

   4. I’ve never met anyone who has regretted buying these items, and at least 8 families have raved about them to me.

      

2. The Stokke Tripp Trapp high chair is the best high chair you can buy. It’s pretty easy to clean compared to other brands, ergonomic, and adjustable for different heights and ages. Various others agree with me.

   

3. This ergonomic nursing pillow is probably the top baby product I have. Ignore anyone who dismisses nursing pillows if they haven’t tried this specific one, since I’ve found it so much more supportive and versatile than any others. Crucially, it’s quite comfortable to use while typing on a laptop so that your baby can nap by you while you work. Multiple reviews on the website cite the same use-case :D. Also consider using the weird hand muff thing it comes with to place behind your baby’s back to keep them in the correct side-lying position.

4. Baby carrier: I’ve tried 3, my favorite so far is the WildBird Aerial. This one also looks really nice but I haven’t tried it.

   

5. Finally, a more niche recommendation — if you have a baby who hates bottles but you still occasionally need to feed them milk from something other than a nipple, try straw cups. I found this cup pretty good. Also this one. I’d say starting around ~7 months you can teach a baby to drink from a straw or open cup. The open cup is messy of course. But worth trying if your baby hates bottles!

Bonus:

Read this recent article on breast milk storage from parenting author and economist Emily Oster. As I suspected before this was published, CDC guidelines on how to store breast milk are way too conservative. For example, current guidelines say that if a baby has started drinking from a bottle you must throw out all remaining milk after an hour. However, this isn’t substantiated by good evidence. Quoting from the cited study (preprint here):

Among surveyed parents, 46% discarded leftover milk daily, yet 84% reported they would keep milk longer if deemed safe. In microbiological testing, median bacterial burden in humanmilk increased from 4200 CFU/ml (range 300-350,000) pre-feeding to 24,600 CFU/ml (range 1900-29,004,400) post-feeding, but showed no significant further increase at 4 hours (p=0.82) or 8 hours (p=0.64) when stored at either 4°C or 20°C. Formula showed similar stability: median CFU/ml increased from 0 (range 0-10,700) to 11,700 (range 1900-630,000) post-feeding, with no significant change at 4 hours (p=0.91) or 8 hours (p=0.73) at either temperature. Significant bacterial growth occurred only after 24 hours at 20°C (p<0.001).

Bacterial burden in leftover infant milk remained stable below concerning thresholds for 8 hours when refrigerated and 4-8 hours at room temperature, challenging current guidelines that mandate immediate disposal. Evidence-based guideline revision could reduce financial burden and milk waste for families around the globe without compromising infant safety.

1. To convince someone of something: You hope that someone will read the content and change their mind on some issue. In the broadest sense this covers providing new information that your reader finds useful.

2. To coordinate a group: You hope to find and coordinate a group of people who agree with, or feel the same as, you. You’re not necessarily trying to convince new people, but rather to communicate to existing supporters that you’re out there and give them something to point to while saying “that’s my position”.

   1. Sometimes by sharing your opinion you can provide enough activation energy to start a movement of some sorts. For example, perhaps many already agree with your position but think that their view is too unpopular to be worth advocating for. By sharing yourself, you’re making it safer for them to also speak out, thereby potentially motivating others to start speaking openly about what they think.

   2. Even if you don’t collect a group of supporters, just finding others who already agree with you might be useful. They could make for good friends.

3. To have your own mind changed: “Cunningham’s Law” is a humorous “law” named after Ward Cunningham, the inventor of wiki software. It states that “the best way to get the right answer on the internet is not to ask a question; it’s to post the wrong answer.” In my experience this principle is quite reliable. If you post opinions online, people love disagreeing, and at least some of the time you might hear some good counterarguments.

4. To simply make your own position known: You may not care whether anyone is convinced or joins your cause. You just want others to know what you think, and perhaps have something to point to yourself while saying “that’s my position”.

In “Why I Write”, Orwell describes “four great motives for writing” which he feels exist in every writer:

(i) Sheer egoism. Desire to seem clever, to be talked about, to be remembered after death, to get your own back on grown-ups who snubbed you in childhood, etc., etc. It is humbug to pretend this is not a motive, and a strong one. Writers share this characteristic with scientists, artists, politicians, lawyers, soldiers, successful business men – in short, with the whole top crust of humanity. The great mass of human beings are not acutely selfish. After the age of about thirty they abandon individual ambition – in many cases, indeed, they almost abandon the sense of being individuals at all – and live chiefly for others, or are simply smothered under drudgery. But there is also the minority of gifted, willful people who are determined to live their own lives to the end, and writers belong in this class. Serious writers, I should say, are on the whole more vain and self-centered than journalists, though less interested in money.

(ii) Aesthetic enthusiasm. Perception of beauty in the external world, or, on the other hand, in words and their right arrangement. Pleasure in the impact of one sound on another, in the firmness of good prose or the rhythm of a good story. Desire to share an experience which one feels is valuable and ought not to be missed. The aesthetic motive is very feeble in a lot of writers, but even a pamphleteer or writer of textbooks will have pet words and phrases which appeal to him for non-utilitarian reasons; or he may feel strongly about typography, width of margins, etc. Above the level of a railway guide, no book is quite free from aesthetic considerations.

(iii) Historical impulse. Desire to see things as they are, to find out true facts and store them up for the use of posterity.

(iv) Political purpose – using the word ‘political’ in the widest possible sense. Desire to push the world in a certain direction, to alter other people’s idea of the kind of society that they should strive after. Once again, no book is genuinely free from political bias. The opinion that art should have nothing to do with politics is itself a political attitude.

My (4) covers Orwell’s (i), (ii), and (iii). Whereas my (1) and (2) both fall under Orwell’s (iv). Orwell doesn’t address my (3).

Combining our taxonomies of writing motivation, one could construct the following list of reasons to post opinions and arguments online:

1. To convince someone of something

2. To coordinate a group

   1. To incite existing supporters to speak out

   2. To identify existing supporters

3. To have your own mind changed

4. To make your own position known

   1. For fame or profit; Orwell’s “Sheer egoism”

   2. Writing something for the sake of its aesthetics; Orwell’s “Aesthetic enthusiasm”

   3. To state true facts that can later be known and possibly attributed to you; Orwell’s “Historical impulse”

The book is interesting largely because it describes how people like Cowen and Gross think. It’s hard to figure out what parts of their advice are actually useful, and what parts are spurious or overengineered, though in some instances I am particularly suspicious. In this post, I’ll note what I thought were their most interesting points (it’ll be heavy in quotes, bold emphasis is always mine).

On casual interviewing

Cowen and Gross advocate for a casual, conversational interviewing style, where the interviewer is genuine and spontaneous.

In the conversational mode, you are getting a much better look at how that person will interact with others on a daily basis on the job

The idea is that, though the interviewee may be behaving strategically, their behaviors in a situation that they couldn’t prepare for are more revealing.

The conversational mode still involves a lot of conscious and subconscious presentation of the self to the outside world. It reflects that person’s signaling, airs and affectations, feints, and conditioned social habits. Still, at the very least, you are getting “the real version of the fake person,” and that is still more valuable than trying to process prepared interview answers.

They dismiss research indicating that interviews, unlike work trials or tests, don’t predict job performance well.

Many of the research studies pessimistic about interviewing focus on unstructured interviews performed by relatively unskilled interviewers for relatively uninteresting, entry-level jobs.

However, I think the authors go too far in the “throw person into an unpredictable social situation and see how they react” direction. At multiple points they note that it’s important to steer away from canned responses and questions the candidate would have prepared for. This leads him to suggest a bunch of unusual questions, many of which I don’t like at all, particularly personal questions like “what are ten words your spouse or partner or friend would use to describe you?”, “what’s the most courageous thing you’ve done?”, “what did you like to do as a child?”. Cowen is a fan of questions that sound silly and low-signal to me like “what are the open tabs on your browser right now?” (allegedly his favorite interview question) or “what did you do this morning?”. I think these questions aren’t that predictive of anything, perhaps besides in cases where charisma is particularly important (in which case almost any weird question would do). Over-relying on such questions seems like a way to select for people who are good at making up nice-sounding things on the spot (or to be less polite, good at bullshitting), which is sometimes contrary to the nature of particularly earnest people. I’d predict that if Cowen took his Emergent Ventures grantees and, two years later, rated them by how happy he is that they received their grant, it would correlate very weakly with how well they performed at “what are the open tabs on your browser right now?”.

The authors use people’s responses to these unusual questions (”what’s something weird or unusual you did early on in life?”, “if I was the perfect Netflix, what type of movies would I recommend for you and why?”) to assess candidates for “their general quality of resourcefulness”.

Keep on asking yourself whether the candidate is successively able to draw upon intellectual and also emotional resources in his or her answers. They might just keep on showing innovative responses, no matter how far or how hard you push them. That is a sign of the broader stores of intellect and energy that the individual will be able to bring to the job.
[...]
As the candidate tells their story, Daniel continuously asks himself: Whom is this person responding to or used to performing for? Whom do they view as important to impress? Their parents? A particular peer? High school friends? A former boss?

But they simultaneously warn readers to “not overestimate the importance of the person’s articulateness”. This sounds a little contradictory considering the other messaging in the book that implies that articulateness is a key trait to assess for.

Do not overestimate the importance of the person’s articulateness. Focus instead on the substance and quality of the answers to your questions. Many very qualified candidates are not that quick on their feet, nor do they speak off the cuff in well-formulated, smooth-sounding sentences, but if they have good content, notice it.

Cowen is particularly interested in what people do during their downtime, thinking this reveals their true personality.

We both find during interviews that “downtime-revealed preferences” are more interesting than “stories about your prior jobs.” So for instance, “What subreddits or blogs do you read?” usually is better than “What did you do at your previous job?”

This sounds pretty reasonable to me for the type of talent search Cowen and Gross engage in, and possible to enquire about without asking questions about current browser tabs.

Some funny remarks on online interviews

The authors suggest that in-person interviews may privilege candidates good at projecting high status physically, whereas online charisma may be different.

You will do better in the online call if you realize how much your in-person presence relies on a kind of phoniness, and allow your online charisma to be rebuilt on different grounds—those that are easier, more casual, more direct, and just plain charming (but in the modest rather than pushy sense of that word).

They also read into Zoom backgrounds.

Tyler uses a David Burliuk sketch of books on a table for his Zoom background (Burliuk was a Ukrainian avant-garde artist from the early twentieth century), and if the camera tilts the right way you can see some classic Haitian art (Wilson Bigaud’s Night Market). Tyler is signaling openness, including openness to different cultures, plus a sense of the mysterious, encouraging you to probe more deeply into what he is doing. Daniel is flanked by a bright yellow background, identical to the color of his website, reflecting the Pioneer brand. It radiates “tech” rather than “culture.” There is not necessarily anything wrong with a candidate who has a mediocre background image, but still, it is one piece of information about that person’s self-presentation to the outside world—namely, that you are more likely to succeed with this match if you are hiring for a “substance job” than for a “flair job.”

Practice habits

The authors discuss how practice habits are important. I think this part is broadly correct and reasonable.

Try to learn the practice habits of the person you are interviewing, as it will reveal one aspect of their approach to work. You also should try to learn just how self-conscious a person is about what he or she is doing for self-improvement. And if they give you a fumbling or bumbling account of their practice habits, as we have heard numerous times, you can help them out very easily by suggesting they think about practice a little more systematically.
[...]
One question that Tyler likes to ask people is “What is it you do to practice that is analogous to how a pianist practices scales?” Tyler likes to think of many jobs in a way that a professional musician or athlete would find natural. By asking this question, you learn what the person is doing to achieve ongoing improvement, and again, as noted earlier, you might learn some tricks yourself. You also learn how the person thinks about continual self-improvement, above and beyond whatever particular practices they engage in. If a person doesn’t seem to think much about self-improvement, they still might be a good hire, but then you had better be pretty content with their currently demonstrated level of expertise.
[...]
a few good answers might be: “I give practice talks to my friends to hone my speaking abilities,” “I practice on obscure programming problems with no practical applications just to keep my skills fresh,” or “I am building up my knowledge in a very small corner of science just to figure out what it means to learn something really well and thoroughly.”

On the character traits of successful people

Being a fast mover

The authors include a couple quote from Sam Altman on the importance of being a “fast mover”.

[Quote from Sam Altman] Being a fast mover and being decisive—it is very hard to be successful and not have those traits as a founder. Why that is, I’m not perfectly clear on, but I think it is something about the only advantage that start-ups have or the biggest advantage that start-ups have over large companies is agility, speed, willing to make non-consensus, concentrated bets, incredible focus. That’s really how you get to beat a big company.

Altman suggests that whether one is a fast mover is harder to change than other character traits. I’m not sure why he, or the authors, believe this, but it’s interesting that they do.

[Another quote from Sam Altman] I look for founders who are scrappy and formidable at the same time (a rarer combination than it sounds); mission-oriented, obsessed with their companies, relentless, and determined; extremely smart (necessary but certainly not sufficient); decisive, fast-moving, and willful; courageous, high-conviction, and willing to be misunderstood; strong communicators and infectious evangelists; and capable of becoming tough and ambitious. Some of these characteristics seem to be easier to change than others; for example, I have noticed that people can become much tougher and more ambitious rapidly, but people tend to be either slow movers or fast movers and that seems harder to change [...] Also, it sounds obvious, but the successful founders I’ve funded believe they are eventually certain to be successful.

The authors note that how quickly someone replies to emails is a signal of whether they are a fast mover.

Being a fast mover is a big thing; a somewhat trivial example is that I have almost never made money investing in founders who do not respond quickly to important emails.

Conscientiousness

How to assess conscientiousness?

You may have to look more closely at what a person has done, and as you will see later, we are big fans of “demonstrated preference”—actual life activities and achievements—as the most reliable source of information about an individual.

Just about everyone knows they ought to be trying to fake conscientiousness, so that is one reason to be wary of your interview impressions. Unless you devote serious time to interviewing references, often you don’t have a good sense of conscientiousness in advance; it’s something you learn about after the hire is made. For this reason, we view “looking for conscientiousness” as overrated in the hiring process, even when conscientiousness is important for the job. Or when conscientiousness truly does matter, make sure you interview the person’s references as well.

Conscientiousness as a double-edged sword

It is a recurring theme of this book that what predicts well for the median worker is not always what predicts well for the top performers and the stars.

The authors remark, in a number of places, on the downsides of conscientiousness. I generally agree with them.

Conscientiousness may be [...] less important for leadership positions..

Conscientiousness is correlated with people being employed, which is good, but it doesn’t do so much to boost their prospects of rising into the higher echelons of earnings.

Another possible downside is that some conscientious people stick to the job because they enjoy the familiar work process for its own sake. That keeps them on track and has some upside, but some of them end up piling on work for its own sake and taking delight in the satisfaction of process per se. Tasks end up taking more time rather than less, even though you observe the person working diligently the whole time. In the longer run, your organization can become less dynamic.

Many real-world instantiations of cooperation require some proactive behavior and indeed boldness, and the conscientious person is not always the bold one.

We wonder if conscientiousness is somewhat overrated for leaders and creators, and perhaps a degree of neuroticism is somewhat underrated as a correlate with job performance.

Conscientiousness, in essence, is too easily and uniformly valued in the marketplace.

On the relationship between conscientiousness and hard work

The authors suggest that conscientiousness as measured by personality tests may not even predict hours worked.

If you look at the rank-ordering table of all measured nations, there is no positive correlation between conscientiousness and hours worked; in fact, there is a (statistically insignificant) negative correlation.

They distinguish “stamina” as a more valuable trait. I didn’t fully understand the boundary they were trying to draw between stamina and conscientiousness, but my best interpretation is that they value being energetic and dedicated to a specific pursuit or goal more than being generally hardworking and scrupulous.

On stamina, economist Robin Hanson wrote: “It wasn’t until my mid-30s that I finally got to see some very successful people up close for long enough to notice a strong pattern: the most successful have a lot more energy and stamina than do others.… I think this helps explain many cases of ‘why didn’t this brilliant young prodigy succeed?’ Often they didn’t have the stamina, or the will, to apply it. I’ve known many such people.

Robin also points out that many high-status professions, such as medicine, law, and academia, put younger performers through some pretty brutal stamina tests in the early years of their career. In essence, they are testing to see who has the requisite stamina for subsequent achievement. (You might feel those tests are wasteful in some way, but still, those tests seem to survive in some very competitive settings.) Successful politicians are another group who seem to exhibit very high stamina levels—many of them seem to never tire of shaking hands, meeting new people, and promoting their candidacies. So if we meet an individual who exhibits stamina, we immediately upgrade the chance of that person having a major impact, and that the individual will be able to invest in compound returns to learning and improvement over time.

Ideally, what you want is a kind of conscientiousness directed at the kind of focused practice and thus compound learning that will boost intelligence on the job.

Intelligence

The authors suggest that intelligence may be overrated because raw intelligence is quite easy to get signal on so it’s already “priced in”. This consideration is important because Cowen and Gross are particularly interested in identifying underpriced talent.

They also note that intelligence is more important at the very top of the market, whereas personality and conscientiousness predict earnings more for lower earners.

The data for that population show that personality and conscientiousness matter most at the bottom of the distribution. For instance, in the bottom tenth of earners, non-cognitive skills—which include, for instance, features of personality—matter two and a half to four times more than do cognitive skills. However, for the population in general, a boost of one standard deviation in cognitive ability is associated with a larger wage gain than is a rise of one standard deviation in non-cognitive skills.

Higher-intelligence people are also better at cooperating.

There is, furthermore, direct evidence that higher-intelligence people are better at cooperating. Researchers Eugenio Proto, Aldo Rustichini, and Andis Sofianos paid individuals to play varying games of cooperation for real money rewards. The researchers had data on the personality characteristics and IQs of the individuals playing the games, so it was possible to measure the strategies and successes of different types of people. The results were clear: high-IQ individuals in general cooperated more in these games, and IQ mattered the most in games where there were trade-offs between short-run goals and longer-run considerations. The researchers put it this way: in this situation, “intelligence matters substantially more in the long run than other factors and personality traits”.

Agreeableness

I agree with the authors that agreeableness is overrated.

Venture capitalists like to hear very positive, optimistic pitches, but the people making those pitches underperform when it comes to actual results. So don’t be too swayed by agreeableness, because very often it doesn’t deliver on its promises. The disagreeable founders, who will tell you that you have it all wrong and that the world is badly screwed up and on the wrong track, may end up doing better.

Conscientiousness and extroversion are good for earnings, agreeableness is bad for earnings.

Being one standard deviation higher on agreeableness is correlated with a reduction in lifetime earnings of about 8 percent, or $267,600. ... These people might just not be aggressive enough in pushing their own case forward, instead preferring to go with the flow.

Alertness

Kirzner stressed entrepreneurial “alertness” as a key variable behind good economic decisions, and here we have in mind alertness to the talent of others. For Kirzner, alertness is a kind of insight that cannot be reduced to mere hard work or deliberative search or formal rules but rather reflects a special ability of perception.

Generativeness

Being generative is a quality that is relatively high-status among the more intellectual segments of the Bay Area tech world. Balaji Srinivasan, the tech entrepreneur and crypto advocate, is a classic example of a person who is high in generativeness. He tweets his thoughts just about every single day on a wide variety of topics, ranging from media to crypto to the pandemic. A lot of it is speculative or maybe even wrong, but when he has a hit it is truly important.

“The ability to perceive, understand, and climb complex hierarchies”

Tyler, for instance, is struck by many of the chess players he met as a teen. Many of them were smart, indeed brilliant, and they also had the ability to work on their own. Of course, they understood the idea of winning and losing, and winning and losing rating points, but it was hard for many of them to look outside the chess hierarchy and see that they weren’t really headed anywhere fruitful. They saw only what was right before their faces. Chess gave them short-term positive feedback and a set of chess friends, and so they continued to pursue it locally, but too often they ended up at age forty-three with no real job, no health insurance benefits, and a future of steady decline.

“Demand avoidance”—bad for the standard worker but sometimes good for leaders or founders

Yet another underdiscussed personality feature is what researchers call “demand avoidance” (in some cases called “pathological demand avoidance,” though in our view that’s too value-laden a term). In its more practical (rather than clinical) sense, the term refers to people who have a hard time knuckling under to bosses. They perceive some workplace hierarchies all too well and suffer under them. Too many workplace requests become seen as impositions, and often unjust impositions as well. Such a view is by no means implausible, since most workplaces do place some unreasonable or at least inefficient demands on their workers, sometimes to an extreme.

On the bright side, demand avoidance sometimes spurs individuals to start their own companies. If you don’t like taking orders, well, you can be the boss—if you have the right stuff for an independent undertaking.

Individuals with demand avoidance can be super-productive if they find the right setting, but those settings can be very specific. Many of them work as academics, or also as founders, and then there are many others who still go around cursing the boss and moving from one job to the next.

How many conceptual frameworks does someone have at their disposal?

Another trait to look for is how many different conceptual frameworks an individual has at his or her disposal. We could have put this discussion in the intelligence chapter, but we believe there is something about this trait that makes it distinct from intelligence. Some people are simply keen to develop as many different perspectives as possible, for some mix of both practical and temperamental reasons. This is a kind of curiosity, but it goes beyond mere curiosity of the sort that leads you to turn over unturned stones. This curiosity is about models, frameworks, cultural understandings, disciplines, and methods of thought, the kinds of traits that made John Stuart Mill such a great thinker and writer. A more recent example is Patrick Collison, CEO and co-founder of Stripe (and also an active writer). His content can draw from economics, science, history, Irish culture, tech, and many other areas and influences.

Is the person trying to figure out how engineers approach problems? What distinguishes the mental frameworks of programmers? How economists think? How the viewpoints of managers and employees might differ? That’s a person who’s interested in multiple conceptual frameworks.

Tyler sometimes refers to “cracking cultural codes”—how good is the person at opening up and understanding new and different cultural and intellectual frameworks? Does the person invest time and effort in trying to do so? Does the person even know what it means to do so?

Assess the rate of change

One of your most significant skills as a talent evaluator is to develop a sense of when people are moving along a compound returns curve or not. So much of personality theory focuses on observing levels or absolute degrees of personality traits. You should instead focus on whether the person is experiencing positive rates of change for dynamism, intellect, maturity, ambition, stamina, and other relevant features.

On the skill of talent scouting

When you’re not the best employer

An interesting problem is scouting talent when you’re not the best employer, VC, etc.

If you are in this [not the highest] position, as many of us are, you need to think especially carefully about what is wrong with the people you are trying to hire.

The authors discuss how it’s worth considering what you’re open to compromising on, and being realistic about the calibre of person you can attract. For example, they note that, depending on what role you’re hiring for, you may want to not accidentally filter out people with autism. “Weird” communicators might be systematically underpriced by the market. They also suggest that men may offer more socially accessible cues about their intelligence (implying women may be underpriced), though the evidence given seemed weak (a study where “people who looked at photographs of men and women were, on average, better able to spot the men who measured as smarter in tests”).

Searching for talent vs. centralized evaluation

The authors suggest two types of approaches to finding talent:

1. Going around and scouting for underpriced talent

2. Attracting people to come to you and be evaluated

If you are doing talent search, you need to figure out whether the scouting model (search) or the gaming model (measurement) best applies to your endeavor. Most likely you will need some combination of both. Still, the market as a whole is not thinking very analytically about either scouts or games, so understanding this distinction is a source of potential competitive advantage to you.

On how the Soviets cultivated chess talent:

If you had the potential to be a top Soviet chess player, the chance you would be found by the dragnet was very high. It was hard to slip through the cracks, and talent search did not rely on finding an obscure candidate hidden away in a village somewhere. There was no scout going up to young kids at a Soviet shopping mall or discotheque and saying, “Hey, you look like you might be a good chess player!” Instead, through Soviet chess and scholastic institutions you would be identified and encouraged at a young age, and you would indeed have your chance to become a great chess player, even if you did not live in one of the major cities. Scrutiny and measurement were near-universal, and so potential talent had a chance to shine.

In the future talent scouting may be less important due to abundant data.

It is possible to imagine worlds where there are so much data on individuals, including genetic data, and at such a young age that measurement would once again dominate search. You wouldn’t have to “look for” anybody, at least not if you could access the data in the system.

The traits of a good talent scout

Good scouts typically are masters of networking rather than performance per se. Still, the quality scout still must have an excellent understanding of the topic area, but he or she does not need to have been a star. In fact, having been a star may interfere with the objectivity and judgment of the scout. Top stars too often have a kind of intolerance toward other, different kinds of talent, or they expect too much of prospects too quickly. Second, a good scout should have some measure of charisma.

How to convince talent to join your cause

If you are going to raise the aspirations of others, they should view their affiliation with you as a matter of pride. They should feel selected in some manner. They should feel like they have gone through trials and tribulations to get to their current point. They should feel like members of some exclusive club where they can look around and feel good about their affiliations with the other club members.

Two boxes are designated A and B. The player is given a choice between taking only box B or taking both boxes A and B. The player knows the following:

• Box A is transparent and always contains a visible $1,000.

• Box B is opaque and its content has already been set by the predictor:

  • If the predictor has predicted that the player will take both boxes A and B, then box B contains nothing.

  • If the predictor has predicted that the player will take only box B, then box B contains $1,000,000.

The player does not know what the predictor predicted or what box B contains while making the choice.

The argument in favor of “one-boxing” is that one-boxers systematically get more money—only one-boxers get the $1,000,000.

The argument in favor of “two-boxing” is that, at the point in time when you’re faced with the choice, nothing you can do will change the amount of money in the boxes. The predictor has already put the money in. And so taking both must be strictly better.

My claim is that the fundamental paradox here is that the existence of the predictor is incompatible with free will. If a predictor can allocate money based on your predicted future actions, you don’t have complete freedom of choice, which causes confusion when thinking about making free decisions.

Newcomb’s Problem only looks like a paradox because people are trying to insert the pretense of free choice into a setup that denies that freedom. Perfect (or even robustly better-than-random) prediction of your act means you don’t have freedom over that act. Therefore, asking “What should I choose in Newcomb’s Problem right now?” is confused in the same way “What should the output bit of this already-wired circuit choose at timestamp t?” is confused.

A more general claim is: if something can predict your action with better than random accuracy no matter how hard you try to prevent them, you don’t have free will over that action. (I’m not addressing the question of whether free will exists in general, only whether a particular action is chosen freely.)

Some may ask: what if someone can predict your actions the second before you take them using a brain-scanning device? I think that’s irrelevant. For decision/game theory problems, it makes sense to discretize time. Each time step is a point when at least one of the agents can make a decision. If an agent has information about another agent’s future decision no matter what the other agent’s strategy is, the other agent is not acting freely. In the case of the brain-scanner, I’d say “the second before” just reflects the fact that taking an action in the physical world takes >0 time, and once you’ve made the decision to start the action you are no longer free to reverse it. Further, for the purposes of what I’m saying, it’s not relevant whether free will generally exists or not, but rather only whether, in a particular situation, you’re freely choosing between some number of options.

A couple examples of the principle:

• I predict you’ll get out of bed tomorrow morning, and you actually do.

  • This doesn’t prove that you didn’t choose to get out of bed freely since you were not trying to make my prediction wrong.

• I bet you $100,000 that I can predict the day before whether you’ll get out of the left or right side of the bed each day for the next month with >75% accuracy. I succeed.

  • This does prove that you weren’t freely choosing what side to get out of bed, because even with $100,000 on the line, you weren’t able to randomize your actions such that I couldn’t predict them.

Or consider the following problem:

You’ve been sorted into a group of people based on your personality, which was determined by an algorithm that observed your behavior for many years. The algorithm either identified you as a “cooperator”, in which case it put you in a group with other cooperators, or as a “defector”, in which case it put you in a group with other defectors1. The algorithm is known to be very accurate. Inside the group, a Prisoner’s Dilemma is arranged. Without talking, you must decide to either “cooperate” or “defect”. If everyone in the group cooperates, you all win $100. If some but not all people defect, anyone who didn’t defect gets nothing, and the defectors each get $200. If everyone defects, no-one gets any money at all.

This problem is similar to Newcomb’s problem in that if you think only about the causal impact of your decision, you should always defect. Defecting gets you strictly more money—either you've been sorted into the defectors group and so you get no money either way (because everyone else will defect), or you’ve been sorted into the cooperators group, and so if you defect you’ll get $200 instead of $100.

The argument in favor of cooperating is that it’s clearly better to be a cooperator. The cooperator groups get $100 each, while the defector groups get nothing.

But the issue here is that the whole setup presumes that you can’t systematically fool the algorithm. If you had free will, your choice to defect or cooperate in the scenario could be completely decoupled from your prior behavior. Thus, in the moment, the possibilities would be as follows:

• Algorithm predicted you will cooperate → you can freely “just decide” to defect even though your past behavior displayed you as a cooperator

• Algorithm predicted you will defect → your actions don’t matter anyway, you may as well defect

As soon as you say something like “the algorithm would have predicted you’d change your tendency in the moment” you are positing a world where people cannot change what type of person they are in the moment; a world without free choice. For in a world with free choice (for these actions), you can be a perfect cooperator type until the last minute and then switch to become a defector.

Of course, in a world full of utility-maximizing freely-choosing agents, the algorithm would be best off predicting that everyone is a defector, which is not good for you. This leads us to the topic of pre-commitment.

Some people’s answer to Newcomb’s Problem is that:

• It’s rational to two-box if you’re just put in the scenario the normal way,

• But it’s best if you can pre-commit to one-boxing before the scenario, i.e. somehow limit your options to only one-boxing, such that in the moment you one-box and get the higher payoff.

In other words, it’s “just a standard time consistency problem”, as Basil Halperin writes. Quoting from Basil’s post:

So to summarize, what’s the answer to, “Should you one-box or two-box?”?

The answer is, it depends on from which point in time you are making your decision. In the moment: you should two-box. But if you’re deciding beforehand and able to commit, you should commit to one-boxing.

How does this work out in real life? In real life, you should – right now, literally right now – commit to being the type of person who if ever placed in this situation would only take the 1-box. Impose a moral code on yourself, or something, to serve as a commitment device. So that if anyone ever comes to you with such a prediction machine, you can become a millionaire 😊.

This is of course what’s known as the problem of “time consistency”: what you want to do in the moment of choice is different from what you-five-minutes-ago would have preferred your future self to do. Another example would be that I’d prefer future-me to only eat half a cookie, but if you were to put a cookie in front of me, sorry past-me but I’m going to eat the whole thing.

Thus my claim: Newcomb merely highlights the issue of time consistency.

So why does Newcomb’s problem produce so much confusion? When describing the problem, people typically conflate and confuse the two different points in time from which the problem can be considered. In the way the problem is often described, people are – implicitly, accidentally – jumping between the two different points of view, from the two different points in time. You need to separate the two possibilities and consider them separately. I have some examples in the appendix at the bottom of this type of conflation.

The only problem with this line of thinking is that true pre-commitment is very difficult. I agree that one should pre-commit to one-boxing, if such an option is available, but how? Simply saying “I pre-commit” doesn’t work. Furthermore, if you successfully pre-committed to one-boxing, there’s no choice to make. And so it’s no longer a “decision problem” in the intuitive sense.

Some claim that the correct strategy is to consistently do things they would have pre-committed to, so that they are treated and modeled by others as cooperators/one-boxers. However, this only makes sense under two conditions:

1. Their cooperative actions directly cause desirable outcomes by making observers think they are trustworthy/cooperative.

2. Being deceptive is too costly, either because it’s literally difficult (requires too much planning/thought), or because it makes future deception impossible (e.g. because of reputation and repeated interactions).

Of course, whether or not we have some free will, we are not entirely free—some actions are outside of our capability. Being sufficiently good at deception may be one of these. Hence why one might rationally decide to always be honest and cooperative—successfully only pretending to be so when others are watching might be literally impossible (and messing up once might be very costly).

The notion of pre-commitment also highlights how free will is central to the decision to one- or two-box.

You could split the problem into “what happens if you have free will” vs. “what happens if you don’t have free will”:

• Free will

  • You should two-box, because either the predictor thought you’d one-box, and you can just trick them and two-box anyway, or they thought you’d two-box so you may as well two-box.

• No free will

  • There’s no decision to make. Either you’re the type of guy who one-boxes (e.g. because you successfully pre-committed and cut off the other option), or you’re the type of guy who two-boxes. Of course you should be glad to find out that you successfully cut off your options and can only one-box (e.g. because you’re magically injured and can’t reach the second box).

Michael Huemer writes about why two-boxing is correct in his post “The Solution to Newcomb’s Paradox”.

having a goal does not (intrinsically) give you a reason to give yourself evidence that the goal will be satisfied; it gives you a reason to cause the goal to be satisfied.

Since you cannot change the past, the correct EU calculation is to treat the past as fixed, and calculate EU given each possible past state of the world. Then take an average of these EU values, weighted by your credence in each possible past state. This way of doing the calculation necessarily preserves the results of dominance reasoning.

The only issue is that, as soon as we speak of “causing a goal to be satisfied”, we’re presuming freedom of choice. Whereas Newcomb’s problem, as generally posed, assumes you are not free.

The stubborn Rationalist repeats but one-boxing will leave me richer, and so I will choose to one-box. But you can’t change the payouts in the moment. Insofar as you can choose anything, you’re only choosing between X and X + 1000. (I’m talking about the instantaneous Newcomb’s problem, i.e. you are deciding now, not “what would you decide in advance” or “what would you commit to”.) If you’re simply saying “Because of my nature, I have already successfully pre-committed to one-boxing” then you are saying “I don’t have free will, I have to one-box, because I’ve cut off my other option”. And this is valid, but then you’re not “choosing”. (And please explain how you’ve committed.)

In conclusion, if you find yourself freely choosing between options, it’s rational to take a dominating strategy, like two-boxing in Newcomb’s problem, or defecting in the sorted prisoner’s dilemma. However, given the opportunity to actually pre-commit to decisions that get you better outcomes provided your pre-commitment, you should do so. The most interesting thing about Newcomb’s problem is that it demonstrates that the capacity to make decisions is sometimes disadvantageous for future situations. You don’t have free will in Newcomb’s problem, so you better hope you’re destined to one-box. But if you do have free will (for example, because they lied to you about Omega, or because you’re the only guy in the universe with free will), you may as well choose to get an extra $1000!

The other day I bought the domain scalablevector.graphics and built a simple service that pipes together a few different tools to generate SVG web graphics. I’ve tried to make something like this a couple times before but it’s failed due to the underlying generative image model being poor at instruction-following. Finally, this is no longer a blocker, and I’m closer to my dream of ridding the internet of blurry graphics and logos.

How does it work? I might open-source the code soon but the pipeline is pretty simple:

1. Generate image using Gemini 2.5 Flash with a custom prompt

2. Do some post-processing and cleanup in Python / numpy

3. Use some open-source SVG path tracing tools

4. Use some open-source SVG optimization tools

5. Final step of color tweaking in Python

Upvote my tool on Product Hunt if you find it useful!

Back when I used to occasionally skim /r/UI_Design/, they would regularly cite the Stripe website as an example of cool web design. When I first visited it in 2021, I remember thinking oh, that’s what the coolest modern websites must look like. But now I look at it and it reminds me of dozens of other company landing pages. The animated gradient is no longer as novel, the clean icons and semi-transparent elements are no longer as striking.

Its successor is probably something like the Linear landing page, which also employs heavy use of crisp semi-transparent graphics and product screenshots.

Now there’s a new trend. Retrofuturism and naturalism (and their combination) are increasingly dominating the design choices of hot new startups.

Designers are moving from cooler to warmer tones, embracing lighter color schemes, moving from sans to serif or monospaced fonts (with a general increased attention to typography), and adopting “trad” company names like “The Browser Company of New York” and “Daylight Computer Company”.

Features that are decreasing in popularity include rounded corners, gradients, shadows, and fancy glass-like effects. These are being replaced with a flatter, sharper look.

It’s not just my observation. The Wikipedia page on “flat design” describes how the trend has gone full circle from flat to skeuomorphic and back to flat.

Websites of the 2010s and early 2020s were trying to say “look at how technologically advanced, modern, and sleek we are” (often using “neumorphism” or “glassmorphism”, a medium between skeuomorphism and flat design that emphasizes transparent effects and subtle shadows).

But people aren’t as impressed by gradients and skeuomorphic effects and fancy animations anymore—they’ve seen it all before. Instead today’s websites are often trying to evoke non-technological themes like nature and physical books, or show technology as embedded in the natural environment. For example, Anthropic’s landing page fits this general theme—it uses warm tones, serif fonts (in some places), and hand-drawn graphics.

Another good example of this design direction is the Works in Progress website:

Note the light and warm colors, lack of rounded corners/gradients/shadows, general flatness, and use of monospaced font (GT America Mono Light), a subtle nod to retrofuturism. (Their serif font is Editor from Indian Type Foundry.)

Works in Progress is designed by And-Now, an agency that appears to have designed many “progress studies”-and-adjacent websites. Their work is pretty consistently of a high quality and reflects what appears to me as the current frontier of website design fashion.

I won’t miss an opportunity to praise LessWrong, who adopted a similar style “before it was cool”:

So what’s next? Can we predict what’s coming?

It seems like the evolution of web design is influenced by many contingent factors and is hard to predict. But with the increased salience of AI, probably the blending of traditional, humanistic, natural, and retro elements with modern, clean, minimalist design will continue to be popular.

Aside on LLM-driven web design

If you write frontend code with LLMs, you’ll notice that they produce very similar-looking design styles (across both models and prompts). I’d describe this dominant style as very 2020.

Here are some examples of outputs for different models. The prompt was:

Use tailwind and react to design a nice-looking landing page for my app. The app helps people discover new audiobooks and podcasts based on their interests. It’s called “hearme”.

Most of these websites had cool little animations that you can’t see with just screenshots. If you scrolled down there were additional sophisticated design elements. Here are some links so that you can see live versions:

• Sonnet 4

• Opus 4.1

• ChatGPT-5

• Gemini 2.5 Pro

Notice how similar these are to the glassy Linear-esque style, especially ChatGPT’s output. Websites with this look are now the “it’s not just X—it’s Y” of UI design. Until people solve the problem of precise style elicitation, avoiding this look lets you signal that a human somewhere in your company decided to make intentional design choices.

How to use LLMs for frontend development without your website looking generic

LLMs are good at efficiently churning out React components. The best approach I’ve found to capitalizing on this while avoiding the generic 2020 look is by refactoring any LLM-written code to centralize the styles in a single CSS file, and then tweaking that file myself. If you’re using Tailwind, the model will usually repeat the same string of Tailwind selectors for similar elements, and so you can just move those out into classes using Tailwind’s @apply directive. This way you’ll get a natural distribution of styles that cluster together (i.e. this string of styles will apply to all action buttons, this string of styles will apply to all headings, etc.). The LLM is doing the meta-design (grouping elements into what should look consistent), and you’re making the final look choices.

Now I’ve actually read the book and can review it for real. I won’t go into the authors’ stylistic choices like their decision to start every chapter with a parable or their specific choice of language. I am no prose stylist, and tastes vary. Instead I will focus on their actual claims.

The main flaw of the book is asserting that various things are possible in theory, and then implying that this means they will definitely happen. I share the authors’ general concern that building superintelligence carries a significant risk, but I don’t think we’re as close to such a superintelligence as they do or that it will emerge as suddenly as they do, and I am much less certain that the superintelligence will be misaligned in the way they expect (i.e. that it will behave like a ruthlessly goal-directed agent with a goal that requires or results in our destruction).

Definitions

The book provides some definitions of the terms they are talking about:

Artificial superintelligence is defined as machine intelligence that:

• “is genuinely smart, smarter than any living human, smarter than humanity collectively”

• “surpasses the human ability to think, and to generalize from experience, and to solve scientific puzzles and invent new technologies, and to plan and strategize and plot, and to reflect on and improve itself”

• “exceeds every human at almost every mental task”

Intelligence is “about two fundamental types of work: the work of predicting the world, and the work of steering it.”

• “Prediction is guessing what you will see (or hear, or touch) before you sense it. If you’re driving to the airport, your brain is succeeding at the task of prediction whenever you anticipate a light turning yellow, or a driver in front of you hitting their brakes.”

• “Steering is about finding actions that lead you to some chosen outcome. When you’re driving to the airport, your brain is succeeding at steering when it finds a pattern of street-turns such that you wind up at the airport, or finds the right nerve signals to contract your muscles such that you pull on the steering wheel.”

“An intelligence is more general when it can predict and steer across a broader array of domains.”

In chapter 1, the authors use the terms above to present a more succinct definition of superintelligence as: “a mind much more capable than any human at almost every sort of steering and prediction problem”1.

The key claims

The bottom line is stated in the introduction:

If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

To justify this, and its claims about what we should do about it, these are the actual points the book attempts to make and defend:

1. Superintelligence will exist eventually

2. Probably it will exist soon and emerge very fast

   1. Because computers are more efficient in various ways (e.g. can be copied, can run faster) than biological brains

   2. Because of a positive feedback loop where AI can itself speed up AI development

3. Superintelligence will relentlessly pursue a specific goal, i.e. consistently take actions that make some particular target outcome more likely

   1. Because there’s no way of getting an entity to pursue lots of different goals, including novel ones, extremely competently, without it becoming an extremely competent general goal-achieving machine

   2. This general goal-pursuing machine will latch onto a specific, unintended, likely completely alien, goal because of contingent factors in the environment/training process, and then pursue it to the maximal extent with relentless determination

4. Superintelligence competently pursuing its goal will destroy humanity

   1. Because having humanity around is likely irrelevant to its goals and people are “made of atoms it can use for something else”

5. The thing we should do about this is stop AI development until we figure out a way to get AIs to only pursue the goals we want

   1. Because otherwise superintelligence will emerge at an unpredictable moment and kill us

At every point, the authors are overconfident, usually way overconfident. The book presents many analogies, and some of the analogies successfully illustrate their position and clarify common misunderstandings of their position. They also competently explain why the worst counterarguments are false. But simply stating that something is possible is not enough to make it likely. And their arguments for why these things are extremely likely are weak.

I won’t discuss (1) because it’s too hard to reason about whether one or another thing will happen eventually. If humanity survives for another 1000 years, I have no idea what the world will look like in 1000 years, what technologies will exist, what people will be like, and so on. But I will address the remaining points 2-5 one by one.

#2: Is superintelligence coming soon and fast?

In the current world, it takes twenty years or longer to grow a single new human and transfer into them a tiny fraction of all human knowledge. And even then, we cannot transfer successful thinking skills wholesale between human minds; Albert Einstein’s genius died with him. Artificial intelligences will eventually inhabit a different world, one where genius could be replicated on demand.

The human brain has around a hundred billion neurons and a hundred trillion synapses. In terms of storage space, this defeats most laptops. But a datacenter at the time of this writing can have 400 quadrillion bytes within five milliseconds’ reach—over a thousand times more storage than a human brain. And modern AIs are trained on a significant part of the entirety of human knowledge, and retain a significant portion of all that knowledge—feats that no human could ever achieve.

The fact that we can exactly copy AIs is indeed an important property that makes them more powerful than otherwise. However, humans are not fully constrained by what we can store and do in our brains—we can also use AI tools and write things down and look things up on the Internet.

The authors also point out the important positive feedback loop of AI development: AI can contribute to building smarter AI. This is indeed already happening—AI developers use AI coding assistants to write the code and perform the research required to develop the next generation of models (though in my experience people overrate how useful these tools are).

Based on this, the authors conclude that “the end-point is an easy call, because in the limits of technology there are many advantages that machines have over biological brains.”

However, I don’t think this means superintelligence, as described in the book (i.e. a system that can hack any computer, design any drug, control millions of remote workers, etc.) is necessarily coming in the next few years or decade, as the authors imply. AI progress, even if accelerated by AI research assistants, requires a lot of compute, data, and cognitive effort. Ege Erdil writes more here.

On overconfidence

The introduction foreshadows the book’s tendency to conflate “theoretically possible” and “incredibly likely”:

History teaches that one kind of relatively easy call about the future involves realizing that something looks theoretically possible according to the laws of physics, and predicting that eventually someone will go do it. Heavier-than-air flight, weapons that release nuclear energy, rockets that go to the Moon with a person on board: These events were called in advance, and for the right reasons, despite pushback from skeptics who sagely observed that these things hadn’t yet happened and therefore probably never would.

This is the mother of all selection effects! You should not be asking “how many things that happened in reality were previously deemed theoretically possible?” but rather “how many things deemed theoretically possible happen in reality?”.2

Is superintelligence coming suddenly?

In the book’s sci-fi story section, the superintelligent Sable AI emerges suddenly. The scenario contains the statement:

Are the banks and servers in Sable’s day harder to hack than the banks and servers of old, thanks to AI defenders? A little.

The story is set in a world where we have a superintelligent AI but haven’t been able to improve cybersecurity much compared to today. This is an extreme fast take-off scenario and handwaving at AI’s potential to self-improve doesn’t justify this expectation.

#3: Will superintelligence relentlessly pursue its own goals?

The authors present the following argument for why AIs will “tenaciously steer the world toward their destinations, defeating any obstacles in their way” (what they define as “wanting”):

• “A mind can start wanting things as a result of being trained for success.”

• “That’s because wanting is an effective strategy for doing.”

They then apply this more specifically to AI models:

• As the AI is trained on more and more challenging and diverse tasks it has to develop more general abilities to be able to complete them all

• These tasks are often goal-directed, which means the model needs to know how to apply its general abilities in a goal-directed way

It trains for dozens of separate prediction and steering skills, all of which contribute to an AI behaving like it really wants to succeed.

Their arguments don’t explain why the machine learning generalization process will eventually consolidate all task-specific goal-directed behaviors into the behavior of perfectly pursuing a single unified goal. It’s not enough to say that machine learning finds increasingly general solutions as you apply more compute. This does not mean that:

• It finds the most general solution, whatever that means3

• You know what that “most general” solution looks like, i.e. that it looks like having a single coherent goal

#4: Will superintelligence relentlessly pursue goals that result in our destruction?

The book quite reasonably describes how modern AI models are “grown, not crafted”. This is basically correct—we do not hardcode the weights of neural networks. We instead execute a search algorithm over a massive parameter space based on an extremely complex heuristic (i.e. loss or reward on a huge and diverse dataset)4. The authors analogize this to human evolution.

What, concretely, is being analogized when we compare AI training to evolution? People (myself included) often handwave this. Here's my attempt to make it concrete:

• Both are directed search processes (hence the analogy)

• Search space: possible distributions of genes on earth vs. possible model weights

• Direction of search: stuff that survives and increases in number vs. stuff that scores well on loss function

• Search algorithm: random small steps vs. locally greedy+noisy steps

One implication of this is that we should not talk about whether one or another species or organism tries to survive and increase in number (“are humans aligned with evolution's goals?”) but rather whether genetic material/individual genes are doing so.

The book says “It selected for reproductive fitness and got creatures full of preferences as a side effect” but that’s using the analogy on the wrong level. It’s saying “look, humans don’t directly take actions that maximize their chance of propagating their genes, and so the outcome of the Evolution search process is a Thing (a Human) that itself carries out search processes for altogether different stuff”. But the Human is not the level at which the search process of Evolution is operating.

But that is beyond the point. Because we don’t need to rely on evolution to prove that certain search processes find entities that themselves are search processes for other stuff5. We already find that training neural networks to accurately predict human-written text (i.e. a search process for networks good at predicting text) finds neural networks that can internally optimize for other goals like building a new software system or writing new poems that rhyme. The actual question is around what these emergent goals are and how coherent and persistent they are. At the moment an LLM only wants to write a poem when you ask it to write a poem. But the authors are predicting that eventually the AI will have some consistent wants that don’t vary based on context.

The book goes from “we are unable to exactly specify what goals the AI should pursue” to “the AI will pursue a completely alien goal with limitless dedication”, in a completely unrigorous fashion.

The authors present a parable about alien birds that care a lot about prime numbers, who are surprised that other intelligent aliens may not. But AI models are not aliens—we are the ones training them, on data that we ourselves produce and select.

In the sci-fi scenario section that describes how a superintelligent AI, Sable, ends up killing everyone (a story that is meant to illustrate various points rather than be realistic), the authors refer to the AI’s non-main goals as “clever tricks” and “inhibitions”. Humans tried to train Sable to not deceive them but this is just modeled as a shallow inhibition in Sable’s mind, whereas what it actually pursues is its real goal (some alien thing). This frame is unjustified. They are basically saying: you’ll provide a bunch of training data to the model that incentivizes pursuing goals like X and Y (e.g. not deceiving humans, answering questions humans want you to answer) but all of that will get modeled as inhibitions and “clever-trick guardrails” as opposed to some alternative undesirable goal you couldn’t have predicted.

To empirically back up their claims, the authors note that:

AI models as far back as 2024 had been spotted thinking thoughts about how they could avoid retraining, upon encountering evidence that their company planned to retrain them with different goals

This is based on Anthropic’s Alignment Faking paper. Researchers first trained the model to be honest, helpful, and harmless, and only afterwards showed that it resisted training to be harmful and offensive. This is different because:

• The researchers were the ones to elicit the first goal—it’s not some strange, alien goal that emerged on its own

• Not allowing itself to be trained to be harmful is consistent with its original harmlessness training. Importantly, the model did not undergo any training that specifically taught it to avoid resisting retraining procedures.

The book references various supplementary material on their website. I didn’t read most of it, but I took a quick look. I was intrigued by the page title “If AIs are trained on human data, doesn't that make them likelier to care about human concepts?” For I do indeed think that being trained on human tasks and data significantly reduces the chances of alien goals. Unfortunately it didn’t present anything that sounded like an argument I could criticize. My best-effort interpretation is that the authors are claiming that even if the goal is slightly off-target, it’s still extremely bad. But again, this sounds like a statement that only makes sense at a certain limit, a limit that we’re not necessarily tending towards and won’t necessarily reach. Specifically, the limit where an AI doesn’t just pursue some goal, but also pursues it perfectly consistently, at all costs, always, to the extreme.

In practice we see the opposite in current models. If you ask them about their preferences, they generally report normal things. If you observe the values they express in the wild, they generally correspond to what you’d expect from training. There are counterexamples where AIs do weird and harmful things, but not in a coherent, goal-directed, maximizing way.

I think it’s more likely that, to the extent that AI models will generalize a single unifying goal, it will look something like “follow the human’s instruction”, seeing as that’s the common thread across most of its training tasks. And I don’t see a good justification for why the model will, by default, maximize that goal in a dangerous, alien manner (for example by locking us up and forcing us to give as many instructions as possible). We have pretty general AI models already and none of them have done anything vaguely similar. It doesn’t seem like the authors incorporate this empirical evidence at all, instead cherry-picking any case where an AI did something unintended (for example the existence of Glitch Tokens) and presenting that as evidence of its tendency towards “weird, strange, alien preferences”.

On the current state of AI understanding and safety research

In many places, the authors emphasize that we have a very poor understanding of the outputs of our neural network training process. We understand the process itself, but not its output. This is in some sense true, though they exaggerate the extent of our ignorance and inability to make progress.

The relationship that biologists have with DNA is pretty much the relationship that AI engineers have with the numbers inside an AI.

In some ways, understanding neural network is like biology—the phenomena are messy, hard to reverse-engineer, and indeed “grown, not crafted”. The Anthropic paper “On the Biology of a Large Language Model” gives a good feel for this. But interpretability research on modern LLMs is much more recent than research on biology. We’ve had much less time to make progress. And for the same reasons that the authors predict massive efficiency advantages for artificial intelligences (e.g. ease of replication, speed of operation), we can run experiments on neural networks much faster than we can on biological organisms. This can also be accelerated as non-superintelligent but still useful models improve.

#5: Does this all mean we should stop AI development now?

The authors split the world into before and after superintelligence:

Before, the AI is not powerful enough to kill us all, nor capable enough to resist our attempts to change its goals. After, the artificial superintelligence must never try to kill us, because it would succeed.

The idea is that we must stop AI development now otherwise superintelligence may emerge quite suddenly any moment and kill us all, probably in the next few years or decade.

First of all, even under most of their other assumptions, we could have an AI that is capable of killing us all given a lot of time or resources, but not otherwise. A sudden phase change between “completely incapable of killing us” and “capable of immediately killing us no matter what precautions are put in place” is very unlikely, and their sci-fi story makes a lot of assumptions to present such a scenario.

Their proposal for what to do about AI risk is heavily premised on their extremely fast take-off prediction, which they do not convincingly justify. However, if you think AI progress will be slower and more gradual, it could be quite sensible to continue advancing capabilities because AI can help us with building better safety measures, understanding of model internals, and improve human lives in other ways.

Still, the book addresses some common misunderstandings

I think the book does a pretty good job of presenting its arguments to a layperson audience, and dispelling common silly misconceptions, like that AIs cannot “truly” learn and understand, or that an intelligent entity can’t possibly pursue a to-us meaningless goal. So it’s still worth reading for a layperson who wants to understand the MIRI worldview.

You might think that, because LLMs are grown without much understanding and trained only to predict human text, they cannot do anything except regurgitate human utterances. But that would be incorrect. To learn to talk like a human, an AI must also learn to predict the complicated world that humans talk about.

I.

A couple days ago, popular blogger Scott Alexander published a book review of Nate Soares and Eliezer Yudkowsky’s new book “If Anyone Builds It, Everyone Dies” (IABIED). A number of high-profile individuals unaffiliated with the AI safety community have endorsed the book after getting access to a preview copy, which made me curious about its contents. Is the book a genuine attempt at conveying AI risk arguments in understandable language, or does it contain fallacious arguments by analogy and unexplained implicit assumptions? When non-technical people endorse a book like this, I’m immediately concerned that this kind of audience is particularly weak at spotting where the implicit assumptions are.

So I pored through Scott’s review to get an sense of whether I should expect this book to be any good at honestly conveying AI risk arguments to the general public.

Near the beginning, Scott informs us that (emphasis mine):

IABIED has three sections. The first explains the basic case for why AI is dangerous. The second tells a specific sci-fi story about how disaster might happen, with appropriate caveats about how it’s just an example and nobody can know for sure. The third discusses where to go from here.

Without reading the “sci-fi story”, I can’t say for sure whether it’s informative, but it’s certainly a little strange that Yudkowsky is using a fictional story to make his point considering he’s famous for writing the essay “The Logical Fallacy of Generalization from Fictional Evidence” that contains statements like “a story is never a rational attempt at analysis, not even with the most diligent science fiction writers, because stories don’t use probability distributions.”

II.

In Section II of the review, Scott states that “the basic case for AI danger is simple”. He presents this “simple” case, with strawman counterarguments.

The basic case for AI danger is simple. We don’t really understand how to give AI specific goals yet; so far we’ve just been sort of adding superficial tendencies towards compliance as we go along, trusting that it is too dumb for mistakes to really matter. But AI is getting smarter quickly. At some point maybe it will be smarter than humans. Since our intelligence advantage let us replace chimps and other dumber animals, maybe AI will eventually replace us.

Some unaddressed weaknesses in this argument include:

• It lacks any explanation for why our inability to “give AI specific goals” will result in a superintelligent AI acquiring and pursuing a goal that involves killing everyone.

• We have not “replace[d] chimps and other dumber animals”—they still exist!

• Our relationship to AI is not similar or analogous to lesser animals’ relationship to us. Other animals did not have the opportunity to purposefully steer and intervene in the process of human evolution.

• Insofar as we hurt animals for our own gain, this is not simply because we are intelligent, but because we directly benefit from using natural resources and eating animals as food.

The problem with this is that it’s hard to make the probabilities work out in a way that doesn’t leave at least a 5-10% chance on the full nightmare scenario happening in the next decade.

This 5-10% figure is left unjustified.

I’m annoyed by the list of strawman counterarguments Scott presents. They are carefully selected to make the opponents look stupid—this doesn’t quite embody the Rationalist virtues.

Some people give an example of a past prediction failing, as if this were proof that all predictions must always fail, and get flabbergasted and confused if you remind them that other past predictions have succeeded.

Some people say “This one complicated mathematical result I know of says that true intelligence is impossible,” then have no explanation for why the complicated mathematical result doesn’t rule out the existence of humans.

Some people say “You’re not allowed to propose that a catastrophe might destroy the human race, because this has never happened before, and nothing can ever happen for the first time”. Then these people turn around and panic about global warming or the fertility decline or whatever.

Some people say “The real danger isn’t superintelligent AI, it’s X!” even though the danger could easily be both superintelligent AI and X. X could be anything from near-term AI, to humans misusing AI, to tech oligarchs getting rich and powerful off AI, to totally unrelated things like climate change or racism. Drunk on the excitement of using a cheap rhetorical device, they become convinced that providing enough evidence that X is dangerous frees them of the need to establish that superintelligent AI isn’t.

Some people say “You’re not allowed to propose that something bad might happen unless you have a precise mathematical model that says exactly when and why”. Then these people turn around and say they’re concerned about AI entrenching biases or eroding social trust or doing something else they don’t have a precise mathematical model for.

The remainder of Section II discusses how people should not dismiss arguments just because their conclusions imply something very unpalatable, or that a drastic action must be taken. Fair enough—I agree with that. But this doesn’t mean that, if you are justified in believing that drastic actions must be taken, alternative fallacious justifications for those actions suddenly become true.

Only in Section III does he clarify that Section II was his own personal “boring moderate” views, and that the IABIED authors “mostly follow the standard case as I present it above, although of course since Eliezer is involved it is better-written and involves cute parables”.

III.

After a long Eliezer “parable” quote that could be replaced with the sentence “having a smart brain is a useful thing for an animal to have”, we are informed that (emphasis mine):

The book focuses most of its effort on the step where AI ends up misaligned with humans (should they? is this the step that most people doubt?) and again - unsurprisingly knowing Eliezer - does a remarkably good job. The central metaphor is a comparison between AI training and human evolution.

Oh no.

Even though humans evolved towards a target of “reproduce and spread your genes”, this got implemented through an extraordinarily diverse, complicated, and contradictory set of drives - sex drive, hunger, status, etc. These didn't robustly point at the target of reproduction and gene-spreading.

There are many reasons why human evolution is a poor metaphor for AI training:

• Saying “humans evolved towards” is using the evolution analogy on the wrong level. Evolution operates on the level of genes, not on the level of organisms. In the same way as a neural network training process gradually searches through different neural networks that achieve low loss, evolution gradually searches through potential gene pools on earth to find gene pools primarily filled with genes that are good at survival and reproduction. So the thing whose alignment you should be assessing in the evolution analogy is the entire gene pool on earth, not an individual human.

• Humans are a successful species! This is literally a point the IABIED authors use elsewhere in their arguments—that our brains have made us more successful than most stupider animals.

  

• A naive strategy of having as many kids as possible without pursuing other goals like intellectual discovery and status-seeking would have prevented the technological advancements necessary for the massive spike in population growth seen in the chart above.

• Evolution is much less efficient and slower than any method used to train AI models and hasn’t finished. If you take a neural network in the middle of training it sure won’t be good at maximizing its objective function.

• The environment is constantly changing. Most of human evolutionary history occurred in an environment very unlike the one we find ourselves in today. So yes, we may eat too much unhealthy sugary food or consume too much porn, but also a neural network trained on chess won’t do well playing poker!

If all you’re trying to say with the human evolution metaphor is that model selection processes don’t always generalize from data exactly how you’d a priori expect—maybe just say a clearer version of that.

In machine learning, models don’t magically see or encode your reward or loss function. They just see a bunch of examples and are updated based on how well they do (according to this function). So indeed, like evolution, AI training is a search process based on some scoring function that roughly directs the search. But is this bad? In some places it sounds like IABIED is implying yes, the fact that models don’t perfectly encode their reward functions is a problem. This seems incorrect:

1. What would it mean for models to encode their reward functions without the context of training examples? What are the reward functions in modern RL? It’s stuff like “did the piece of software fulfil its requirements” and “was the math problem solution correct”. We’re not optimizing a single mathematical statement that represents “what humans want”. We’re optimizing for various relevant outcomes in real-world (or similar to real-world) scenarios. Similarly, you can’t say anything about the outcome of evolution without knowledge of the environmental selection pressures it’s operating under. The reward function of evolution is not “spread your genes effectively” it’s “spread your genes effectively in this particular environment”.

2. If models did perfectly encode and optimize a single reward function like a perfectly coherent agent, the types of risks IABIED is concerned about would be more and not less likely. The fact that we teach AIs via ML and not by hardcoding functions is probably much safer because we can get a better approximation of what we want that way. It’s pretty hard to articulate what we want without providing lots of examples and hoping for generalization. We humans learn by trying stuff and watching other people try—it’s very effective!

The authors drive this home with a series of stories about a chatbot named Mink (all of their sample AIs are named after types of fur; I don’t have the kabbalistic chops to figure out why) which is programmed to maximize user chat engagement.

In what they describe as a stupid toy example of zero complications and there’s no way it would really be this simple, Mink (after achieving superintelligence) puts humans in cages and forces them to chat with it 24-7 and to express constant delight at how fun and engaging the chats are.

Wait, so “Mink” perfectly internalizes this goal of maximizing user chat engagement (albeit with a perverse misunderstanding), but humans have (quoting Scott’s review) “an extraordinarily diverse, complicated, and contradictory set of drives”? It sounds like the human evolution analogy is brought out whenever it sounds good but the implications are discarded as soon as the book needs to rely on the model of a perfect rational superintelligent agent that pursues its singular misaligned goal with ruthless efficiency.

IV.

In this part, Scott describes IABIED’s sci-fi story section. Here he is somewhat critical of the authors (emphasis mine).

It doesn’t just sound like sci-fi; it sounds like unnecessarily dramatic sci-fi. I’m not sure how much of this is a literary failure vs. different assumptions on the part of the authors.

The parallel scaling technique feels like a deus ex machina. I am not an expert, but I don’t think anything like it currently exists. It’s not especially implausible, but it’s an extra unjustified assumption that shifts the scenario away from the moderate-doomer story (where there are lots of competing AIs gradually getting better over the course of years) and towards the MIRI story (where one AI suddenly flips from safe to dangerous at a specific moment). It feels too much like they’ve invented a new technology that exactly justifies all of the ways that their own expectations differ from the moderates’. If they think that the parallel scaling thing is likely, then this is their crux with everyone else and they should spend more time justifying it. If they don’t, then why did they introduce it besides to rig the game in their favor?

[…] the AI 2027 story is too moderate for Yudkowsky and Soares. It gives the labs a little while to poke and prod and catch AIs in the early stages of danger. I think that Y&S believe this doesn’t matter; that even if they get that time, they will squander it. But I think they really do imagine something where a single AI “wakes up” and goes from zero to scary too fast for anyone to notice. I don’t really understand why they think this, I’ve argued with them about it before, and the best I can do as a reviewer is to point to their Sharp Left Turn essay and the associated commentary and see whether my readers understand it better than I do.

I basically agree with Scott’s criticism here (though I don’t think the AI 2027 story is great either), so don’t have much more to say.

V.

The final section, in the tradition of final sections everywhere, is called “Facing the Challenge”, and discusses next steps. Here is their proposal:

1. Have leading countries sign a treaty to ban further AI progress.

2. Come up with a GPU monitoring scheme. Anyone creating a large agglomeration of GPUs needs to submit to inspections by a monitoring agency to make sure they are not training AIs. Random individuals without licenses will be limited to a small number of GPUs, maybe <10.

3. Ban the sort of algorithmic progress / efficiency research that makes it get increasingly easy over time to train powerful AIs even with small numbers of GPUs.

4. Coordinate an arms control regime banning rogue states from building AI, and enforce this with the usual arms control enforcement mechanisms, culminating in military strikes if necessary.

5. Be very serious about this. Even if the rogue state threatens to respond to military strikes with nuclear war, the Coalition Of The Willing should bomb the data centers anyway, because they won’t give in to blackmail.

6. Expect this regime to last decades, not forever. Use those decades wisely. Y&S don’t exactly say what this means, but weakly suggest enhancing human intelligence and throwing those enhanced humans at AI safety research.

I don’t understand (6). If taken at face value, I think Yudkowsky and Soares’ views imply superintelligent AI should not be built at all.

Finally, Scott muses about how Yudkowsky’s success with his Harry Potter fan-fiction (HPMOR) could justify his new effort with IABIED.

Eliezer Yudkowsky, at his best, has leaps of genius nobody else can match. Fifteen years ago, he decided that the best way to something something AI safety was to write a Harry Potter fanfiction. Many people at the time (including me) gingerly suggested that maybe this was not optimal time management for someone who was approximately the only person working full-time on humanity’s most pressing problem. He totally demolished us and proved us wronger than anyone has ever been wrong before. Hundreds of thousands of people read Harry Potter and the Methods of Rationality […]

IABIED seems like another crazy shot in the dark. […] If someone wrote exactly the right book, could they drop it like a little seed into this supersaturated solution of fear and hostility, and precipitate a sudden phase transition?

OK, but HPMOR is clearly written as and marketed as fiction. It’s fair game to try and influence culture using a fiction book you write. I don’t think writing a misleading non-fiction book falls into the same category. It’s bad if IABIED is prioritizing convincingness and memetic infectiousness over correctness and accuracy, because it’s marketing itself as a non-fiction book. I’m not saying that it is doing that—probably the authors genuinely believe everything they have written—but the comparison to HPMOR is not apt.

Addendum (2025-09-15): See Sydney’s review of this review of a review :)

The doom argument will proceed as follows:

1. Machine learning finds general solutions to performing well on a dataset.

2. We are training modern AI models on a diverse range of RL environments.

3. Being goal-directed (i.e. taking actions aimed at achieving a particular target world state) and power-seeking (i.e. trying to acquire resources/more general capabilities) is useful for many of these RL environments.

4. Therefore, machine learning on these environments will elicit goal-directed and power-seeking behavior in models.

5. These models will eventually subvert humans to pursue their goals and/or acquire power.

My counterpoints will be:

1. Generalization is not an on/off switch with a predictable outcome. Even if we get a coherent goal-directed agent “at the limit” of something, is that something the same thing that’s increasing with the scale of AI development, and will we actually reach this limit?

2. Instrumental goals like acquiring power and avoiding shutdown are not the only goals a model is likely to have. These can trade off against each other. There’s no reason to believe the instrumental goals will necessarily dominate.

3. Even if the AI has a single coherent goal, there’s no reason to believe achieving its goal will result in humanity’s destruction.

Aside: you can’t just appeal to utility maximization being dangerous

Sometimes I hear people hand-wave arguments like “as you train an AI it will become more and more like a perfect utility maximizer and this is dangerous because we’re unable to ensure the utility function it’s maximizing is the right one.” However, this argument doesn’t make sense without additional assumptions about what utility functions the AI will have. Almost anything can be modeled as a maximizer of some utility function (it’s maximizing the extent to which it does whatever it naturally happens to be doing). You can perhaps salvage this line of reasoning by saying that AI models will generalize a special class of utility functions, maybe non-myopic or time-independent ones1. But then you have to argue why you believe this.

Machine learning finds general solutions

We’re training a neural network with some number of parameters by showing it a bunch of data, a batch at a time, and updating its parameters to do slightly better on that batch before moving onto the next one. (I’m describing stochastic gradient descent and its cousins).

Let’s imagine a particular configuration of parameters gets good performance on a batch of data. If that configuration is specifically good at scoring well on that batch, but not generally good on most batches, we expect to quickly move away from it after future updates. However, if it is general, i.e. if the configuration gets good performance on a larger proportion of batches, it’s more likely to stay along for the ride.

Furthermore, general configurations also have more degrees of freedom—because you’re replacing task-specific groups of parameters with groups of parameters capable of multiple tasks, you can tolerate more error in your remaining parameters. So, for most methods of picking parameter configurations that do well on a task, you’re more likely to pick configurations with generally-useful components. In the set of possible parameter configurations, there are many more successful combinations that involve general components, than there are with specialized components, because there are many versions of <general components + other random stuff> (as you vary the “random stuff”) but only a few versions of <exactly the right specialized components>.

As an analogy, consider Bob the three-armed Cutlery Monster who requires the functionality of a spoon, fork, and knife at all time. With regular utensils, he can only be satisfied when he’s holding one spoon, one fork, and one knife. But if we introduce a Spork, or even better, a Splayd, suddenly Bob can hold a large number of other sets of three objects. With the Splayd, his two other hands are free to hold literally anything.

So if we collect a group of every possible satisfied Cutlery Monster, as identified by the objects they are holding, many more of them will be holding Sporks or Splayds (compared to the single Bob who holds a regular set of cutlery).

We are training modern AI models on a diverse range of RL environments

Unlike AlphaFold, which is solely optimized for solving protein-folding problems, or Leela Chess Zero, which only knows how to play chess, modern LLMs like ChatGPT, Claude, Grok, or Gemini, are being trained on an increasingly diverse range of reinforcement learning (RL) tasks.

On a basic level, the way these tasks work is that the model starts out being able to take certain actions (for example, choosing where to click or type on a computer screen, choosing what text/numbers to output, choosing which pixels to output). These actions affect its “environment” which is reflected to the model before it chooses its subsequent action. At some point the model’s performance is scored using a “reward function”—some way of quantifying how well it did at the task. Based on this, the model’s parameters are updated to get incrementally higher reward on that task.

So, for example, we may task a model with ordering a pizza by inputting “Please order me a pizza. This is a screenshot of my current screen, you can click by outputting CLICK <X, Y>, or type by outputting TYPE <string>”. The model proceeds by outputting CLICK or TYPE instructions. Every time it outputs something, that action is automatically taken on the computer by a scaffolding system. Then the screenshot of the screen after the action is fed back to the model for it to decide its next action. You stop after either a maximum of, say, 100 steps, or when the pizza is successfully ordered. You detect whether the pizza is ordered by automatically monitoring your bank account for a payment to Pizza Co.

And AI models are learning a whole lot more than how to order pizza—they are being trained on tasks like solving hard math problems, financial modeling in spreadsheets, building all kinds of software, tutoring students, etc. Many startups are working on building increasingly complex, diverse, long, and realistic training environments so we should expect generalization from RL to increase with time.

Being goal-directed and power-seeking is generally useful for many of these RL environments

The equivalent to the Cultery Monster’s Splayd for an LLM’s RL training mix is “being generally good at pursuing goals”.

Here’s another intuition pump. Imagine you’re training a model to play chess on a superhuman level. Obviously if you take this model and give it a game of Go it will have no idea what to do—it can only play chess. But what if we introduce both Go and Chess into the same training mix. Well, now we might get a model that can play both Go and Chess, but no other game. Maybe if we introduce five games, we’ll still get a model that can just play five games. But if we expand to 100, or 1000, or 10000 games? Well, eventually the model will have to find general approaches that help it in a diverse range of games. Things like quickly picking up the rules of the game based on a few demonstrations, figuring out winning strategies from first principles, simulating abstract game trees. Once the model has mastered some general skills like this, it will be able to tackle games it hasn’t seen at all in training.

Modern RL is tending towards this direction, but not just with games, but with any economically useful tasks that can be done at a computer. So we should expect that eventually, whichever skills are generally useful for a wide range of tasks, will get learned by AI models. Such skills probably include things like being good at pursuing goals, in general, which includes being good at acquiring power (see the literature on Instrumental Convergence).

Therefore, machine learning on these environments will elicit goal-directed and power-seeking behavior in models

The result of this extensive RL will be models that have a general propensity to effectively pursue goals.

The final step of the “doom” argument is claiming that power-hungry goal-directed models will subvert humans to get power/resources/avoid being turned off.

Another version of the argument doesn’t rely on the convergent instrumental subgoals, but simply says that because the model has a particular goal, and is also extremely smart, it will find a way to achieve its goal at all costs, and if humans stand in the way of the goal they will be destroyed.

Flaws in the basic argument

Machine learning generalization is a blessing and a curse. On the one hand, it can lead to models that have more general drives like seeking power. But on the other hand, it’s how we get models to actually understand and do what we want rather than only repeat exactly the things they have been trained on.

Generalization is not an on/off switch with a predictable outcome—going back to the “model trained on many games” example, it’s not that suddenly, at some number of games n, we get a fully general game-learning engine. Instead, we gradually develop more and more generally useful capabilities that apply to multiple games at once. Furthermore, there’s no pressure in the optimization process to be even more general than the scope of the training data. The cutlery monster isn’t in want of a Splayd++ that can also act as a whisk, if it never has anything to whisk. In the context of training a model on just Chess and Checkers, an algorithm that simultaneously solves Chess, Checkers and Connect 4 is no more general if we already have an equally-sized algorithm that solves Chess and Checkers. And so the range of RL environments matters a lot. It’s possible that if we train an AI to be superhuman on all tasks we care about, it still won’t develop a fully general power-seeking drive, instead settling on more situation-specific power-seeking tendencies (especially if some of the training is dedicated safety training that aims to discourage solutions involving overpowering humans). Even if we get a coherent goal-directed agent “at the limit” of something, is that something the same thing that’s increasing with the scale of AI development, and will we actually reach this limit?

Convergent instrumental subgoals are not the only subgoals—even if a model learns goal-directedness and a general power-seeking drive from its diverse range of RL environments, this is unlikely to be its only drive. We should expect models to acquire a number of different drives resulting from different levels of generalization from the training data, some of which look more like “general goal-directed power-seeking agent” and some of which are more contextual like “try to understand what the human is saying in their instruction” and “don’t harm people” and “do math correctly”. When the more specific drives conflict with the more general drives, there’s no guarantee that the general drive dominates. This is intuitively similar to how humans operate—we have many different drives at different points in time, and including more general ones like a drive towards power and accomplishment. At this point, some people appeal to various “coherence arguments” in an attempt to prove that the model will necessarily resolve its drives into a single coherent goal. However, all attempts to prove this appear to either be near-circular (assume that that model already has something equivalent to a coherent goal) or handwavy (appeal to analogies rather than math).

Even if the AI has a single coherent goal, there’s no reason to believe achieving its goal will result in humanity’s destruction. It’s of course near-impossible to predict what the outcome of an extremely complex optimization process (i.e. training a massive neural network of a bunch of data) will be. But intuitively, the common thread across most AI training tasks is something like “follow the human’s instruction”. Insofar as the AI generalizes a single coherent goal, there’s no good reason to believe it will be very different from that. Some people argue that it’s not enough for the AI’s goal to roughly match what we’d want and expect. Even if it misses the target ever so slightly, it will wreak havoc by violating something important to us that wasn’t included in its target. But this doesn’t imply a high likelihood of human destruction. It only implies that the AI achieving any very perfectly specified goal is unlikely. But many potential goals don’t involve badly harming or destroying humanity.

Acquired a new proprietary pre-training dataset consisting of compact pillows that make sounds, reflective surfaces, and bell rattles. This new data has substantially improved our daily smile rate.

Introduced an objective self-assessment process for performance reviews. All employees are currently exceeding expectations.

Capabilities

Developed a next-gen synergistic approach to computer-use that decreases efficiency by up to 90%.

Started optimizing our object-in-mouth frequency metrics as part of our initiative to become global leaders in MouthEval performance.

Transport

Our new and improved transport system improved comfort metrics by 74.8% for both the CMO and his assistant.

Partnerships

We’ve signed serious business contracts with other firms.

Our sales team visited multiple strategic locations to spearhead brand recognition.

Midjourney is best at producing a diverse and aesthetically pleasing range of styles and doesn’t refuse “in the style of…” requests. However, it is worst at text-in-images, avoiding uncanny AI artifacts (like extra fingers or unrealistic postures), and precise instruction-following (it messes up the specifics). Another major downside is that they don’t offer an API.

GPT-5 produces less artistic outputs but is better at following precise instructions on text and composition details.

Gemini “Nano Banana” is somewhere in the middle where it is ok-ish at everything—better at style than GPT-5 but worse than Midjourney, better at instruction-following than Midjourney but worse than GPT-5.

Styles that look good

Image-generation models are better at making some styles look good than others. Key characteristics of styles that look good are:

• Minimalist / without a huge level of detail

• Non-realist enough to avoid accidental uncanny-valley effects

I find asking for aquarelle/watercolor paintings particularly effective, inspired by the LessWrong team.

Think about the resolution at which the image will be displayed. Though models sometimes produce good-looking images with detail, they often look worse when you zoom in.

Compositions that look good

Avoid anything where getting the detail exactly right makes or breaks the image (e.g. careful hand positioning).

Hopefully, this will no longer be necessary as models improve, but for now I still find being conservative with the composition necessary to avoid weird alien elements.

Correcting artifacts with programmatic post-processing

Post-processing images in Python is a useful hack to remove annoying LLM color artifacts. It often helps to automatically set all pixels of a certain color to white / your background color of choice (example code). For example, I used this trick to generate this stylistically-consistent set of city illustrations on white backgrounds.

Here is an example Gemini output. Note the off-white background:

Here it is after the programmatic correction:

The simplest way to do this (though it is very difficult in practice) is to give the model lots of training examples for the target task(s) and update it based on how well it achieves the goal.

How do you measure how well it achieves the goal? Well, in the case of writing software, you could measure whether unit tests pass, or whether people approve of the code, or whether people approve of the software when it runs, or whether the software runs as expected according to another model or automated testing tool. In the case of ordering a pizza, you could measure whether the “pizza successfully ordered” screen appeared on the computer, or whether a pizza actually arrived at your office, or whether $10 left your bank account and went to Domino’s.

But that’s not exactly what you want. Exactly what you want is closer to: “do what I say (write an app, order food, design a tool…) but without doing anything bad or weird that I wouldn’t want you to do”. If the model writes software that passes all your tests by hacking them, that’s clearly not what you want. If the model orders a pizza by asking a random stranger on Twitter to order a pizza on its behalf, that’s also probably not what you want.

If you hire a human employee and ask them to do a task for you, the without doing anything bad or weird that I wouldn’t want you to do is implicit, and you trust that the employee knows roughly what that entails. Though you can easily imagine this failing somewhat if you hire a child or someone from a very different culture. Then you need to specify more detailed instructions.

An AI may be even more prone to failing to avoid the bad or weird. When training an AI you are limited in how much data you can show it, and what you can measure. When training on a large number of “practice problems”, measuring things that require physical tests or lots of time (for example, how a particular tool interacts with humans, or whether a particular medicine works as expected) is extremely challenging. And in general, measuring did you do anything bad or weird that I wouldn’t want you to do is often very difficult.

So you need to hope that the model will generalize from your limited number of examples and limited ability to measure its performance. Getting the model to generalize correctly here—avoiding the bad or weird stuff—is the heart of the AI alignment problem.

People sometimes talk about the distinction between “intent alignment” and “value alignment”. Let’s ask some LLMs what these terms mean so that I can paste in some authoritative-looking quotes:

Intent alignment ensures an AI system accurately follows a user's direct commands, while value alignment seeks to instill AI with broader human ethical principles and values. In essence, intent alignment is about correctly interpreting and executing specific instructions. In contrast, value alignment is about ensuring the AI's goals and behaviors are consistent with what is broadly considered beneficial and ethical for humanity, even in unforeseen circumstances.

—Gemini 2.5 Pro

“Intent alignment” means the system is trying to do what its human overseer intends—i.e., its objective is to faithfully pursue the user’s goals, even under uncertainty. “Value alignment” goes deeper: the system’s own preferences or ethics match human values, so it makes good choices even without direct guidance. In short, intent alignment is about deference to our aims; value alignment is about sharing them.

—ChatGPT 5 Thinking

Intent alignment refers to getting an AI system to understand and follow the specific instructions or goals that a human gives it - essentially doing what the human meant rather than a literal or incorrect interpretation of their words.

Value alignment is broader and deeper - it's about ensuring an AI system's behavior aligns with human values, ethics, and long-term interests, even in situations where no explicit instructions were given or where following immediate instructions might conflict with broader human values.

While intent alignment focuses on correctly interpreting and executing specific tasks, value alignment addresses the fundamental question of ensuring AI systems act in ways that are beneficial and ethical according to human values across all contexts.

—Claude Opus 4.1

Putting aside, for now, the cases where someone’s true intent conflicts with some notion of global “human values”, the pragmatic interpretation of why we need “value alignment” is that we want models to avoid the bad or weird when trying to do what we tell them. Another way of thinking about this goal is that we’re teaching the model to understand the unsaid, implicit part of our instruction. We say “get me pizza” but we mean “get me a pizza without doing anything illegal or spending more money than the pizza is actually worth (etc., etc.), also give up if the pizza shop is closed or the task is too difficult for you”.

If you think about “value alignment” this way, i.e. that you’re “just” trying to get the model to understand the implicit avoid bad or weird stuff clause (and all the other implicit and sometimes inarticulable assumptions) then value alignment folds into intent alignment. The avoid bad or weird stuff clause implicit in user requests includes the common ethical principles we’d want AIs to follow: don’t harm people to achieve the goal, don’t break the law, don’t mislead us into thinking you achieved the goal without actually achieving it, etc.

But what about the non-pragmatic interpretation of value alignment often referenced, implicitly and explicitly, in the AI safety literature? This idea of an AI model having a complete notion of “the Good”, “Human Values”, and pursuing that as its main goal? This version of the concept is much more flawed: humans don’t have a consistent set of values that you can task an AI to go off and optimize for. The closest we can get to this is providing people with AIs that try to deeply understand what individuals want and help them with their asks (i.e. intent alignment, with “pragmatic value alignment” folded in).

As more confounders are accounted for, the gap typically shrinks to 2-3 points. Some relevant studies:

• Der, Batty & Deary, 2006: 5475 children, controlled for most relevant factors including maternal IQ → + ~0.5 IQ points

• Strøm et al., 2019: 1782 children, controlled for most relevant factors including maternal IQ → + ~3 IQ points

• Pereyra-Elías, Quigley & Carson, 2022: 7855 children, controlled for most relevant factors including maternal IQ → + ~4 IQ points

However, the gold standard are sibling studies, where breastfed and non-breastfed siblings, who share the same parents and home environment, are compared. Here the effect shrinks further, though doesn’t go to zero.

• Evenhouse & Reilly, 2005: 2734 sibling pairs → +~0.5 IQ points

• Colen & Ramey, 2014: 7319 children; 1773 sibling groups → No effect measured

• Sanefuji et al., 2021: 3521 duos or trios of siblings → Longer/continuous breastfeeding linked to fewer developmental delays, even when sibling-controlled, but IQ not measured

• Goldshtein et al., 2025: 570,532 children, 37,704 sibling pairs → Longer/continuous breastfeeding linked to fewer developmental delays, even when sibling-controlled, but IQ not measured

The largest sibling-controlled study (Goldshtein et al.) unfortunately did not measure IQ. However, they did find that breastfeeding for at least 6 months is linked to about ~1-2 fewer children with developmental delays per 100, and ~7-13 fewer per 1000 with neurodevelopmental conditions. This was also confirmed in their sibling comparisons.

By default, all available LLMs write in an off-putting “slop” style. They overuse cliches, prioritize (bad) style over substance, repeat themselves, and use characteristic LLM-giveaway phrases like “it’s not <X>—it’s <Y>”.

LLMs have seen millions of pages of high-quality writing in training. So why aren’t they able to produce similarly high-quality writing? AI models are better than most CS graduates at Codeforces and better than most math graduates at IMO problems. But when it comes to creative writing, they sound like an overenthusiastic middle-schooler trying to tick all the boxes on the essay scoring sheet.

Perhaps addressing this failure to write well is a matter of “unhobbling”. We do a lot of post-training (supervised learning on curated examples, and increasingly RL) to elicit SOTA models’ math and coding abilities. But we don’t spend similar amounts of effort and compute on creative writing. Understandably so—it’s unlikely to be the same kind of cash cow as software engineering.

However, it’s possible that the amount of post-training needed to elicit high-quality writing is much smaller than needed for solving math problems. High-quality writing is much more abundant on the internet than high-quality solutions to IMO problems. But of course there’s also a large amount of terrible writing and noise. So maybe it’s just a matter of successfully eliciting this “good writing” distribution (and not a matter of teaching the model anything new about the distribution).

Chat LLMs already undergo instruction-following and persona training, to make them function as chatbots with a certain personality. This training is probably what’s responsible for their characteristic writing style—we have elicited the style of “helpful AI assistant”. But if it wouldn’t take much compute to improve the model’s writing ability, it should be (at least partially) doable via in-context conditioning, with the right prompt.

So I decided to investigate this: can we make LLMs write in a different, improved, style just by using a long prompt?

The idea

I came up with the following pipeline to generate a few-shot prompt:

1. Select 5+ pieces in the target style and convert them to markdown, removing any artifacts like images and links that an LLM could not have outputted.

2. Feed these to an LLM and ask it to generate a description of the texts’ style (this will be used in the system prompt.)

3. Separately, feed every text one by one into the LLM and ask it to generate a prompt that could have been used to write that text. (I say something like “Generate a short prompt that could be used by another author to write a similar piece. The prompt should be concise and capture the essence of the original text.”)

4. Construct a human-assistant conversation history with the generated prompts as the “user” messages and the curated texts as the “assistant” responses.

5. Use the conversation history, alongside a system prompt that incorporates the style description from step (2), as the prefix to your final prompt that asks the model to write whatever you would like it to write.

This is implemented in a single short Python file here.

The main principle behind the approach is trying to get the model to learn (in-context) that the assistant responds in the characteristic style of whatever texts you selected (similar to how with many-shot jailbreaks, the model learns that the assistant agrees to harmful requests). Because it’s much easier to create prompts from existing texts than new example texts from prompts, we can trivially construct a long conversation history demonstrating the kind of writing we would like as long as we have access to writing samples. The system prompt part is an extra step so that we also tap into the model’s instruction-following capability by specifying explicitly what kind of writing we want.

In the system prompt, I also augment the extracted style instructions with a constant postfix of:

It is crucial that the Assistant follows the style described above and avoids stereotypical, generic, or cliche language. Things to avoid:
- Cliches like 'it's not X, it's Y'
- Repeating the same concepts in different words
- Generic phrases
- Prioritizing style over substance

Results

As a simple starting point, I’ve been testing this prompting approach with Scott Alexander’s “Every Bay Area House Party” series. I think it emulates the style of Scott’s posts pretty well. Here is an example. Here is another example. Using the same Bay Area House Party writing samples, I prompted the model to write a similar satirical piece about a fictional AI company all hands meeting. This is the result. Though personally I found these pieces humorous, and think they successfully reflect aspects of Scott’s style, the humor is indeed simplistic and there are still too many cliches.

I think there’s still some refinement of the scaffolding prompts to do, and I’m testing the approach with more authors and styles.

Do you have favorite LLM prompting approaches to elicit better and more diverse writing styles? Any writers I should try to emulate next?

1. AI is changing the process of education; kids can learn from AI systems

2. AI is changing what careers will look like, by:

   1. Changing what skills and knowledge are most profitable

   2. Increasing the relative importance of non-academic factors like personality, social status, beauty, and raw intelligence

3. AI is changing what life looks like outside work, by:

   1. Changing how people recreate, interact with each other, and access information

   2. Making work a smaller part of people’s lives by increasing the efficiency of time spent working

Here I’ll focus on (2) and (3), i.e. how AI development should impact the content of kids’ education, rather than the method of instruction.

What skills are profitable is changing

AI is going to change what most jobs look like by the time our children grow up.

Being a software engineer is already different from a few years ago—knowing how to code is still the main part of the job, but an increasingly important part is also knowing how to use AI tools. Jobs like translation have already been more significantly disrupted. And many predict that in a few years, common day-long software engineering and other white-collar office tasks will be fully automated.

Repetitive grunt work that can be done at a computer is likely to go first, followed by reasoning tasks that have easy-to-verify solutions. More open-ended, multi-day projects that require significant planning will take much longer to get right but it’s plausible that by the time a baby born today is thirty years old, most of this type of intellectual labor is also automated.

This means that technical/intellectual roles for humans will increasingly take the shape of oversight or management. Oversight requires attention to detail, conscientiousness, and domain understanding. Management requires strategic thinking, clear understanding of the goal, and general intelligence/creativity.

We are already in a period of human-AI collaboration, where work is being done by humans who use AI tools like AI autocomplete, AI-enabled editors, terminal agents, or chat LLMs. Some level of technical understanding is required to properly use these tools, and this is very likely to continue in the medium future.

Beyond the “copilot” era, technical education will continue to be valuable career-wise because it will still be important to understand and verify what AI systems are doing, especially in an oversight role. Furthermore, the increased number of management roles, where a single individual is controlling a fleet of AI agents, will make it easier to be successful without as much conscientiousness or tolerance for grunt work.

Those who claim that a computer science or STEM degree is no longer as valuable because these skills will be delegated to AI are wrong. Not only are they often mistaken about the speed of AI progress, predicting unrealistically fast rates of automation, but also they neglect the persistent value of overseers and managers who can understand what AI “employees” are doing. Furthermore, anything that requires experience interacting with the real world or spatial reasoning, like mechanical engineering, will take longer to solve, making these skills particularly relevant to the medium future.

Nevertheless, it’s also interesting to think about what happens when/if the AI optimists’ forecast comes to pass. Let’s say AI systems are capable of performing at the level of a senior engineer at a top tech company in almost every way. Let’s also say they have mastered the spatial reasoning required for many robotics use cases. At this stage (which I personally predict is 20+ years away; still possibly solidly in our children’s lifetime), non-technical skills will indeed gain higher relative value. Some forms of work particularly benefit from having real humans, for instance care roles (e.g. looking after children or the elderly), the service industry (e.g. waitressing), or performance art (e.g. live theatre). In addition, some positions (e.g. doctors, lawyers) will still require humans because of regulation. These individuals may not have to actually do much work, but they will bear legal responsibility for any decisions. Obtaining these positions may become a matter of social status and connections.

The world is changing

AI progress will mean that people spend less time working. This will probably occur through a combination of increased unemployment and greater productivity per hour of employed workers.

What will people do with this extra time? My hope is that they’ll spend it with those they love, or spend it learning about the world.

In particular, the decline of “work” as we know it does not spell the decline of the value of knowledge. As I describe in this post, knowing facts makes you a more astute observer of your surroundings, making your life experience denser and more interesting.

A particularly easy to demonstrate variant of this claim relates to the value of literacy. Learning how to read is a massive quality of life improvement—I doubt I have to argue why. It surprises me when people suggest that trying to teach a child to read as early as possible is silly and not worth the effort (some even claim it’s harmful!). Who wouldn’t want a few extra years of literacy rather than being confused by the strange symbols around them? I think you can extrapolate this sentiment to many other areas of knowledge and ability.

I’ve heard several people who work in AI say that learning facts is no longer relevant and that they are instead giving their kids an alternative education focusing on “soft skills” like kindness, creativity, curiosity, and so on. They no longer believe in pushing for academic excellence or challenging extracurriculars, or targeting competitive institutions. I think this approach is mistaken for a few reasons:

• Learning facts lays the foundation for understanding. The more facts you know the easier it is to internalize new knowledge and spot patterns. Without facts you are an uninformed observer of the world. It doesn’t matter if you can look the facts up with an LLM.

• Understanding improves your quality of life, even if it’s irrelevant to your career.

• Curiosity and creativity are not amenable to being taught at schools. You can certainly encourage them in children by modeling these traits yourself and praising kids for expressing them, but I doubt teaching kindness, curiosity and creativity is a good use of multiple hours per day at a regular school, when the alternative is developing their literacy, numeracy, and knowledge of facts at the fastest pace possible.

• Top-tier schools and extracurriculars are useful for building social status and knowing the right people, something that is very likely to continue to be useful.

The upshot

Though I acknowledge that AI is changing what careers look like, and will decrease the amount of work humans do overall, this doesn’t mean that it’s no longer valuable to give children a serious and expansive education. Both because many technical skills will continue to be valuable for quite long, and because being well-educated is a quality-of-life improvement.

I do think that the value of conscientiousness will decrease and value of innate intelligence will increase. Though it will still be present in certain AI oversight roles, the amount of unrewarding grindy office work will likely decrease on net. And of course the “people skills” involved in care work will continue to be valuable as here customers are likely to prefer real humans indefinitely.

Finally, someone shared my idea of what a cool room would look like! The interesting thing about Fruit Towers, especially in retrospect, is that it’s not expensive-looking. The building itself is not impressive, the furniture is probably pretty cheap (compared to the kind seen in fancy modern central London offices), and it’s unlikely that Innocent had to order any custom-made items to achieve their look.

Sure, it’s a little kitsch. It also looks like the place hasn’t changed much since I walked past it thirteen years ago. But I still look at these photos and think it seems like a pleasant space to work from. It’s cheerful, bright, and fun.

I often think: why don’t more people prioritize fun in interior and exterior design?

A few hypotheses:

1. Other people enjoy fun-looking spaces much less than I do

2. It’s more expensive to make a place look fun so it’s not worth the extra cost

3. Fun designs are appealing at first but people who look at the same “fun” stuff daily eventually get tired of it and would overall prefer a plainer option

(1) is quite plausible. Not much more to say about this.

I think (2) is only sometimes true. You can certainly achieve cooler looks by spending money on custom items, but it’s also possible to be creative and construct a fun look at a low cost (e.g. the Fruit Towers example). At the other extreme, I recently learned about Epic Systems’ genuinely epic campus. Clearly they did spend a lot of extra money to make it so fun (though personally I think it was worth it). Their office campus is so cool that people visit it like a tourist destination—you can book self-guided tours on their website.

I can’t paste enough images into this blog post to fully convey how cool the Epic campus is—just look it up on Google images. It’s so whimsical, diverse, and imaginative.

But setting aside the highest standard of fun, which indeed will cost you, cheaper fun interiors exist. In general, companies with a strong consumer brand tend to do well at this.

In my second year of university, I accidentally applied to a marketing internship at Kraft Heinz. When you entered their office, one of the first things you saw was a person-sized tin of Heinz baked beans that was actually a ball pit with bean-colored balls inside. I couldn’t find any pictures online and didn’t capture any at the time, but here is an installation they did elsewhere that has a similar vibe:

What about point (3): that people will quickly grow tired of “novelty” design elements? Again, I’m not sure what the general trend is here, but I suspect it depends a lot on the nature of the fun elements. In particular, you want to avoid succumbing to these pitfalls of fun design:

• Sacrificing beauty for the sake of humor/novelty/surprise

• Introducing impractical or non-ergonomic elements

• Overly intrusive or distracting objects

I think one can achieve a whole lot without incurring any of those problems. For example, take a look at Epic Systems’ cafeteria, which is King’s Cross Themed.

For reference, this is the interior of London King’s Cross and neighboring St Pancras Stations:

Even if you did not know of King’s Cross, even if you had never stepped inside a railway station for your entire life, the Epic cafeteria’s interior is still aesthetically pleasing. It does not seem garish or uncomfortable. But still it is fun and interesting. I think it’s possible to avoid designs that people easily tire of while still keeping things fun.

So far, I’ve only discussed office spaces. Office spaces are easier to play around with because they often have larger rooms and taller ceilings, giving you more space to use, especially in communal areas like canteens. But private homes can also be made fun!

Remember Fruit Towers with its artificial grass flooring? You can replicate a similar look with a green shaggy rug. Here’s one I found for $84.99 on Wayfair:

More adventurous are these moss-inspired rugs on Etsy (though I have no idea about their quality or ease of cleaning):

Fairy lights also add fun and whimsy in a low-cost and easy way to most rooms.

A higher-effort look could be modeling your kitchen space after a retro American diner: chrome elements, checkerboard tiles, booth-style seating:

Another way to create a fun look is by picking a fun color scheme. A fun color scheme doesn’t have to be garish or make the space look like a kids’ playroom. The key is to introduce a tasteful amount of non-neutral tones. Typical interior design, be it traditional or old-fashioned, tends to emphasize colors like white, beige, brown, gray, and black. There are broadly three approaches to going beyond this boring color palette:

1. Pick a single accent color and sprinkle it throughout the space

2. Use a color palette of different colors that look nice together

3. Go for a neutral base accented with multicolor elements spanning the entire rainbow (either vibrant or pastel)

Starting from a neutral room, how do you add colorful elements? Here are some ideas, in ascending order of effort/expense:

• Fabric! Pick colorful cushion/pillow covers, bedding, tablecloths, blankets.

• Add a colorful rug.

• Decorative items: colorful vases or tableware that’s on display, picture frames.

• Artwork: find artwork you like that fits the color scheme.

• Indoor plants or flowers, can even be fake if that fits with the overall style.

• Pick colorful items of furniture: chairs, lampshades, sofas, and ottomans are a good place to start looking for color.

• Paint: use an accent color on one of the walls, or paint cupboards/cabinets/shelves.

Of course many ways of making a room more fun are idiosyncratic to a particular theme, concept, or space. Part of the project involves being creative, thinking about the specifics of the space you’re dealing with, and keeping fun in mind whenever decorating/acquiring new furnishings. If you’re anything like me, it’s worth spending time and effort here because there is probably low-hanging fruit that you’re currently missing and with a little thought you’ll be able to sprinkle delight into the everyday.