The Rationalist Conspiracy

(By Eliezer Yudkowsky; posted here with permission)

Building an intellectual edifice requires ongoing conversation, and that ongoing conversation needs four layers of speech to be successful.

(Yes, four. Not three. Later I’m going to pretend I didn’t say that, but right now I’m serious and this is important.)

There is a widespread traditional notion that the total absence of critique is bad; that it is a bad sign to have a conversation consisting of people saying X and nobody saying “hey maybe not-X”.

Why is this bad? Well, because people could say stupid things about X, and nobody would call them on the stupidity. Yikes!

Okay, so here’s the thing: If the people saying “hey maybe not-X” don’t anticipate losing points from being called out on stupid critiques, that doesn’t create a conversation either. I am speaking here from awful experience and many people reading this will have seen the same.

A conversation that successfully builds an intellectual edifice has four *necessary* layers. I’m not saying “necessary” as an emphasis-word for how nice it is to have more layers. I mean, “If you eliminate the fourth layer, the mechanism falls apart.”

You need:

0: People saying X.
1: Critics saying “hey maybe not-X”.
2: Responses/counter-critiques saying “well maybe X after all”.
3: Counter-responses saying “but maybe not-X after all”.

If you eliminate layer 3, that means the conversational record doesn’t include critics responding to critiques of their criticism.

In other words, the critics saying “not-X” won’t anticipate needing to defend their “not-X” claims.

Layers 0-2 being visible in the record, but not layer 3, is the sort of situation you have when biologists speak of evolution (0, object-level claim), and a priest says something about evolution being true but only God being able to create the first life that started it (1, critique), and biologists reply with a detailed account of the current thought on abiogenesis (2, response/counter-critique); and the priest does not reply with detailed thoughts explaining why the current thinking on abiogenesis is technically flawed (absence of 3, the counter-response).

0-2 is what you have when, say, Eric Drexler is writing detailed technical books about molecular nanotechnology; and a famous chemist says something profoundly stupid indicating they have not read very far into this literature (e.g. “but the sticky-fingers problem!”); and Eric Drexler writes a response dissecting the critique, which doesn’t receive as much media attention; and the chemist doesn’t care or replies with something that is visibly not very detailed or thought-out.

Conversational layers 0-2 being visible in the record, but not much layer 3 or an unimpressive layer 3, generally represents a situation where the critic doesn’t expect that their criticism will come under harsh scrutiny to which the critic will be socially obliged to respond intelligently as part of the widely seen public record. On the critical side, that’s just as bad as there being just layer 0.

When I say layers 0-3 need to be there, I mean that there must be a social incentive to do them well; people must lose status points for saying dumb things at any of these layers. When as an outsider you look at the conversational record, all of these layers should not just be merely present as a checkmark on a list. You should be looking for the same standards of impressive technical-sounding words, or abiding by epistemic norms and discourse norms or whatever, as you would demand of the ground-level statements.

Now here’s the dire part: the current academic journal system, in practice, operates at layers 0-2. You submit your paper, and the reviewers offer a response, and you’re expected to have an intelligent response to the review. But these reviewers (often anonymous) do not expect to lose huge social brownie points if their critique is stupid. Even if the reviewer is supposed to offer some kind of counter-response for the record, it can be a casual and stupid counter-response, and nobody will go “Hey what the hell are you doing” at that.

Absent any incentive to be smart, the reviews are often really, really stupid; especially if the original paper doesn’t look to be authored by a high-status person. Though I’ve heard from more than one person with very high status in their field that even the reviews *they* get are stupid.

Modern academics treat stupid bad reviews as an environmental hazard. It’s not *conversation*. It’s not building the edifice of knowledge.

So where does the real conversation happen, in scientific fields, when there’s a proposition worthy of debate and not just another unquestioned fact to be recorded?

Maybe it happens in the bar at conferences, where people are speaking in realtime in front of their friends, and would actually lose status points if they uttered a dumb critique that was shot down and their counter-response looked stupid. Or maybe it happens on email lists. It could be happening on Facebook, for all I know of any particular field.

But the journals merely record an intellectual edifice that was built elsewhere. The real conversation that creates the intellectual edifice in the first place couldn’t happen with the journals as a medium.

The only time I’ve seen a stream of journal articles that looked like they were seriously *building*, not just *recording*, an intellectual edifice, it’s been in analytic philosophy. Analytic philosophy is about debating, qua debating, in a way that chemistry isn’t. I could be wrong, but I expect that editors of analytic-philosophy journals *do* expect intelligent counter-responses; and that reviewers expect to lose status points if they can’t come up with intelligent counter-responses. (Though this could be confounded by analytic philosophers having the highest IQ of any graduate group in academia, yes that happens to be a thing that is true.)

Unfortunately analytic philosophy lacks the ability to settle on any answer and declare it settled. So be it noted that just because you ought to demand high standards of both counter-critiques and counter-responses, it doesn’t mean that nobody’s right. Or that there isn’t such a thing as one side having an overwhelming weight of argument. Or that so long as both sides are writing long technical arguments they must have equally socially respectable positions which is all there is to epistemics, etcetera.

Mainstream media that pretends to be serious pretends to have layers 0-1, though journalists often just make up their own critiques, or twist the quoted critics to make the criticism look like the cliche they expect the reader to expect. And when the media is not pretending to be serious they operate at unvarnished layer 0.

It’s sad, it’s really sad, to compare the current academic conversation about AGI alignment–not that the academics know they should be calling it that nowadays–with the informal conversations I saw on email lists in the late nineties. Email lists where you knew that if you said something dumb, even if it was an ohmygosh virtuous “critique”, Robin Hanson might reply with an email pointing out the flaw, and everyone else on the mailing list would see that reply. On those mailing lists there was a real conversation, and that’s what built up the early edifice of thought about AI alignment. There’s been more theory built up since those days, but almost everything the public got to see in Bostrom’s _Superintelligence_ just records the edifice of thought built up on those email lists where a real, actual conversation took place.

By comparison, academic discourse on AGI comes from the void and is cast into the void. It shows little awareness of previous ideas, it is not written as if to anticipate obvious responses, the obvious responses go unpublished in the same public record, and certainly there is no detailed and impressive counter-response.

When you publish a journal article claiming to have shot down the so-called notion of the intelligence explosion once and for all, and your article is about hypercomputation being impossible, then you are clearly not operating in an environment where you expect to be socially obliged to come up with an intelligent response to counter-critique. Perhaps the thought crossed your mind that somebody might say “Hey maybe the intelligence explosion doesn’t require hypercomputation, and you made little or no effort to establish that it did, and if you say that’s true by definition then this is not the definition anyone else in the field uses.” But if so it was a fleeting thought and you didn’t expect to be troubled, to lose reputation, if any of your prey tried to reply that way to your predation. When you published your “critique”, you were done scoring all the points you expected to score off them, and you didn’t expect to lose any points for responding casually or not at all to their counter-critique.

So the academic conversation hasn’t gotten anywhere near as far as the informal conversation on those old email lists in the late 90s, never mind everything built up since then.

Unfortunately, the traditional scientific upbringing speaks only of the importance of criticism.

EAs used to ask: “Has there been critique of MIRI’s ideas? Who are the critics?” If you take this literally, they were asking to see a record of a conversation that included layers 0-1. Implicitly, they were asking to see 0-2; they would have been surprised if I showed them critique but couldn’t point to anything that responded to the critique.

But if you want to know that critics are a part of the conversation, you need to be able to point to serious-looking counter-responses by critics. Back in the old days I’d always reply “Robin Hanson is the serious critic, there isn’t really anyone else worth pointing to”, because nobody else was writing detailed counter-responses to detailed counter-critiques.

Keep that in mind the next time you’re trying to judge the strength and health of an ongoing conversation… or, this is very important, *or* when you’re wondering how seriously to take a critic. Don’t ask, “Is there a forum where both sides of the story can be heard?” Rather ask, “Is there back-and-forth-and-back-*and*-forth?” Don’t ask, “Has somebody performed the duty of critique?” Ask, “How impressed am I by the counter-response to the counter-critique?”

(Facebook discussion)