Four Layers of Intellectual Conversation

(By Eliezer Yudkowsky; posted here with permission)

Building an intellectual edifice requires ongoing conversation, and that ongoing conversation needs four layers of speech to be successful.

(Yes, four. Not three. Later I’m going to pretend I didn’t say that, but right now I’m serious and this is important.)

There is a widespread traditional notion that the total absence of critique is bad; that it is a bad sign to have a conversation consisting of people saying X and nobody saying “hey maybe not-X”.

Why is this bad? Well, because people could say stupid things about X, and nobody would call them on the stupidity. Yikes!

Okay, so here’s the thing: If the people saying “hey maybe not-X” don’t anticipate losing points from being called out on stupid critiques, that doesn’t create a conversation either. I am speaking here from awful experience and many people reading this will have seen the same.

A conversation that successfully builds an intellectual edifice has four *necessary* layers. I’m not saying “necessary” as an emphasis-word for how nice it is to have more layers. I mean, “If you eliminate the fourth layer, the mechanism falls apart.”

You need:

0: People saying X.
1: Critics saying “hey maybe not-X”.
2: Responses/counter-critiques saying “well maybe X after all”.
3: Counter-responses saying “but maybe not-X after all”.

If you eliminate layer 3, that means the conversational record doesn’t include critics responding to critiques of their criticism.

In other words, the critics saying “not-X” won’t anticipate needing to defend their “not-X” claims.

Layers 0-2 being visible in the record, but not layer 3, is the sort of situation you have when biologists speak of evolution (0, object-level claim), and a priest says something about evolution being true but only God being able to create the first life that started it (1, critique), and biologists reply with a detailed account of the current thought on abiogenesis (2, response/counter-critique); and the priest does not reply with detailed thoughts explaining why the current thinking on abiogenesis is technically flawed (absence of 3, the counter-response).

0-2 is what you have when, say, Eric Drexler is writing detailed technical books about molecular nanotechnology; and a famous chemist says something profoundly stupid indicating they have not read very far into this literature (e.g. “but the sticky-fingers problem!”); and Eric Drexler writes a response dissecting the critique, which doesn’t receive as much media attention; and the chemist doesn’t care or replies with something that is visibly not very detailed or thought-out.

Conversational layers 0-2 being visible in the record, but not much layer 3 or an unimpressive layer 3, generally represents a situation where the critic doesn’t expect that their criticism will come under harsh scrutiny to which the critic will be socially obliged to respond intelligently as part of the widely seen public record. On the critical side, that’s just as bad as there being just layer 0.

When I say layers 0-3 need to be there, I mean that there must be a social incentive to do them well; people must lose status points for saying dumb things at any of these layers. When as an outsider you look at the conversational record, all of these layers should not just be merely present as a checkmark on a list. You should be looking for the same standards of impressive technical-sounding words, or abiding by epistemic norms and discourse norms or whatever, as you would demand of the ground-level statements.

Now here’s the dire part: the current academic journal system, in practice, operates at layers 0-2. You submit your paper, and the reviewers offer a response, and you’re expected to have an intelligent response to the review. But these reviewers (often anonymous) do not expect to lose huge social brownie points if their critique is stupid. Even if the reviewer is supposed to offer some kind of counter-response for the record, it can be a casual and stupid counter-response, and nobody will go “Hey what the hell are you doing” at that.

Absent any incentive to be smart, the reviews are often really, really stupid; especially if the original paper doesn’t look to be authored by a high-status person. Though I’ve heard from more than one person with very high status in their field that even the reviews *they* get are stupid.

Modern academics treat stupid bad reviews as an environmental hazard. It’s not *conversation*. It’s not building the edifice of knowledge.

So where does the real conversation happen, in scientific fields, when there’s a proposition worthy of debate and not just another unquestioned fact to be recorded?

Maybe it happens in the bar at conferences, where people are speaking in realtime in front of their friends, and would actually lose status points if they uttered a dumb critique that was shot down and their counter-response looked stupid. Or maybe it happens on email lists. It could be happening on Facebook, for all I know of any particular field.

But the journals merely record an intellectual edifice that was built elsewhere. The real conversation that creates the intellectual edifice in the first place couldn’t happen with the journals as a medium.

The only time I’ve seen a stream of journal articles that looked like they were seriously *building*, not just *recording*, an intellectual edifice, it’s been in analytic philosophy. Analytic philosophy is about debating, qua debating, in a way that chemistry isn’t. I could be wrong, but I expect that editors of analytic-philosophy journals *do* expect intelligent counter-responses; and that reviewers expect to lose status points if they can’t come up with intelligent counter-responses. (Though this could be confounded by analytic philosophers having the highest IQ of any graduate group in academia, yes that happens to be a thing that is true.)

Unfortunately analytic philosophy lacks the ability to settle on any answer and declare it settled. So be it noted that just because you ought to demand high standards of both counter-critiques and counter-responses, it doesn’t mean that nobody’s right. Or that there isn’t such a thing as one side having an overwhelming weight of argument. Or that so long as both sides are writing long technical arguments they must have equally socially respectable positions which is all there is to epistemics, etcetera.

Mainstream media that pretends to be serious pretends to have layers 0-1, though journalists often just make up their own critiques, or twist the quoted critics to make the criticism look like the cliche they expect the reader to expect. And when the media is not pretending to be serious they operate at unvarnished layer 0.

It’s sad, it’s really sad, to compare the current academic conversation about AGI alignment–not that the academics know they should be calling it that nowadays–with the informal conversations I saw on email lists in the late nineties. Email lists where you knew that if you said something dumb, even if it was an ohmygosh virtuous “critique”, Robin Hanson might reply with an email pointing out the flaw, and everyone else on the mailing list would see that reply. On those mailing lists there was a real conversation, and that’s what built up the early edifice of thought about AI alignment. There’s been more theory built up since those days, but almost everything the public got to see in Bostrom’s _Superintelligence_ just records the edifice of thought built up on those email lists where a real, actual conversation took place.

By comparison, academic discourse on AGI comes from the void and is cast into the void. It shows little awareness of previous ideas, it is not written as if to anticipate obvious responses, the obvious responses go unpublished in the same public record, and certainly there is no detailed and impressive counter-response.

When you publish a journal article claiming to have shot down the so-called notion of the intelligence explosion once and for all, and your article is about hypercomputation being impossible, then you are clearly not operating in an environment where you expect to be socially obliged to come up with an intelligent response to counter-critique. Perhaps the thought crossed your mind that somebody might say “Hey maybe the intelligence explosion doesn’t require hypercomputation, and you made little or no effort to establish that it did, and if you say that’s true by definition then this is not the definition anyone else in the field uses.” But if so it was a fleeting thought and you didn’t expect to be troubled, to lose reputation, if any of your prey tried to reply that way to your predation. When you published your “critique”, you were done scoring all the points you expected to score off them, and you didn’t expect to lose any points for responding casually or not at all to their counter-critique.

So the academic conversation hasn’t gotten anywhere near as far as the informal conversation on those old email lists in the late 90s, never mind everything built up since then.

Unfortunately, the traditional scientific upbringing speaks only of the importance of criticism.

EAs used to ask: “Has there been critique of MIRI’s ideas? Who are the critics?” If you take this literally, they were asking to see a record of a conversation that included layers 0-1. Implicitly, they were asking to see 0-2; they would have been surprised if I showed them critique but couldn’t point to anything that responded to the critique.

But if you want to know that critics are a part of the conversation, you need to be able to point to serious-looking counter-responses by critics. Back in the old days I’d always reply “Robin Hanson is the serious critic, there isn’t really anyone else worth pointing to”, because nobody else was writing detailed counter-responses to detailed counter-critiques.

Keep that in mind the next time you’re trying to judge the strength and health of an ongoing conversation… or, this is very important, *or* when you’re wondering how seriously to take a critic. Don’t ask, “Is there a forum where both sides of the story can be heard?” Rather ask, “Is there back-and-forth-and-back-*and*-forth?” Don’t ask, “Has somebody performed the duty of critique?” Ask, “How impressed am I by the counter-response to the counter-critique?”

(Facebook discussion)

Radar Detector-Detector-Detector-Detector Almost Certainly a Hoax

The Wikipedia article on radar detector detectors used to say:

“In 1982 the US military funded a project, codenamed R4D (radar detector-detector-detector-detector), in order to develop a device capable of detecting radar detector-detector-detectors.”

This has been widely shared online, by (among others) the rationalist blog Slate Star Codex, and the /r/wikipedia and /r/TIL subreddits. However, it’s almost certainly a hoax. The evidence:

– First, this “fact” was added to Wikipedia by an unregistered, anonymous IP address. No source was provided.

– Second, similar sentences have been added to this article before. These edits are always anonymous, always unsourced, and tell contradictory stories:

“In response, a few people also employ the use of a radar detector detector detector to detect the detection of their radar protector, but that is rare. At this time, the police are developing a radar detector detector detector detector to counter-act this.” (Aug. 2008)

“Furthermore, it is now known that a radar detector detector detector detector detector is being developed by military organizations in many countries.” (Sep. 2008)

“Scientists are currently working on a radar detector detector detector detector detector detector detector detector detector which is in the early stages of prototyping. There were plans to create a radar detector detector detector detector detector detector detector detector detector detector, but these were scrapped due to gross stupidity.” (Sep. 2008)

“Scientists at Cambridge University are currently working on a radar detector detector detector detector detector detector detector detector detector which is in the early stages of prototyping. There were plans to create a radar detector detector detector detector detector detector detector detector detector detector, but these were scrapped due to gross stupidity.” (May 2009)

“This technology was countered with the invention of the radar-detector-detector-detector-detector. However, due to escalation, the development of the radar-detector-detector-detector-detector-detector was deemed necessary.” (Feb. 2011)

– Third, there seem to be no real sources for a radar detector-detector-detector-detector, or for an “R4D” project. All the Google hits for this supposed “military project” trace back to Wikipedia. Google Books, Google News, and Google Scholar turn up nothing, except for some references to the Douglas R4D cargo plane. There are a few posts on forums, but they’re obvious jokes. Eg.:

“More importantly.. Are there radar detector detector detector detectors? That Meanz we Need 2 Stay 1 Step Ahead of the Game. Some1, Quick, Create a Radar Detector Detector Detector Detector Detector *Head Explodes*” (link)

– Fourth, the military doesn’t seem that interested in radar detectors, never mind radar detector-detector-detector-detectors. Military radar towers are generally large, powerful, and obvious (see eg. this Distant Early Warning station); a radar system weak enough to avoid detection wouldn’t be very useful against stealth planes at long ranges, or against jamming devices. The Wikipedia article on radar detectors is exclusively about civilian use. There’s plenty of military interest in radar jamming, or electronic countermeasures, but that’s a different thing. (Radar jammers are illegal for civilians under FCC rules.)

– Fifth, there’s really no reason to build a radar detector-detector-detector-detector. A radar detector-detector-detector is useful if you have a radar detector, because it lets you distinguish between radar (which the detector can ignore) and radar detector detectors (which mean the detector has to shut down, in places where detectors are illegal). However, the only reason someone would have a radar detector detector detector is if they also had a radar detector. Hence, it’s easier and equally useful to simply detect the original radar detector.

Dark Patterns by the Boston Globe

After years of falling revenue, some newspapers have resorted to deception to boost their subscription numbers. These dishonest tactics are sometimes called “dark patterns” – user interfaces designed to trick people.

For example, this is a Boston Globe story on Bernie Sanders:

Screenshot from 2016-04-24 18-06-33.png

Before you can read the article, there is a pop-up ad asking you to subscribe. By itself, this is annoying, but not deceptive. The real dark pattern is hidden at the top – the ‘Close’ button (circled in red) uses a very low contrast font, making it hard to see. It’s also in the left corner, not the standard right corner. This makes it likely that users won’t see it, causing them to subscribe when they didn’t have to.

One the ‘Close’ link is clicked, deception continues:

Screenshot from 2016-04-24 18-06-57.png

At the bottom, there’s a non-removable, high-contrast banner ad asking for a paid subscription. Again, this is annoying, but honest. However, the circled text “for only 99 cents per week” is not honest. It’s simply a lie, as later pages will show.

Clicking the “Subscribe now” button brings up this page:

Screenshot from 2016-04-24 18-07-17

Here, it becomes obvious that $0.99 per week isn’t the real price. It’s common for companies to have initial discounts, which isn’t itself a dark pattern. The problem on this page is that the real price is never stated. This misleads the consumer.

Clicking the “Sign Up” button reveals yet more dark patterns:

Screenshot from 2016-04-24 18-08-14

This is the first signup form. It shows the amount charged, but only for the first month ($3.96). The real price is below that, in smaller font, and made less obvious by the red highlighting on the previous line. At first glance, it looks like the same price ($3.99), but the real rate is actually $3.99 per week, while the number in red is $3.96 for the entire month. In addition, in the left column, three of the marketing email signups are checked “yes” by default, so people will subscribe without noticing.

The next page is pretty similar, it’s a standard credit card form:

Screenshot from 2016-04-24 18-58-51.png

And this page is the last one you see before ordering:

Screenshot from 2016-04-24 19-00-27.png

It isn’t visible, but this page is yet another dark pattern, because even right before the purchase it never shows the real price. To find the real price, one must click the little “FAQs” link on the right:

Screenshot from 2016-04-24 19-04-15.png

Then, hidden among questions about crosswords, obituaries, and horoscopes, the user has to click the circled link to discover:

Screenshot from 2016-04-24 19-06-43.png

Yes, the real price isn’t the $0.99 per week in the banner ad, or even the $3.99 per week in fine print on the purchase page. It’s $6.93 per week, almost twice as much as the purchase page rate, and seven times as much as the banner. Since this price only kicks in after a year, it’s almost impossible for average users to notice, unless they carefully check each and every bank statement.

If they do find out and try to cancel, they’ll discover this catch, which isn’t stated or even implied during signup:

Screenshot from 2016-04-24 19-09-44.png

A Boston Globe reader can subscribe online. If they have a question, they can ask over email, or through a convenient live chat service. But if they want to stop paying, they have to call and ask on the phone, no doubt after a long hold time and mandatory sales pitches. There’s no plausible reason for this, other than forcing people to pay when they’d rather cancel the service.

In the short term, these dishonest tricks raise revenue for newspapers that use them. But in the longer term, they do even more damage, by giving the whole industry a reputation for bad business practices. Cable companies can get away with it because of government-granted monopolies; newspapers won’t be able to.

Vegetarian Advocacy Is Ineffective

Most pro-vegetarian advocacy is not very effective. The problem isn’t the goal of making animals happy – it’s likely that farm animals have moral value, and almost everyone agrees that factory farm conditions are horrible. Instead, the problem is the most common strategy used to achieve that goal: namely, emotionally-charged rhetoric to convince people, either in person or on the Internet, that they should personally not eat meat.[1] This category of solution to animal rights problems is likely ineffective at best, and downright harmful at worst. As GiveWell says, non-profits shouldn’t “point to a problem so large and severe (and the world has many such problems) that donors immediately focus on that problem – feeling compelled to give to the organization working on addressing it – without giving equal attention to the proposed solution, how much it costs, and how likely it is to work.”

This essay doesn’t address whether animals have nonzero moral value, which has been thoroughly discussed elsewhere. Nor does it look at other solutions to factory farming, like government legislation, or scientific research to develop meat substitutes. It simply tries to show that a lot of pro-vegetarian advocacy, as it’s currently practiced, is ineffective or outright counterproductive. Since there’s a wide range of arguments to consider, this writeup has been broken up into chunks, of which this is the first. This chunk looks at the simplest sub-question: is vegetarian advocacy a cost-effective way of reducing the amount of meat eaten?

First, we should check: is any kind of activism ever cost-effective? The answer seems to be yes. Eg., everyone knows about the campaign for gay marriage, which won a full victory in the US in 2015 (though after many decades of work). However, there seems to be a clear pattern in which activism campaigns are successful, and which aren’t. Psychologist Daniel Kahneman divides the brain into two systems: System 1, which is fast and intuitive, and System 2, which is slow and reflective. System 1 evolved before System 2, and is more connected to the physical actions we take, while System 2 is more closely linked with what we say, write, and think. The historical record shows that activism aimed at System 2 is difficult, but can sometimes be effective. On the other hand, activism aimed at System 1 is usually a waste of effort.

[Edit: The pattern of successful vs. unsuccessful activism still seems real, but the distinction being drawn here is not what Kahneman meant by System 1 and 2. Apologies for the mistake, further clarification to follow.]

For example, consider the history of activism against racism. In 1955, Rosa Parks started the Montgomery Bus Boycott, which triggered a wave of anti-racist advocacy across the US. Though it was a tough battle, after nine years, Congress and President Lyndon Johnson passed the Civil Rights Act, a sweeping bill that outlawed almost all racial discrimination. Over the next fifty years, advocates kept pushing harder and harder for an end to racism. On a System 2 level, this campaign was so successful that virtually no public figure now advocates for segregation, a radical change from 1950. When businessman Donald Sterling was caught being racist, it was such a big deal that it became the focus of his entire Wikipedia page. However, System 1 has been much more stubborn. After sixty years of advocacy, a psychology metric called the Implicit Association Test shows that most white Americans still have negative System 1 assocations with black faces.

There have been many other campaigns to persuade people’s System 1s through rhetoric, advertising, peer pressure, graphic images, and so on, but they usually get negligible or marginal results, compared with the effort invested. Consider smoking as another test case. There’s near-universal agreement that smoking is very, very bad for health, in both the short term and long term, and there’s been enormous efforts to convince smokers to quit. On one side, most smokers themselves know darn well how bad smoking is, and many make heroic efforts to stop. On the other side, governments, nonprofits, and smoking-cessation-aids companies spend billions researching how to help people stop smoking. Thousands of studies have been done on the effectiveness of anti-smoking programs, so we’ve put a lot of effort into finding the very best strategies.

The results of this enormous, expensive, fifty-year effort have been modest at best. The US smoking rate has fallen from ~40% to ~20%, a decline of ~50%, or a bit over 1% per year. Of that decline, much of it was caused by fewer people taking up smoking in the first place. Much of the remainder was caused by laws that make cigarettes more expensive and difficult to use, such as taxes, restrictions on sales, public smoking bans, restaurant smoking bans, and so on. Hence, all anti-smoking programs, cessation aids, addiction research, PR campaigns, etc. combined have given us a few tenths of a percent decline per year.[2] More generally, Scott has a long essay on how these types of programs are ineffective:

“We figured drug use was “just” a social problem, and it’s obvious how to solve social problems, so we gave kids nice little lessons in school about how you should Just Say No. There were advertisements in sports and video games about how Winners Don’t Do Drugs. And just in case that didn’t work, the cherry on the social engineering sundae was putting all the drug users in jail, where they would have a lot of time to think about what they’d done and be so moved by the prospect of further punishment that they would come clean. And that is why, even to this day, nobody uses drugs. (…)

What about obesity? We put a lot of social effort into fighting obesity: labeling foods, banning soda machines from school, banning large sodas from New York, programs in schools to promote healthy eating, doctors chewing people out when they gain weight, the profusion of gyms and Weight Watchers programs, and let’s not forget a level of stigma against obese people so strong that I am constantly having to deal with their weight-related suicide attempts. As a result, everyone… keeps gaining weight at exactly the same rate they have been for the past couple decades.”

To create a quantitative model, we can look at the test case of online ads, which Animal Charity Evaluators (ACE) have done substantial research on. ACE says that “online vegetarianism and vegetarianism ads are currently our most cost-effective intervention recommendation”.[3] In economic terms, one should naively expect that one dollar of ad purchases causes about one dollar of money moved, where “money moved” equals the change in (retail price of goods purchased – marginal cost of goods sold), summed over all relevant goods. If each dollar of ads caused more than one marginal dollar of money moved, companies would just buy more ads to make more money, until decreasing marginal returns brought gains back down to $1.

Of course, that’s only a rough approximation. Any given ad campaign might be more or less effective, for any number of reasons. However, in this case, the sheer magnitude of the gap is cause for great concern. ACE estimates that the cost-per-click (CPC) of pro-vegetarian ads is about two to twenty cents, and that based on survey data, around 2% of ad clickers become vegetarian or vegetarian. American adults spend around $5,000 to $10,000 on food per year,[4] so total money moved through becoming vegetarian is on the order of $100,000. Hence, under the naive economic model, the chance of people becoming vegetarian because of an ad click is roughly 0.00002% – 0.0002%, a massive four to five orders of magnitude smaller than ACE’s estimate. A likely explanation for this, as ACE themselves note, is that people only click on the ads if they were thinking about becoming vegetarian anyway. About two thousand people typically see an online ad for each person who clicks, so even a very small number of existing proto-vegetarians in the ad audience fully account for the survey data.

[Edit: 0.00002% is inaccurate, even within this model. Two cents per click is only available in poorer countries, which have much less total money moved, bringing the estimated odds back to around 0.0002%. However, poorer countries also consume much less meat, which largely compensates for this effect in terms of benefit per dollar.]

What empirical data we have backs this model up. After decades of vegetarian advocacy, PETA says that “society is at a turning point” for veganism. But polling data suggests that only 5% of Americans are vegetarian, and that this percentage has gone down since the 90s. Only 2% consider themselves vegan. Further, these numbers are likely overestimates. Polls often show that a few percent will support any idea, no matter how crazy, like “all politicians are secretly alien lizards” (really!); in addition, most people who said “yes” to the vegan question said “no” to the vegetarian question, which suggests lots of confusion. Even among self-described vegetarians, more detailed surveys show that most still eat an average of one serving of meat per day, which nicely confirms the System 1/System 2 model. It’s easier to convince System 2 that vegetarianism is a good idea than System 1, creating a paradox where two-thirds of ‘vegetarians’ ate meat yesterday.

In addition, even if advocacy is successful, the benefits from one person going vegetarian are not very large compared to their cost. Statistics from vegetarian advocacy groups usually cite the large numbers of animals killed. However, each individual animal life is very short, because meat becomes cheaper when farmers breed animals for rapid growth. Consider chickens as an example. The average American eats 27.5 chickens per year; since a broiler chick takes about five weeks to grow, this gives us 2.64 chicken-years of life prevented by one year of vegetarianism. To evaluate the cost of not eating chicken, we must look at not the price of the chicken, but the “consumer surplus” – how much benefit the customer derives from the product. With some rough math, this comes out to around $18.84 per chicken;[5] this is averaged over both people who like chicken a lot, and people who only like it barely enough to buy it. That gives us a total value-from-chicken (after the cost of the chicken) of $518 per person per year, which can be given up to save 2.64 years of chicken suffering.

Comparing this to human charity, GiveWell estimates a cost-per-child-saved from malaria nets of $2,838. Since GiveWell’s numbers only count children saved, given developing-world life expectancy, each life saved creates about 60 extra person-years. That equals a cost of $47 per person-year, compared to the average cost of $196 per chicken-year from not eating chicken. Therefore, the person-years are a much cheaper buy, even if we assume that chicken lives are so incredibly bad that preventing one chicken-year is as good as saving one person-year.

In fact, this estimate is still biased in favor of chickens, for two main reasons. The first is that GiveWell’s estimate doesn’t include the benefits of mosquito nets beyond saving children; these include saving adults from death by malaria (though adults have a much lower fatality rate), preventing many more non-fatal cases of malaria, preventing mosquito-borne disease in general, and of course preventing mosquito bites, which (ignoring everything else) can be done at a cost of hundredths of a penny per bite. The second reason is that Against Malaria Foundation is selected for being extremely low-risk; given a donation to AMF of $X dollars, one can be extremely confident in at least Y lives being saved. GiveWell thinks it’s likely that if we take on riskier projects, like scientific research and policy reform, the expected cost per life saved will be even lower. Indeed, one of these riskier projects is actually US policy reform to improve farm animal welfare.

Several people have suggested this is an unfair comparison. Most donors have a limited “charity budget”, and $10 that they spend on one charity is $10 which they don’t spend on another. However, what if going vegetarian increases people’s willingness to do other kinds of good? Unfortunately, psychology research suggests this is unlikely – people often have a “do-gooding budget” in addition to their financial budget, and doing one altruistic act will decrease their willingness to do others. To quote Nick Cooney, founder of animal charity The Humane League:

“The contribution ethic refers to the feeling many people have that “I’ve done my part on issue A, so it’s okay for me to ignore issues B, C, and D.” During a Humane League campaign to get restaurants to stop purchasing products from a particularly cruel farm, owners would often tell us that they already do something to help animals (“We buy our eggs from a local farm,” or “I donate to the ASPCA”) so we shouldn’t be bothering them. This phenomenon worked across issues too, as we often had owners or chefs tell us how they supported some other social cause so we should leave them alone about this one.

In addition to feeling like they’ve done their part and therefore don’t need to do anything more, people often overestimate the amount of good they’ve done. Combined, these phenomena make it hard to move people beyond small actions for their one or two preferred causes (Thogersen and Crompton 2009).”


1. Of course, there are many different animal-rights diets. Other types of ethics-based dietary restrictions include “vegan”, “lacto-ovo vegetarian”, “pescatarian”, “flexitarian”, “reducetarian”, and so on. Since keeping track of all the different labels is unwieldy, for the most part I’ll simply say “vegeterian”, even though the main arguments also apply to many other ethics-based diets. I’ve seen people use “veg*n”, but that’s also unwieldy due to ‘*’ not being a letter.

2. One possible complication is that nicotine is chemically addictive. However, treatments to fight the chemical part of addiction have been available for decades, which I’d expect to largely cancel out this effect. In addition, similar problems (alcoholism, gambling, sugar consumption, etc.) have generally seen similar results.

3. Although I disagree with them on several issues, ACE should be commended for trying to be quantitative.

4. Data from this Gallup poll. Note that this data is based on self-reports, and is per-household rather than per-adult, so it is only approximate.

5. The cost of a whole chicken is roughly $1.50 per pound per this article, or $9 for a six-pound chicken. From this paper, the price elasticity of demand of chicken is around -0.8, so we can naively model the demand-price curve with the ODE dy/dx = -0.8*y/x, y(9) = 1, which has the solution y(x) = 5.8/x^0.8. Integrating from x = 9 to, say, 100, we get 27.84, which subtracting the $9 for the chicken’s price gives us a consumer surplus of $18.84. This is, of course, just a rough guess.

Asking Good Questions

Aumann’s Agreement Theorem says two perfect rationalists can’t “agree to disagree”. Therefore, when two people disagree, a good question is one that makes either the asker or askee change their minds. Some examples of bad questions are:

– Why don’t you agree that abortion kills innocent babies?
– Why don’t you support welfare programs that help children with cancer?
– Isn’t it obvious that Politician Bob is corrupt?
– Do you deny supporting the poaching of baby seals?

These aren’t really questions. They’re more like attacks, with a question mark tacked on at the end. Instead, a good question tries to roll back a chain of inferences. A person might support E because they support A, and they also believe in the chain of arguments A (therefore) B (therefore) C (therefore) D (therefore) E. A good question tries to bring the debate about E back to a debate about D, and ultimately all the way back to A.

Some thoughts on how to ask better questions:

– Try to be quantitative. For example, someone might say “I think idea X is too expensive.” So you might reply, “About how much do you think X will cost?”. Sometimes this is a bit more difficult, like if your co-worker said “I think Bob would be a terrible person to hire”. But you can still be semi-quantitative; eg., “On a scale from 1 to 10, where 1 is the worst candidate and 10 is the best, where do you think Bob is?”.

– Ask for examples. Your friend might say, “Policy X has always been a disaster.” So a good next question might be, “Can you talk about some times when people tried X, and it turned out really badly?”. A lot of words are vague enough that two people will hear them, and imagine totally different things in their heads, often without realizing it. So examples can help clear up semantic misunderstandings.

– Talk about probabilities. Eg., a lot of people will say that X is a serious threat, for various different values of X. But sometimes X is very unlikely to happen; the most famous examples are media sensations like terrorist bombings, shark attacks, and stranger abductions. So a good question might be, “If I did X, about how likely do you think it is that <bad thing Y> would happen?”.

– Investigate where ideas come from. Even when people are wrong, it’s rarely because they made something up out of their heads. Much more often, they’ll hear something from Alice, who heard it from Bob, who heard it from Carol, and so on, and the original (correct) idea got lost in a long game of “telephone”. (The Science News Cycle shows this process in action.) So if you can find the original source for an idea, you might both wind up agreeing with it.

– Ask what a supporter thinks about an idea’s downsides. Sometimes, they might disagree that a downside exists; sometimes, they might think a downside exists, but that it’s very small; and sometimes, they might think the downsides are large, but the benefits are so big that it’s worth it. So if eg. someone supports using a new chemical in agriculture, you might ask “How dangerous do you think chemical X is?”. (Don’t let this become rhetorical, though. A question like “But won’t idea X kill millions of puppies?” is back in attack territory.)

– Find comparisons to other examples. A person who really liked X might say things like “X is the best Y ever”. So you might ask, “what are some other Ys, and what makes X better than them?”. Luke Muehlhauser’s post on textbooks used this technique very successfully – people who liked a book also had to name two other books they thought were worse. Otherwise, people might recommend something just because it was the only book they’d read in the field.

Why you should focus more on talent gaps, not funding gaps

This website focuses on original content, or content that would otherwise be non-Googleable. However, I am making an exception for Ben Todd’s excellent essay, “Why you should focus more on talent gaps, not funding gaps“, both because of how critically important it is and how thorough and well-written it is. The main focus of Ben’s essay is that solving problems is most often limited by human capital (which Ben calls “talent”, although it’s much broader than the word “talent” might normally imply) and social capital (combining talented people into a team that works well), not by funding. I agree with almost everything he says, and I’ve tried to write up bits and pieces of the arguments Ben makes before, but he really does a much better job. The essay is targeted at effective altruists, but I think it’s a must-read even for people who wouldn’t consider themselves EAs.

A Debate on Animal Consciousness

This debate is from the comments to a public Facebook post by Eliezer Yudkowsky. Some comments were omitted for brevity. Links were inserted for clarity.

Eliezer Yudkowsky: [This is cross-]posted [with] edits from a comment on [the] Effective Altruism [Facebook group], asking who or what I cared about.

I think that I care about things that would, in your native mental ontology, be imagined as having a sort of tangible red-experience or green-experience, and I prefer such beings not to have pain-experiences. Happiness I value highly is more complicated.

However, my theory of mind also says that the naive theory of mind is very wrong, and suggests that a pig does not have a more-simplified form of tangible experiences. My model says that certain types of reflectivity are critical to being something it is like something to be. The model of a pig as having pain that is like yours, but simpler, is wrong. The pig does have cognitive algorithms similar to the ones that impinge upon your own self-awareness as emotions, but without the reflective self-awareness that creates someone to listen to it.

It takes additional effort of imagination to imagine that what you think of as the qualia of an emotion is actually the impact of the cognitive algorithm upon the complicated person listening to it, and not just the emotion itself. Like it takes additional thought to realize that a desirable mate is desirable-to-you and not inherently-desirable; and without this realization people draw swamp monsters carrying off women in torn dresses.

To spell it out in more detail, though still using naive and wrong language for lack of anything better: my model says that a pig that grunts in satisfaction is not experiencing simplified qualia of pleasure, it’s lacking most of the reflectivity overhead that makes there be someone to experience that pleasure. Intuitively, you don’t expect a simple neural network making an error to feel pain as its weights are adjusted, because you don’t imagine there’s someone inside the network to feel the update as pain. My model says that cognitive reflectivity, a big frontal cortex and so on, is probably critical to create the inner listener that you implicitly imagine being there to ‘watch’ the pig’s pleasure or pain, but which you implicitly imagine not being there to ‘watch’ the neural network having its weights adjusted.

What my model says is that when we have a cognitively reflective, self-modely thing, we can put very simple algorithms on top of that — as simple as a neural network having its weights adjusted — and that will feel like something, there will be something that it is like that thing to be, because there will be something self-modely enough to feel like there’s a thing happening to the person-that-is-this-person.

If the one’s mind imagines pigs as having simpler qualia that still come with a field of awareness, what I suspect is that their mind is playing a shell game wherein they imagine the pig having simple emotions and that feels to them like a quale, but actually the imagined inner listener is being created by their own minds doing the listening. Since they have no complicated model of the inner-listener part, since it feels to them like a solid field of awareness that’s just there for mysterious reasons, they don’t postulate complex inner-listening mechanisms that the pig could potentially lack. You’re asking the question “Does it feel like anything to me when I imagine being a pig?” but the power of your imagination is too great; what we really need to ask is “Can (our model of) the pig supply its own inner listener, so that we don’t need to imagine the pig being inhabited by a listener, we’ll see the listener right there explicitly in the model?”

Contrast to a model in which qualia are just there, just hanging around, and you model other minds as being built out of qualia, in which case the simplest hypothesis explaining a pig is that it has simpler qualia but there’s still qualia there. This is the model that I suspect would go away in the limit of better understanding of subjectivity.

So I suspect that vegetarians might be vegetarians because their models of subjective experience have solid things where my models have more moving parts, and indeed, where a wide variety of models with more moving parts would suggest a different answer. To the extent I think my models are truer, which I do or I wouldn’t have them, I think philosophically sophisticated ethical vegetarians are making a moral error; I don’t think there’s actually a coherent entity that would correspond to their model of a pig. Of course I’m not finished with my reductionism and it’s possible, nay, probable that there’s no real thing that corresponds to my model of a human, but I have to go on guessing with my best current model. And my best current model is that until a species is under selection pressure to develop sophisticated social models of conspecifics, it doesn’t develop the empathic brain-modeling architecture that I visualize as being required to actually implement an inner listener. I wouldn’t be surprised to be told that chimpanzees were conscious, but monkeys would be more surprising.

If there were no health reason to eat cows I would not eat them, and in the limit of unlimited funding I would try to cryopreserve chimpanzees once I’d gotten to the humans. In my actual situation, given that diet is a huge difficulty to me with already-conflicting optimization constraints, given that I don’t believe in the alleged dietary science claiming that I suffer zero disadvantage from eliminating meat, and given that society lets me get away with it, I am doing the utilitarian thing to maximize the welfare of much larger future galaxies, and spending all my worry on other things. If I could actually do things all my own way and indulge my aesthetic preferences to the fullest, I wouldn’t eat any other life form, plant or animal, and I wouldn’t enslave all those mitochrondria.

Tyrrell McAllister: I agree with everything you say about the inadequacy of the “pigs have qualia, just simpler” model. But I still don’t eat pigs, and it is for “philosophically sophisticated ethical” reasons (if I do say so myself). When I watch pigs interact with their environment, they seem to me, as best as I can tell, to be doing enough reflective cognition to have an “inner listener”.

Eliezer: Tyrell, what looks to you like a pig’s brain must be modeling itself?

Tyrell: “Modeling itself” is probably too weak a criterion. But pigs do seem to me to do problem-solving involving themselves and their environments that is best explained by their working with a mental model of themselves and their environment. (See, e.g., Pigs Prove to Be Smart, if Not Vain.)

I acknowledge that this kind of modeling isn’t enough to imply that there is an inner listener, so there I am being more speculative.

Also, I should have written, “they seem to me, as best as I can tell, with sufficient probability given the utilities involved, to be doing enough reflective cognition to have an ‘inner listener'”.

(I do eat fish, because the benefits of eating fish, and the probability of their being conscious, seem to make eating them the right thing to do in that case.)

Eliezer: Tyrell: This looks to me like environment-modeling performing a visual transform, and while that implies some degree of cognitive control of the visual cortex it doesn’t imply brains modeling brains.

If my model is correct then the mirror test is actually an ethically reasonable place to put a “do not eat” barrier; passing the mirror test may not be sufficient, but it seems necessary-ish (leaving aside obvious caveats about using the right modality).

Jamie F Duerden: Pigs seem reasonably ‘smart’, insofar as recognising names and processes, solving puzzles and so on. I don’t know whether they recognise themselves in a mirror and are aware of their own awareness of that fact, but I would not be especially surprised to discover it was so. Yet I still eat bacon, because it is filling, very tasty, and a great source of protein. I would not, however, eat a pig which I had been keeping as a pet.

This distinction seems to be consistently applied by whichever part of my brain makes intuitive ‘moral’ judgements, because I experience no psychological backlash when contemplating eating people I don’t know, but am disturbed by the idea of eating someone I was friends with. Comparing those sensations to the equivalent responses for ‘farmed/wild animal’ and ‘pet’ yields negligible difference. I have hunted animals for meat, so this is not a failure to visualise unknown animals correctly. I am forced to conclude that ‘does it have qualia?’ is not an important variable in my default ‘is it food?’ function. (Nor apparently in my ‘would it make a good pet?’ function.) As I routinely catch myself empathising with hypothetical AIs, I suspect this may be a more general complete failure to have separate categories for the various sorts of ‘mind’.

Luke Muehlhauser: I think my probability distribution over theories of consciousness (in animals, humans, and machines) looks something like this:

~30%: Basically Eliezer’s model, described above
~30%: Apes and maybe some others have an inner listener, but it results in less salient subjective experience than the human inner listener due to its less integrated-y self-modely nature, and this subjective salience drops to 0 pretty sharply below a certain kind/degree of self-modely-ness (ape level? pig level?), rather than trailing off gradually down to fish or whatever.
< 5%: pansychism, consciousness is fundamental, consciousness is magical, and similar theories
~35%: other theories not close to the above, most of which I haven’t thought of

I try to limit my meat intake due to how much mass I have on theory categories #2 & 4, but I’m not strictly vegetarian or vegan because I’m choosing to devote most of my “be ethical” willpower/skill points elsewhere. But this does mean that I become more vegetarian/vegan as doing so becomes less skill+willpower-requiring, so e.g. I want that all-vegan supermarket in Germany to open a branch in Berkeley please.

David Pearce: Some errors are potentially ethically catastrophic. This is one of them. Many of our most intensely conscious experiences occur when meta-cognition or reflective self-awareness fails. Thus in orgasm, for instance, much of the neocortex effectively shuts down. Or compare a mounting sense of panic. As an intense feeling of panic becomes uncontrollable, are we to theorise that the experience somehow ceases to be unpleasant as the capacity for reflective self-awareness is lost? “Blind” panic induced by e.g. a sense of suffocation, or fleeing a fire in a crowded cinema (etc), is one of the most unpleasant experiences anyone can undergo, regardless or race or species. Also, compare microelectrode neural studies of awake subjects probing different brain regions; stimulating various regions of the “primitive” limbic system elicits the most intense experiences. And compare dreams – not least, nightmares – many of which are emotionally intense and characterised precisely by the lack of reflectivity or critical meta-cognitive capacity that we enjoy in waking life.

Anyone who cares about sentience-friendly intelligence should not harm our fellow subjects of experience. Shutting down factory farms and slaughterhouses will eliminate one of the world’s worst forms of severe and readily avoidable suffering.

Jai Dhyani: [Eliezer], how confident are you about this? I would like to bet on the proposition: “In 2050 you will not be willing to eat a cow-as-opposed-to-vat grown steak in exchange for $50 inflation-adjusted USD.” I think there is at least a 10% chance of this occurring.

Eliezer: Jai Dhyani, if I was accustomed to vat-grown steak I probably would value that ethical combo continuing more than I valued $50.

Mason Hartman: I think you’ve laid the groundwork for a useful model of consciousness, but it’s not clear why you’ve arrived at the position that pigs probably don’t have it. It seems unlikely to me that only a very small handful of species have been or are “under selection pressure to develop sophisticated social models of conspecifics.” Basically, a little red flag quietly squeaks “Check whether this person actually knows much about animal social behavior!” whenever someone says something that seems to imply that humans are the pinnacle of social-animal-ness.

Also, I think consciousness probably “unlocks” a lot of value aside from better dinner conversation – predation, for example, seems like the sort of thing one might be better able to prevent with the ability to model minds, including one’s own mind.

Brent Dill: Incidentally, baboons have quite a rich set of social scheming instincts. All of this guy’s stuff is amazing.

Eliezer: I wouldn’t eat a BABOON. Eek.

Brent: Well, yeah. Eating primates is an excellent way to get really really sick.

Eliezer: They recognize themselves in mirrors! No eat mirror test passers! Anything which has the cognitive support for that could have an inner listener for all I currently know about inner listeners.

Mason: The mirror test has some other problems with it – the big one being that a lot of non-Western kids don’t pass it. I assume Kenyan 6-year-olds will remain off the menu.

William Eden: I am personally more inclined towards the panpsychism view, for various reasons, not least of which is Eliezer’s post, ironically enough.

I care about subjective experience apart from concepts of personhood. Imagine a superintelligence discovered humans, decided that we lacked some critical component of their cognition, and because of this they felt justified in taking us apart to be used as raw atoms, or experimented on in ways we found distressing.

Eliezer: Or even worse, they might not think that paperclips had ethical value? I think if you’re going to go around caring about conscious beings instead of trade partners, you need to accept that your utility function looks weird to a paperclip maximizer. Saying that you care about subjective experience already relegates you to a pretty weird corner of mindspace relative to all the agents that don’t have subjective experience and had different causal histories leading them to assign terminal value to various agent-like objects.

Brent: I think that all ‘pain’, in the sense of ‘inputs that cause an algorithm to change modes specifically to reduce the likelihood of receiving that input again’, is bad.

I think that ‘suffering’, in the sense of ‘loops that a self-referential algorithm gets into when confronted with pain that it cannot reduce the future likelihood of experiencing’.

Social mammals experience much more suffering-per-unit-pain because they have so many layers of modeling built on top of the raw input – they experience the raw input, the model of themselves experiencing the input, the model of their abstracted social entity experiencing the input, the model of their future-self experiencing the input, the models constructed from all their prior linked memories experiencing the input… self-awareness adds extra layers of recursion even on top of this.

One thought that I should really explore further: I think that a strong indicator of ‘suffering’ as opposed to mere ‘pain’ is whether the entity in question attempts to comfort other entities that experience similar sensations. So if we see an animal that exhibits obvious comforting / grooming behavior in response to another animal’s distress, we should definitely pause before slaughtering it for food. The capacity to do so across species boundaries should give us further pause, as should the famed ‘mirror test’. (Note that ‘will comfort other beings showing distress’ is also a good signal for ‘might plausibly cooperate on moral concerns’, so double-win).

At that point, we have a kind of consciousness scorecard, rather than a pure binary ‘conscious / not conscious’ test.

Eliezer: I think you just said that a flywheel governor on a steam engine, or a bimetallic thermostat valve, can feel pain.

Brent: I did, and I intended to. “Pain”, in and of itself, is worthless. “Suffering” has moral weight. “Pain x consciousness = suffering”, and a flywheel governor just isn’t complex enough to be very conscious.

David Brin: See this volume about how altruism is actually very complicated. (I wrote two of the chapters.)

Mason: I have some other thoughts, assuming my understanding of Eliezer’s theory is fairly accurate. These are the claims I’m assuming he’s making:

(a) Meaningful consciousness (i.e. that which would allow experiences Eliezer would care about) in the animal kingdom is exceedingly rare.

(b) Animals only evolve meaningful consciousness when under selection pressure to develop sophisticated social models.

(c) The mirror test is a decent way to determine whether a species possesses, or has the capacity for, meaningful consciousness.

Assuming these claims were true, I’d make some predictions about the world that don’t seem to have been borne out, including:

(1) Given the rarity of meaningful consciousness, animals who pass the mirror test will likely be closely related to one another – but humans, elephants, dolphins, and magpies have all passed the mirror test at one point.

(2) Animals who pass the mirror test should be exceptionally social or descended from animals who are/were exceptionally social. I admittedly don’t know much about magpies, but a quick Google doesn’t seem to imply that they’re exceptionally more social than other birds.

Eliezer: I hadn’t heard of magpies before. African grey parrots are surprisingly smart and apparently play fairly complex games of adultery. Dolphins are the only known creatures besides humans that form coalitions of coalitions. Not sure what elephants do socially but they have large brains and the balance of selection pressures is plausibly affected thereby (i.e., it is more advantageous to have good brain software if you have a larger brain, and if you have a bigger body and brain improvements buy percentages of body-use then you can afford a bigger brain, etc.)

Mason: I think all of the test-passing species are pretty social (some of them possibly more so than us, depending on how you measure social-ness), but they don’t seem exceptionally so. Many, many animals play very complex social games – in my opinion (and much to my surprise) even chickens have fascinating social lives.

The question isn’t “Do we have evidence these animals are highly social?” but “Do we have evidence these animals tend to be social to an extent/in a way which other animals aren’t?”

Rob Bensinger: I don’t believe in phenomenal consciousness. I think if you try to put quotation marks around a patch of your visual field (e.g., by gesturing at ‘that patch of red in the lower-left quadrant of my visual field’), some of the core things included in your implicit intension will make the gesture non-referring. Asking about ‘but what am I really subjectively experiencing?’ is like going: ‘but what is my deja-vu experience really a repetition of?’ The error is trickier, though, because the erroneous metarepresentation is systematic and perception-like (like a hallucination or optical illusion, but one meta level up) rather than judgment-like (like a delusion or hunch).

Like Luke said, believing I’m a zombie in practice just means I value something functionally very similar to consciousness, ‘z-consciousness’. But ‘z-consciousness’ is first and foremost a (promissory note for a) straw-behaviorist, third-person theoretical concept. Thinking in those terms — starting with one box, some of whose (physical, neuronal, behavioral…) components I sometimes misdescribe as ‘mind’, rather than starting with separate conceptual boxes for ‘mind’ and ‘matter’ and trying to glue them together as tightly as possible — has been a really weird (z-)experience. It’s had some interesting effects on my intuitions over the past few years.

1. I’m much more globally skeptical that I can trust my introspection and metacognition.
2. Since (z-)consciousness isn’t a particularly unique kind of information-processing, I expect there to be an enormous number of ‘alien’ analogs of consciousness, things that are comparable to ‘first-person experience’ but don’t technically qualify as ‘conscious’.
3. I’m more inclined to think there are fuzzy ‘half-conscious’ and ‘quarter-conscious’ states in between z-consciousness and z-unconsciousness.

I entertained limited versions of those ideas in my non-eliminative youth, but they’re a lot more salient and personal to me now. And as a consequence of 1-3, I’m much more skeptical that (z-)consciousness is a normatively unique kind of information-processing. Since I think a completed neuroscience will overturn our model of mind fairly radically, and since humans have strong intuitions in favor of egalitarianism and symmetry, it wouldn’t surprise me if certain ‘unconscious’ states acquired the same moral status as ‘conscious’ ones.

The practical problem of deciding which alien minds morally ‘count’ will become acute as we explore transhuman/posthuman mindspace, but the principled problem is already acute; if we expect our ideal, more informed selves to dispense with locating all value in consciousness (or, equivalently, if we expect to locate all value in a bizarrely expansive conception of ‘consciousness’), we should do our best to already reflect that expectation in our ethics.

So, I’m with Eliezer in thinking pig pain isn’t just ‘human pain but simpler’, or ‘human pain but fainter’. But that doesn’t much reassure me: to the extent human-style consciousness is an extremely rare and conjunctive adaptation dependent on complex social modeling, I become that much less confident that that’s the only kind of information-processing I should be concerned for, on a Coherent-Extrapolated-Volition-style reading of ‘should’. My four big worries are:

(a) Pigs might still be ‘slightly’ conscious, if consciousness (ideally conceived) isn’t anywhere near as rare and black-and-white as our non-denoting folk concept of consciousness.

(b) If consciousness is rare and exclusive, that increases the likelihood that our CEV would terminally value some consciousness-like unconscious states. Perhaps the pig lacks a first-person perspective on reality, but has a shmerspective that is strange and beautiful and rich, and we ought ideally to abandon qualia-chauvinism and assign some value to shmuffering.

(c) Supposing our concept of ‘consciousness’ really does turn out to be incoherent unless you build in a ton of highly specific unconscious machinery ‘behind the scenes’, that further increases the likelihood that our CEV would come to care about something more general than consciousness, something that captures the ‘pain’ aspect of our experience in a way that can also apply to systems that are ‘free-floating pain’, pain sans subject. (Or sans a subject as specific and highly developed as a human subject.)

(d) Once we grant pigs might be moral patients, we also need to recognize that they may be utility monsters. E.g., if they’re conscious or quasi-conscious, they might be capable of much more acute (quasi-)suffering than ordinary humans are. (Perhaps they don’t process time in the way we do, so every second of pain is ‘stretched out’.) This may be unlikely, but it should get a big share of our attention because it would be especially bad.

I bounce back and forth between taking the revisionist posthuman consciousness-as-we-think-of-it-isn’t-such-a-big-deal perspective very seriously, and going ‘that’s CRAZY, it doesn’t add up to moral normality, there’s no God to make me endorse such a ridiculous “extrapolation” of my ideals, I’ll modus tollens any such nonsense till I’m blue in the face!’ But I’m not sure the idea of ‘adding up to moral normality’ makes philosophical sense. It may just being a soothing mantra.

We may just need to understand the principles behind consciousness and what-makes-us-value-consciousness on an especially deep level in order to avoid creating new kinds of moral patient; I don’t know whether avoiding neuromorphism, on its own, would completely eliminate this problem for AI.

David: Rob, you might want to explore (d) further. Children with autism have profound deficits of self-modelling as well as social cognition compared to neurotypical folk. So are profoundly autistic humans less intensely conscious than hyper-social people? In extreme cases, do the severely autistic lack consciousness’ altogether, as Eliezer’s conjecture would suggest? Perhaps compare the accumulating evidence for Henry Markram’s “Intense World” theory of autism.

Eliezer, I wish I could persuade you to quit eating meat – and urge everyone you influence to do likewise.

Kaj Sotala: Eliezer, besides the mirror test, what concrete functional criteria do you have in mind that would require the kind of processing that you think enables subjective experience? In other words, of what things can you say “behavior of kind X clearly requires the kind of cognitive reflectivity that I’m talking about, and it seems safe to assume that pigs don’t have such cognitive reflectivity because they do not exhibit any behaviors falling into that class”?

Also, it seems to me that this theory would imply that we aren’t actually conscious while dreaming, since we seem to lack self-modeling capability in (non-lucid) dreams. Is that correct?

I would also agree that “pigs have qualia, only simpler” seems wrong, but “pigs have qualia, only of fewer types” would seem more defensible. For example, they might lack the qualia for various kinds of sophisticated emotional pain, but their qualia for physical pain could be basically the same as that of humans. This would seem plausible in light of the fact that qualia probably have some functional role rather than being purely epiphenomenal, and avoiding tissue damage is a task with a long evolutionary history.

It would feel somewhat surprising if almost every human came equipped with the neural machinery that reliably correlated tissue damage with the qualia of physical pain, and the qualia for emotional pain looked like outgrowths of the mechanism that originally developed to communicate physical pain, but most of our evolutionary ancestors still wouldn’t have had the qualia for physical pain despite having the same functional roles for the mechanisms that communicate information about tissue damage.

Michael Vassar: My best guess is that there is moral subjectivity and possibly also moral objectivity, that moral subjectivity works pretty similarly to Eliezer’s description here. However, moral subjectivity is a property of mirror test passers some of the time, but not very much of the time, and how much depends on the individual and varies over quite a large range. It probably also varies in intensity quite a lot, probably in a manner that isn’t simply correlated with its frequency of occurrence but probably is simply correlated with integrated information. It’s probably present in various types of collective. A FAI would probably have this. Neither a mob nor the members of the mob have it.

There’s also probably moral objectivity. Dreamers, pigs, flow-states, and torture victims have this, but most AGIs probably don’t. Most people, most of the time, do have it, but, e.g. certain meditators may not. It’s harder to characterize its properties. My best guess is that it’s a set of heuristics for pruning computational search trees. “measure” might refer to the same thing.

Luke: On the topic of Rob’s skepticism about introspection, see my post. I should also note that when I mentioned “integrated-y” above, I wasn’t endorsing IIT at all. I have the same reaction to IIT as Aaronson, and encouraged Aaronson to write his post on that subject.

Rob: If you’re skeptical about whether you’re conscious at times you aren’t thinking about consciousness (even though you can in many cases think back and remember those experiences later, and consider what their subjective character was like at the time — as you remember it — and learn new things, things which seem consistent with your subjective experiences at other times), it’s possible you should also be skeptical about whether you’re conscious at times you are thinking about consciousness. Especially when you’re merely remembering such times.

If you can misremember your mental state of five minutes ago as a conscious one, what specifically forbids your misremembering your mental state of five minutes ago as a conscious-of-consciousness one?

My own worries go in a different direction. I’m fine with the idea that I might be confabulating some of my experiences. I’m more concerned that large numbers of other subjects in my brain may be undergoing experiences as a mechanism for my undergoing an experience, or as a side-effect. What goes into the sausages?

Some people fear the possibility that anesthetics prevent memory-of-suffering, instead of preventing suffering. I’m a lot more worried that ordinary human cognition (e.g., ordinary long-term memory formation, or ordinary sleep) could involve large amounts of unrecorded ‘micro-suffering’. Subsystems of my own brain are mysterious to me, and I treat those subsystems with the same kind of moral uncertainty as I treat mysterious non-human brains.

Brent: Note that dogs have evolutionary pressure to express depression, because they have spent the last 40,000 years co-evolving with humans, being selected explicitly for their capacity to emulate human emotional communication and bonding.

Mason: Brent – I’m not convinced that dogs have ever been bred to emulate negative human emotions. Until recently, many breeds were used primarily for work. Many – e.g. herding breeds – don’t actually make very good companions unless made to work or perform simulated work (e.g. obedience/agility/herding trials) to manage their physical/intellectual needs. A dog that would cease to work as effectively during periods of emotional stress (e.g. by displaying symptoms of depression) would probably not be selected for. And yet these breeds are often the most expressive across the board (and the most capable of reacting to emotional expression in humans), as evidenced by their extensive use as therapy/emotional support animals.

It seems very likely to me that we either created actual emotional complexity through selective breeding, or that we just took animals that already had very complex emotional lives and bred for a communication style that was intuitive to us. If we had only been breeding for behavior that simulated emotional expression, we would probably have avoided behaviors that aren’t conducive to the work dogs have done throughout their history with humans. Keeping dogs primarily as pets is a very new thing.

Eric Schmidt: Responding to the initial post:

IMO [Eliezer’s] making a huge, unfounded leap in assuming that all qualia only arise in the presence of an intelligent “inner listener” for some mysterious reason. For all we know, you could engender qualia in a Petri dish. For all we know, there are fonts of qualia all around us. You are restricting your conception of qualia to the kind you are most familiar with and which you hold in highest regard: the feeling of human consciousness and intelligence, the feeling of your integrated sense of self. But if qualia can originate in a Petri dish, it could certainly originate in a pig. I used to think like you do when I was younger, but IMO it’s just an unfounded bias towards the familiar, towards ourselves. AFAIK, if you poke a pig w[ith] a pointy stick, some pain feelings will engender in the universe. No, no intelligent consciousness will be there to stare at them, reflect on them, it won’t lead to the various sorts of other qualia that it can for humans (e.g. the qualia of noticing the pain, dwelling on it, remembering it, thinking about it, self pity, whatever), but AFAIK it’s still there in our universe, tarnishing it slightly (assuming pain is in some sense negatively valued and ought to be minimized).

On vegetarianism: Well, demand for meat keeps livestock species’ populations artificially way high. So as long as those livestock are living net-positive-qualia lives, then great. The more the better. (Aside: maybe [the] Fermi paradox is [because] earth is a farm for Gorlax who eats advanced civilizations in a single bite as an exquisite delicacy. I’m totally okay with that: the TV here is that good.) So I think eating slaughtered animals is fine, so long as the animals aren’t miserable. I’d like to see some data on that. In general, I’d like to see a consumer culture that pressured the meat industry to treat the animals decently, somehow assure us that they’re living net-positive-qualia lives.

EDIT: By “as far as I know”/”as far as we know” I mean that it hasn’t been disproven and there’s no compelling reason to believe it’s false.

Eliezer: Nobody knows what science doesn’t know. The correct form of the statement you just made is “For all I [Eric Schmidt] know, you could engender qualia in a Petri dish.”

My remaining confusion about consciousness does not permit that as an open possibility that could fit into the remaining confusion. I am not thinking about an intelligent being required to stare at the quale, and then experience other quales about being aware of things. I am saying that it is a confused state of mind, which I am now confident I have dissolved, to think that you could have a “simple quale” there in the first place. Those amazingly mysterious and confusing things you call qualia do not work the way your mind intuitively thinks they do, and you can’t have simple ones in a petri dish. Is it that impossible to think that this is something that someone else might know for sure, if they had dissolved some of their confusion about qualia?

Confusion isn’t like a solid estimating procedure that gives you broad credibility intervals and says that they can narrow no further without unobtainable info, like the reason I’m skeptical that Kurzweil can legitimately have narrow credibility intervals about when smarter-than-human AI shows up. Confusion means you’re doing something wrong, that somebody else could just as easily do right, and exhale a gentle puff of understanding that blows away your bewilderment like ashes in the wind. I’m confused about anthropics, which means that, in principle, I could read the correct explanation tomorrow and it could be three paragraphs long. You are confused about qualia; it’s fine if you don’t trust me to be less confused, but don’t tell other people what they’re not allowed to know about it.

To be clear, pigs having qualia does fit into remaining confusion; it requires some mixture of inner listeners being simpler than I thought and pigs having more reflectivity than I think. Improbable but not forbidden. Qualia in a petri dish, no.

In ethical terms, where society does not derogate X as socially unacceptable, and where naive utilitarianism says X is not the most important thing to worry about / that it is not worth spending on ~X, I apply a “plan on the mainline” heuristic to my deontology; it’s okay to do something deontologically correct on the mainline that is not socially forbidden and which is the best path according to naive utilitarianism. Chimps being conscious feels to me like it’s on the mainline; pigs being conscious feels to me like it’s off the mainline.

David: Eliezer, we’d both agree that acknowledged experts in a field can’t always be trusted. Yet each of us should take especial care in discounting expert views in cases where one has a clear conflict of interest. [I’m sure every meat eating reader hopes you’re right. For other reasons, I hope you’re right too.]

Eliezer: What do they think they know and how do they think they know it? If they’re saying “Here is how we think an inner listener functions, here is how we identified the associated brain functions, and here is how we found it in animals and that showed that it carries out the same functions” I would be quite impressed. What I expect to see is, “We found this area lights up when humans are sad. Look, pigs have it too.” Emotions are just plain simpler than inner listeners. I’d expect to see analogous brain areas in birds.

David: Eliezer, I and several other commentators raised what we see as substantive problems with your conjecture. I didn’t intend to rehash them – though I’ll certainly be very interested in your response. Rather, I was just urging you to step back and reassign your credences that you’re correct and the specialists in question are mistaken.

Eliezer: I consider myself a specialist on reflectivity and on the dissolution of certain types of confusion. I have no compunction about disagreeing with other alleged specialists on authority; any reasonable disagreement on the details will be evaluated as an object-level argument. From my perspective, I’m not seeing any, “No, this is a non-mysterious theory of qualia that says pigs are sentient…” and a lot of “How do you know it doesn’t…?” to which the only answer I can give is, “I may not be certain, but I’m not going to update my remaining ignorance on your claim to be even more ignorant, because you haven’t yet named a new possibility I haven’t considered, nor pointed out what I consider to be a new problem with the best interim theory, so you’re not giving me a new reason to further spread probability density.”

Mark P Xu Neyer: Do you think people without developed prefrontal cortices – such as children – have an inner listener?

Eliezer: I don’t know. It would not surprise me very much to learn that average children develop inner listeners at age six, nor that they develop them at age two, and I’m not an expert on developmental psychology nor a parent so I have a lot of uncertainty about how average children work and how much they vary. I would certainly be more shocked to discover that a newborn baby was sentient than that a cow was sentient.

Brian Tomasik: I wrote a few paragraphs partially as a response to this discussion. The summary is:

There are many attributes and abilities of a mind that one can consider important, and arguments about whether a given mind is conscious reflect different priorities among those in the discussion about which kinds of mental functions matter most. “Consciousness” is not one single thing; it’s a word used in many ways by many people, and what’s actually at issue is the question of which traits matter more than which other traits.

Also, this discusses why reflectivity may not be ethically essential:

In Scherer’s view, the monitoring process helps coordinate and organize the other systems. But then privileging it seems akin to suggesting that among a team of employees, only the leader who manages the others and watches their work has significance, and the workers themselves are irrelevant.[2] In any event, depending on how we define monitoring and coordination, these processes may happen at many levels, just like a corporate management pyramid has many layers.

Buck Shlegeris: BTW, Eliezer, AFAICT pigs have self awareness according to the mirror test: they only fail it because pigs don’t care if they have mud on their face. They are definitely aware that the pig in the mirror is not another pig. Is that enough uncertainty to not eat them?

From Wikipedia: “Pigs can use visual information seen in a mirror to find food, and show evidence of self-recognition when presented with their reflection. In an experiment, 7 of the 8 pigs tested were able to find a bowl of food hidden behind a wall and revealed using a mirror. The eighth pig looked behind the mirror for the food.[28]”

Eliezer: What I want to see is an entity not previously trained on mirrors, to realize that motions apparent in the mirror are astoundingly correlated to motions that it’s sending to the body, i.e., the aha! that I can control this figure in the mirror, therefore it is me. This seems to me to imply a self-model. If you train pigs to use a mirror generically in order to find food, then what you’re training them to do is control their visual imagination so as to take the mirror-info and normalize it into nearby-space info. This tells me that pigs have a visual imagination which is not very surprising since IIRC the back-projections from higher areas back to the visual cortex were already a known thing.

But if the pig can then map what it sees in the mirror onto its spatial model of surrounding space, and as a special case can identify things colocated in space with itself, you’ve basically trained the pig to ‘solve’ the mirror test via a different pathway that doesn’t need to go through having a self-model. I’m sorry if it seems like I’m moving the goalpost, but when I say “mirror test” I mean a spontaneous mirror test without previous training. There’s similarly a big difference between an AI program that spontaneously starts talking about consciousness and an AI program that researchers have carefully crafted to talk about consciousness. The whole point of the mirror test is to provoke (and check for) an aha! about how the image of yourself in the mirror is behaving like you control it; training on mirrors in general defeats this test.

Buck: I don’t quite buy your reasoning. Most importantly, the pigs are aware that the pig in the mirror is not a different pig. That seems like strong evidence of self awareness. One of the researchers said “We have no conclusive evidence of a sense of self, but you might well conclude that it is likely from our results.”

So pigs haven’t passed or failed the mirror test, but they seem aware that a pig in a mirror is not a different pig, and experts in the field seem to think pigs are likely to have self awareness.

And again, I think that the burden of proof is on the people who are saying that it’s fine to torture the things! Like, I’m only 70% sure that pigs are conscious. But that’s still enough that it’s insane to eat them.

(Also, when you say “an entity not previously trained on mirrors”: literally no species can immediately figure out what a mirror is. Even humans have to have a while around them to figure them out, which is as much training as we give to pigs.)

Andres Gomez Emilsson: Harry would say: We must investigate consciousness and phenomenal binding empirically.

Let us take brain tissue in a petri dish, or something like that, and use bioengineered neural bridges between our brains and the petri dish culture to find out whether we can incite phenomenal binding of any sort. Try to connect different parts of your brain to the petri dish. E.g what happens when you connect your motor areas to it, and what happens when you add synapses that go back to your visual field? If you are able to bind phenomenologies to your conscious experience via this method, try changing the chemical concentrations of various neurotransmitters in the petri dish. Etc.

This way we can create a platform that vastly increases the range of our possible empirical explorations.

Eliezer: Um, qualia are not epiphenomenal auras that contaminate objects in physical contact. If you hook up an electrical stimulator to your visual cortex, it can make you see color qualia (even if you’re blind). This is not because the electrical stimulator is injecting magical qualia in there. The petri dish, I predict with extreme confidence, would seem to you to produce exactly the same qualia as an electrical stimulator with the same input/output behavior. Unless you have discovered a new law of physics. Which you will not.

Andres: I’m thinking about increasing the size of the culture in the petri dish until I can show that phenomenal binding is happening by doing something I would not be able to do otherwise:

If I increase the size of my visual field by adding to my brain a sufficient number of neurons in a petri dish appropriately connected to it, I would be able to represent more information visually than I am capable of with my normal visual cortex.

This is thoroughly testable. And I would predict that you can indeed increase the information content of a given experience by this method.

Eliezer: You’re not addressing my central point that while hooking up something to an agent with an inner listener may create what that person regards as qualia, it doesn’t mean you can rip out the petri dish and it still has qualia in it.

David: A neocortex isn’t needed for consciousness, let alone self-consciousness. Perhaps compare demonstrably self-aware magpies, who (like all birds) lack a neocortex.

Eric, many more poor humans could be fed if corn and soya products were fed directly to people rather than to factory-farmed nonhuman animals we then butcher. Such is the thermodynamics of a food chain. Qualia and the binding problem? I’ve put a link below; but my views are probably too idiosyncratic usefully to contribute to the debate here.

Eliezer: The PLOS paper seems to be within the standard paradigm for mirror studies. Pending further confirmation or refutation this is a good reason not to eat magpies or other corvids.

David: Eliezer, faced with the non-negligible possibility that one might be catastrophically mistaken, isn’t there a powerful ethical case for playing safe? If one holds a view that most experts disagree with, e.g. in my case, I’m sceptical that a classical digital computer will ever be conscious, I’d surely do best to defer to consensus wisdom until I’m vindicated / confounded. Or do you regard the possibility that you are mistaken as vanishingly remote?

Francisco Boni Neto: Empathic brain-modeling architecture as an conditio sine qua non for an ‘inner listener’ that is a requirement for context-dependent qualia or what-is-likeness that-it-is-like-that-thing-to-be seems like an exaggeration of the “simulation” and “mirroring” theory of affective cognition, overstating top-bottom processes: “there must be a virtualization process, helped by socially complex networks of interactions that are positively selected in modern humans, so I can simulate other humans in my brains while I sustain my own self-modelying thingy that adds a varied collection of qualia types in the mammalian phenotype that are lacking in other less complex phenotypes (e.g. pigs)”.

It seems like a very adaptationist thought that overstates the power and prevalence of natural selection towards the certain recent branches of the phylogenetic tree against the neural reuse that makes ancient structures so robust and vital in processing vivid what-is-likeness and that makes bottom-up processing so important despite the importance of the pre-frontal cortex in deliberative perceptual adaptation and affective processing. That is why I agree when David Pearce points out that many of our most intensely conscious experiences occur when meta-cognition or reflective self-awareness fails. Super vivid, hyper conscious experiences, phenomenic rich and deep experiences like lucid dreaming and ‘out-of-body’ experiences happens when higher structures responsible for top-bottom processing are suppressed. They lack a realistic conviction, specially when you wake up, but they do feel intense and raw along the pain-pleasure axis.

Eliezer: It is impossible to understand my position or pass any sort of Ideological Turing Test on it if you insist on translating it into a hypothesis about some property of hominids like “advanced reflectivity” which for mysterious reasons is the cause of some mysterious quales being present in hominids but not other mammals. As long as qualia are a mysterious substance in the version of my hypothesis you are trying to imagine, of course you will see no motivation but politics for saying the mysterious substance is not in pigs, when for all you know, it could be in lizards, or trapped in the very stones, and those unable to outwardly express it.

This state of confusion is, of course, the whole motivation for trying to think in ways that don’t invoke mysterious qualia as primitive things. It is no use to say what causes them. Anything can be said to cause a mysterious and unreduced property, whether “reflectivity” or “neural emergence” or “God’s blessing”. Progress is only made if you can plausibly disassemble the inner listener. I am claiming to have done this to a significant extent and ended up with a parts list that is actually pretty complicated and involves parts not found in pigs, though very plausibly in chimpanzees, or even dolphins. If you deliberately refuse to imagine this state of mind and insist on imagining it as the hypothesis that these parts merely cause an inner listener by mysteriousness, then you will be unable to understand the position you are arguing with and you will not be able to argue against it effectively to those who do not already disbelieve it.

David: Eliezer, I’ve read Good and Real, agree with you on topics as varied as Everett and Bayesian rationalism; but I still don’t “get” your theory of consciousness. For example, a human undergoing a state of blind uncontrollable panic is no more capable of reflective self-awareness or any other form of meta-cognition than a panicking pig. The same neurotransmitter systems, same neurological pathways and same behavioural responses are involved in the panic response in both pigs and humans. So why is the human in a ghastly state of consciousness but the pig is just an insentient automaton?

Eliezer: One person’s modus ponens is another’s modus tollens: I’m not totally sure people in sufficiently unreflective flow-like states are conscious, and I give serious consideration to the proposition that I am reflective enough for consciousness only during the moments I happen to wonder whether I am conscious. This is not where most of my probability mass lies, but it’s on the table. I think I would be equally surprised to find monkeys conscious, or people in flow states nonsentient.

Daniel Powell: If there existed an entity which was intermittently conscious, would the ethics of interacting with it depend on whether it was conscious at the moment?

What about an entity that has never been conscious, but might become so in the future – for example, the uncompiled code for a general intelligence, or an unconceived homo sapiens?

I’m having a hard time establishing what general principle I could use that says it is wrong to harm a self-aware entity, but not wrong to harm a creature that isn’t self-aware, but wrong to cause “an entity which used to be self-aware and currently isn’t, but status quo will be again” to never again be self aware, but not wrong to cause a complex system that has the possibility to create a self-aware entity not to do so.

I thought I held all four of those beliefs from simple premises, but now I doubt whether my intuitive sense of right and wrong cares about sentience at all.

Eliezer: Daniel, I think the obvious stance would be that having unhappy memories is potentially detrimental / causes suffering, or that traumatic experiences while nonsentient can produce suffering observer-moments later. So I would be very averse to anyone producing pain in a newborn baby, even though I’d be truly shocked (like, fairies-in-the-garden shocked) to find them sentient, because I worry that might lose utility in future sentient-moments later.

Carl Shulman: I find [Eliezer] unconvincing for a few reasons:

1. Behavioral demonstration of self-models along the lines of the mirror test is overkill if you’re looking for the presence of some kind of reflective “thinking about sense inputs/thoughts”, the standard mirror test requires other additional processing and motivation to pass the standard presentation. So one should expect that creatures for which there isn’t yet a wikipedia mirror test entry could also pass idealized tests for such reflective processing, including somewhat less capable relatives of those who pass. [Edit: see variation among human cultures on this, even at 6 years old, and the dependence on grooming behavior and exposure to mirrors, as discussed in the SciAm article here.]

2. There is a high degree of continuity in neural architecture and capacities as one moves about the animal kingdom. Piling on additional neural resources to a capacity can increase it, but often with diminishing returns (even honeybees with a million neurons can do some neat tricks). If one allows consciousness for magpies and crows and ravens, you should expect with fairly high probability that some less impressive (or harder to elicit/motivate/test) versions of those capacities are present in other birds, such as chickens.

3. You haven’t offered neuroscience or cognitive science backing for claims about the underlying systems, just behavioral evidence via the mirror test. The claim that other animals don’t have weaker versions of the mechanisms enabling passage of the mirror test, or capable of reflective thought about sense inputs/reinforcement, is one subject to neuroscience methods. You don’t seem to have looked into the relevant neuroscience or behavioral work in any depth.

4. The total identification of moral value with reflected-on processes, or access-conscious (for speech) processes, seems questionable to me. Pleasure which is not reflected on or noticed in any access-conscious way can still condition and reinforce. Say sleeping in a particular place induced strong reinforcement, which was not access-conscious, so that I learned a powerful desire to sleep there, and not want to lose that desire. I would not say that such a desire is automatically mistaken, simply because the reward is not access-conscious.

5. Related to 4), I don’t see you presenting great evidence that the information processing reflecting on sense inputs (pattern recognition, causal models, etc) is so different in structure.

“Now, a study published September 9 in The Journal of Cross-Cultural Psychology is reinforcing that idea and taking it further. Not only do non-Western kids fail to pass the mirror self-recognition test by 24 months—in some countries, they still are not succeeding at six years old.

What does it mean? Are kids in places like Fiji and Kenya really unable to figure out a mirror? Do these children lack the ability to psychologically separate themselves from other humans? Not likely. Instead researchers say these results point to long-standing debates about what counts as mirror self-recognition, and how results of the test ought to be interpreted.” (link)

Eliezer: More seriously, the problem from my perspective isn’t that I’m confident of my analysis, it’s that it looks to me like any analysis would probably point in the same direction — it would give you a parts list for an inner listener. Whereas what I keep hearing from the opposing position is “Qualia are primitive, so anything that screams probably has pain.” What I need to hear to be persuaded is, “Here is a different parts list for a non-mysterious inner listener. Look, pigs have these parts.” I don’t put any particular weight on prestigious people saying things if this does not appear to be the form of what they’re saying — I would put significant weight on Gary Drescher (who does know what cognitive reductionist philosophy looks like) private-messaging me with, “Eliezer, I did work out my own parts list for inner listeners and I’ve studied pigs more than you have and I do think they’re conscious.”

Carl: Quote from [Stanford Encyclopedia of Philosophy] summarizing some of my objection to Eliezer’s use of the mirror test and restrictive [higher-order theory] views above:

“In contrast to Carruthers’ higher-order thought account of sentience, other theorists such as Armstrong (1980), and Lycan (1996) have preferred a higher-order experience account, where consciousness is explained in terms of inner perception of mental states, a view that can be traced back to Aristotle, and also to John Locke. Because such models do not require the ability to conceptualize mental states, proponents of higher-order experience theories have been slightly more inclined than higher-order theorists to allow that such abilities may be found in other animals.”

Robert Wiblin: [Eliezer], it’s possible that what you are referring to as an ‘inner listener’ is necessary for subjective experience, and that this happened to be added by evolution just before the human line. It’s also possible that consciousness is primitive and everything is conscious to some extent. But why have the prior that almost all non-human animals are not conscious and lack those parts until someone brings you evidence to the contrary (i.e. “What I need to hear to be persuaded is,”)? That just cannot be rational.

You should simply say that you are a) uncertain what causes consciousness, because really nobody knows yet, and b) you don’t know if e.g. pigs have the things that are proposed as being necessary for consciousness, because you haven’t really looked into it.

Carl: Seems to me that Eliezer is just strongly backing a Hofstadter-esque HOT [higher-order theory] view. HOT views are a major school of thought among physicalist accounts of consciousness. Objections should be along the lines of the philosophical debate about HOT theories, about how much credence to give them and importance under uncertainty about those theories, and about implementation of the HOT-relevant (or Higher-Order-Experience relevant) properties in different animals.

BTW, Eliezer, Hofstadter thinks dogs are conscious in this HOT way, which would presumably also cover chickens and cows and pigs (chickens related to corvids, pigs with their high intelligence, cows still reasonably capable if uninterested in performing and with big brains).

“Doug [Hofstadter], on the other hand, has a theory of the self, and thinks that this is just the same as talking about consciousness. Note that this concern with consciousness is not the same concern as whether there is a “subject” that “has” experiences over and above the public self; you can believe that talk of consciousness is irreducible to talk of the built self without thereby positing some different, higher self that is the one that is conscious. As Sartre puts it, consciousness is a primary feature of our experience and the self is built later. For Doug, we should consider an animal conscious only insofar as it’s built up this kind of self-symbol. So consciousness will be a matter of degree: there is probably little-to-nothing that it is “like to be” a mosquito, yet certainly a dog has a conception of self, and once you get language in there you get the whole deal.” (link)

Robert: Thanks [Carl] for all that useful info. I have no particular objection to HOT of consciousness. I just think people should be really unsure about what is going on with consciousness, because there is no overwhelming case for one view over another and it’s such a hard question to study with current technology. I am not even confident of physicalism, which seems to be popular in our circles as much for sociological as sound reasons. Perhaps consciousness is another primitive part of the universe? Perhaps it is an epiphenomenon that is generated by a very large number of processes? We should be more humble about what we do and do not know (more).

Appearing to jump to the conclusion that non-humans do not meet the standards required for HOT of mind also looks to me like sloppy motivated reasoning. As you point out, even if HOT of mind are correct, a wide range of animals may well have the required capabilities.

Eliezer: So there’s this very common thing where it looks to me like other people are failing to distinguish (from my perspective) between confidence about P vs. Not-P and confidence within P. Like, we can be extremely confident about the general category of natural selection vs. intelligent design without any of the open questions of evolutionary biology creating uncertainty that slops over outside the whole category of natural selection and lets [intelligent design] advocates say, “Oh, how can you be so certain of things?”

In this case, the reason I feel this way is that, for example, I don’t particularly feel like I’m confident of Higher-Order Thought vs. Higher-Order Perception or rather it seems like a Wrong Question, and the whole presentation of the theory within SEP seems like not at all the way I would present an attempted reduction; like it’s the sort of consciousness theory you would make up if you were confident that reflection had something to do with it, but were rushing a bit too fast to say exactly what, before you’d really started taking apart reflection into parts or explaining exactly which aspects of our beliefs about consciousness were originating in exactly which reflective algorithms as seen from the inside.

On the other hand, when I see someone who seems to be reasoning that pain qualia are the cause of screaming, so things that scream and look distressed are probably doing so for the same reason, namely pain qualia, then all of my uncertainty lives in worlds where this is definitely, no doubt about it false, and my remaining uncertainty does not slop over.

The Slopover Fallacy really needs a more standard name.

This doesn’t make me confident that pigs lack qualia, and I have scheduled to look at Carl’s thing on the mirror test because that is the sort of thing that affects what I think I know about pigs. Gary Drescher understands what does or does not constitute dissolution of a seemingly mysterious thing as a cognitive algorithm as seen by inside (and has either developed the same theory of consciousness independently or something very close to it), so if he calls me up and says, “I’ve studied pigs, I think they have enough empathic brain-controlling architecture to have a pig-that-is-this-pig with perceived and imaginable properties and this is what my model says is the inner listener”, then I would stop eating pigs until I’d talked it over with him. People who don’t live by the rules of reductionism, regardless of what their job title says or what other people respect them, don’t get a seat at my table. They live outside my space of uncertainty, and although it feels to me like I’m uncertain about lots of things, none of that uncertainty Slops Over into their mental world and grants it any credibility.

It also seems to me that this is not all that inaccessible to a reasonable third party, though the sort of person who maintains some doubt about physicalism, or the sort of philosophers who think it’s still respectable academic debate rather than sheer foolishness to argue about the A-Theory vs. B-Theory of time, or the sort of person who can’t follow the argument for why all our remaining uncertainty should be within different many-worlds interpretations rather than slopping over outside, will not be able to access it.

Robert: The less strong is the case behind our existing theories of consciousness, the more credence we should put on ‘something else’.

Physicalist monism doesn’t lend itself to an easy solution to the hard problem of consciousness. As I am more confident that I have subjective experience than that the external world exists, this is a drawback. Of course idealism, dualism, panpsychism, and ‘something else entirely’ have serious drawbacks too. But until we are closer to a theory of mind that produces widely acceptable results, I don’t want to make belief in any of these theories part of my identity.

Of course, the probability that consciousness has some causal connection to what’s going on in the brain is pretty close to 100% (though even then, we may be in a simulation and in fact that brain is just put there to confuse us).

Eliezer: Robert, is there some part of you that knows that in real life of course the real answer is going to be a physicalist one, even if your System 2 is saying not to be confident of that?

Robert: Of course my gut instinct is that there is only one kind of stuff and it’s all physical. This is the opposite of the normal evolved human intuition. But I am instinctively a physicalist, not least because I’ve spent my adult life being surrounded by people who make fun of anyone who says otherwise. Believing this is a handy identity marker that the skeptics/rationalists use to draw the boundaries on their community. And sure, physicalist science has done a great job explaining things.

But consciousness is the natural counter. I am almost sure consciousness exists, but it doesn’t seem to be a physical thing, at least not as physical stuff is understood now. Yes, physical stuff seems to be involved, and pinning down that involvement would be great, but unless I’m being massively fooled, there seems to be something else there as well. Maybe my subjective experience can be satisfactorily squared with physicalism, but if so, I haven’t seen it done yet.

Eliezer: Do you think there is a relatively straightforward piece of knowledge which might convince you that consciousness is physical? Like, is it plausible to you that such an insight exists and that it is not supernaturally sacred, that afterward you would look back and be surprised you ever were confused?

My point is that it shouldn’t be very weird to you if someone else seems much more confident than you of physicalist consciousness, since you register a high probability that it is true, and also a high probability that given that it is true there’s some simple insight that makes it seem much more obvious. Like this is not a case where you’re entitled to conclude that if you don’t know, someone else who claims to know is probably mistaken. (Example case where that inference is valid: AI timelines.)

Robert: You have that insight? If so, please share. Of course it is never weird to see people overconfident in their theories, that is the nature of being human. But that doesn’t make it any less important to push back.

Eliezer: (points to Reductionism sequence on LW) … nobody said it had to be condensed into a sentence.

Robert: Any particular part where it explains how to reconcile physicalism with my seeming to have subjective experience?

Eliezer: The most direct address might be in Zombies! Zombies? which can plausibly be read out of order. It’s not going to give you the full experience, and it’s more like telling you that physicalism has to be true than giving you a sense that it can be true despite the apparent impossibility. Good and Real might get you further especially if you are allergic to LW posts.

Robert: In large part I’m taking my lead from Chalmers, who is very knowledgable of all the arguments in this area, isn’t convinced of physicalism as far as I can see, and seems unusually intellectually honest and modest.

Eliezer: If I can make you read more:

How An Algorithm Feels From Inside

Dissolving the Question

Hand vs. Fingers

Heat vs. Motion

Angry Atoms


Excluding the Supernatural

Robert: OK, I’ll read a decent amount and see if I’m convinced.

Of course even if I am convinced of physicalism, that would still leave a lot of uncertainty about what precise physical structures are required. And even assuming something like a HOT of consciousness, it doesn’t seem safe to assume animals are not conscious (based on priors, common sense, and Carl’s links above).

Eliezer: We’re dealing with a relatively long inferential chain here. Trying to get all the way to the end is out of the cards. At best it will seem to you plausible that, extrapolating from earnest-tokens provided, it’s not implausible that I could occupy the epistemic state I claim to occupy.

Robert: The longer is the inferential chain, and the less other experts have chosen the same inferential chain, the more likely that you’ve made a mistake somewhere! But sure, I appreciate that you’re making your case in good faith.

Rob Bensinger: “Physicalist monism doesn’t lend itself to an easy solution to the hard problem of consciousness. As I am more confident that I have subjective experience than that the external world exists, this is a drawback.”

I used to think this way, but no longer do. I could list a few things that persuaded me otherwise, Robert, if that helps.

(Backstory: I’ve studied Chalmers’ arguments in some detail, and these days I don’t think they refute physicalism. I do find them very interesting, and I think they’re evidentially weak and outweighed-by-stronger-considerations, rather than thoughtless or confused. I also agree Chalmers is one of the best philosophers out there. (He discovered a lot of the best arguments against his view!) But I don’t think there’s any human institution that’s up to the task yet of consistently incentivizing ‘getting the right answer to the hardest philosophical questions in a timely fashion’. Top-tier analytic philosophers like David Lewis, David Armstrong, Donald Davidson, Peter van Inwagen, or Thomas Nagel — or David Chalmers — don’t receive esteem and respect in academic philosophy because they’ve proven that they’re remarkably good at generating true beliefs. They receive esteem and respect because they have a lot of creativity and ingenuity, are very careful close readers of other prominent philosophers, and give meticulous and thought-provoking arguments for exotic theses. In some ways, it matters more that their writing is intellectually fun than that it’s strictly accurate.)

Changes that caused me to break with Chalmers’ basic perspective:

(1) I came to believe that ‘consciousness’ is not a theory-neutral term. When we say ‘it’s impossible I’m not conscious’, we often just mean ‘something’s going on [hand-wave at visual field]’ or ‘there’s some sort of process that produces [whatever all this is]’. But when we then do detailed work in philosophy of mind, we use the word ‘consciousness’ in ways that embed details from our theories, our folk intuitions, our thought experiments, etc.

As we add more and more theoretical/semantic content to the term ‘consciousness’, we don’t do enough to downgrade our confidence in assertions about ‘consciousness’ or disentangle the different things we have in mind. We don’t fully recognize that our initial cogito-style confidence applies to the (relatively?) theory-neutral version of the term, and not to particular conceptions of ‘consciousness’ involving semantic constraints like ‘it’s something I have maximally strong epistemic access to’ or ‘it’s something that can be inverted without eliminating any functional content’.

E.g., if you think the inverted qualia argument against physicalism is as certain as a minimalist cogito, you may be startled to learn that the argument doesn’t work for any known color experience — see this. But we were so certain! We were certain of something, maybe. But how certain are we that we can correctly articulate what it is that we’re certain about?

Possibly that particular argument can be repaired, but it does tie in to a broader point…

(2) As we learn new things, that can feed back in to our confidence in our premises. Evidence about our own mental states may have temporal priority, but that doesn’t mean that we retain more confidence in introspective claims than in physics even after we’ve finished examining the psychology research showing how terrible humans are at introspection. (Or even after examining physics.)

How good an agent is at introspecting is a contingent fact about its makeup. You can build an AI that’s much better at introspecting its own mental states than at perceiving sensory objects, a RoboDescartes. Such an agent would be better off relying on its reflective reasoning than at doing science, and it would be foolish to trust science over phenomenology and introspectionist psychology. Empirically, however, we aren’t that kind of mind; we screw up all the time when we introspect, and our reflective judgments seem a lot more vulnerable to overconfidence and post-facto rationalization than is our ordinary perception of external objects.

(3) On top of that, we should be especially suspicious of our introspection in cases where there’s no cognitive or evolutionary account for why it should track reality. Evolution won’t waste effort adding extra truth-tracking faculties unless it actually helped our ancestors survive and reproduce. If Chalmers’ arguments are right, not only is there no such account available, but there can never be one. In fact, our brains are systematically deluded about consciousness, though our acausal minds are somehow systematically correct about our world’s metaphysics. (This is the main point Eliezer focuses on in his objections to Chalmers.)

We can’t provisionally forget the history of science, the available literature from the sciences of brain and mind, etc., when we do philosophy or phenomenology. We have to actually bear in mind all that science when we critically examine cogito-like evidence. It’s all of a kind. That sort of perspective, and a perspective that focuses on ‘what kind of work is this cognitive system doing, and what mechanism could allow it to do that?’ a la A Priori, is I think a lot of what gets under-emphasized in philosophy of mind. Philosophers who call themselves ‘reductionists’ have more practice asserting reductionism as a metaphysical thesis, than they have actually performing reductions on empirical phenomena, or thinking of their intuitions as physical processes with physical explanations. (Even Chalmers can accept this point, since he thinks for every intuition our consciousness has, there’s a corresponding intuition in our brains that phenomenal consciousness has absolutely no explanatory relevance to.)

Eric Bruylant: As a moral vegetarian since the age of four, this by a considerable margin the most relevant argument against my reasons for vegetarianism which I have encountered so far, though I’m not likely to switch over primarily due to ethical caution, and to some extent inertia. From my experience with animals it seems at least plausible that, at minimum, many social mammals possess reflective self-awareness (>5% probability that several branches of mammals other than apes have it), and so long as the opportunity costs of avoiding eating meat remain low and my aversion to encouraging potential death of sentient life is high sticking with vegetarianism seems like a reasonable precaution. I would be interested in more details of why you think key parts of the machinery which generates consciousness are missing from most social mammals, since reflective self-awareness and similar mind-modeling tools seem like they would be extremely useful for any creature engaging in politics.

Eliezer: Eric, if an animal has coalitions of coalitions (bottlenose dolphins) or is doing anything that looks like political plotting (chimps) then that is an instant stop eating order. A book called PIG POLITICS: Only Slightly Less Complex Than Chimpanzee Politics would get me to stop eating bacon until I had learned whether the title was an exaggeration. I also want to emphasize that the “why so confident?” is a straw misquestion from people who can’t otherwise understand why I could be unconfident of many details yet still not take into account the conflicting opinion of people who eg endorse P-zombies. I guess that pigs are lacking parts whose presence I see little positive reason to expect. The chimpanzee politics that produced the selection pressure for their renowned cognitive abilities is actually pretty damned impressive. I never heard of pigs having anything like that.

Robert, curiosity: Suppose I claimed to be able to access an epistemic state where (rather than being pretty damn sure that physicalism is true) I was pretty damn sure that P-zombies / epiphenomenalism was false. How does the intuitive plausibility of my being able to access such an epistemic state feel to you? Does this equally feel like “Eliezer shouldn’t be so sure, so many experts disagree with him” or is it more plausible that “Hm, it seems probable that epiphenomenalism is false, and if it is false it seems plausible that there’s some legitimate way for Eliezer to know it’s false.”

Robert: Your believing it is false is certainly evidence against, but in a field with lots of epistemic peers who aren’t yet convinced by your line of argument it isn’t overwhelming evidence. Humans just aren’t very good at doing philosophy, so I can’t put too much trust in any one person’s views/intuitions.

Eliezer: Robert: I’m pretty happy to take a stand on it being okay for me to not update on other people believing in P-zombies. If we agree that these issues are of the same kind, then this case is far far more accessible to me — much much easier to communicate. Like “not P-zombies” is 3% of the way to the finish line of the previous issue.

Rob Bensinger: Robert [Wiblin]: ‘Zombies! Zombies?’ probably doesn’t hit hard enough outside the context of the sequences. Inside the sequences, it’s one of dozens of posts hammering on one of Eliezer’s most basic theses: ‘Beliefs are physical states of the brain, useful because we’ve evolved to be able to put those states in physical correlation with configurations of stuff outside the brain. Acquiring veridical beliefs, then, is a special case of thermodynamic work, and requires the emission of waste heat. Any specific reliable belief about any matter of fact must (in principle) admit of a causal explanation that links the brain state to the worldly circumstance it’s asserting.’

Now we’re not just picking on Chalmers. We have a general causal theory of evidence, and as a side-effect it predicts that brains can’t know about epiphenomenalism (even if it’s true!). No event in the brain, including any thought experiment or intuition or act of introspection, can causally depend in any way on epiphenomenalism, the character of (epiphenomenal) phenomenal consciousness, etc. Our brains’ miraculously guessing correctly that they are attached to an epiphenomenal ghost would be a pure lucky coincidence. Like guessing ‘there’s a ghostly wombat 200 feet under my house that has never causally impacted anything’ and getting it right; or like guessing ‘every diamond lattice in the universe is attached to an epiphenomenal feeling of regret’. The greater causal proximity of our brains to our consciousness shouldn’t make any difference, because the causal arrows only go away from the brain, not toward it.

If you assign a low prior to ‘all diamonds are epiphenomenally regretful’, you should also assign a low prior to ‘my brain is currently epiphenomenally attached to a visual experience of seeing this sentence, which has no impact on my brain’s beliefs or assertions to that effect’. And if you have no prior reason to believe there’s a Matrix God who created our world, loves epiphenomenalism, and would build dualistic minds with just the right mind-matter bridging laws (a view called occasionalism), then no argument or experience you have can lend evidential support to epiphenomenalism, for the exact same reason it can’t send support to the diamonds-are-regretful thesis. If you can’t establish a causal link inside the system, then you need to posit a force outside the system to establish that causal link — else the physical process we call ‘evidence’ just won’t work.

Robert Wiblin: Why do you think you should trust your judgement on that question over the judgement of similarly qualified people? I don’t get it.

Rob Bensinger: I think it’s pretty widely accepted by philosophers that traditional epiphenomenalism is one of the more fraught and hard-to-believe views on consciousness. (E.g., it was abandoned a decade ago by its most prominent modern defender, Frank Jackson.) What Chalmers would say if you brought this criticism to him (and has said before, on LW) is that his view isn’t ‘traditional epiphenomenalism’. His view allows for phenomenal consciousness to be ‘causally relevant’ to our phenomenal judgments, because his view identifies consciousness (or its underpinnings) with the fundamental thingies (‘quiddities’) that play the functional roles described in physics. It’s mysterious what electromagnetism is ‘in itself’ (above and beyond the relations it enters into with other patterned empirical thingies), and it’s mysterious what consciousness is, so we combine the two mysteries and sort-of-solve them both. (Which feels elegant and gratifying enough that we’re willing to abide the lingering mysteriousness of the view.) This is also Strawson and David Pearce’s view. See this.

I don’t think this is an adequate response to Eliezer’s objection. Aside from occasionalism, I haven’t yet seen any adequate response to Eliezer’s objection in the literature. The problem is that this view treats ‘causal relevance’ as a primitive, like we can just sprinkle ‘causality’ vaguely over a theory by metaphysically identifying phenomena really really closely, without worrying about exactly how the physical structure of a brain ends up corresponding to the specific features of the phenomena. The technical account of evidence Eliezer is giving doesn’t leave room for that; ‘causal relevance’ is irrelevant unless you have some mechanism explaining how judgments in the brain get systematically correlated to the specific facts they assert.

If the zombie argument works, quiddities can’t do anything to explain why we believe in quiddities, because our quiddities can be swapped out for nonphenomenal ones without changing our brains’ dynamics. If the qualia inversion argument works, quiddities can’t explain why we have accurate beliefs about the particular experiences we’re having (e.g., as William James noted, that we’re experiencing phenomenal calm as opposed to phenomenal agony), because the quiddities can be swapped out for other phenomenal quiddities with a radically different character. The very arguments that seek to refute physicalism also refute all non-interactionist dualisms. I think this has been unclear to professional philosophers because philosophers of mind mostly treat ‘causality’ and ‘evidence’ as black boxes, rather than having a detailed theory of how they work (such that they would need to constrain their philosophy-of-mind views to accord with that theory).

Systematic philosophy is relatively unpopular in modern academic analytic philosophy, so different fields often carry on their debates in isolation from each other. And systematic philosophy is especially unpopular these days among hard-nosed reductionists — the sorts of academics most likely to share Eliezer’s interests, background, and intuitions.

If you want to know how a blind person guessed that you’re holding a diamond, it’s not enough to say that the diamond is ‘causally relevant’ to the blind person (e.g., photons are bouncing off of the diamond to hit the blind person). You need at least a schematic account that allows the blind person’s brain to systematically correlate with the physical structure and location of the diamond. If your assumptions are telling you that a certain systematic correlation between belief and reality is a coincidence, then, vague ‘causal relevance’ or no, at least one of your assumptions must be wrong.

Eliezer: I agree with Rob [Bensinger]’s reply.