This article was written for the website Less Wrong, and is cross-posted here.
I won’t try to argue that papers aren’t worth publishing. There are many reasons to publish papers – prestige in certain communities and promises to grant agencies, for instance – and I haven’t looked at them all in detail. However, I think there is a conclusive case that as a discussion forum – a way for ideas to be read by other people, evaluated, spread, criticized, and built on – academic papers fail. Why?
1. The time lag is huge; it’s measured in months, or even years.
Ideas structured like the Less Wrong Sequences, with large inferential distances between beginning and ending, have huge webs of interdependencies: to read A you have to read B, which means you need to read C, which requires D and E, and on and on and on. Ideas build on each other. Einstein built on Maxwell, who built on Faraday, who built on Newton, who built on Kepler, who built on Galileo and Copernicus.
For this to happen, ideas need to get out there – whether orally or in writing – so others can build on them. The publication cycle for ideas is like the release cycle for software. It determines how quickly you can get feedback, fix mistakes, and then use whatever you’ve already built to help make the next thing. Most academic papers take months to write up, and then once written up, take more months to publish. Compare that to Less Wrong articles or blog posts, where you can write an essay, get comments within a few hours, and then write up a reply or follow-up the next day.
Of course, some of that extra time lag is that big formal documents are sometimes needed for discussion, and big formal documents take a while. But academic papers aren’t just limited by writing and reviewing time – they still fundamentally operate on the schedule of the seventeenth-century Transactions of the Royal Society. When Holden published his critique of the Singularity Institute on Less Wrong, a big formal document, Eliezer could reply with another big formal document in about three weeks.
2. Most academic publications are inaccessible outside universities.
This problem is familiar to anyone who’s done research outside a university. The ubiquitous journal paywall. People complain about how the New York Times and Wall Street Journal have paywalls, but at least you can pay for them if you really want to. It isn’t practical for almost anyone doing research to pay for the articles they need out-of-pocket, since journals commonly charge $30 or more per article, and any serious research project involves dozens or even hundreds of articles. Sure, there are ways to get around the system, and you can try to publish (and get everyone else in your field to publish) in open-access journals, but why introduce a trivial inconvenience?
3. Virtually no one reads most academic publications.
This obviously goes together with point #2, but even within universities, it’s rare for papers, dissertations or even books to be read outside a very narrow community. Most people don’t regularly read journals outside their field, let alone outside their department. Academic papers are hard to get statistics on, but eg., I was a math major in undergrad, and I can’t even understand the titles of most new math papers. More broadly, the print run of most academic books is very small, only a few hundred or so. The average Less Wrong post gets more views than that.
4. It’s very unusual to make successful philosophical arguments in paper form.
When doing research for Personalized Medicine, I often read papers to discover the results of some experiment. Someone gave drug X to people with disease Y. What were the results? How many were cured? How many had side effects? What were the costs and benefits? All useful information.
However, most recent Singularity Institute papers are neither empirical (“we did experiment X, these are the results”) or mathematical (“if you assume A, B, and C, then D and E follow”). Rather, they are philosophical, like Paul Graham’s essays. I honestly can’t think of a single instance where I was convinced of an informal, philosophical argument through an academic paper. Books, magazines, blog posts – sure, but papers just don’t seem to be a thing.
5. Papers don’t have prestige outside a narrow subset of society.
Several other arguments here – the time lag, for instance – also apply to books. However, society in general recognizes that writing a book is a noteworthy achievement, especially if it sells well. A successful author, even if not compensated well, is treated a little like a celebrity: media interviews, fan clubs, crazy people writing him letters in green ink, etc. (This is probably related to them not being paid well: in the labor market, payment in social status probably substitutes to a high degree for payment in money, as we see with actors and musicians.)
There’s nothing comparable for academic papers. No one ever writes a really successful paper, and then goes on The Daily Show, or gets written up in the New York Times, or gets harassed by crowds of screaming fangirls. (There are a few exceptions, like medicine, but philosophy and computer science are not among them.) Eg., a lot of people are familiar with Ioannidis’s paper, Why most published research findings are false. However, he also wrote another paper, a few years earlier, titled Replication validity of genetic association studies. This paper actually has more citations – over 1300 at least count. But not only have we not heard of it, no one else outside the field has either. (Try Googling it, and you’ll see what I mean.)
6. Getting people to read papers is difficult.
Most intellectual people regularly read books, blogs, newspapers, magazines, and other common forms of memetic transmission. However, it’s much less common for people to read papers, and that reduces the affordances that people have for doing so, if they are asking “hey, this thing is a crazy idea, why should I believe it?”. Papers are, intentionally, written for an audience of specialists rather than a general interest group, which reduces both the tendency and ability of non-specialists to read them when asked (and also violates the “Explainers shoot high – aim low” rule)
7. Academia selects for conformity.
The whole point of tenure is to avoid selecting for conformity – if you have tenure, the theory goes, you can work on whatever you want, without fear of being fired or otherwise punished. However, only a small (and shrinking) number of academics have tenure. In order to make sure fools didn’t get tenure, it turns out academia resorted to lots and lots of negative selection. The famous letter by chemistry professor Erick Carreira illustrates some of what the selection pressure is like, similar to medicine or investment banking: there’s a single, narrow “track”, and people who deviate at any point are pruned. Lee Smolin has written about this phenomenon in string theory, in his famous book The Trouble with Physics.
Things may change in the future, but as it stands now, many ideas like the Singularity are non-conformist, well outside the mainstream. They aren’t likely to go very far in an environment where deviations from the norm are seen negatively.
8. Papers have a tradition of violating the bottom line rule.
In a classic paper, one starts with the conclusion in the abstract, and then builds up an argument for it in the paper itself. Paul Graham has a fascinating essay on this form of writing, and how it came to be – it ultimately derives from the legal tradition, where one takes a position (guilty or innocent), and then defends it. However, this style of writing violates the bottom line rule. Once something is written on the paper, it is already either right or wrong, no matter what clever arguments you come up with in support of it. This doesn’t make it wrong, of course, but it does tend to create a fitness environment where truth isn’t selected for, just as Alabama creates a fitness environment where startups aren’t selected for.
9. Academic moderation is both very strict and badly run.
All forums need some sort of moderation to avoid degenerating. However, academic moderation is very strict by normal standards – in a lot of journals, only a small fraction of submissions get approved. In addition, academic moderation has a large random element, and is just not very good overall; many quality papers get rejected, and many obvious errors slip through.
As if that wasn’t enough, most journals are single-blind rather than double-blind. You don’t know who the moderators are, but they know who you are, raising the potential for all kinds of obvious unfairness. The most common kind of bias is one that hurts most of us unusually badly: people from prestigious universities are given a huge leg up, compared to people outside the system.
Reblogged this on Solar Physics Task Group and commented:
Beautiful list of reasons why academic paper are such a broken system… and a few more could be added, like authorship trading, maximizing minimal publishable content, reciprocal citations, …
This is an excellent post. To at least partly overcome these problems I think we need something like structured discussion on the web, as supplements to, or alternatives for, academic papers.
Completely agree. A wiki for all scientific work would be a much better platform. All edits should be saved but must be peer-reviewed to be accepted. Also has the advantage of seeing everyone’s work on the topic in the context of all other work being done.
A wiki would be great…but the chances of one being built organically is next to impossible…what is needed is a platform which bridges from the current method to the new paradagm; ie. a platform that builds dynamic content around static work like academic articles.
I don’t understand no. 8. What is your evidence that most academic papers start with a conclusion which is then fixed and justified? In my experience the process is usually the reverse: read a lot of things, think about them, come up with an interesting idea, read and think some more about the problem and idea and consider whether the idea needs revision, send those thoughts in, think about the audience’s response and consider whether to revise my opinion, … I think you are confusing the structure of a written-up paper with the structure of thoughts that produced it. I also think that things are different in different fields, so generalizing about “academe” may not be very trustworthy.
A majority of academic work is deductive in nature…the process you describe is inductive. Inductive logic may be useful in discovering potential research questions…however an existing body of literature is most often approached with a deductive method.
I think this is implied in your post, but may be worth pointing out explicitly: many of your points are more valid against _publishing_ academic papers than against _writing_ them in the first place.
And a minor remark: You write “I was a math major in undergrad, and I can’t even understand the titles of most new math papers”, which I think is perfectly reasonable: there is a lot of math! I don’t think it’s reasonable to expect that as an undergrad (or even as a graduate student) you’ll learn about more than a small fraction of the interesting topics in mathematics. This is a good thing, it means humans are good at inventing math.
The Author makes many good points. To improve academic publishing it may be wise to give more control to the individual scientist-authors and individual readers. There are a number of self-publishing website-experiments underway. I have a project called Scitopedia to test and develop a more open and free self-publishing paradigm.
While it’s true that there are plenty of problems related to getting good scholarship published quickly and accessibly, there are people working to make these things easier.
For instance, Scholastica (www.scholasticahq.com) is a platform providing the infrastructure scholars need to manage and publish journals on their own (full disclosure, I’m a founder).
So in terms of your concerns about the time it takes to publish scholarship, an editor with a journal on Scholastica could publish new work whenever they wanted to. Scholastica allows articles to be published whenever an editor deems them worthy. If an editor wishes, they can also still publish work in old-fashioned issues. Also, journals publishing on Scholastica may do so Open Access, which is a feature aimed make scholarship more available. You can see an example of what a journal publishing Open Access content looks like here: http://scholasticahq.com/the-scholastica-example-journal.
Finally, you mention that most journals are single blind rather than double. Scholastica allows journal to have single, double, and even triple blind processes (in triple blind, only the editor-in-chief knows the identity of the author, not reviewers or even other journal editors).
Since improving the dissemination of scholarship is an important to you, we’d love to talk more about what we’re doing.
If I’ve got an idea where I really want some debate, then I blog about it. I have started doing this prior to giving a talk or writing a paper, if I think the ideas need honing and want to check that they will stand scrutiny. The academic paper can then sit in the archives as a permanent and accessible source (accessible, as Wellcome Trust pay for all my papers to be open access, thank heavens).
Perhaps the best example I’ve seen of this process is this blogpost by Daniel Bor with associated comments, which was sparked off by a blogpost that I wrote.
Some of the top people in the field commented on the post, but an additional benefit is that you can also get comments from people outside academia, whose views can be very pertinent.
My field, History of Art, is slowed down further by the need to get permission to reproduce images, which are often slow to get and incredibly expensive. So I’d love to think about different frameworks for academic publishing. One of the reasons I blog is just to get ideas out there more quickly and organically than possible in a journal article. So, I do think there’s several good points here – but from a UK point of view (with the Research Excellence Framework looming, http://www.ref.ac.uk/) traditional peer-reviewed journals will always be the first point of call for publishable work. They are a safe bet, prestigious, and it’s easier to get 4 articles out than 4 books. They are easier to assess in terms of quality, as well, when for some academics publishing online remains a foreign idea. How to assess, say, academic blogging (fast but not formally peer reviewed) or other types of websites etc?
I agree with your point “Getting people to read papers is difficult”. I am not sure how many researchers are reading journals on a regular basis, particularly so because there are so many of them in every field of study. It would be a good idea to come with some sort of standard taxonomy so that researchers can easily search and find article relevant to their interest. As of now scholars have to search/ read through massive content to get to interesting articles.