How to Fix the Referee Crisis in Professional Philosophy

It has come to the point that the most suitable referee(s) for a paper are almost never available. It takes us so long to find reviewers, sometimes a month, six weeks or more (this situation is also exacerbated by the fact that many people don't respond to referee requests at all). This lengthens the span of time even more for the total review process. All sorts of small fixes don't work anymore, e.g., shortening the time that people get to review, asking for alternatives (this is still very much appreciated, but unfortunately alternative referees are just as unavailable). 

Of course, it could be that my editorial experience is atypical--I'm just one scholar.

But it is my strong suspicion that the peer review system is finally broken beyond reasonable repair. We've seen a slow worsening of the situation and the pandemic has finally broken the system. People are burnt out and overburdened, job candidates increasingly desperate.--Helen de Cruz "Seems like the peer review system has given up the ghost" @The Philosophers' Cocoon

De Cruz's piece has generated a huge discussion, not just among the Cocooners, but also (here) at Daily Nous. The situation she describes has been a long time coming (and for once I can claim I partially predicted it (recall here; here)). There are a number of major interlinked reasons for the current status quo (extreme referee shortage): (i) an extreme scarcity of decent tenure track jobs relative to number of PhDs minted. This is a major source of anxiety throughout the profession. (ii) The unreasonably low acceptance rate of philosophy journals such that perfectly competent papers have to be refereed multiple times by a fixed number of referees. (iii) The perceived role that prestige and quality of publications play in hiring. And, perhaps, (iv) the relative ease of free-riding in the current system. The real free riders are universities (who outsource quality control without rewarding it internally--my own university never even asks to list refereeing anywhere) rather than particular individuals. In what follows, I take for granted that the low acceptance rates are functional, that is, maintained in order to produce prestige (recall this post). And the combination of (i-iii) has generated a veritable arms race in publication. In addition, (v) the long delays in refereeing time, have made it prudent to have multiple papers under review at the same time, thus, exacerbating the underlying problem.

In what follows, I take for granted that the norm of regular publication has been good for professional philosophy. Reasonable people can disagree about this. But in my view it has allowed for more fine-grained discussion and for specializations to emerge that have improved quality. Yes, it risks generating hyperspecialized echo-chambers with lowest common denomination salami publications, but these, in turn, generate arbitrage opportunities for people who read widely or who practice what I call synthetic philosophy.

I also take for granted that (i) is entrenched in the political economy of modern universities. I am complicit in it, if you find a way to create a revolution, good for you. But what we can do, as a profession, is to cut the ties between prestige, journal publication, and hiring decisions and so reduce the need for referees. And we can do that if we destroy the role of pre-eminence of journals in manufacturing prestige, and restore them to the role of disseminators of knowledge and sites of discussion. This can be achieved in two steps: first, all the major philosophy journals should commit to publishing, say, 25% of articles received. That's considerably higher than the status quo (Phil Quarterly, publishes just 4% of over 850 papers received.) This will have six obvious effects: it will reduce the load on referees; it will end the excessive role of chance in where your paper is published; it will reduce (but not eliminate) the steep prestige hierarchy of our current journal system; it will reduce the excessive delays in getting work into print caused by the difficulty of finding referees; it will turn most major journals into paper mills; it will increase citation rates within the profession.

One may worry that once journals can't play the role of extreme prestige generators, then it's entrenches the roles of PhD granting departments. I agree, and that would be bad. So, and this is the second step: another institution has to take its place. Luckily, such a practice already exists in neighboring disciplines and is present in incipient form in ours: the article (and book) prize. What we need is a proliferation of prizes in philosophy: conference prizes (which are pre-publication); professional association prizes (of the major and smaller societies); best article prizes by journals themselves; and we need more versions of The Philosopher’s Annual.:) In fact, as Don Ainslie reminded me, there should be more prizes on different grounds too: philosophy has far too few of these, which in turn impacts success in interdisciplinary competitions (e.g. national scholarly societies, grant competitions, etc.).+ 

At this point one may worry that such prizes are easily shaped by politics, and that our time will be spent adjudicating prizes rather than refereeing. On the former, I don't see how politics can be removed from the equation entirely (as it is, the journal system has a major 'trust us' component recall). I do not deny that genuine masked review, when it happens, is a real potential equalizer. And notice that on my proposal, we don't eliminate that. People who come from non prestigious intellectual backgrounds can still use journal publication to become noticed. (But the initial signal's power may well be reduced.) And one can use masked review also in prize committees. Over time prizes that show a good track record in their judgment will become more prestigious, and so complement journals in prestige generation and quality control. 

In a way, my proposal is a variant on, and complements, the post-peer crowd review advocated by Marcus Arvan, Liam Kofi Bright, and Remco Heesen which they have defended in print and at Daily Nous (here),* and the ideas promoted by Katzav and Vaessen in a paper in Philosopher's Imprint. (Recall also my post here during the Hypatia controvery, but I had limited the idea for work that might be controversial in various ways and in order to internalize political considerations in a transparent and fruitful way in the review process.) Using prizes is a mechanism to solve the coordination and collective action problem inherent in such proposals, and to assuage worries that non-experts may overwhelm the wisdom of our own crowds.

On the latter concern, this is undeniably true. But I suspect it's more fun to evaluate papers that have already been published -- and so have undergone non trivial quality and editorial control. Also, while writing a prize report is a non-trivial exercise, it's only required for the winning paper(s).  So, on balance I would expect it should reduce the load on those in the profession that are the back-bone of our current referee-system.** 

 

 

+This comment was added after initial publication of the post.

*I also suspect my proposal solves an objection against those that think Condorcet jury theorem is not apt here, because prize awarding is standard setting not discovery of truth. But that's for another time.

**I thank Helen de Cruz and Neil Levy for discussion.

 

 

 

 

Comments

[F. Contesi]

This may well be a (few) step(s) in the right direction but some of the problems (i.e. opacity and referee assignment) with the refereeing process would seem to remain---or even be exacerbated. Publishing more papers and adding prize selection without changing the refereeing process would do that. Changing the refeering process would still be required, e.g. by way of outsourcing refereeing to a transparent and public Internet hive mind, such as the one advocated by https://doi.org/10.1086/719117 , and implementable through https://freelosophy.github.io/ .