As any journal editor will tell you (at length, possibly via the medium of rant), the trickiest part of the job is not the papers, not the authors, and not even the typesetters. It’s the referees. It is no mean feat to secure referees who are, first, reliable in their academic judgment, second, responsive to emails, and third, willing to return reports when they say they will. But the frustrations of editors aside, the far more pressing concern is for the career prospects of early-career researchers. Jobs and funding can depend on timely decisions. Indeed, whether an early-career researcher gets to become a mid- or late-career researcher can depend on whether a decision is made in a reasonable amount of time....Whatever the answer, authors need more from referees than they ever have done; more depends on papers being reviewed in a professional, timely manner. And at the very least, there’s a ‘pay it forward’ case to be made:...
But what to do about the bad referees, the system’s free riders? Relentless pestering and various forms of emotional blackmail fall on deaf ears. At the BJPS, we operate a flag system for persistent offenders, but all this amounts to is bad referees being asked to perform fewer reviews, while good referees carry more of the load.
So here’s a radical suggestion, using the only weapon/motivational device editors have: If someone fails to fulfill their duties as referee, the journal will not accept submissions from that referee, for some period of time to be determined. The time period should reflect the severity of the dereliction of duties. For instance, agreeing to act as a referee but then disappearing off the radar might warrant the most substantial ban. Delivering a meager report that’s extremely late, and without communicating with the relevant editor about the delay, might mean some shorter period of time on the bench. First-time offenders surely deserve different treatment to persistent re-offenders. And the embargo period will need to be substantial enough to be effective (too short and it will have no real impact; too long and it’s probably not practical due to the changes in the editorial team). The details can be ironed out.--Elizabeth Hannon "Peer Review or Perish: The Problem of Free Riders in Philosophy" @Dailynous
About once a year somebody proposes -- remember last Summer David Velleman's proposed moratorium on graduate submissions [recall my response] -- a radical new solution to the problems facing the way professional philosophers handle publication. As I suggest below, how to characterize the nature of the problem is itself part of the problem. [Trigger Alert: the key, scandalous part of this post is a few paragraphs down!] Let me grant to Hannon that there is a free rider problem: it's possible not all would be referees are chipping in equitably from a system they benefit from. For example, I know senior peers that only review for 'top four journals' [hereafter 'T4J'] or (more popular to admit) that take lengthy sabbaticals from reviewing. (No worries I won't rat you out.) Let's stipulate they free ride. They are not the only such free riders in the system: grant agencies and universities everywhere want to see (increased) academic 'productivity' and 'output' (that is refereed publications), but increasingly refuse to spend money on hosting journals or rewarding referee work by their staff.*
Before I get to the scandalous part of my post, let me note that Hannon's proposal will lead to a predictable consequence. As Thorsten Wilholt noted -- and he like me referees for lots of journals he has no hope/desire or expectation ever to be published in --, "the option of being able to submit to them in the future is of little value to me. I fear that the proposed system of disincentives, if widely adopted, would turn refereeing into a quid pro quo thing in the minds of most academics. As such, it would benefit some top journals...but work to the disadvantage of others....So people would reserve their refereeing time for journals that they are interested in submitting to." (That's already the de facto the practice of my friend who only referees for T4J!)
What's notable of Hannon's proposal is that it neither really looks at how other disciplines handle such problems nor shows any signs of reflecting on the institutional and social sources of the problem. Trigger Warning: for the dirty secret of philosophy is that we have insanely ridiculously low acceptance rates -- often well under 10% -- for papers. This low rate is only defensible if you think that publication in philosophy has the kind of inductive risk that any false positive leads to society's catastrophe. Nobody thinks that. (And I actually think philosophy is really important to society's flourishing!) This low acceptance rate is coupled with an insanely steep hierarchy (recall) pertaining to which journals are worth publishing in.
In the old days (not so long ago in professional philosophy), when our purportedly prestigious journals were run unprofessionally, as clubhouses serving an in-crowd, this low acceptance rate was not perceived as a real problem because insiders could get their PhD students and junior faculty friends published. But since we have moved to more minimal professional standards (although few are as good as Ergo), the high rejection rate has made publication de facto a lottery where those who need publication most are at the mercy not just of the timeliness but also humors of the least appreciated element in the system, the anonymous referee. Because there is a veritable arms race in publication, this means that papers are now routinely submitted early and then improved through a dialectic with the very same anonymous referees (some of whom (recall) end up being the personal gatekeeper of a particular paper).
So, rather than punishing free riding referees (which creates its own collective action and procedural fairness problems), the more rational response [AND [later insert--thank you Justin Weinberg] I HAVE BEEN ANTICIPATED BY NEIL SINHABABU] is to increase the acceptance rate in professional philosophy from the the top journals down to about, say, 30% or so. (Lots of fields have even higher acceptance rates.) Given, that these days most publication is really electronic publication, there is no reason to keep the number of articles and journal space so limited. My proposal would eliminate a good chunk of the problem in the profession: a lot fewer papers would be shopped around (releasing valuable editorial and referee time); young scholars, who meet the minimal professional competence the discipline expects, would not be part of an unfair lottery anymore; and the sense of mystique and prestige surrounding publication in a top journal would disappear. Multiple publication would still be a sign of (would-be-)productivity and professional competence, but journal publication would stop being the proxy for extremely fine-grained (and in my mind absurd) differentiation of judgments of quality. That's how it should be. Also, it will halt the excessive nature of the arm's race; with that higher acceptance rate people will be less impressed by any extra publication beyond what is taken to be the new normal.
There are two obvious objections to my proposal: first, it would devalue being published in T4J (and lower) to some degree. That's not a bug, but a feature of my proposal. Admittedly, in a transition period, there will be some confusion about the value of a T4J publication (not everybody reads blogs), but after some time people will adjust to slightly less bright halo effects (see Helen de Cruz on prestige bias in Ergo). Second, my proposal faces a collective action problem: the leading journals have to embrace the new norm. Luckily, it's a very small collective action problem because, given the steep prestige hierarchy, this only involves very few journals to establish a new norm.+
So, bottom line: before you blame and sanction referees [the cogs that keep the system flowing], why not change the enabling, background conditions first? Okay, now let me get back to my referee-duties.
*I am ignoring the for-profit presses because in professional philosophy they tend be less dominant than in other professions.
+Of course, one may suspect it goes against the self-interest of those that benefit most from controlling the steep prestige hierarchy to give up one of their key levers.