It's Not Referees; it's The Journals that are the Problem (again, part MXCVIII)

As any journal editor will tell you (at length, possibly via the medium of rant), the trickiest part of the job is not the papers, not the authors, and not even the typesetters. It’s the referees. It is no mean feat to secure referees who are, first, reliable in their academic judgment, second, responsive to emails, and third, willing to return reports when they say they will. But the frustrations of editors aside, the far more pressing concern is for the career prospects of early-career researchers. Jobs and funding can depend on timely decisions. Indeed, whether an early-career researcher gets to become a mid- or late-career researcher can depend on whether a decision is made in a reasonable amount of time....Whatever the answer, authors need more from referees than they ever have done; more depends on papers being reviewed in a professional, timely manner. And at the very least, there’s a ‘pay it forward’ case to be made:...

But what to do about the bad referees, the system’s free riders? Relentless pestering and various forms of emotional blackmail fall on deaf ears. At the BJPS, we operate a flag system for persistent offenders, but all this amounts to is bad referees being asked to perform fewer reviews, while good referees carry more of the load.

So here’s a radical suggestion, using the only weapon/motivational device editors have: If someone fails to fulfill their duties as referee, the journal will not accept submissions from that referee, for some period of time to be determined. The time period should reflect the severity of the dereliction of duties. For instance, agreeing to act as a referee but then disappearing off the radar might warrant the most substantial ban. Delivering a meager report that’s extremely late, and without communicating with the relevant editor about the delay, might mean some shorter period of time on the bench. First-time offenders surely deserve different treatment to persistent re-offenders. And the embargo period will need to be substantial enough to be effective (too short and it will have no real impact; too long and it’s probably not practical due to the changes in the editorial team). The details can be ironed out.--Elizabeth Hannon "Peer Review or Perish: The Problem of Free Riders in Philosophy" @Dailynous

About once a year somebody proposes -- remember last Summer David Velleman's proposed moratorium on graduate submissions [recall my response] -- a radical new solution to the problems facing the way professional philosophers handle publication. As I suggest below, how to characterize the nature of the problem is itself part of the problem. [Trigger Alert: the key, scandalous part of this post is a few paragraphs down!] Let me grant to Hannon that there is a free rider problem: it's possible not all would be referees are chipping in equitably from a system they benefit from. For example, I know senior peers that only review for 'top four journals' [hereafter 'T4J'] or (more popular to admit) that take lengthy sabbaticals from reviewing. (No worries I won't rat you out.) Let's stipulate they free ride. They are not the only such free riders in the system: grant agencies and universities everywhere want to see (increased) academic 'productivity' and 'output' (that is refereed publications), but increasingly refuse to spend money on hosting journals or rewarding referee work by their staff.*

Before I get to the scandalous part of my post, let me note that Hannon's proposal will lead to a predictable consequence. As Thorsten Wilholt noted -- and he like me referees for lots of journals he has no hope/desire or expectation ever to be published in --, "the option of being able to submit to them in the future is of little value to me. I fear that the proposed system of disincentives, if widely adopted, would turn refereeing into a quid pro quo thing in the minds of most academics. As such, it would benefit some top journals...but work to the disadvantage of others....So people would reserve their refereeing time for journals that they are interested in submitting to." (That's already the de facto the practice of my friend who only referees for T4J!)

What's notable of Hannon's proposal is that it neither really looks at how other disciplines handle such problems nor shows any signs of reflecting on the institutional and social sources of the problem. Trigger Warning: for the dirty secret of philosophy is that we have insanely ridiculously low acceptance rates -- often well under 10% -- for papers. This low rate is only defensible if you think that publication in philosophy has the kind of inductive risk that any false positive leads to society's catastrophe. Nobody thinks that. (And I actually think philosophy is really important to society's flourishing!) This low acceptance rate is coupled with an insanely steep hierarchy (recall) pertaining to which journals are worth publishing in.

In the old days (not so long ago in professional philosophy), when our purportedly prestigious journals were run unprofessionally, as clubhouses serving an in-crowd, this low acceptance rate was not perceived as a real problem because insiders could get their PhD students and junior faculty friends published. But since we have  moved to more minimal professional standards (although few are as good as Ergo), the high rejection rate has made publication de facto a lottery where those who need publication most are at the mercy not just of the timeliness but also humors of the least appreciated element in the system, the anonymous referee. Because there is a veritable arms race in publication, this means that papers are now routinely submitted early and then improved through a dialectic with the very same anonymous referees (some of whom (recall) end up being the personal gatekeeper of a particular paper).

So, rather than punishing free riding referees (which creates its own collective action and procedural fairness problems), the more rational response [AND [later insert--thank you Justin Weinberg] I HAVE BEEN ANTICIPATED BY NEIL SINHABABU]  is to increase the acceptance rate in professional philosophy from the the top journals down to about, say, 30% or so. (Lots of fields have even higher acceptance rates.) Given, that these days most publication is really electronic publication, there is no reason to keep the number of articles and journal space so limited. My  proposal would eliminate a good chunk of the problem in the profession: a lot fewer papers would be shopped around (releasing valuable editorial and referee time); young scholars, who meet the minimal professional competence the discipline expects, would not be part of an unfair lottery anymore; and the sense of mystique and prestige surrounding publication in a top journal would disappear.  Multiple publication would still be a sign of (would-be-)productivity and professional competence, but journal publication would stop being the proxy for extremely fine-grained (and in my mind absurd) differentiation of judgments of quality. That's how it should be. Also, it will halt the excessive nature of the arm's race; with that higher acceptance rate people will be less impressed by any extra publication beyond what is taken to be the new normal.  

There are two obvious objections to  my proposal: first, it would devalue being published in T4J (and lower) to some degree. That's not a bug, but a feature of my proposal. Admittedly, in a transition period, there will be some confusion about the value of a T4J publication (not everybody reads blogs), but after some time people will adjust to slightly less bright halo effects (see Helen de Cruz on prestige bias in Ergo). Second, my proposal faces a collective action problem: the leading journals have to embrace the new norm. Luckily, it's a very small collective action problem because, given the steep prestige hierarchy, this only involves very few journals to establish a new norm.+ 

So, bottom line: before you blame and sanction referees [the cogs that keep the system flowing], why not change the enabling, background conditions first?  Okay, now let me get back to my referee-duties. 

*I am ignoring the for-profit presses because in professional philosophy they tend be less dominant than in other professions.

+Of course, one may suspect it goes against the self-interest of those that benefit most from controlling the steep prestige hierarchy to give up one of their key levers.

Comments

[Catarina Dutilh Novaes]

I hope this doesn't sound self-serving, but I believe Synthese is already implementing the suggestion. We have 12 issues per year, often going up to 400 pages per issue. Space is never a consideration regarding which papers to accept: we accept all of those that receive positive reports from referees. (We also send 90% or more of our papers to referees, and with well over 1.000 submissions per year, that's a lot of referee work!) I don't recall our exact acceptance rate at this point, but I believe it is about 20% (including special issue submissions).

[Elizabeth Hannon]

Thanks for giving this more coverage. We're keen get as wide a sample of opinion as possible, before we decide about whether to implement this proposal.

1. I’m not clear on how a higher acceptance rate will solve the problem of the free rider. If the thought is that with fewer papers circulating, referees will have fewer demands on their time and thus be better behaved when reviewing, then I’m not at all confident that this follows. It doesn’t appear that the worst offenders here are also the busiest, by any means. Busy people will often decline reviews, it's true; but flaking on reviews is a different matter. And we’re not proposing any penalty for simply declining invitations.

2. It's true that I didn’t discuss the ‘social and institutional sources of the problem’. A lot of the problems in academic publishing are outrageous, but they are also well known. What’s less well discussed are refereeing norms and how we might come to some agreement about what these ought to be; how we might minimize the harm done to authors (often junior colleagues, often referees) and avoid making the often invisible labour of editors (also colleagues, also referees) any more burdensome. Stick two fingers up at Springer by all means (other evil publishers are available) and perhaps being a flake will turn out to be the best way to fight all the problematic elements of the system. All the same, free riders harm the other cogs keeping the system going, and so perhaps alternatives ought to be considered first.

3. We are limited in how many papers we can accept by our page budget (though we frequently overshoot this limit). If we were to accept many more papers, we'd need to increase the number of pages that we’re allocated each year. To pay for the extra pages, we’d have to increase the journal’s price. We’re significantly cheaper than Synthese; indeed, the BJPS is one of the cheapest out there (and is free in some parts of the world). We’d like to keep it that way. And of course the internet solves nothing (evergreen): one third of our readers are print subscribers.

I chose the term ‘free rider’ because there are a group of people getting something for nothing out of academic publishing besides the profit-making publishers: bad referees get the benefit of the labour of their colleagues without returning the favour, and in doing so they frequently increase their colleagues' labour.

[Eric Schliesser]

Thank you for your engagement with your critics, Elizabeth.
1.The problem of free-rider is, in large part, the problem of the overstretched referee, who is part of a system where there is a lot of gate-keeping of prestige/journal space and free-riding by institutions who do not reward/acknowledge his work. That's why I call this a framing issue. There is huge amount of demand on our time from lots of directions, most of whom are in the dark about each other's demands (and then make claims about to what degree we are busy or not according to their lights).
2. I understand that you qua editor want to solve a very localized problem. Nothing stops you from implementing your solution. Go for it. I am not against more experimentation among philosophy journal editors. I wish there were more of it.
3. I think it is clear that philosophy journal editors are keeping access to their pages relatively low and so generating extra work for everybody else and making publication a weighted lottery. The only benefit I see from the status quo is that editors are primarily in the business to be able to confer prestige. But that element of my argument you refuse to address let alone acknowledge.
4. Look, if you extol the cheap price of BJPS, you should at least note the even cheaper price editors seem to put on the time of their referees. I really prefer not to look at these things in monetary fashion, but if we do let's take in the wider perspective.

[Keith Green]

Just one brief though: re the issues with refereeing from 'Synthese' to 'Hypatia': I wonder if part of the problems might be at least contained by allowing reviewers to see other reviews, but still only under the veil of anonymity, and only after a final and irreversible editorial decision has been made. This would allow reviewers to 'check' their own reviews by reference to others (without knowing who the reviewers are, one would still have to focus on the comparative merits of reviews). Even under the veil of anonymity, I think it would make reviewers more careful about several things, including the tone and language of their review, but not only this. In general, it would recognize and underscore the accountability of reviewers to authors and editors.

[Eric Schliesser]

Some journals do that already. I like it a lot.

[Matt]

" I wonder if part of the problems might be at least contained by allowing reviewers to see other reviews, but still only under the veil of anonymity, and only after a final and irreversible editorial decision has been made."

I've refereed for a fairly large number of journals (not all strictly in philosophy, but mostly in philosophy or closely related areas) and the only two that I've refereed for that did this automatically were the Journal of Politics and Legal Theory. In both cases, it was done as suggested and, like Eric, I liked it, finding it very useful. I'd be quite happy if it became more common.