If the system is broken blame the referees

While reviewing papers might seem thankless, please know that your service to the profession does not go entirely unnoticed. For many journals, there are some aspects of the process that are automated—such as reminders about deadlines—but others go directly through the editors, such as requests to review, recommendations to reject and accept, reports, and so on. This means that there is another person on the end of all of those e-mails you’re getting who is grateful for your careful, thoughtful work. Moreover, data is often retained in the online system, showing an individual’s history of reviewing: how many requests have been sent, how many have been accepted, the average number of days for a review, and so on. I am always struck by how some members of our profession are true heroes in these respects.--Jennifer Lackey, "A Journal Editor’s Wish List"@APA Blog [emphasis in original]

As I write this post, I am overdue on two referee requests [mea culpa].  During the next four weeks, I am supposed to read three PhD dissertations (one of which I have read before) as an external referee. Since January 1, 2015, I have reviewed three more PhDs, one tenure dossier (that's below my average), I have acted as an external referee to at least half a dozen big grant proposals, served on a grant award committee, refereed about 20 journal articles, refereed two book manuscripts, and refereed countless paper abstracts for a number of conferences. I am pretty sure my numbers understate because I have a tendency to forget to keep track of this on my CV. It's not that I am unaware that editors have trouble finding referees: as I write this I am co-editing two special issues for a journal, and a forty chapter Handbook. (I have edited several more collections during the last few years.) I try to decline requests when I can and offer alternative names, but some editors are very persistent, or despairing. I am not especially proud about my service to the profession, and it certainly does not feel heroic. (Well since I am often late-ish with my reports, I probably don't count as a hero.)

I have to admit that, say, a decade ago qua referee, (a) I enjoyed, for a while, the role of assistant-gate-keeper. I enjoyed (b) learning what folk in the discipline were discussing in my area of specialization before it was published. And, (c) I especially found it important to call attention to overlooked, relevant work by people in the field (no, not necessarily me). Now, I see refereeing as drudgery (no doubt this serves me right for being an Adam Smith expert), and I am now seriously entertaining the idea of a  year-long moratorium on refereeing. The opportunity costs to refereeing are fairly large, and the benefits low.

Jennifer Lackey's post is all about what prospective referees can do to smooth the process. It's good that the APA blog generates discussion about this, and all of her suggestions are sensible. But despite the fact that I am in awe of her service to the profession, the piece annoyed me; it's a bit puzzling that there is no mention of what journals and the profession can do to improve the situation. For example,

[A] I have often wondered why journals don't give a free subscription to the timely, journals' referees (if cash is out of the question).

[B] A few journals have started to share their decision (and even other referees's reports) with a paper's referees--I found that very inspiring--that should be common practice by now.

[C] Why not increase the publication rates in journals? Philosophy has insanely low publication rates. This is odd now that we have moved to electronic publishing. With higher publication rates, early career folk don't need to keep shopping around the same (very fine) papers to numerous journals. I can see why the journal status quo benefits some (senior) insiders and those  junior folk whose senior buddies have captured journals, but honestly, a good chunk of our problems would disappear if top journals didn't reject 90% of the material.

In addition, some journal editors are eliding their responsibilities, too. I have come to suspect that a non-trivial-bunch of editors don't read the papers they send out to referees (and I have also wondered how many of them actually read what they publish). [If you are an editor who reads the papers before you send them out and then again before you publish them, swell, and thank you so much!] I say this because I rarely get an editor say, 'yes, ignore that idiot referee.' (Rather, I increasingly have editors tell me that they have a rule-based-policy of accepting only after two or three positive reviews.) I understand that reading papers would be extremely time consuming, and that editors' research times would be eaten up altogether, but...ahhh...why farm this out  to overworked referees who get no credit. 

In what follows I focus only on routine editing. (I have been in the profession long enough to have seen some really crazy editing shit--with editors enabling plagiarizing and insulting referees.) Anyway, in the spirit of charity, here are some proxies for papers that journals can return to the author before sending it to a referee:

• (nearly) all the citations are to high status men.

This is always evidence for a less than thorough literature search and lack of generosity to existing work in the field (not to mention that it reinforces gender biases). No, seriously, journal editors are farming even minimal quality control out to referees.

• more than half the citations are 'omitted due to blind review.'

I love progressive research programs, too, but...ah....if the author is more than 50% of the field then maybe the field is less than exciting.

• is the thesis clear after the first page.

Yes, I also love mystery novels, and I recognize that there are disciplinary differences. (I read a lot of political theory these days, and it's clear they love a 'see it's relevant signal' at the start of the paper.) But a lot of journal articles that are culled from dissertation chapters suffer from too much bloat and a kind of desire to cover all bases. Seriously, I spend a good bit of time trying to figure out what's at stake in a paper.

• A revise & resubmit that makes cosmetic changes in response to my report.

Lackey writes: "When suggesting a revise and resubmit, it is often ideal to also agree to review a revised and resubmitted version of the paper." Sure, by why can't editors not compare the report with the revised paper (with say track changes), and check the nature of the revisions? If it's cosmetic why bother me. And, also, if I am reading a paper for a third time (which is often the case with a R&R), then why be surprised I find new troubles in it?

• If a referee accepts an assignment and says, well, 'I will do it between-so-and-so' or 'please nudge me around so-and-so'

Then the automatic software nudge generator ignores 'so-and-so' altogether, and starts sending you an insane number of emails before so-and-so.

Let me repeat, I am grateful to the hard and often thankless work by editors. But they get the pleasure to see their names on the masthead, often get other nice goodies (teaching release, secretarial support, in some cases serious cash), and they can shape the field in non-trivial ways (if they wish). At the moment, referees get negligible credit yet, oddly enough, the responsibility for improving the system is on our overworked shoulders?

Comments

[Muhammad Velji]

I still don't quite understand the argument against paying referees. Of course for online journals and open access journals this would be prohibitive. I get that. One could even say the labour that goes into reviewing could be thought of as contributing to the open access, but let's put that to one side.

With the obscene profits these companies make, the LEAST they could do is pay for labour. It feels inherently wrong that this should be volunteer work. I've never been considered a contract theorist but at least with money involved maybe the referee will feel obligated to finish on time and give a reasonable amount of feedback, rather than put it on the absolute back burner. It's almost irrational NOT to put refereeing on the back burner when it's a voluntary activity, because I am being compensated to do the other work I agreed to do.

I'm really trying hard to be charitable toward the argument against paying referees but I am having a hard time linking conflict of interests with money here, especially with the new triple blind system going on these days.