It is our position that the harms that have ensued from the publication of this article could and should have been prevented by a more effective review process. We are deeply troubled by this and are taking this opportunity to seriously reconsider our review policies and practices. While nothing can change the fact that the article was published, we are dedicated to doing what we can to make things right. Clearly, the article should not have been published, and we believe that the fault for this lies in the review process. In addition to the harms listed above imposed upon trans people and people of color, publishing the article risked exposing its author to heated critique that was both predictable and justifiable. A better review process would have both anticipated the criticisms that quickly followed the publication, and required that revisions be made to improve the argument in light of those criticisms.... We are a scholarly journal committed to an anonymous peer review process. We want readers to feel free to offer their honest feedback on manuscripts submitted to Hypatia. Anonymous peer review is important for the scholarly reputation of Hypatia; mistakes in particular instances should not compromise the commitment to anonymous peer review in scholarship.--"To our friends and colleagues in feminist philosophy," Hypatia’s Board of Associate Editors [HT Ásta Kristjana Sveinsdóttir]
The details of the review process at Hypatia are described in the remainder of Associate Editors' statement. These do not live up to the discipline's very best practices (established recently) at Ergo, but -- if impartiality is our aim -- they are much better than how many purportedly 'top' (clubby) philosophy journals were being edited in recent memory (and some still are edited).
Even so, one need not be a student of psycho-analysis to note the anxious manner in which scholarly and anonymous peer review are being coupled in the statement. This confuses the process with the product. (In fact, there is a more general tendency among editors to farm out their judgment, procedurally, to overworked and under-appreciated referees, including the ridiculous practice of giving even transparently incompetent or biased referees de facto veto-power over the publication of a paper.) Anonymous peer review may be a means to being scholarly and other important ends, it is not itself an end (even if various journal ranking agencies, whose metrics subtly structure the incentives of so much of professional academic life, treat it as such). But in what follows, I accept, for the sake of argument, that anonymous peer review is a condition on being thought scholarly these days.
As the Hypatia debacle is unfolding, Neil Levy notes (on Facebook) that risk averse editors of many journals (an overworked and unappreciated lot) will be less likely to accept papers on trans issues and other issues that potentially attract controversy: if even Hypatia got it wrong, what chance do I have? Such editors will worry that they don't have the expertise to choose reviewers, let alone assess the papers themselves, and rather than risk time consuming and reputation shattering controversy will simply desk reject.+ Levy's observation is important because it reminds us that self-censorship in light of would be social or political controversy is always a live option in philosophy (I routinely discuss this as the Socratic Problem in philosophy).
Now, one response to Levy's worry is specialization and ever more fine-grained scholarly niche construction. Potentially controversial papers would only land at extremely specialized journals with the expertise to handle them. Now, while I think philosophy would have higher utility if a whole bunch of work on infanticide had been marginalized, such a strategy (hyper-specialization) would be undesirable; such specialization would reinforce marginalization of morally and politically important topics. It would also be self-defeating because it would push talent away into other disciplines and practices.
But there is another way to think about specialization here. Rather than specialization being driven by topic, it could be driven by expected fallout* (<-- sorry that sounded too much like an atom bomb) potential, public controversy. That is, some journals can foresee that the papers they publish have a higher than mere chance probability to touch a powerful nerve in scholarly disputes and be taken up in the public culture, public policy, and (alas) the culture wars. Hypatia is, in fact, some such a journal. It is pretty clear that its current review process is ill-suited to Hypatia's participation in would-be-public philosophy. (On what public philosophy might be, recall here.) No amount of tinkering will prevent future debacles. It does not follow it should become more risk averse nor does it follow it needs to give up anonymous review.**
For one can make the anonymous review process public and combine it with crowd-review. What I have in mind is not uncommon in journals devoted (not surprisingly) to climate science. (What follows is inspired by the practice of Earth Science Dynamics; take a look at the flow chart of its review process here. [HT Charlotte Werndl; Joel Katzav))*** Here's how it works. A paper is handled as it always is handled and sent out to referees. These then submit their honest feedback. After minor review by the editor (to check that the report is not abusive, does not remove confidentiality, etc.) the paper and the anonymous report are put up on the journal's website, where the author can respond to reviewer's objections, even re-submit the paper (with new reports, etc.), and where crowd-reviewers can participate on the paper (non-anonymously [in order to prevent the worst excesses familiar from the internet]). The exact details do not matter here. There are lots of ways to mix the process and many are pretty standard in various sciences. Luckily for Hypatia, it can draw on the expertise of a thriving and innovative community of feminist philosophers of science (some of whom sit on its various boards), many of whom have, in fact, scrutinized peer review practices ([see also this classic empirical paper by Wenneras &Wold] and recent work by Lee et al.)
As an aside, philosophy is a remarkably conservative and clubby professional discipline. (How clubby and boy-centric has long been known [recall here and here], but it has been now amply demonstrated in the Healy data and Weatherson's recent analysis.) This is especially problematic as it attempts to tackle issues that impinge on the lived experience(s) of just a few of its (ahh) gatekeepers.
What matters is that some such mixed referee process can (i) maintain anonymity of the main peer referees (and, thus, keep being thought scholarly), while simultaneously (ii) draw on a wider community of would be experts even interested/informed lay-people and (iii) alert the editors, authors, and referees to potential controversies that (this is important) one would wish to prevent. (The point is not to avoid controversy, but rather to be thoughtful and wise about them.) It also has the nice feature of (iv) creating interest in some of a journal's publications. And, it puts the responsibility of publication back where it belongs: the editor and her team. Of course, such a would-be-technocratic solution does not end political controversy. But it would internalize political considerations in a transparent and fruitful way in the review process and it would address the systematic nature of the problems (for useful comments of their systematicity see thus post @Feministphilosophers). Of course, such mixed review practices were not practical (time, cost, etc.) in the pre-internet days. But that's no excuse now.
+I should say that this post was prompted by a discussion with Lewis Michael Powell about journal practices generally. I hope he forgives me for stealing his thunder. But I trust he allows me to say that at any given time, there is plenty of thunder left in LMP.
*No, I am not making fun of Heidegger's fourfold!
**My loyal readers know that I have long advocated a switch to the co-publication of referee reports with accepted papers. (Damn, I have been writing about this stuff for a long time now!)
***UPDATE: An earlier version of this post unfairly failed to acknowledge Katzav. I had, in fact, read (and commented on) an as of yet unpublished work by Katzav and Vaesen, 'Pluralism and peer review in philosophy,’ that offered a proposal along the lines suggested in this post.
I think we need experiments in peer review - not the replacement of what we have with another system, but a diversity of different systems. The one you suggest might indeed be an appropriate one for journals that expect to routinely handle difficult and sensitive topics. If Tuvel's article had gone through an open review process, my sense is that it would have been published in an improved form. This kind of process would address the need for expertise on the part of editors: there would be a check on them getting it wrong.
I do want to speak up for aspects of the traditional system. For most topics and most journals, it would work fine if people did their fair share of reviewing and did it in a timely manner. Sending a paper out should be a throw in a multi-round game. If the paper is good, then the chances are it will be accepted somewhere decent eventually. The problem with the current system is "eventually" is often a long time away. If reports were timely, most of the problems would be eliminated.
I also want to speak up for hiding behind referees. I *want* to be able to say "my hands are tied; two referees recommended rejection". Of course, they are not literally tied: as an editor I can and very occasionally do override reports. But I have a policy of not doing so. The reason is obvious: papers are submitted by people I know (triple blind reviewing is preferable, but neither of the journals I have significant editorial responsibilities have instituted it, and in any case I can often identify the author from internal evidence). Authors sometimes pressure the editor: I get regular emails along the lines of "Hey Neil, it was great to meet up at X. We should have a drink soon...oh, by the way I just submitted a paper to your journal". I want the shield of deferring to referees, and I want to be able to limit the effects of my own liking or disliking for individuals. As for "ransparently incompetent or biased referees", of course they exist. But are they more common than incompetent or biased editors? Using referees in a multi-round game is better I think than having editors make the decisions. Moreover, when people share their stories of biased or incompetent referees, I sometimes find myself thinking that the bias or incompetence is not on the referee's side at all.
Posted by: Neil | 05/04/2017 at 04:50 AM