What do Hoax papers really show? (Ten theses)

(1) The refereeing institutions of philosophy are vulnerable to fraud. This is something I have  been blogging about since September 23, 2010; in one my first dedicated post for NewAPPS I wrote:

The world of Renaissance and Early Modern philosophy and scholarship has been shaken by the "Martin Stone case".... Before his fall from grace, Martin Stone was a widely admired and highly respected professor at Leuven University....I want to call attention to the rather lukewarm, if not indifferent, reaction of the editors and journals/presses that have published his work....What does not seem to have happened is a general discussion about referee-ing. It is well known that one can play the referee-process in various ways that increases chances for favorable review....The Martin Stone case suggests that the referee process is broken in rather serious fashion. Stone published in leading specialist journals and also with leading presses. It means that lots of people missed a considerable amount of plagiarism. What is even more striking is that this happened in a small-ish sub-field where folk pride themselves on learning, their languages, and scholarship.... Anyway, peer-review is not working, and we need to have an open and honest conversation about this.

Since then, by reading retractionwatch religiously for a few years, I learned that other disciplines also can't prevent serious academic fraud (google "diederik stapel") and that the reporting of it used to be seriously understated (hopefully that has changed).

Catching fraud is not the only function of referee process (and not the only known problem with the process), and since, I blogged about a number of proposals to improve philosophical refereeing (none has had any uptake). When the Tuvel/Hypatia affair broke, I wrote critically, "we are all supposed to take on trust that the review process of Tuvel's article was dandy--to best of my knowledge there is no public evidence." Echoing work by Katzav/Vaessens, and responding to the Tuvel/Hypatia affair, I also offered a proposal on how to structure the referee process for papers that may "touch a powerful nerve in scholarly disputes and be taken up in the public culture, public policy, and (alas) the culture wars." 

Of course, (2) what we don't know is how many attempted frauds get caught by existing referee practices. The latest hoax, which presents itself as a study, is not properly designed to contribute to learning that (except for telling us that some journals and fields do detect some fraud; HT Dailynous; see the discussion there.) It weirdly assumes there is no "problem with peer review itself ." Plagiarism software is increasingly normal at academic journals, but the unwillingness to change dramatically existing practices by philosophy's community of editors suggests that while publication is a big deal for individual academics (since it helps determine their career trajectory), (3) the risks associated with having bad work published are thought not worth the bother/cost to do anything about. And that's, alas, because most published papers have negligible impact on much other work and the field. 

I had come to the conclusion that papers have no impact because I had argued in one of my papers, perhaps too en passant, that it was way too easy to produce results in all kinds of social sciences. Writing journal articles about that struck me as not the right way to call people's attention to the  problem. To be sure, I do not claim to have foreseen the extent of the replication crisis in other fields, But alerted by two methodologically very astute economists, Deirdre McCloskey and David Levy, I started blogging about the problems with statistical significance in November (also or 20101 (see here)). That is, at some point the costs associated with bad publication practices may induce changes. But that point is reached when folk feel they can't rely on the best work being done not through hoaxing the system.

An impatient reader may well feel that I am missing the point today. The latest frauds were perpetrated for purportedly high minded reasons including combating urgent problems. For example, the first reason given is that some "scholars increasingly bully students, administrators, and other departments into adhering to their worldview."* Let's stipulate this problem exists. But publications don't bully. In fact, if publication ended altogether, the bullying would almost certainly still exist. So, here the means (fraud) are almost comically out of sync with the purported aims.

The second reason given is that this scholarship has a worldview that is "is not scientific." We are never told where we can do experiments on a scientific worldview, or what its identity conditions may be. 'Worldview' is itself not an especially scientific concept. It has roots in my people (Montesquieu, Hume, and Herder), but it was made popular by folk who had read Hegel and wished to turn it into (ahh) a science (thankfully Wikipedia agrees with me) and then at a later point it became central to philosophy of science (Kuhn, Goodman) via another Hegelian, Koyré. This allowed science to be turned into an object of inquiry. By which time it was ripe to be used by those who wish to raise critical questions about the role of science today. Hint: if you wish to defend the sciences against constructivists, don't appeal to constructivist-friendly notions.+ In fact, this hoax provides evidence for those that treat the sciences as an all-too-human, fallible process in which non-epistemic preferences can play a non-trivial role along the way. So, (4) taken at face value the hoax is self-undermining.** 

This is related to a further problem, the perpetrators of this hoax claim that "open, good-faith conversation around topics of identity such as gender, race, and sexuality (and the scholarship that works with them) is nearly impossible, our aim has been to reboot these conversations."  Let's stipulate this is all true. If that's the problem wouldn't the proper response be to create places and institutions that (ahh) facilitate "open, good-faith conversation around topics of identity such as gender, race, and sexuality (and the scholarship that works with them)." (5) Open and good-faith conversations among folk that disagree are not easy at all. It is, in fact, constitutive of civilization. So, it would be a genuine achievement to contribute to that goal. But it is peculiar to think that such a good-faith conversation would start well by way of disparagement and (ahh) academic fraud. Open and good-faith discussions need to be nurtured, and require delicacy, they require tact, and a willingness to recognize other viewpoints.

If you don't like the scholarship that works with gender, race, and sexuality, why not engage with the very best of it and improve the field? If you think the very best of it is not worth engaging with, start writing informed methodological critiques. If you prefer to ignore all previous existing scholarship, start your own journal and write your own seminal contributions. Of course, one may think the very topic should be ignored. By calling the intended target of the hoax, 'grievance studies,' one may come away thinking the subject is worthless. But prima facie there is nothing wrong with their 'common goal of problematizing aspects of culture in minute detail in order to attempt diagnoses of power imbalances and oppression rooted in identity.' (6) The authors give not a single reason, let alone an argument, for thinking there is anything wrong with such a goal.

Now preventing a noble goal from "corrupting scholarship," is not easy.  (Just examine medical research and its incentive structure.)What would be interesting to learn is given the nobility of the goals, what kind of institutional/incentive structure would facilitate scholarship toward the worthy goal. But the method adopted --  they call it 'reflexive ethnography' -- is ill-suited to that goal. It's not really ethnography because these days the research ethics of real ethnography of the reflexive kind would prevent hoaxing. (7) At most the adopted method can call attention to the existence of flaws in the referee process, but that's old news (no need to cite me, lots of others who have made the same point.) The idea that "peer-reviewed journals are the absolute gold standard of knowledge production" is a strange claim. They are a means to communicate that knowledge and to facilitate community discussion about it. These day this standard primarily exists, I fear, to facilitate job decisions.

At one point the authors of the hoax explain that "to make absurd theses acceptable was central to the project." The problem is (8) being permissive to making absurd theses acceptable is also a means of how scholarship can advance. So, this is not a bug, but, as Liam Kofi Bright suggests, a feature of scholarship. Undermining this principle generates (even more) status quo bias. The authors of the hoax claim that they are defending 'liberalism, progress, modernity, open inquiry, and social justice.' I suspect, but can't prove that (9) Undermining receptivity toward entertaining absurd theses is a good way to undermine 'progress, modernity, open inquiry, and social justice.' 

 The real underlying method of these hoaxes is explicitly acknowledged. It's 'sophism,' which was adopted in order to fit in with what were taken to be other sophists. with the field and then sought to bend the existing scholarship to support it. There is ingenuity and humor in these hoaxes--all the ingredients of effective satire. But the problem is, the hoaxes are not satire (see here how it should be done). By contrast, (10) if you adopt the methods of the sophists to combat the sophists, you contribute to a culture of sophistry and you become a sophist. To avoid that fate, Socrates was willing to die.   

 

(1) The refereeing institutions of philosophy are vulnerable to fraud. This is something I have  been blogging about since September 23, 2010; in one my first dedicated post for NewAPPS I wrote:

The world of Renaissance and Early Modern philosophy and scholarship has been shaken by the "Martin Stone case".... Before his fall from grace, Martin Stone was a widely admired and highly respected professor at Leuven University....I want to call attention to the rather lukewarm, if not indifferent, reaction of the editors and journals/presses that have published his work....What does not seem to have happened is a general discussion about referee-ing. It is well known that one can play the referee-process in various ways that increases chances for favorable review....The Martin Stone case suggests that the referee process is broken in rather serious fashion. Stone published in leading specialist journals and also with leading presses. It means that lots of people missed a considerable amount of plagiarism. What is even more striking is that this happened in a small-ish sub-field where folk pride themselves on learning, their languages, and scholarship.... Anyway, peer-review is not working, and we need to have an open and honest conversation about this.

Since then, by reading retractionwatch religiously for a few years, I learned that other disciplines also can't prevent serious academic fraud (google "diederik stapel") and that the reporting of it used to be seriously understated (hopefully that has changed).

Catching fraud is not the only function of referee process (and not the only known problem with the process), and since, I blogged about a number of proposals to improve philosophical refereeing (none has had any uptake). When the Tuvel/Hypatia affair broke, I wrote critically, "we are all supposed to take on trust that the review process of Tuvel's article was dandy--to best of my knowledge there is no public evidence." Echoing work by Katzav/Vaessens, and responding to the Tuvel/Hypatia affair, I also offered a proposal on how to structure the referee process for papers that may "touch a powerful nerve in scholarly disputes and be taken up in the public culture, public policy, and (alas) the culture wars." 

Of course, (2) what we don't know is how many attempted frauds get caught by existing referee practices. The latest hoax, which presents itself as a study, is not properly designed to contribute to learning that (except for telling us that some journals and fields do detect some fraud; HT Dailynous; see the discussion there.) It weirdly assumes there is no "problem with peer review itself ." Plagiarism software is increasingly normal at academic journals, but the unwillingness to change dramatically existing practices by philosophy's community of editors suggests that while publication is a big deal for individual academics (since it helps determine their career trajectory), (3) the risks associated with having bad work published are thought not worth the bother/cost to do anything about. And that's, alas, because most published papers have negligible impact on much other work and the field. 

I had come to the conclusion that papers have no impact because I had argued in one of my papers, perhaps too en passant, that it was way too easy to produce results in all kinds of social sciences. Writing journal articles about that struck me as not the right way to call people's attention to the  problem. To be sure, I do not claim to have foreseen the extent of the replication crisis in other fields, But alerted by two methodologically very astute economists, Deirdre McCloskey and David Levy, I started blogging about the problems with statistical significance in November (also or 20101 (see here)). That is, at some point the costs associated with bad publication practices may induce changes. But that point is reached when folk feel they can't rely on the best work being done not through hoaxing the system.

An impatient reader may well feel that I am missing the point today. The latest frauds were perpetrated for purportedly high minded reasons including combating urgent problems. For example, the first reason given is that some "scholars increasingly bully students, administrators, and other departments into adhering to their worldview."* Let's stipulate this problem exists. But publications don't bully. In fact, if publication ended altogether, the bullying would almost certainly still exist. So, here the means (fraud) are almost comically out of sync with the purported aims.

The second reason given is that this scholarship has a worldview that is "is not scientific." We are never told where we can do experiments on a scientific worldview, or what its identity conditions may be. 'Worldview' is itself not an especially scientific concept. It has roots in my people (Montesquieu, Hume, and Herder), but it was made popular by folk who had read Hegel and wished to turn it into (ahh) a science (thankfully Wikipedia agrees with me) and then at a later point it became central to philosophy of science (Kuhn, Goodman) via another Hegelian, Koyré. This allowed science to be turned into an object of inquiry. By which time it was ripe to be used by those who wish to raise critical questions about the role of science today. Hint: if you wish to defend the sciences against constructivists, don't appeal to constructivist-friendly notions.+ In fact, this hoax provides evidence for those that treat the sciences as an all-too-human, fallible process in which non-epistemic preferences can play a non-trivial role along the way. So, (4) taken at face value the hoax is self-undermining.** 

This is related to a further problem, the perpetrators of this hoax claim that "open, good-faith conversation around topics of identity such as gender, race, and sexuality (and the scholarship that works with them) is nearly impossible, our aim has been to reboot these conversations."  Let's stipulate this is all true. If that's the problem wouldn't the proper response be to create places and institutions that (ahh) facilitate "open, good-faith conversation around topics of identity such as gender, race, and sexuality (and the scholarship that works with them)." (5) Open and good-faith conversations among folk that disagree are not easy at all. It is, in fact, constitutive of civilization. So, it would be a genuine achievement to contribute to that goal. But it is peculiar to think that such a good-faith conversation would start well by way of disparagement and (ahh) academic fraud. Open and good-faith discussions need to be nurtured, and require delicacy, they require tact, and a willingness to recognize other viewpoints.

If you don't like the scholarship that works with gender, race, and sexuality, why not engage with the very best of it and improve the field? If you think the very best of it is not worth engaging with, start writing informed methodological critiques. If you prefer to ignore all previous existing scholarship, start your own journal and write your own seminal contributions. Of course, one may think the very topic should be ignored. By calling the intended target of the hoax, 'grievance studies,' one may come away thinking the subject is worthless. But prima facie there is nothing wrong with their 'common goal of problematizing aspects of culture in minute detail in order to attempt diagnoses of power imbalances and oppression rooted in identity.' (6) The authors give not a single reason, let alone an argument, for thinking there is anything wrong with such a goal.

Now preventing a noble goal from "corrupting scholarship," is not easy.  (Just examine medical research and its incentive structure.)What would be interesting to learn is given the nobility of the goals, what kind of institutional/incentive structure would facilitate scholarship toward the worthy goal. But the method adopted --  they call it 'reflexive ethnography' -- is ill-suited to that goal. It's not really ethnography because these days the research ethics of real ethnography of the reflexive kind would prevent hoaxing. (7) At most the adopted method can call attention to the existence of flaws in the referee process, but that's old news (no need to cite me, lots of others who have made the same point.) The idea that "peer-reviewed journals are the absolute gold standard of knowledge production" is a strange claim. They are a means to communicate that knowledge and to facilitate community discussion about it. These day this standard primarily exists, I fear, to facilitate job decisions.

At one point the authors of the hoax explain that "to make absurd theses acceptable was central to the project." The problem is (8) being permissive to making absurd theses acceptable is also a means of how scholarship can advance. So, this is not a bug, but, as Liam Kofi Bright suggests, a feature of scholarship. Undermining this principle generates (even more) status quo bias. The authors of the hoax claim that they are defending 'liberalism, progress, modernity, open inquiry, and social justice.' I suspect, but can't prove that (9) Undermining receptivity toward entertaining absurd theses is a good way to undermine 'progress, modernity, open inquiry, and social justice.' 

 The real underlying method of these hoaxes is explicitly acknowledged. It's 'sophism,' which was adopted in order to fit in with what were taken to be other sophists. with the field and then sought to bend the existing scholarship to support it. There is ingenuity and humor in these hoaxes--all the ingredients of effective satire. But the problem is, the hoaxes are not satire (see here how it should be done). By contrast, (10) if you adopt the methods of the sophists to combat the sophists, you contribute to a culture of sophistry and you become a sophist. To avoid that fate, Socrates was willing to die.   

 

 

*No evidence is given for this claim, let alone gold-standard social scientific evidence.

+Yes I know that in Germany the humanities are thought a science, and that plenty of humanities have had a non-trivial impact on the sciences or even use all kinds of scientific methods, 

 

Comments

[D. Des Chene]

As an exercise in statistical reasoning, I don’t think it proves much. On the other hand, as propaganda given a scientific sheen it will no doubt work well enough.

The sample size is very small (no doubt because it’s genuinely laborious to write even fake papers). Moreover, one crucial piece of information is missing: the background acceptance rates of the journals in question. If the Journal of Disingenuous Scholars accepts everything that comes over the transom, then it’s not surprising that one of the “grievance studies” papers was published.

To show in a statistically responsible manner that pandering has a significant effect it is not sufficient to show that papers that pander get accepted. You have to show that the acceptance rate is significantly higher than the rate one would expect given background acceptance rates. Otherwise all you have provided evidence for is lax reviewing.

One would like to know that basis of the following claim: “We improved this ratio [of reaching the peer-review stage] from 0% at first to 94.4% after a few months of experimenting with much more hoaxish papers”. That last percentage is approximately equivalent to getting 19 of every 20 papers accepted. How you get that number from a sample well under twenty is a mystery.

Furthermore: notice that the acceptance rate at the outset was 0%. Presumably the first papers they wrote pandered as much as the later papers. Only the obviousness of their being hoaxes changed. Isn’t it possible that the increase in success rate owes as much to “academic virtue signalling” as to pandering? After all, that may well have been the only quality that varied through the trials. Social constructionsm would be irrelevant.

One might suppose that the papers are such that in a best of all worlds at most one paper would have been accepted; that seven (or 10, as they estimate) were accepted is evidence that we do not live in the best world. If, on the other hand, the noise level is high enough, then again it has not been proved that pandering has an effect, but only that the reviewing process isn’t filtering out enough noise.