I have received no email invitation to complete a PGR survey. My (not modest) vanity is a bit hurt -- 'surely, I should qualify as a rater on 17th and 18th century philosophy as well as philosophy of social science by now? I am fully at home in the PGR ecology; yet because I work in a small far away place, I would be an impartial spectator!' --, but I am also relieved not to have to face the decision to complete the survey.
Despite all the non-trivial social and intellectual harms that the PGR generates, the discipline needs a well-functioning PGR (and some competitors) given the institutional metrics-focused environments we inhabit (recall my defense).* So, I would welcome contributing to a social good. Even so, there are also compelling reasons to withhold one's services to the current PGR, which almost certainly inherits all the known serious methodological flaws and biases from previous generations (recall this post).* I am hopeful that the new co-editor, Brit Brogaard, who is also an accomplished practicing scientist, will address these in the design of future editions. While the current (2011 survey) PGR is getting dated, a further delay to fix some of the most glaring problems would have been the more optimal approach.
Finally, Ichikawa notes that "since the reason his [Brian Leiter] control [of PGR] was relevant in the first place was his social power as a result of the general perception of his control, and the current un-clarity will allow that perception to continue, I don't consider it consistent with the September Statement to participate." Oddly enough, perhaps, I have always been un-bothered by the ways in which Brian Leiter leveraged the perceived linkage between PGR and Leiterreports as a means of (perceived) social power; this 'social power' always rested in large part on Leiter being able to articulate a more widely held (if inchoate) common sense. That this common sense is animated by social justice and adherence to high intellectual standards as well as often by an adversarial and unreceptive to others conception of philosophy (which is also implicated in non-trivial patterns of exclusion) is no surprise to those of us within the ecology (there is a very nice analysis by Bharath Vallabha of the "double talk regarding Leiter Reports"). All professional worlds have some socially powerful. This is not to deny that one could wish for more gentle rule in the future and, especially, the encouragement of the development of multiple measures/rankings of quality and employability.
Rather, the problem goes in the other direction; the design of the PGR serves not just the general interests of the PGR ecology by providing it a measure that serves its professional autonomy (at the partial expense, of course, of certain outsiders), but it also serves the particular scholarly and disciplinary interests of its originator and owner in violation of the ordinary canons of impartiality (incentives matter, after all, also to professional philosophers). Perhaps, such an appearance of a conflict of interest is unavoidable as long as the editor of PGR comes from within the PGR ecology (no, I am not volunteering, funny one!), but filling out the survey makes one complicit in it.***
So, what would I do if I had been invited?
I would procrastinate and wait to see if anybody's argument could sway me in one direction or another,**** and then probably complete the survey under the motto a 'terribly flawed PGR is better than no PGR, maybe.'
UPDATE: I find Eric Schwitzgebel's analysis in terms of an "exercise in the sociology of philosophy," compelling.
* In brief: my defense does not rely on the purported benefits to prospective students, but rather on the way decision-making works in modern universities.
**At NewAPPS, Jonathan Ichikawa also cites a methodological concern in his open letter declining to complete the survey.
***Note while Brian Leiter sometimes makes fun of me (recall here and here), I have always been eager to stress my sense of his personal and intellectual integrity (again recall). So, my criticism here of Leiter's role as editor-owner of PGR is a functional not a personal one.
****I have not found decisive the moral arguments against the PGR as such offered by, for example, Mitchell Aboulafia in a series of posts. Others may see it his way.
Eric, I am really surprised to find that you thought Eric Schwitzgebel's post compelling. I found it a weak defense of the PGR. Have you checked out the comments section? Many criticisms. I supplied a rather detailed one. He completely avoided answering my criticisms in his response to me, and they were pretty serious in terms of claims that have been made for the PGR.
Regarding your support for the PGR in terms of decision making in modern institutions, I'm going to address this shortly on UP@NIGHT, but let me say here, I think you are seriously underestimating how much damage a flawed rankings system can do. If departments in the past were able to use the PGR in some helpful way, and I say IF, that game is coming to an end. The
discussion in philosophy about the PGR is now being viewed by people in other disciplines. They will
know that we have seriously flawed rankings, and believe me, they aren't going to allow philosophy
departments to get more resources based on them. If you think I'm mistaken about this, just start googling the PGR with other disciplines. The process has begun. (Political science people are
actually making fun of us in one series of comments.) And it will not end when the rankings come out, because those of us who know how flawed they are will continue to make the case. The best thing
that could happen now is for the PGR evaluators to back off, so we don't put out a seriously defective "product" that will embarrass us. (Btw, just to be clear, my criticisms of the PGR have not only been ethical. I have posts up about other problems.)
Posted by: Mitchell Aboulafia | 10/30/2014 at 06:14 PM
It's certainly possible that because PGR is flawed (and Schwitzgebel and I are both fairly clear about what we take these to be) it may be viewed with suspicion by outsiders in an institutional context. But ALL the metrics I am familiar with are flawed and have perverse effects, so it's not obvious that PGR (which certainly can be improved w/o losing its existing essential characteristics) is really all that much worse than ANY other metrics. Of course, if the Meme you endorse gathers steam in institutional contexts, then that would spell the end of PGR fairly quickly.
Posted by: Eric Schliesser | 10/30/2014 at 07:20 PM