On Participating in the PGR

Not long after I published this post, I was invited to participate in the PGR survey (funny how that goes). Despite the incisive, ongoing criticism, especially by Mitchell Aboulafia, Jessica Wilson, and John Protevi, I signed up as an evaluator in two of my areas of specialization (early modern and philosophy of social science). [To be clear: I recognize the truth of many of their criticisms some of which I share, but I also feel they never managed a fully adequate response to my own and Schwitzgebel's defense. Either way, I hope rankings will proliferate in the future.] Despite pre-existing misgivings, I also participated in the general evaluation. Because I was traveling and presenting, I ended up doing much of the survey in between other commitments. Here follow some first-personal reflections.

With hindsight I don't regret participating in the 17th and 18th century philosophy surveys. I host and attend a lot of workshops in the area, referee and review a lot, and still try to keep up with scholarship. Undoubtedly I did not immediately recognize some names that work in various sub-fields (e.g., Leibniz, where I don't try to keep up), or that I missed scholarship by people that themselves don't travel much, but often I could 'spot' the 'early modernist' easily in each list of names. I'll be curious to see who the other evaluators are and if my judgments track theirs. Undoubtedly, I have been influenced by various stereotypes and biases (a serious danger, of course, given how many of our judgments are shaped by personal contacts), including, of course, my own idiosyncratic qualitative judgments, but I found ranking easy. What did I rank? Well, some vague mixture of (a) quality of work; (b) personality; (c) sense/evidence of advising capability. One notable feature of my ranking practice -- and this shows my self-interest -- is that I was not afraid to give a '0' to departments that fall short of early modern coverage worthy of a PhD department. (This also includes some purportedly top departments that have a M&E/Ethics/science focus.)

I found evaluating 'philosophy of social science' much harder. It's a very diffuse area with lots of different niches and as I spent time on scoring, I became increasingly less confident that outside of philosophy of economics, I could recognize all the relevant names. When in doubt, I did go to department websites to check out people's areas of work. (I did not have  time to read new work, alas.) Much to my own surprise, when I was deciding on some of my final scores, I even checked the past PGR to see if others had also ranked a dept in philo social science. So, I ended up with serious misgivings about my participation despite the fact that I think it is a very important area of research and that I try to stay abreast on parts of the area. While I think the field important, I could not bring myself to insist that each department ought to have coverage in it; so rather than giving  a lot of 'O's, I left a lot of rankings blank. Anyway, I probably should not have participated in this part of the ranking.

Finally, on the department-wide scores: part of me still strongly thinks it is an extremely dubious exercise -- my knowledge of the give-or-take-1000+-names is superficial --, I also came to think that it was not so hard to distinguish between, say, the obviously understaffed or very narrow departments from the departments filled with reasonably well known and good even excellent people. What I found most difficult, in fact, was to rank the departments that I know really well (because of a recent visit or stay) and that have some terrific people but that are also uneven in various ways (coverage, quality of members, known-to-me-predators on faculty, etc.).

I close with two general thoughts, one about my perspective and another about the non-inclusion of continental heavy departments.

First, here's one thing that made me uneasy about my own department-wide rankings.  I noticed that a whole number of departments have not jumped on the revival of metaphysics train. As a 'HPS-style person' I ought to feel sympathetic to those departments, which often house my intellectual friends. But I also feel these departments do not serve their students well if they do not provide detailed training in recent metaphysics. I feel this is true the other way around, too, but I did feel I betrayed 'my side' a bit by not giving them higher marks generally. (Oddly enough, I don't feel this about recent epistemology, much of which I find sterile.)

Second, I do not see the argument for excluding any PhD granting department. The survey is long, indeed, as is, but the current list includes by my count about a dozen non-excellent departments (small, narrow, uneven faculty, etc.). It would be better for the profession if all the departments had a chance at a ranking, and we might all even agree that we can recognize that some of the continental-excluded are fine places to do graduate training.

 

 

 

Comments

[Brian Leiter]

Evaluators complain about the length of the survey, but in each iteration we rotate in departments that haven't been recently evaluated, including SPEP departments. There were several "Continental-heavy" departments in the surveys this year, including UC Riverside, Warwick, Chicago, and Emory. Thanks for participating.

[Keith DeRose]

I think of not having all PhD-granting programs surveyed as being underwritten largely by a pretty deep structural feature of the PGR: that it seeks to report which programs (overall in various areas) are good and how good they are (at least acc. to the PGR evaluations), but not which are bad and how bad they are. I see there is also the concern about survey length. I guess it's the two combined.

As for that structural feature: I guess there's a lot that can said both for and against reporting the bad (as evaluated by the PGR machine) as well as the good, but I think just the added fervent enemies the PGR would gain by doing that makes the idea completely infeasible.

[Eric Schliesser]

Keith, the survey is lengthy, but given the stakes involved I don't see why adding, say, an hour's work is an undue burden.

Given the level of existing enmity I doubt much can be added to it (perhaps I am naive?) whereas some experiments with modest reform and inclusivity in the evaluation and evaluated pool might lead to a healthier conversation. Anyway, I encourage trial and experimentation!