Choice & Inference

We observed before that the 2011 PGR is unbalanced with respect to educational background.  That point can be captured graphically by merging the two earlier bubble plots—one representing the distribution of Home institutions (blue) and another for the distribution of PhD institutions (green)—into one.

(Click image to enlarge, or download the full-sized image in pdf or png format.  Also, the 2011 PGR data set is available as an excel file. If you cannot open .xlsx files and need .xls instead, please email me.)

The structure of the bias in educational background is robust. The median years of seniority for PGR raters, determined by the year the most advanced degree was awarded to each evaluator, is 22 years. Examining the younger segment of the rater pool (0 – 21 years of seniority) also finds half (47.3%) of the rankings submitted by alumni from just 8 universities.  And that pattern repeats, though with less dramatic effects as the pool gets sliced into smaller and smaller segments.  The composition of this top group is relatively stable through time as well, with some minor shuffling to make way for the rise of Rutgers. So, educational bias is a curiously durable feature of the PGR evaluator pool.

To this one might counter that the rater pool is, despite appearances, an accurate representation of the larger community of research active philosophers.  As I mentioned in my previous post, this position is hard to reconcile with objective measures of research activity, like the one that Jon Kvanvig put together at Certain Doubts.   But we don’t even need to bother wading into the controversies surrounding bibliometrics to clear this objection. Instead, simply look at the PGR itself and consider whether its own judgment of research excellence across its own 33 areas of specialization is reflected in the composition of the PGR evaluator pool.  Put simply, does the PGR walk the PGR talk?

The answer is, No, not really.

Read the rest of this entry »

The Philosophical Gourmet Report (PGR) purports to be a ranking of faculty reputation based on the opinions of “research-active” faculty, which it surely is. But the legitimacy of the PGR as an accurate measure of opinion relies in part on how well the PGR pool of evaluators reflects the population of research-active faculty.   Addressing this point, Brian Leiter remarks (here) on the selection procedure for the PGR evaluator pool:

Evaluators were selected with an eye to balance, in terms of area, age and educational background—though since, in all cases, the opinions of research-active faculty were sought, there was, necessarily, a large number of alumni of the top programs represented. Approximately half those surveyed were philosophers who had filled out the surveys in previous years; the other half were nominated by members of the Advisory Board, who picked research-active faculty in their fields.

However, the PGR evaluator pool is not balanced with respect to educational background, and the claim that the educational imbalance observed in the 2011 PGR is a necessary consequence of soliciting the opinion of research-active faculty is false.

Now, to criticize the composition of the PGR evaluator pool is not to criticize the individual members making up the pool. The focus of the discussion at C&I (here, here, here, here, and even here; see also Andrew Gelman’s post here) has been on methodology.  The point instead is this: if you are interested in an accurate picture of professional opinion but oversubscribe from some parts of the profession and systematically omit other parts altogether, you might end up with an accurate assessment. But knowing this much about your methods, there is no reason to believe that you will. That, in a nutshell, is the nature of the PGR sampling problem.
Read the rest of this entry »