Workshop on Foundations for Ockham’s Razor

All are welcome to attend.

June 22-24, 2012

Adamson Wing, Baker Hall 136A, Carnegie Mellon University

Workshop web page and schedule

Contact:  Kevin T. Kelly (kk3n@andrew.cmu.edu)

Rationale:  Scientific theory choice is guided by judgments of simplicity, a bias frequently referred to as “Ockham’s Razor”. But what is simplicity and how, if at all, does it help science find the truth? Should we view simple theories as means for obtaining accurate predictions, as classical statisticians recommend? Or should we believe the theories themselves, as Bayesian methods seem to justify? The aim of this workshop is to re-examine the foundations of Ockham’s razor, with a firm focus on the connections, if any, between simplicity and truth.

Speakers:

• Vladimir Cherkassky (University of Minnesota, Computer and Electrical Engineering),
• Peter Gruenwald (Leiden and Amsterdam, Machine Learning),
• Malcolm Forster (University of Wisconsin, Philosophy),
• Kevin Kelly (Carnegie Mellon, Philosophy),
• Hannes Leeb (University of Vienna, Statistics),
• Deborah Mayo (Virginia Tech, Philosophy),
• Oliver Schulte (Simon Fraser, Computer Science),
• Cosma Shalizi (Carnegie Mellon, Statistics),
• Elliott Sober (University of Wisconsin, Philosophy),
• Vladimir Vapnik (Columbia University, Center for Computational Learning Systems),
• Larry Wasserman (Carnegie Mellon, Statistics and Machine Learning).

Also, a video of Kahneman’s recent 2012 Sackler lecture, “The science of communication,” is here.

Formal Ethics 2012
Munich Center for Mathematical Philosophy
Ludwig-Maximilians-Universität München

The application of formal tools from logic and rational choice theory to the analysis of ethical concepts and theories is a rapidly growing field of research. It has shed new light on a variety of concepts that are central to ethical theory, such as freedom, responsibility, values, norms, and conventions. We invite submissions to Formal Ethics 2012, to be held at the Munich Center for Mathematical Philosophy on October 11th to 13th, 2012. The workshop aims to bring together researchers at the crossroads of ethical theory and formal methods.

Aside from the contributed talks, the workshop will feature keynote addresses from:

John F. Horty (University of Maryland-College Park)

Simon Huttegger (University of California, Irvine)

Gerhard Schurz (Heinrich-Heine-Universität Düsseldorf)

Martin van Hees (University of Groningen)

Instructions for submission.

Authors should send an extended abstract (1000 words max, pdf or postscript format) together with their name, institutional affiliation(s) and current position(s) to the Organizing Committee (organization@formalethics.net) with “Submission” in the email subject line.

Submitted by Olivier Roy

Modality and Modalities  is a two day CADILLAC event on all things Modal Logical. It will be held on Tuesday, 29th May and Wednesday, 30th May at the University of Copenhagen. The keynote speakers are Max Cresswell, Adriane Rini, and Krister Segerberg, and there will be a special tutorial presentation by Valentin Goranko.

All are welcome to attend, and attendance is free. But please: email CADILLAC if you plan on coming, so that they have some idea of how many to expect.

Those flaws and others like them can be fixed, however, which is a starting point for reforming the PGR. But the PGR itself can be repaired only if there is a will to make the PGR work they way it is advertised to work.  Further, even if one were to start fresh with an alternative ranking system that ran on the level, one might still find that it leads to more harm than good. Each of these points offer grounds for abolishing the PGR.

Surrey Before The Horse

The Leiter Reports switched web platforms several years ago, which is unfortunate because the early exchanges over the PGR in the mid- to late- 1990s were simpler to follow than today’s commentary.  Back then, like now, the strategy was to defend the PGR by attacking its critics.  But at the beginning there was no luxury of pretending that all the objections of the past had been already addressed and that critics were simply flogging away on dead horses. Instead, in response to the first PGR going up online, people wrote in, annoyed or bemused, pointing out one shortcoming or another with seat-of-the-pants methods.  Then, those complaints would turn up on the Leiter Reports in some variation of the form “Professor X, from a third tier department, wrote to complain about my reporting that his is a third tier department.”  It was a beautifully simple circle.  And a sensation!  I remember thinking at the time that nobody could take this stuff seriously and that everyone had to see the points that Richard Heck raised.  Boy, was I wrong.

Today it is essentially the same circular path but dressed up with a bit more overhead and advertising to finance it.  Persistence, time, and an impressive infrastructure has given the PGR a patina of legitimacy.  The spin has become more sophisticated sounding, too, with reassurances like

“results were checked for evidence of strategic voting; there was none. Evaluators were admirably responsible and honest in their assessments.”

Impressive sounding, but total nonsense.  Checked against what? The editor’s ‘gestalt’ sense of programs?  If there was a method for detecting strategic voting on a ballot with 87 departments, by which I mean a method that did not already assume an answer to the very question the PGR rankings are supposed to supply, the very best advice to anyone with knowledge of such a method is to publish it.  The Royal Swedish Academy is very interested in this sort of thing.

And therein lies the trouble with the reform agenda: the assumption that the PGR is reformable. For the PGR  has all the markings of data manipulation carried out in support of a preconceived conclusion.  A reformed PGR that actually worked the way it pretends to work would risk yielding a result that did not agree with those preconceived notions, which begins with the very idea that philosophy departments are the sort of things which yield a sensible ordering. The PGRs own cross-field consensus data casts doubt on even that, a point we will return to in a moment.

Years of vitriolic attacks on critics, but a reluctance or inability to engage with the substance of their criticisms; false claims about its own design; a battery of dubious methods;  quality control assurances that are not only false but impossible to perform; a history of stubborn resistance to meaningful reform.  Standing in the way of the reform agenda is this history.

Heck was Right

One still might ask whether reforming the PGR is a worthwhile goal to pursue.  After all, having a strong preferences among some alternatives, like where to go on a vacation, does not ensure that the axiom of total ordering applies. Vacation spots are shot through with incomparabilities, and to pretend otherwise—to insist that there is a single round vacation hole through which all the pegs of your family must pass, facets be damned—does not end well, as any veteran of family vacations will attest to. More generally, uncritically embracing an imaginary structure for a decision as a reflection of reality is a mistake which can lead to foolish decisions. The PGR assumes that departments of philosophy are things which can be sensibly ordered, even though there is little cross-specialty agreement within its own highly biased evaluator pool. The health of the field may well turn on embracing this lack of consensus rather than trying to paper over it.

It was an innovation to think of philosophers as show dogs, and an extraordinary achievement to presuade a good part of the profession to think of themselves in these terms, or at least to go along with it. A few have benefited a great deal from this, but most have not. There has always been more of the stick to running this operation than carrots, and a culture of fear of what might be lost rather than hope for what might be gained has settled in the profession, particularly in North America.  Nowhere has this trend been covered  better than at The Leiter Reports itself. 

The last decade was a decade of bubbles, a decade of people knowing better but acting otherwise. From greed. From fear. From anything but a sober assessment. Enron. WMD. Madoff. A new home with no money down. Why? Because everyone else was going along with it, too.  It is simply how things were done, or in some cases because everyone believed this is what everyone else was doing.

The Philosophical Gourmet Report is philosophy’s bubble.  It is no longer so much a question of whether it will pop, but when. (Although I am not holding my breath.)  And when it does pop, when the collective scales fall from our eyes and forehead smacks are heard in faculty lounges across the land, all of us, but particularly our elders, will owe Richard Heck an apology for not having had his back 15 years ago.  He was right.

(Click image to enlarge, or download the full-sized image in pdf or png format.  Also, the 2011 PGR data set is available as an excel file. If you cannot open .xlsx files and need .xls instead, please email me.)

The structure of the bias in educational background is robust. The median years of seniority for PGR raters, determined by the year the most advanced degree was awarded to each evaluator, is 22 years. Examining the younger segment of the rater pool (0 – 21 years of seniority) also finds half (47.3%) of the rankings submitted by alumni from just 8 universities.  And that pattern repeats, though with less dramatic effects as the pool gets sliced into smaller and smaller segments.  The composition of this top group is relatively stable through time as well, with some minor shuffling to make way for the rise of Rutgers. So, educational bias is a curiously durable feature of the PGR evaluator pool.

To this one might counter that the rater pool is, despite appearances, an accurate representation of the larger community of research active philosophers.  As I mentioned in my previous post, this position is hard to reconcile with objective measures of research activity, like the one that Jon Kvanvig put together at Certain Doubts.   But we don’t even need to bother wading into the controversies surrounding bibliometrics to clear this objection. Instead, simply look at the PGR itself and consider whether its own judgment of research excellence across its own 33 areas of specialization is reflected in the composition of the PGR evaluator pool.  Put simply, does the PGR walk the PGR talk?

The answer is, No, not really.

The PGR covers 33 areas of specialization (AOS), and the Group 1 departments of an AOS are those judged by experts in those specialties to be the best in class.  Only 42 departments have one or more Group 1 AOSs. Half (20/42) of those departments have a single Group 1 AOS, a quarter (11/42) have two, a bit more than an eighth (6/42) have three, which leaves five remaining departments with 4 or more Group 1 areas of specialization:  Notre Dame (4), Rutgers (5), Michigan (5), NYU (6), and Oxford (8).  The following table orders those 42 departments by their share of Group 1 AOSs.
In addition, the table lists the number of alumni from each university who participated in the PGR specialty rankings (PhD raters), the number of individual rankings each university’s alumni submitted (PhD votes), and the same for each university’s faculty, i.e., the number of participants (Home raters) and number of rankings those faculty submitted (Home votes).   Zeros in the table are highlighted in red.

Surprisingly, well over a third (15/42) of alumni from Group 1 departments have no representation in the PGR evaluator pool.  Of those universities with a single Group 1 AOS, about half (45%) are without alumni in the evaluator pool.  The same story holds for universities with two Group 1 specialties (45.4%). Two universities are omitted entirely from the rater pool—that is, zero alumni and zero faculty: Penn State and Carnegie Mellon.  These two are missing despite having one (PSU) and two (CMU) Group 1 AOSs.  Duke has three Group 1 specialties but no alumni raters. Notre Dame has four Group 1 specialties but merely two alumni and one faculty in the rater pool.

By comparison, the next table lists four departments well represented in the PGR but without any Group 1 AOSs.

Curiously, while Berkeley graduates are well represented in the evaluator pool, the faculty are nowhere to be seen in the PGR evaluator pool. Taken together, no Berkeley faculty, no CMU faculty, and no LSE faculty are in the rater pool. Note their relative position on Healy’s original PCA plot. Assuming this PCA is a sound heuristic for viewing relative position of departments with respect to field, by the internal logic of the PGR the omission of these three faculty is evidence for an unrepresentative evaluator pool.[1]

These tables focus on cases of zero representation, since this is the most glaring and uncomplicated example: you cannot gainsay the null set. Simply put, having this many holes in the PGR evaluator pool beggars belief in its “educational balance” and the claim that, “necessarily, a large number of alumni of top programs are represented” is preposterous.

A more refined analysis will find a disconcertingly low correlation between expert opinion and the composition of the rater pool for those who are represented, a discrepency that Professor Healy alludes to in his remark about the rough fit between high performance in specialty areas and position in the over all ranking. Although Healy suggests that the raters as a whole assign different weights to the importance of the specialties, the plausibility of this charitable interpretation is undermined by the fact that, with the exception of a few at the top and a few at the bottom, there is not a lot of cross-field consensus in the rater pool.[2]

So far we have been focusing on the non-representativeness of the PGR evaluator pool by focusing on omissions from the pool that introduce bias. One could instead investigate other sources of bias, such as by looking at the actual composition of the PGR evaluator pool.  That is a more controversial line to pursue, but not a pointless one.  I leave this as an exercise for the reader.

***

One might consider reforming the PGR.  The PGR sampling problem and several of the other problems which plague it can be fixed, at least in principle. The profession could have a better working reputational survey, one that stood on its own merits and would not require dragooning the field to following along.  But I am of two minds on whether it is worth trying.  On Monday, Wednesday, and Friday, I am a PGR reformer.  Think of all that could be done to make this better!  An open data model; a separation of the business of measuring faculty reputations from the hack work of lauding and defaming—a move that would help reduce the dubious practice of push-polling, since recognition is at the heart of the decision making heuristic of raters;  a broadening of the pool of experts, perhaps through the use of PhilPapers or another publicly available data set; and a much better security protocol for protecting evaluator anonymity, since the systematic biases in the rater pool that make the PGR fundamentally unsound can be exploited to break the anonymity of evaluators should anyone have both the skill and motivation to try. Who besides bullies and con artists wouldn’t be all for that?

But on the off days I am a PGR abolitionist.  Explaining why is next.

NOTES:

[1] This point buries an assumption that I highlight here. In the case of CMU and the LSE, the point can be seen immediately by looking at the relative positions of those universities within Healy’s principle component analysis plot.  By including Berkeley too I am focusing on the faculty comprising the logic and methodology program at Berkeley, which is not separated in the PCA (just as Pitt and Pitt HPS aren’t separated, either, for example). If this assumption troubles you, then omit Berkeley and rerun the paragraph with just CMU and the LSE instead.

[2] As a reminder, Healy analyzed the 2006 PGR rater pool and my dataset is built from 2011 data. See the previous post, Manufacturing Assent: The Philosophical Gourmet Report’s Sampling Problem, for an explanation of how the two are being used together.

Evaluators were selected with an eye to balance, in terms of area, age and educational background—though since, in all cases, the opinions of research-active faculty were sought, there was, necessarily, a large number of alumni of the top programs represented. Approximately half those surveyed were philosophers who had filled out the surveys in previous years; the other half were nominated by members of the Advisory Board, who picked research-active faculty in their fields.

However, the PGR evaluator pool is not balanced with respect to educational background, and the claim that the educational imbalance observed in the 2011 PGR is a necessary consequence of soliciting the opinion of research-active faculty is false.

Now, to criticize the composition of the PGR evaluator pool is not to criticize the individual members making up the pool. The focus of the discussion at C&I (here, here, here, here, and even here; see also Andrew Gelman’s post here) has been on methodology.  The point instead is this: if you are interested in an accurate picture of professional opinion but oversubscribe from some parts of the profession and systematically omit other parts altogether, you might end up with an accurate assessment. But knowing this much about your methods, there is no reason to believe that you will. That, in a nutshell, is the nature of the PGR sampling problem.

The PGR sampling problem is not breaking news.  It has been a stubbornly durable feature of the survey since its inception, as Richard Heck pointed out long ago, and Zachery Ernst has elaborated on—remarking, and I paraphrase, that the snowball sampling method used by the PGR, while suitable for surveying pimps and dope dealers, is indefensible for academics practicing philosophy out in the open.

What might be new is a data set for the 2011 PGR rater pool for you to see for yourself, and a visualization of the degree of the educational background bias within the PGR rater pool, presented below.

***

We start with Kieran Healy’s excellent series on the 2006 PGR data set, which provides several insights into how to think about the data for the PGR. In those posts he is pretty careful to signal that his remarks are descriptive (i.e., based on the properties of the rater pool) and not inferential (i.e., representative of corresponding properties in the profession), but he does consider some objections to using the PGR to draw such inferences and the bulk of the posts are taken up with investigating whether the data set can rule those objections out.  And in some cases, the data does just that. Although it would be better to move to an open data model for the PGR, these posts are the next best thing.

In his Ratings and Specialties post, Professor Healy looks at various ways to measure “cross-field” consensus.  In one exercise he reconstructs the PGR ranking from the point of view of each specialty. What would the PGR ranking of departments look like if just the Ethicists were in charge? What would it look like if just the Philosophers of Language were? The Kantians? And so on. Those rankings were then aggregated to see how much variation there is across the specialties, which gives a sense of  how much (or how little) consensus there is across specialties. The box and whisker plots (top 25: .png, .pdf; total population: .png, .pdf) give a picture of this.  Except for the top 6 departments, and a handfull rounding out the bottom, the answer is that the rankings vary quite a lot:  people vote according to whom they recognize, and invariably those are the people working in their area(s) of specialization(s).  The flip side of this appears to be a version of the closed world assumption: if a rater hasn’t heard of you, then you probably aren’t worth hearing from.  This was my favorite of Healy’s posts, although I suspect that the observed volatility is attenuated by the unrepresentative composition of the evaluator pool.  So, let’s turn to that.

Healy considers whether voting patterns in the 2006 data are correlated to the “social location” of the evaluators, which is their place of employment or “Home” institution. But, another question is to examine whether the voting patterns are correlated to the “imprinting location” of raters (i.e., where they earned their PhDs), once the cross-specialty volatility is controlled for.   Through some reverse engineering of the data that is available,  it would be surprising if Ph.D. institution and voting patterns did not turn out to be correlated. In short, while it may be true that where you stand does not depend on where you sit, it still may be that where you stand depends on from whom you learned to stand on your own.

This point can be illustrated by comparing two graphs. This first graph, with the blue ink blots, overlays the distribution of the Home Institutions in the 2011 PGR pool with Healy’s PCA of the 2006 data set.[1]   In Healy’s terms, what this graph does is overlay the distribution of “social locations” (but for the 2011 data set) over his heuristic for viewing how departments and subfields differ from one another (based on his 2006 analysis).

(Click to enlarge, or view in full size: pdf and png)

This picture is consistent with a happy view of the PGR, for it paints the picture of a wide and reasonably diverse sample of the profession. Even the centers of concentration of the raters are outside the center, which further contributes to a sense of balance.  I suspect that something like this is what some have in mind when they counter that the PGR simply tells it like it is, like it or not, and is what bucks up their confidence to label dissenters as idiosyncratic cranks on the fringe of the field longing to be accepted by the mainstream.  Although more telling of the critic than those criticized, one can be forgiven for thinking that there is some truth behind the bluster.

But that case starts to crumble when one takes a look instead at where the raters earned their PhDs.  For here we see an unusually high concentration around a small cluster of universities, and this fact runs counter to the claim that the evaluator pool is “educationally balanced”.

(Full size: pdf and png)

Here we see not only a high concentration around a small number of imprinter institutions, but a cluster around the center of the PCA. This invites several questions, most of which are unanswerable without open PGR datasets. One immediate question in this context is the degree to which the oversubscription of a handful of ‘imprinter institutions’ is driving this PCA analysis and the clustering of green ink blots around the center, as this seems to be evidence for an effect from the educational imbalance in the evaluator pool.[3]

To put a finer point on this, there are 299 [4] evaluators and 126 universities represented in the 2011 PGR rater pool—113 individual Home institutions (blue)  and 58 PhD institutions (green).  But of the 736 individual rankings submitted for the 33 areas of specialization covered by the PGR, half (48.5%) were submitted by alumni from just 8 universities.

Now, you might maintain that a hard-nosed look at the research-active faculty will bear out that concentration, and that it will also bear out the 8 institutions in that set. But there are gaps in that set and holes in that argument—holes that appear even by the PGR’s own measure.  Exploring that point is next.

____

NOTES:

[1]  Healy’s caveats apply here, as well as the additional warning that I am overlaying the 2011 rater pool with the departmental positions determined by the 2006 data. Leiter reports that the PGR is remarkably stable, so this shouldn’t be too far off from a new PCA constructed from the 2011 data.

[2]  I have taken the subfield vectors out of my graphs to make them less cluttered. See Healy’s original (png or pdf), or email me if you would like to see the version with these vectors back in.

[3] Note that evaluators are prohibited from directly ranking both their home institution and their PhD institution, but this does not eliminate the effects of an unbalanced evaluator pool.

[4] There are some discrepancies on the PGR website between the main list of evaluators and the evaluators listed for the individual specialties. The total number of evaluators that I can account for in the specialty rankings is 299, but Leiter’s list of “nearly 270 evaluators who completed the overall faculty quality survey” numbers 302, and there is one missing. Professor Leiter was contacted about discrepancies on the PGR website but did not reply.