The college rankings marketplace became a little bit more crowded this week with the release of a partial set of rankings from an outfit called The Alumni Factor. The rankings, headed by Monica McGurk, former partner at McKinsey & Company, rank colleges primarily based on the results of proprietary alumni surveys. The site claims to have surveyed approximately 100-500 alumni across the age distribution at 450 universities in creating these rankings. (By this point in time, a few alarms should be sounding. More on those later.)
The Alumni Factor uses thirteen measures from their surveys to rank colleges, in addition to graduation and alumni giving rates from external sources. These measures include development of cognitive and noncognitive skills, bang for the buck, average income and net worth, and “overall happiness of graduates.” The data behind the rankings has supposedly been verified by an independent statistician, but the verification is just a description of confidence intervals (which are never again mentioned). They provide rankings for the top 177 schools, which are not visible unless you sign up for their services. As a result, I do not discuss the overall rankings.
While I commend the Alumni Factor for making more information available to consumers (albeit behind a paywall), I don’t see this set of rankings as being tremendously useful. My first concern is how they selected the colleges to be included in the rankings. The list of the top 177 schools consists primarily of prestigious research universities and liberal arts colleges; few to no good public bachelor’s to master’s level universities (such as my alma mater) are included in the list. This suggests that the creators of the ranking care much more about catering to an audience of well-heeled students and parents than making a serious attempt to examine the effectiveness of lesser-known, but potentially higher-quality, colleges.
I am extremely skeptical of survey-based college ranking systems because they generally make every college look better than they actually are. The website mentions that multiple sources were used to reach alumni, but the types of people who respond to surveys are generally happy with their college experience and doing well in life. (Just ask anyone who has ever worked in alumni relations.)
The website states that “we did not let our subjective judgment of what is important in education lead to values-based weightings that might favor some schools more than others.” I am by no means a philosopher of education, but even I know that the choice of outcomes to be measured is a value judgment about what is important in education. The fact that four out of fifteen of the measures used measure income and net worth, while only one reflects bang for the buck, shows their priority on labor market outcomes. I don’t have a problem with that, but they cannot ignore the decisions they made in creating the rankings.
Finally, these survey outcomes are for alumni only and exclude anyone who did not graduate from college. Granted, the colleges on the initial list of 450 all probably have pretty high graduation rates, but it is important to remember that not everyone graduates. It’s another example of these rankings being limited to prestigious colleges, as well as the perception that everyone who starts college will graduate. It’s just not true.
Although no set of college rankings is perfect (and not even the ones that I’ve worked on), The Alumni Factor’s rankings serves a very limited population of students who are likely well-served by the existing college guides and rankings. This is a shame because the survey measures could have been tailored to better suit students who are much closer to being on the margin of graduating from college.
For those who are interested, the 2013 U.S. News & World Report college rankings will come out on Wednesday, September 12. I’ll have my analysis of those rankings later in the week.
Disclaimer: I am the consulting methodologist for the 2012 Washington Monthly college rankings. This post reflects only my thoughts and was not subject to review by any other individual or organization.
Robert Kelchen 