Comments on Federal College Rating Metrics

The U.S. Department of Education (ED) released a document containing draft metrics for the Postsecondary Institution Ratings System (PIRS) today (link via Inside Higher Ed), with a request for comments from stakeholders and the general public by February. Although the release of the metrics was delayed several months (and we were initially expecting ratings this fall instead of just some potential metrics), the potential metrics and the explanations provided by ED provide insights about what the ratings will look like if (and when) they are finalized. Below are some of the key pieces of the released metrics, along with my comments.


Which colleges will be rated, and how will they be grouped? ED is planning to rate degree-granting and certificate-granting two-year colleges separately from four-year colleges. They are still considering whether to have finer gradations among four year colleges. Given the substantial differences in mission and completion rates between associate’s degree-granting and certificate-granting two-year colleges, I strongly recommend separating the two groups. Four-year colleges can all be rated together if input adjustments are used, or they can be put into much smaller peer groups (the latter seems to be what colleges prefer).


Leaving non-degree-granting colleges out of PIRS sounds trivial, but it leaves out a fair number of small for-profit colleges. I think many of the colleges not subject to PIRS will be subject to gainful employment, should that survive its latest legal challenge. Given that gainful employment has financial consequences while PIRS does not at this point, the colleges left out of PIRS are subject to more stringent accountability than many of those in PIRS.


What will the ratings categories and scoring system look like? I’m glad to see ED considering three rating categories: high-performing, in the middle, and low-performing. That’s about all the fine gradation the data can support, in my view, and it is far more politically feasible to have fewer ratings categories. No information was provided about how individual metrics will be weighted or scored, which likely indicates that ED is still in the preliminary stage on PIRS.


What metrics are being considered? And which ones do we already have data on? The metrics fall into three main categories: access, affordability, and student outcomes.


Access: Percent Pell, distribution of expected family contributions (EFC), enrollment by family income quintile, percent first-generation. Percent Pell and enrollment by family income quintile are already collected by the Department of Education, although these measures have gaps because not all students from low-income families file the Free Application for Federal Student Aid (FAFSA). The EFC distribution measure is intriguing, but it’s not currently collected. Perhaps considering the percentage of students with zero EFC (who have the least ability to pay) would make sense. The FAFSA asks students about parental education, so first-generation status could be made available in a few years. There is a question of how to define first-generation status, as it could include a student whose parents have some college but no degree or be limited to those with no college experience.


Affordability: Net price of attendance (overall and by income quintile). The net price reflects the total cost of attendance (tuition, fees, books/supplies, and living costs) less all grant or scholarship aid received. It’s a good measure to include, even if it can be gamed by institutions that cut their living allowances to absurdly low levels or use income from the FAFSA instead of the CSS PROFILE (where more sources are counted). I’m surprised not to see a measure for debt burdens or student borrowing here as a measure of affordability.


Outcomes: Graduation and transfer rates, short-term employment, longer-term earnings, graduate school attendance, and “loan performance outcomes.” As of right now, the only measures available are graduation/transfer rates (for first-time, full-time students) and student loan repayment. ED is working to improve the graduation and transfer metrics by 2017, which is welcome. I’m intrigued by how loan performance was described:


“Relatively simple metrics like the percentage of students repaying their loans on time might be important as consumers weigh whether or not they will be able to handle their financial obligations after attending a specific school.”


This is different from the standard cohort default rate measure, which measures whether a student defaults by not making a payment in the last 270 days. Measuring the percentage in current repayment would show a lower percentage of students having a successful outcome, but it better reflects former students’ performance than a cohort default rate. Kudos for ED for making this suggestion.


I see employment, earnings, and graduate enrollment outcomes as being good things to consider, but they won’t be ready to include in PIRS for several years. The ban on student unit record data makes tracking employment and earnings difficult unless ED relies on colleges to self-report data from their former students. It’s worth emphasizing the importance of including dropouts as well as graduates in these metrics. Graduate enrollment could in theory be done with the National Student Clearinghouse, but colleges may not want to participate in the voluntary system if it is used for accountability.


Any other surprises? I was pleasantly surprised to see ED include a section on considering how to reward colleges for improving their outcomes over time. This might be a way to get around the question of how to adjust for student inputs and institutional resources, or it could be a piece designed to bring more colleges to the discussion table.


What does all of this mean? It appears that PIRS is very much in its infancy at this point, given the broadness of the suggested metrics and the difficulty in getting data on some of them in the next year or two. Putting college ratings together is methodologically quite easy to do, but politically very difficult. The delay in the timeline and the call for additional feedback by February highlight the political difficulty of PIRS. Given the GOP takeover of Congress, I think it’s safe to say that even if a full set of ratings comes out next week, the likelihood of ratings being tied to aid by 2018 (as the President has proposed) is basically nil. (For more on why I think PIRS is a difficult political sell, read my new piece in Politico Magazine.) But even getting draft ratings ready for the start of the 2015-16 academic year will be very difficult. ED has a lot of work to do before then.


But PIRS does have the potential to substantially improve data availability and transparency on a number of important student outcomes, even without becoming a high-stakes accountability system. I expect that college access organizations, higher education publications, guidance counselors, and even those of us in the rankings business will work to get any new data sources out to students and their families in a consumer-friendly format. That may be the lasting legacy of PIRS.


The 2014 “Not Top Ten” List in Higher Education

Earlier this week, I unveiled my list of the top ten higher education policy issues of 2014, with the fascinating saga of Corinthian Colleges getting top billing this year. Now it’s time to turn to the “not top ten list,” highlighting some of the less-than-wonderful happenings of the year. Last year’s “winner,” Georgetown Law’s plan to stick taxpayers with the entire cost of legal education, gets a stern finger wagging again this year along with a one-year reprieve from the list.

10. Colleges spend millions to buy out the contracts of their football coaches. I wish I could be as financially successful as Charlie Weis, who is currently drawing enormous paychecks from Notre Dame and Kansas not to be their football coach. He is due a total of $4.6 million from the two colleges in 2015, and will get nearly $25 million to do absolutely nothing. This month, Nebraska, Florida, and Michigan all fired their coaches at the cost of over $17 million in buyouts. The (awesome) parody Twitter account of former Nebraska coach Bo Pelini (who just became the newest coach of the Youngstown State Penguins) is happy:

But Don Heller, dean of the education school at Michigan State, sees a better use for the money:

9. New Jersey teenagers sue their parents for financial support for college. I live in New Jersey, but I’m not sure what is in the water in the Garden State that has resulted in two teenagers suing their parents for financial support for college. In March, 18-year-old Rachel Canning made news by moving out of her parents’ house and suing for her private high school and college tuition. After a great deal of national scrutiny, she decided to drop her lawsuit and is now enrolled at Western New England University in Massachusetts.

In November, 21-year-old Caitlyn Ricci successfully sued her divorced parents for her $16,000 per year out-of-state tuition at Temple University in Philadelphia. Given that she has been completely estranged from her parents for two years, she might be able to qualify as independent for financial aid purposes. But New Jersey legal precedent actually requires divorced parents to chip in for their adult child’s educational expenses. Legislation has been introduced to effectively overturn past Supreme Court decisions.

8. It’s surprisingly hard to figure out how many students are having trouble repaying their loans. Putting aside concerns with how student loan default rates are calculated (which made my “top ten” list), the Department of Education doesn’t consider a student to be in default unless they have not made a monthly payment in the last 270 days. And their measure of loan delinquency rates actually exclude students in default, with the assumption that the loans will never be repaid. I got into a great discussion with Shahien Nasiripour of the Huffington Post about what percentage of students are actually having difficulties repaying loans. He wrote a piece claiming that about half of all students are not repaying, while my preferred estimate is about 30% and the federal government reports about 17%. Without better data from the feds, it’s hard to tell.

7. The “sexy PhD costume” available on Amazon for Halloween is just sad. For Halloween, PhD holders can finally put away that tweed jacket and attempt to shimmy into the “Delicious Women’s PhD Sexy Costume” before undergoing the peer review process. (Sadly, there is no men’s version, so your humble correspondent stayed home and handed out candy to local children while wearing appropriate attire.) Needless to say, women (and men) with actual doctorates were not amused by the costume, both in the way it denigrated women and did not comport with actual doctoral robes. I shared some of the Amazon reviewer comments via Twitter, and one of those tweets ended up being my most-viewed tweet of the year:

6. Some colleges report net price figures using PROFILE data instead of the FAFSA, possibly making themselves look better. Colleges are required to report net prices (the total cost of attendance less all grant aid received) for five household income brackets each year. These net prices are often used in media coverage of higher education, and they also play an important part in the Washington Monthly ranking of best bang-for-the-buck colleges.

I had always assumed the net prices were based on income reported on the FAFSA, which excludes income from noncustodial parents and business enterprises in certain cases. But this excellent (if graphic-heavy) piece from The Chronicle of Higher Education found that some colleges instead use the CSS PROFILE definition of income, which typically results in fewer students being in the bottom income categories. In addition to making comparisons across colleges difficult (since we don’t know which colleges report PROFILE income versus FAFSA income), students have to fill out the PROFILE in addition to the FAFSA.

5. I feel sorry for negotiated rulemaking panels. Negotiated rulemaking panels are used whenever the Department of Education (or other federal agencies) wish to promulgate new rules. The goal is to build consensus around a set of rules, but what most often happens is that the panel (consisting of representatives from various affected parties) cannot reach a consensus. In this case, the federal agency can go ahead and issue its own rules. Two of the most famous negotiated rulemaking panels this year were for redefining “adverse credit” for PLUS loans and regarding gainful employment. Although the panels do have value (such as the first-ever release of PLUS loan default rates), the members still need a big hug.

4. Some colleges use where students send the FAFSA to shape financial aid packages. While completing the FAFSA, students list up to ten colleges where they would like to send their information. But what most students don’t know is that the listing is shared with other colleges—and that some enrollment management offices base part of a student’s financial aid award on where their college is listed. (Other colleges, such as DePaul, use the data to predict the size of an incoming class, which is benign. I highly recommend Jon Boeckenstedt’s take on the topic.)

3. Nicholas Kristof pokes the bear on #engagedacademics. One of the best ways to upset the academic community is to say that we don’t engage the public and instead stay cloistered in the ivory tower. But Nicholas Kristof of the New York Times said exactly that in a February opinion piece. While there is some truth to the statement, the academic community wasn’t too happy. Chuck Pearson of Tennessee Tech University started an #engagedacademics hashtag on Twitter that got lots of great responses, and this Chronicle piece summarizes the response from the academic community, including my blog post on the topic. But I think this is the best counterexample that academics can point to:

2. Congress raids future Pell Grant funding to pay the bills today. The continuing resolution/omnibus spending bill (or cromnibus, in DC-speak) for the federal government took just over $300 million from future surpluses to the federal Pell Grant program to pay student loan servicers in 2015 for their services performed. Some people are really upset that the money is going to companies like Nelnet and Navient, but in my view, those companies were going to get paid anyway. Congress has a long and rather sordid history of kicking the fiscal can down the road, and this is just another example. If the Pell program is running a shortfall in 2017 or 2018, this shortsighted (bipartisan) action by Congress will partially be to blame.

1. Kean University spent $219,000 on a conference table…and vigorously defended the purchase. Kean, a relatively unknown public university in New Jersey, has gotten a lot of attention in recent weeks—and not of the good kind. (In-state peer NJIT, on the other hand, got great publicity for its vagabond men’s basketball team upending Michigan.) Kean spent a remarkable $219,000 on a 22-foot-long oak conference table with global communication capabilities that was imported from China, where Kean has academic partners. (I’ve heard of endowed chairs in academia, but a table that needs to be endowed? My goodness!)

When the inevitable criticism of the university sprouted up on social media, Kean doubled down on the need for such an expensive table. Kean claimed in a letter that the table “means added value to your Kean degree.” One can only hope that the claim is empirically validated.

Also receiving votes: Rating colleges “like blenders,” conspiracy theories involving higher education foundations, celebrating a touchdown one yard too early, referring to the Department of Education as “DOE” (Energy) instead of “ED,” Pell Grant recipient graduation rate data being delayed yet again, people who make annual “top ten” and “not top ten” lists.


The 2014 Top Ten Higher Education List

As we near the end of 2014, it’s time to look back and reflect upon the events of the past year. I’ve ranked the top ten newsworthy events in the higher education policy world (at least in my view) in 2014—some of these items also made my 2013 list. As always, feel free to take issue with my list and suggest your own items that I missed!

10. “Big data” systems keep getting bigger, but run into privacy concerns. Encouraged by funding from the federal government and private foundations, states continue to build comprehensive unit record data systems in order to better track students from pre-kindergarten through college and the labor market. Some states, such as Florida and Texas, have also allowed researchers to use that data for a host of studies. (I have to mention Virginia’s stellar higher education data system, run by the incomparable Tod Massa.) Private aggregators, such as LinkedIn and PayScale, have collected outstanding (albeit limited) employment and earnings data from a large sample of volunteers—and these data have been used in a number of college rankings.

Although the federal government collects some unit record data through its series of nationally representative surveys, it is prohibited by law from collecting unit record data on all students. In March, Amy Laitinen and Clare McCann of the New America Foundation released College Blackout, a scathing report blaming the lobbying association representing private nonprofit colleges for the lack of a federal unit record system. Yet privacy concerns do exist with unit record data, and research suggests that “deidentified” data (which should contain no information allowing the record to be tied to an individual) may not truly be deidentified. I’m on the side of supporting unit record data—and legislation has been introduced overturning the ban on unit record data—but policymakers should proceed with caution.

9. Faculty, administrators, and the public disagree on the definition of academic freedom. This is best illustrated by two cases from the University of Illinois. The case of Steven Salaita, whose job offer as a tenured professor of American Indian studies was revoked due to tweets he had sent out regarding the Israeli-Palestinian conflict. Salaita had already resigned his tenured position at Virginia Tech, so the revocation of his offer meant that he did not have an academic job for this fall. (In the short run, he has kept busy as a public speaker.)

The second case is of James Kilgore, an adjunct professor who was a member of the Symbionese Liberation Army in the 1970s who spent time in prison for a botched bank robbery. His rehiring became controversial, but the Board of Trustees allowed departments within the university to make the hiring decision. However, a large donor has threatened to withhold a major gift over the rehiring, so this case may not yet be closed.

8. The NCAA enacts governance reforms and gives the most powerful programs more autonomy. 2014 was a largely forgettable year for the NCAA, including a unionization attempt by Northwestern’s football players and a loss in court to Ed O’Bannon and other players who sued for their likenesses being used without compensation. But in addition to adopting a four-team college football playoff in the NCAA’s highest level of competition (so close, Baylor and TCU!), the NCAA adopted rules that gave more autonomy for teams in the “Power Five” conferences (the Big Ten, Big Twelve, ACC, SEC, and Pac-12 conferences plus Notre Dame). This included additional compensation for student-athletes to meet the full cost of attendance—something that is likely unaffordable for most Division I programs without increasing subsidies to athletic programs.

7. “Free community college” programs gain steam. In 2014, Tennessee announced a program in which qualified high school graduates would have tuition and fees for community college waived if they met certain requirements, including full-time attendance. The city of Chicago adopted a more restrictive version of the plan, while Mississippi, Oregon, and Texas also discussed similar programs.

These programs have the potential to benefit students, particularly by providing clear information that college is (relatively) affordable. Tennessee expects two-thirds of high school seniors to sign up for the program, even though many won’t attend community colleges. But, as I noted in an essay in Inside Higher Ed, the program doesn’t actually provide any additional money to students from low-income families. Sara Goldrick-Rab and Nancy Kendall’s proposal for a Free Two-Year College Option would make community college truly free, but I view it as unworkable due to it effectively cutting financial aid availability at all private colleges.

6. “Yes means yes” pledges and sexual assaults on campus are in the spotlight. The Rolling Stone piece alleging gang rapes at the University of Virginia shook me to the core. Even though the story may not be true (and the reporting was substandard), sexual assaults on campus are still a concern. This is true even if sexual assault rates are lower among college students than in society as a whole—one assault is one too many. “Yes means yes” or “affirmative consent” rules regarding intimate relationships have been adopted by many states and universities, but concerns exist about whether these rules are truly effective or protect the rights of the accused.

5. Gainful employment rules come out—and immediately go to court. The Department of Education released its second try at gainful employment in October, after the 2011 rules were struck down in court last year. Most observers had expected two measures to be included in the rules—a cohort default rate measure and a debt-to-income measure—but the default measure was unexpectedly dropped. (For what the change means, I highly recommend Ben Miller’s take on the topic.)

The lobbying association for the for-profit sector filed suit one week after the rules were released, claiming that the rules lacked legal basis and the preceding session of negotiated rulemaking was biased against for-profit colleges. What I view as the for-profits’ strongest point of the lawsuit is their claim that the Department of Education said no single measure would result in a reliable gainful employment metric. After the default measure was dropped, gainful employment may be in trouble once again in the courts.

4. The Higher Education Act is due for reauthorization. Will it happen in 2015? The single most influential piece of higher education legislation was due to be authorized this year, but it did not get done. (Don’t feel too bad, higher ed folks: the Elementary and Secondary Education Act has been in limbo since 2007.) Not surprisingly, Republicans and Democrats can’t agree on how to move forward. Republicans have supported a piecemeal approach, getting three bills on smaller pieces (including supporting competency-based education) through the House with unanimous support. Senate Democrats have supported a comprehensive reauthorization, as evidenced by retiring Senator Tom Harkin (D-IA)’s draft legislation. Will 2015 be the year for reauthorization? I wouldn’t bet the farm on it, but unified GOP control of the House and Senate might get a bill to President Obama’s desk.

3. The Department of Education blinks on cohort default rates. The Department of Education received a lot of criticism (and some praise) for its last-minute exemption of some students who had loans through multiple servicers from the cohort used to calculate default rates. This change allowed a number of colleges to have default rates under 30%–the cutoff for potentially losing federal student aid eligibility. Between loosening the criteria for both gainful employment and cohort default rates, many policy folks and advocates aren’t too happy with the Department of Education.

2. The saga of federal college ratings continues. We should see something from the Department of Education regarding the Postsecondary Institution Ratings System (PIRS) any day now, as multiple officials have said to expect a list of metrics this fall. 2014 began with an expectation that ratings would be released by the middle of the year, but that quickly fell by the wayside as it now appears that no ratings will be released until well into 2015. I’ve been fortunate enough to offer my thoughts on ratings to Department of Education representatives on multiple occasions. This has the potential to provide some useful information, but I don’t see any way that ratings are tied to financial aid in 2018 (the Obama Administration’s stated goal, although they’re gone by then).

Less noticed on the ratings front is the Department of Education’s move to have states develop quality ratings for teacher preparation programs. Again, the goal here is to tie some grant aid to these quality ratings, but this has concerned many in the education community. This could be a political battle to watch in 2015.

1. COCO is no more. The rapid collapse of Corinthian Colleges (COCO), a large for-profit college chain, in the middle of 2014 gets my vote for the biggest higher education policy event of the year. In June, the Department of Education placed Corinthian under additional financial oversight, including a 21-day waiting period on accessing financial aid dollars. This nearly immediately caused Corinthian to collapse, as the company immediately began negotiations with the feds about how to wind down operations. Corinthian still faces criminal lawsuits over its practices—and you can buy a share of its stock for a mere nine cents, down about 95% from this time last year.

Corinthian’s saga was made even more fascinating this fall with the announcement that ECMC Group (a student loan guarantee agency with no history of owning colleges) would buy 56 of the Everest and WyoTech colleges formerly owned by Corinthian. This deal has gotten a lot of criticism, but the Department of Education had a strong incentive to get someone to buy these colleges. I’ll wait to withhold judgment on the deal until I see what ECMC does in terms of bringing in senior administrators and faculty.

Also considered: Large variations in estimated living costs across colleges, Starbucks-Arizona State bachelor’s degree completion partnership, renewed student activism, relaxing tightened credit standards on PLUS loans, Grand Canyon considering going nonprofit, Georgia Tech’s online master’s degree in computer science.

Stay tuned later this week for my annual “not top ten” list of some of the not-so-great happenings of the year—plus a potential analysis of the draft college rating metrics should they come out on time.


Review of “American Higher Education in Crisis? What Everyone Needs to Know”

I recently had the pleasure of reading American Higher Education in Crisis? What Everyone Needs to Know (Oxford University Press) by Goldie Blumenstyk, senior writer at The Chronicle of Higher Education. Goldie has a well-deserved reputation as perhaps the best higher education journalist out there, and this book reflects her ability to summarize complex topics in higher education for a broad audience. Veteran researchers in higher ed finance and policy probably won’t come across too many new things in this book, but it serves as an excellent primer for policymakers, students, and the general public.

The book consists of four main sections, in which Blumenstyk quickly moves through key questions of interest to the higher education community and the public. Each section focuses on an overview of the topics at hand and brings in data and research in an easily accessible manner. In the first section, she examines students, with some of the following questions being considered:

–Who enrolls in college? How have demographics changed over time?

–How does the college admissions process work?

–Are students academically prepared for college, and what is the extent of “undermatching?”

In the second section, Blumenstyk examines higher education finance, paying attention to both student and institutional finance. Some of the questions considered include the following:

–Why does college cost so much? She gets extra credit from me for explaining the difference between cost (the amount of money spent to provide the educational experience) and price (what students and their families actually pay). This is a crucial distinction that is rarely made with the general public in mind.

–What do student loan burdens look like? And are they reasonable?

–Are institutional finances stable? What types of colleges may be in trouble going forward?

The third section of the book centers on leadership and governance. The following questions are considered:

–What is a typical governance structure like? What are the roles of internal stakeholders such as faculty and administrators?

–What are the roles of external stakeholders such as policymakers, foundations, and accreditors?

–What do state and federal accountability policies look like?

In the final section, Blumenstyk looks at some practices and policies that may represent the future of higher education. While some of the items discussed below will probably not have as large of an impact on higher education as we think right now, this section serves as a great reference regarding some of the buzzwords in higher education. These topics include:

–What do MOOCs (massive open online courses) look like, and how have they been adopted to this point?

–What is competency-based education, and what are other ways to earn college credit without sitting in a classroom (or in front of a screen online) for a semester?

–How are “big data” and predictive analytics being used to change educational practices and policies?

–Will traditional campuses survive? And what types of colleges are most at risk?

This book provides a great overview of the state of higher education and directs readers to additional resources for further reading. Not only do I recommend this book to those looking to learn more about higher education, I plan to use parts of this book for my organization and governance and higher education finance classes in the future.

What Will the College Opportunity Summit Mean for Higher Education?

Today, the White House is hosting a second College Opportunity Summit, following up on a summit held in January that was roundly criticized for focusing on elite institutions. Both this summit and the previous summit involved colleges and other organizations making pledges designed to improve college access and completion rates, particularly for underrepresented populations and in STEM. The first round of pledges (and progress made) and the second round of pledges can both be found on the White House’s website.

Several hundred people, including administrators, policy analysts, and researchers, are at today’s summit, which has the potential to generate useful discussions. But it could also be the case that the discussion turns into a stereotypical academic conference, where a lot of items are discussed but no action is ever taken. So what could the summit mean for higher education?

The first thing that jumps out from the list of pledges is the sheer number. The list contains over 600 actions that colleges, associations, and other organizations plan to take—which is admirable. But as a researcher, two key questions should be considered:

(1) Would colleges and organizations have adopted these policies even without a formal pledge? In research language, this is known as the counterfactual—considering what would have happened in the absence of the policy being studied. This list could represent a list of things that colleges already planned to do (but they get good PR and tickets to the White House tree lighting), or this could be a result of colleges setting new goals as a result of the White House’s call for commitments. When considering the impact of this summit, researchers should talk to some college administrators (while promising confidentiality) to see if the pledges were policies already being planned or a new development.

(2) Will these pledges improve student outcomes? This involves thinking carefully about program design and data collection, so it is possible to use experimental or quasi-experimental methods combined with in-depth interviews in order to examine program impacts and potential moderating and mediating factors. The Institute for Education Sciences announced an additional $10 million in funding for postsecondary research, but that amount won’t make much of a difference as funding an intervention and conducting an evaluation can easily cost several million dollars.

I hope the summit helps colleges and organizations develop partnerships similar to the University Innovation Alliance, the Student Achievement Measure, and other organizations that link colleges with similar goals to each other. But it’s worth keeping in mind that many of these pledges are likely things that colleges planned to do anyway.