Using Input-Adjusted Measures to Estimate College Performance

I have been privileged to work with HCM Strategists over the past two years on a Gates Foundation-funded project to explore how to use input-adjusted measures to estimate a college’s performance. Although the terminology sounds fancy, the basic goal of the project is to figure out better ways to measure whether a college does a good job educating the types of students that it actually enrolls. It doesn’t make any sense to measure a highly selective and well-resourced flagship university against an open-access commuter college; doing so is akin to comparing my ability to run a marathon with that of an elite professional athlete. Just like me finishing a marathon is a much more substantial accomplishment, getting a first-generation student with modest academic preparation to graduate is a much bigger deal than someone whom everyone expected to race through their coursework with ease.

The seven-paper project was officially unveiled in Washington on Friday, and I was able to make it out there for the release. My paper (joint work with Doug Harris) is essentially a policymaker’s version of our academic paper on the pitfalls of popular rankings. It’s worth a read if you want to find out more about my research beyond the Washington Monthly rankings.  Additional media coverage can be found in The Chronicle of Higher Education and Inside Higher Ed.

As a side note, it’s pretty neat that the Inside Higher Ed article links to the “authors” page of the project’s website (which includes my bio and information) under the term “prominent scholars.” I know I’m by no means a prominent scholar, but maybe some of that will rub off the others via association.

Is College Cost Certainty a Possibility?

There are only a few certainties in life, such as death, taxes, and Murphy’s Law holding true at the most inopportune times. For nearly everyone, however, knowing the cost of college more than a few months in advance is definitely not one of those certainties. But given the high sticker price of attending college (which is not tremendously useful for most people), what can be done to provide cost certainty for students and their families over the course of several years?

As a part of a nifty series of essays on possible ways to overhaul the college experience, Beckie Supiano of The Chronicle of Higher Education takes a quick look at what has been done to provide better cost information. She focuses on a useful goal—being able to provide students with the net price of education (tuition less grant aid) over the course of several years. While this is a great idea in theory, it quickly runs into several problems in practice:

(1)    Colleges only control a fraction of grant aid, especially for financially needy students. Most federal need-based grants require that a student be eligible to receive the Pell Grant; if family income rises by a small amount, the loss of aid can be devastating. This also makes forecasting net price nearly impossible for middle-income families.

(2)    Most colleges cannot forecast their available resources several years in advance. Public colleges are at the whims of economic circumstances and the state government, while many private colleges rely on a combination on endowment revenue and rear ends in seats in order to make ends meet.

(3)    Colleges, like most of us out there, tend to be what economists call risk-averse. In English, that means that we don’t like being exposed to uncertainty. Students and their families currently bear the brunt of the uncertainty with respect to higher education pricing, but locking in a price regardless of the economic circumstances would shift that risk to the college. Most colleges would likely set a very high initial price in order to account for this uncertainty.

A small number of states have experimented with guaranteed tuition plans through prepaid tuition programs, which helps provide at least some certainty (although they do not guarantee set amounts of financial aid). Alabama’s program ran out of money during the financial crisis and is currently closed to new enrollment while facing legal challenges. A similar program in Illinois is also drastically underfunded, which is a common theme in the Land of Lincoln.

The article makes a key mistake in discussing cost certainty—it ignores the fact that known cost increases still represent cost certainty. If a college guarantees that tuition will go up by five percent per year for the next five years, students and parents can still have an idea of what college will cost in the future. The State University of New York system took a similar path in 2011, allowing each university in the system to raise tuition by up to $300 per year for five years. This proposal is quite useful as it puts increases in dollar terms, which are easier to understand than percentages.

I am interested in providing more information on cost certainty, but my research focuses on the financial aid side of the cost equation instead of the tuition side. I am currently working on two studies examining whether students and their families can receive earlier notifications of their financial aid, with results hopefully to come in the next few months. It is far from perfect, but it does help provide a little more information in an uncertain world.

New Data on the Returns to College

Many people love to hate college rankings, but they have traditionally been one of the most easily digestible sources of information about institutions of higher education. We know very little about the outcomes of students who attend a particular college over time, so we tend to rely on simplistic measures such as graduation rates or measures of prestige. It is difficult to follow and assess the outcomes of students once they leave a given college for multiple reasons:

(1)    A substantial percentage of students transfer colleges at least once. A recent report estimated that about one-third of students who enrolled in fall 2006 were enrolled elsewhere sometime in the next five years. The growth of the National Student Clearinghouse has made following students easier, but it is difficult to figure out how to split the credit for successful outcomes across the colleges that a given student attends.

(2)    While the group of students to be assessed (everyone!) sounds straightforward, most of the push has been to focus on the outcomes of graduates. This makes for a reasonable comparison group across colleges, but colleges have different graduation rates. It makes sense to focus on all students who entered a college, but this would lower the returns to college (and doesn’t fit well with selective colleges, where everyone is assumed to graduate).

(3)    Some people choose to postpone entry into the full-time labor market, whether for good reasons (such as starting a family) or for more dubious reasons (such as getting a master’s degree and working on a PhD). Given the lack of a federal data system, other students will not be observed if they move out-of-state to work.

Even with all of the limitations of measuring student outcomes once they leave college, I am heartened to see states starting to track the labor market outcomes of students who attended public colleges and stay in-state. This requires the merging of two data systems that don’t always exist in some states and don’t talk to each other in others—state higher education data systems and unemployment insurance (UI) records. Two states, Arkansas and Tennessee, just launched websites with labor market information for graduates from their public institutions of higher education. While the sample included is far from perfect, it still provides useful data to many students, families, and policymakers.

Not surprisingly, many in academia are worried about these new measures, as they prioritize one of the purposes of higher education (employment) at the expense of other important purposes (such as critical thinking and higher-order learning). The comments on this recent Chronicle of Higher Education article are worth a read. I am concerned about policymakers solely relying on these imperfect measures of student outcomes, but stakeholders should be able to have more information about the effectiveness of colleges on as many outcomes as possible.

Knowing Before You Go

Knowing Before You Go

The American Enterprise Institute today hosted a discussion of the Student Right to Know Before You Go Act, introduced by Senator Ron Wyden (D-OR) and co-sponsored by Senator Marco Rubio (R-FL). The two senators, both of whom are known for working across party lines, briefly discussed the legislation and were then followed by a panel of higher education experts. Video of the discussion will be available on AEI’s website shortly.

The goal of the legislation, as the senators discuss in a column in USA Today, is to provide more information about labor market and other important outcomes to students and their families. While labor market outcomes are rarely available in any systemic manner, this legislation would support states which release the data both at the school level and by academic programs. This sort of information cannot be collected at the federal level due to a restriction placed in Section 134 of the Higher Education Act reauthorization in 2008, which bans the Department of Education from having a student-level data system of the sort used in some states.

While nearly everyone across the political spectrum agrees that making additional data available is good for students and their families, there are certainly concerns about the proposed legislation. One concern is that the availability of employment data will make more rigorous accountability systems feasible, even though state-level data systems can only track students who stay within that state. This concern is shared by colleges, which tend to loathe regulation, and some conservatives, who don’t feel that the federal government should regulate higher education.

Additionally, measuring employment outcomes does place more of a focus on generating employment over some of the other goals of college (such as learning for learning’s sake). The security of these large unit-record datasets is also a concern of some people; I am less concerned about this given the difficulty of accessing deidentified data. (I’ve worked with the data from Florida, which has possibly the most advanced state-level data system. Getting access is extremely difficult.)

Although I certainly recognize those concerns, I strongly support this piece of legislation. It would reduce reporting requirements for colleges, since they would work primarily with states instead of the federal government. (In that respect, the legislation is quite conservative.) It makes more data available to all stakeholders in education and provides researchers with more opportunities to examine promising educational practices and intervention. Finally, it allows for states to make more informed decisions about how to allocate their scarce resources.

I don’t expect this legislation to go anywhere during this session of Congress, even with bipartisan support. Let’s see what happens next session, by which time I hope we are away from the “fiscal cliff.”

Analyzing UW-Madison’s Accountability Report

Recent legislative changes required the University of Wisconsin-Madison to submit an annual accountability report summarizing the university’s accomplishments over the previous year. While the UW System and UW-Madison already do a commendable job of making basic performance data public, this year’s accountability report nicely summarizes the performance data. A few highlights are below:

–The retention and graduation rates (for first-time, full-time students) are very high, as they should be given students’ academic and financial resources. Nearly 94% of students returned for a second year and 83% graduated within six years using the most recent data available. The retention and graduation rates are lower for targeted minority students (91% and 69%, respectively), but the gap is not nearly as large at UW-Madison as at many other universities.

–Just over half (52%) of all undergraduate students filed the FAFSA in 2011. Of these students, the median family income was just over $99,000. Given that most students who do not file the FAFSA and enroll in a selective college come from high-income families, the median family income of UW-Madison undergraduates is likely well in excess of $100,000 per year. This report does not include retention and graduation rates by Pell Grant receipt, but other UW-Madison data reports do.

–Roughly eight in ten students reported being able to enroll in desired classes most or all of the time (using data from the National Survey of Student Engagement). This is an improvement of roughly ten percentage points in the past five years, but more still needs to be done.

–ECON 101 (principles of microeconomics) was taken by 2,831 students in fall 2010 or spring 2011. That number makes me glad that I am no longer a TA for that course!

–UW-Madison claims an impact of $12.4 billion on the Wisconsin economy and creates or supports over 128,000 jobs. I am skeptical of those numbers, but the impact is clearly large. (But the question remains—what can we be doing better with our available funds?)

The accountability report for the rest of the UW System is available here.