Blog (Kelchen on Education)

Is the Term “College Ratings” Toxic?

In what will come as a surprise to few observers, much of the higher education community isn’t terribly fond of President Obama’s plan to develop a college ratings system for the 2015-16 academic year. An example of this is a recently released Inside Higher Ed/Gallup survey of college provosts and chief academic officers. Only a small percentage of the 829 individuals who returned surveys were supportive of the ratings and thought they would be effective, as shown below:

  • 12% of provosts agree the ratings will help families make better comparisons across institutions.
  • 12% of provosts agree the ratings will reflect their own college’s strengths.
  • Just 9% agree the ratings will accurately reflect their own college’s weakness.

There is some variation in support by type of college. Provosts at for-profit institutions and public research universities tended to offer more support, while those at private nonprofit institutions were almost unanimous in opposition. But regardless of whether provosts like the idea of ratings, the plan seems to be full steam ahead.

The Association of Public and Land-Grant Universities (APLU) took a productive step in the ratings conversation by releasing their own plan for accountability and cost-effectiveness. This plan centers on three components that could be used to allocate financial aid to colleges: risk-adjusted retention and graduation rates, employment/graduate degree rates, and default/loan repayment rates. Under APLU’s proposal, colleges could fall into one of three groups: a top tier that receives bonus Title IV funds, a middle tier that is held harmless, and a bottom tier that loses some or all Title IV funds.

To me, that sounds like a ratings system. But APLU took care not to call their plan a ratings system, and viewed the Administration’s plans as being “extremely difficult to structure.” It seems like the phrase “college ratings” has become a toxic idea; so rather than call for a simplified set of ratings, APLU discussed the use of “performance tiers.” This sounds a little like the Common Core debate in K-12 education, in which some states have considered renaming the standards in an attempt to reduce opposition.

It will be interesting to see how the discussion on college ratings moves forward over the next several weeks, particularly as more associations either offer their plans or decry the entire idea.  The technical ratings symposium previously scheduled for January 22 will now occur on February 6 on account of snow, and I’ll be presenting my thoughts on how to develop a ratings system for postsecondary education. I’ll post my presentation on this blog at that time.

Can Maintenance of Effort Programs Fund Public Higher Education?

The American Association of State Colleges and Universities released a policy paper this week calling for the federal government to enact (and fund) a program designed to encourage states to increase their support for public higher education. The AASCU brief rightly notes that per-student funding for public higher education has fallen over the past three decades (the magnitude of which is overstated somewhat due to their choice in inflation adjustments), and they propose a potential solution in the form of a maintenance of effort provision.

AASCU’s proposal would give colleges a partial match of their higher education appropriations, as long as per-FTE funding to institutions is higher than 50% of the value of the maximum Pell Grant and did not decline from the previous year’s value. The value of the matching funds would go up as state appropriations to institutions increased. They estimate that their hypothesized program would cost something in the neighborhood of $10-$15 billion per year, which could be paid for by cutting waste, fraud, and abuse in current financial aid systems (particularly among for-profits) and by implementing some sort of risk-sharing for student loans—which I’ve written on recently.

However, I view the plan as having a fatal flaw. By only including state appropriations to institutions in the calculation—and not requiring that the matching funds be spent on higher education—states can game the system to get additional money from the federal government. States could reduce funding to their financial aid programs and direct those funds toward institutional appropriations in order to get federal dollars, which could be used for K-12 education, healthcare, or tax cuts.

If states followed the incentive to eliminate all grant aid and fund institutions instead, tuition would likely decrease (something that AASCU institutions would appreciate). The most recent NASSGAP survey of state aid programs found that states spend $9.4 billion per year on grant aid, two-thirds of which is allocated based on financial need. Putting this money into state appropriations would cost the federal government several billion dollars, with no guarantees of any additional funding for students or institutions.

I have a hard time seeing Congress approving this maintenance of effort plan, regardless of the merits. Lobbyists for the private nonprofit and for-profit sectors are likely to strongly oppose this measure, as are lobbying groups for K-12 education, healthcare, and corrections spending (behind the scenes) since higher education is often cut at the expense of higher ed. In addition, this is likely to be a nonstarter in the House due to its placing restrictions on state priorities.

I’m glad to see this proposal from AASCU, but I don’t see it becoming law anytime soon. I would suggest that they follow up with some more details on their proposed risk-sharing program, as well as how elements of this plan could be incorporated into the Obama Administration’s proposed college ratings.

Should College Admissions be Randomized?

Sixty-nine percent of students who apply to Stanford University with perfect SAT scores are rejected. Let that sink in for a minute…getting a perfect SAT is far from easy. In 2013, the College Board reported that only 494 students out of over 1.6 million test-takers got a 2400. Stanford enrolled roughly 1700 students in their first-year class in 2012, so not everyone had a perfect SAT score. Indeed, the 25th percentile of SAT scores is 2080, with a 75th percentile of 2350, for the fall 2012 incoming class according to federal IPEDS data. But all of those scores are pretty darned high.

It is abundantly clear that elite institutions like Stanford can pick and choose from students with impeccable academic qualifications. The piece from the Stanford alumni magazine that noted the 69% rejection rate for perfect SAT scorers also noted the difficulty of shaping a freshman class from the embarrassment of riches. All students Stanford considers are likely to graduate from that institution—or any other college.

Given that admissions seem to be somewhat random anyway, some have suggested that elite colleges actually randomize their admissions processes by having students be selected at random conditional on meeting certain criteria. While the current approach provides certain benefits to colleges (most notably allowing colleges to shape certain types of diversity and guaranteeing spots to children of wealthy alumni), randomizing admissions can drastically cut down on the cost of running an admissions office and also reduces the ability of students and their families to complain about the outcome. (“Sorry, folks…you called heads and it came up tails.”)

As a researcher, I would love to see a college commit to randomizing most of all of its admissions process over a period of several years. The outcomes of these randomly accepted students should be compared to both the students who were qualified but randomly rejected and to the outcomes of the previous classes of students. My sense would be that the randomly accepted students would be roughly as successful as those students who were admitted under regular procedures in prior years.

Would any colleges like to volunteer a few incoming classes?

The College Ratings Suggestion Box is Open

The U.S. Department of Education is hard at work developing a Postsecondary Institution Ratings System (PIRS), that will rate colleges before the start of the 2015-16 academic year. In addition to a four-city listening tour in November 2013, ED is seeking public comments and technical expertise to help guide them through the process. The full details about what ED is seeking can be found on the Federal Register’s website, but the key questions for the public are the following:

(1) What types of measures should be used to rate colleges’ performance on access, affordability, and student outcomes? ED notes that they are interested in measures that are currently available, as well as ones that could be developed with additional data.

(2) How should all of the data be reduced into a set of ratings? This gets into concerns about what statistical weights should be assigned to each measure, as well as whether an institution’s score should be adjusted to account for the characteristics of its students. The issue of “risk adjusting” is a hot topic, as it helps broad-access institutions perform well on the ratings, but has also been accused of resulting in low standards in the K-12 world.

(3) What is the appropriate set of institutional comparisons? Should there be different metrics for community colleges versus research universities? And how should the data be displayed to students and policymakers?

The Department of Education has convened a technical panel on January 22 to grapple with these questions, and I will be among the presenters at that symposium. I would appreciate your thoughts on these questions (as well as the utility of federal college ratings in general), either in the comments section of this blog or via e-mail. I also encourage readers to submit their comments to regulations.gov by January 31.

Will Holding Colleges Accountable for Default Rates be Effective?

As student loan debt continues to climb and Congress enters a midterm election year, three Democrats in the United States Senate (Reed, Durbin, and Warren) recently introduced a piece of legislation designed to hold certain types of colleges and universities accountable for their students’ loan default rates. If enacted, the bill would require colleges to pay a fine of a percentage of its students’ total defaulted loans to the Department of Education, part of which would be used to help borrowers avoid future defaults and the other part would go to a fund to help support the Pell Grant in case of any future funding shortfalls.

The proposed fines are the following:

  • 5% fine if the most recent cohort default rate (CDR) over three years is 15-20%
  • 10% if CDR is 20-25%
  • 15% if CDR is 25-30%
  • 20% is CDR is 30%+

As an example of what these fines could mean, consider their potential implications for the University of Phoenix’s online division. Data from the Department of Education’s Integrated Postsecondary Education Data System (IPEDS) show that Phoenix collected roughly $1.4 billion in student loan revenue during the 2011-12 academic year, while 34.4% of students who took out loans defaulted in a three-year period. This default rate would place them in the 20% fine category, resulting in a fine of roughly $100 million per year based on an estimated $500 million per year in defaulted loans. This would represent roughly four percent of their total tuition revenue ($2.7 billion) in the 2011-12 academic year—which is far from a trivial sum.

Daniel Luzer on Washington Monthly’s College Guide blog (where many of my pieces are cross-posted) notes some of the potential positives of this legislation, including encouraging colleges to spend more time and energy counseling students and providing more information about financial aid.

But, in order for this legislation to actually benefit students, three things must happen:

(1) Some colleges must actually be affected by the legislation. The sanctions in the bill would not apply to community colleges, historically black colleges and universities (HBCUs), and likely other colleges designated as minority-serving institutions. This excludes a substantial number of nonprofit institutions, many of which have higher default rates. A provision in the bill excludes colleges at which fewer than 25% of students take out federal loans, which further diminishes the number of nonprofit institutions on the list.

But even if a college is not exempt from the legislation, it is still possible to avoid fines if default rates are over 15%. The legislation grants the Secretary of Education the authority to grant waivers, which would be the first time the Secretary has ever been granted that authority. (Kidding!) Colleges can submit remediation plans in order to avoid or reduce fines. It will be interesting to see the reaction to the first waiver request, as colleges’ lobbying efforts tend to be well-organized.

A more interesting case will involve the for-profit sector. Given the three Senators’ general distrust of for-profit institutions, it would not surprise me if nearly all of the colleges facing fines are proprietary in nature. But the way the bill is targeted seems to be similar to previous attempts at gainful employment legislation, which have been the subject of massive amounts of litigation. Expect this proposal to face litigation if it ever became law.

(2) Colleges must be able to improve their financial aid offices without restricting students’ access to financial aid. One of the underlying premises of this legislation is that financial aid offices are not helping students make sound financial decisions that help them complete college. Aid administrators would likely disagree with that statement, although additional resources targeted toward financial counseling may be beneficial.

Another concern is that in order to reduce default rates, aid offices will not offer students loans if they perceive the student as having a higher risk of default. While there is a prohibition written into the legislation against denying loans based on the perceived risk of default, this would be extremely difficult to prove and enforce. Colleges are not required to offer students the full amount of loans available in the initial aid package, and indeed some community colleges decline to offer any federal loans to their students. Some colleges would like more authority to limit loan offers to students, and this legislation could reduce access to credit for needy students.

(3) The legislation must adequately address students who transfer. If a student takes out loans while attending multiple institutions, would each college be held responsible for a student’s default—even if most of the debt was at one institution? Consider a student who attends a regional public university for one year and takes out the maximum in subsidized Stafford loans ($3,500). She then transfers to an expensive private college and accrues an additional $30,000 in debt before graduating. If she defaults on her principal of $33,500, should both colleges be held responsible? That is unclear at this point.

So would holding colleges accountable for default rates (in the method of this legislation) help students? I’m skeptical because I don’t see many colleges actually facing sanctions, nor do I see the fines being particularly effective. This is one of those ideas that is great in theory, but may not work as well in practice.

I don’t think this legislation is likely to become law in its current form, but it’s worth keeping an eye on as the Department of Education works to develop the Postsecondary Institution Rating System (PIRS). Many of the potential discussions this legislation raises will certainly come up again once the draft ratings are released.

Is This America’s Coolest College President?

With a few exceptions (such as the eternal leader Gordon Gee, now at West Virginia University), college presidents generally have a reputation for being a stoic, bland bunch of people. Given their job duties of managing a large business enterprise, raising funds, and dealing with often-cantankerous faculty members and students, college leaders rarely have a chance to have fun—and even more rarely show their sense of humor with the general public.

Troy Paino, the president of Truman State University in Kirksville, Missouri (my undergraduate alma mater) stands out from the crowd. Often known around campus as “T-Pain” since his name can be shortened to that of the famous rapper, he is not afraid to be a little goofy in front of the camera. He made a YouTube video over last winter break talking about how much he missed the students, as Kirksville gets a little quiet over breaks. The video was fairly popular, getting over 10,000 views due in part to Paino’s reaction to what ended up being a pack of squirrels.

http://www.youtube.com/watch?v=P01oMLrfqoc

Paino and crew decided to create another video this year, and this one has definitely gone viral with more than 30,000 views in the last week. Titled “T-Pain Misses You,” the video features Paino riding a toy tricycle around campus, measuring the height of the basketball hoops in Pershing Arena, and giving a heartfelt lip-sync rendition of a Miley Cyrus song in the campus radio studio (for more details on the video, see this nice article in the Kirksville Daily Express). This video has gotten coverage from the Huffington Post and may also be mentioned on Good Morning America.

http://www.youtube.com/watch?v=9e_CCOx6wnI&feature=player_embedded

I expect to see more colleges produce videos like this in an effort to create a buzz around their brands and to attract prospective students. But, for now, Truman (ranked third in the Washington Monthly master’s university rankings) can enjoy the attention gained by its cool—and utterly nerdy—president.

The 2013 Higher Education Not Top Ten List

Yesterday, I put out my top-ten list of higher education policy and finance issues from 2013. And today, I’m back with a list of not-top-ten events from the year (big thanks to Justin Chase Brown for inspiring me to write this post). These are events that left me shaking my head in disbelief or wondering how someone could fail so dramatically.

(Did I miss anything? Start the discussion below!)

10. Monsters University isn’t real. The higher education community was abuzz this summer with the premiere of Pixar’s newest movie about one of the few universities outside Fear Tech specializing in scaring studies. The Monsters University website is quite good, and as Jens Larson at U of Admissions Marketing notes, it’s hard to distinguish from many Title IV-participating institutions. I’ll use this blog post to announce my willingness to give a lecture or two at Monsters University. (As an aside, since the two main characters didn’t graduate, their post-college success may not help MU’s scores in a college rating system.)

9. Brent Musburger set men back at least five decades in the course of 30 seconds. His public ogling of the girlfriend of Alabama quarterback A.J. McCarron during January’s BCS championship game instantly became a YouTube sensation. Musburger shouldn’t have listened to his partner in The Waterboy, Dan Fouts, who urged him to not hold anything back in the last game of the season. McCarron, on the other hand, is preparing to play Oklahoma in the Sugar Bowl on January 2.

8. Rankings and ratings are not the same thing. While college leaders tend not to like the Obama Administration’s proposed Postsecondary Institution Rating System, it is important to emphasize the difference between rankings and ratings. Rankings assign unique values to each institution (like the college football or basketball polls), while ratings lump colleges into broad categories (think A-F grades). Maybe since I work on college rankings, I’m particularly annoyed by the confusion. In any case, it’s enough to make my list.

7. Mooooove over: The College Board has another rough year. This follows a rough 2012 for the publishers of the SAT, as more students took the ACT than the SAT for the first time last year. But in 2013, the redesign of the SAT got pushed back from 2015 to 2016, giving the ACT more time to gain market share. The College Board followed that up with a head-scratching example of “brand-ing,” passing out millions of cow stickers to students taking the PSAT. If these weren’t enough, the College Board also runs the CSS Profile, a supplemental (and not free) application for financial aid required by many expensive institutions. Rachel Fishman at New America has written extensively about the concerns of the Profile.

6. Gordon Gee is the most interesting man in higher education. The well-traveled university president began 2013 leading Ohio State University, but left the post this summer after his 2012 comments disparaging Notre Dame, Catholic priests, and the ability of the Southeastern Conference to read came to light. Yet, he and his large bowtie collection will be heading to West Virginia University this spring as he assumes the role of interim president. There is still no word if the Little Sisters of the Poor will show up on WVU’s 2014 football schedule.

5. Rate My Professor is a lousy measure of institutional teaching quality. I’m not going to fully dismiss Rate My Professor, as I do believe it can be correlated with an individual professor’s teaching quality. But a Yahoo! Finance piece claiming to have knowledge of the 25 colleges with the worst professors cross the boundaries of absurd. I quickly wrote a response to that piece, noting that controlling for a student’s grade and the difficulty of the course are essential in order to try to isolate teaching quality. This was by far my most-viewed blog of 2013.

4. Elizabeth Warren’s interest rate follies. The Democratic Senator from Massachusetts became even more of a progressive darling this spring when she announced a plan to tie student loan interest rates to the Federal Reserve’s overnight borrowing rate—0.75%. Unfortunately, this plan made no sense on several dimensions. While overnight borrowing has nearly no risk, student loans (over a ten-year period) have considerable risk. Additionally, if interest rates were set this low, money would have to come from somewhere else. I would much rather see the subsidy go upfront to students through larger Pell Grants than through lower interest payments after leaving college. Fortunately, Congress listened to smart people like Jason Delisle at New America and her plan went nowhere.

3. The Common Application fails early applicants. The Common Application, used by a substantial number of elite colleges, did not work for some students applying in October and November. The reason was that the Common App’s new software didn’t work and they failed to leave the previous version available in case of problems. Although this didn’t affect the vast majority of students who aspire to attend less-selective institutions, it certainly got the chattering classes talking.

2. The federal government shut down and budget games ensued all year long. The constant partisan battle culminated with a sixteen-day shutdown in October, bringing much of the Department of Education to a screeching halt. While the research community used Twitter to trade downloaded copies of IPEDS data and government reports, other disruptions were more substantial. 2013 also featured sequestration of some education spending, although it looks like the budget process might return to regular order for the next two years.

1. Georgetown Law finds a way to stick taxpayers with the entire cost of law school. It is no secret that law school is an expensive proposition, with six-figure debt burdens becoming the norm at many institutions. But some of the loans can be forgiven if students pursue public service careers for a decade, a program that was designed to help underpaid and overworked folks like public defenders or prosecuting attorneys.

Georgetown’s Loan Repayment Assistance Program advertises that “public interest borrowers might now pay a single penny on their loans—ever!” To do this, the law school increased tuition to cover the cost of 10 years’ worth of loan payments under income-based repayment for students making under $75,000 per year. Students take out Grad PLUS loans to fund this upfront, but never have to pay a dime of those loans back as Georgetown makes the payments. Jason Delisle and Alex Holt, who busted this scheme wide open this summer, estimate that students will have over $150,000 in loans forgiven—and put on the backs of taxpayers.  Although Georgetown tries to defend the practice as being good for society, it is extremely hard to make that argument.

Honorable mentions: #Karma, lousy attacks on performance-based funding research, financial stability of athletics at Rutgers and Maryland, and parking at 98% of campuses.

The Year of Higher Education Policy in Review

As 2013 draws to a close, it’s time to take a look back at some of the biggest happenings (or non-happenings) of the year. Some of these items would have been on the list for several years, but others (including the top happening of the year) are brand-new for 2013. Enjoy the list!

10. There is still some hope in the academic job market. In spite of continued concerns about the working conditions of adjuncts (as exemplified in the case of former Duquesne adjunct Margaret Mary Vojtko—read both the original op-ed and a thoughtful retelling of her life story), the tenure-track job market may just be springing back to life after a few lean years. I’m thankful to be one of those success stories, as I got a great job offer from Seton Hall University before defending my dissertation at the University of Wisconsin-Madison. (Look at my faculty webpage…I’m bona fide and I love my job!) But, in other disciplines, the rough market continues.

9. We heard more noise about reauthorizing the Higher Education Act, but no action. The HEA, which dates back to 1965, is supposed to be renewed in 2014. And Congress is saying all the right things about renewing the HEA, including holding a series of hearings on reforming the Pell Grant. However, it is hard to find anyone in academia or the policy community who thinks that is likely. After all, No Child Left Behind (the Elementary and Secondary Education Act) expired in 2007. If I had to put money on a reauthorization date, I would go for 2017.

8. The higher ed policy world gets RADDical. During late 2012 and early 2013, 17 organizations and teams released white papers as a part of the Gates Foundation-funded Redesigning Aid Design and Delivery (RADD) project. The recommendations of the groups ranged widely (see this nice summary from the National Association of Student Financial Aid Administrators, one of the participating organizations), but all groups suggested substantial changes from the status quo. It’s worth noting that the recommendation shared across the largest number of reports is stabilizing or increasing Pell funding, which could be a tough political lift in the current fiscal environment. This effort was not without its skeptics, as this well-commented Chronicle piece on the influence of Gates funds details.

7. The FAFSA changes to recognize same-sex parents, but is still complicated. Despite the push among many of the RADD grantees and at least some interest in Congress, the FAFSA ends 2013 as perhaps being more complicated than it was at the beginning of the year. This is because the venerable form changed to recognize the existence of same-sex marriages after this year’s Supreme Court ruling and political pressure before the ruling took place. The net result is that some students will see less aid. I would also be remiss if I didn’t mention my work with NASFAA on the feasibility of using prior prior year financial data to determine aid eligibility. That might get tied into the next HEA authorization.

6. Congress reached a reasonable solution on student loan interest rates. Put your shocked face on, folks—Congress did accomplish something without causing too much pain to students or financial aid offices. Interest rates on undergraduate subsidized Stafford loans were set to increase from 3.4% to 6.8% on July 1 (and actually did for a few weeks), leading to the hashtag #DontDoubleMyRate. The rates ended up being tied to 10-year Treasury notes, yielding a rate of under 4% this year; however, advocates note that the rate is likely to rise over time. Thankfully, Senator Warren’s plan to set interest rates based on the Federal Reserve discount window (which is nearly riskless) never received serious discussion.

5. MOOCs expand, but their outcomes are questioned. Massive open online courses (MOOCs) are seen by some as having the potential to change how higher education is delivered, but it is safe to say that not all faculty support them—as evidenced at San Jose State. MOOCs have also been hammered for low completion rates, which are often below 10%. The always-astute Kevin Carey notes, however, that the low completion rates are partially due to people who sign up for the course but never really attempt to complete them. Additionally, large numbers of students may still be completing the course, even if completion rates are low. This issue will only get hotter during 2014.

4. Student loan debt grows amid possible reforms. The Institute for College Access and Success (TICAS) recently put out its annual report on student debt loads—and the results aren’t pretty. The average debt load of graduates was $29,400 in 2012, and 71% of students took out debt. (Even more concerning is the fact that TICAS can’t even get data on a lot of colleges’ graduates.) Increasing debt loads have led to innovative plans to make college more affordable. The most-discussed plan is Oregon’s Pay it Forward proposal, which would be a type of income-based repayment covering tuition and fees in that state. While I have serious concerns about whether the program could work (but think it’s worth a demonstration program), my dear friend and dissertation mentor Sara Goldrick-Rab makes her opposition clear.

3. One of the nation’s more prominent community colleges might actually lose its accreditation. The City College of San Francisco is currently slated to lose its accreditation next summer if they do not meet 357 goals set by the Accrediting Commission for Junior and Community Colleges. Since students cannot qualify for federal Title IV financial aid if they attend an unaccredited college, this would effectively shut down an institution that had nearly 100,000 students. Students and faculty went after the accreditor and nearly shut it down, although it was recently announced that the accreditor could operate for another year. I still think that CCSF will keep its accreditation, but the damage (in terms of enrollment) may already be done.

2. Gainful employment continues to be a hot political topic. The Obama Administration proposed gainful employment regulations several years ago, in which vocationally-oriented colleges would lose Title IV eligibility if they had poor employment and loan repayment outcomes. These rules have been in and out of court for several years, and a new set is now being developed. The Department of Education tried to reach consensus with stakeholders last week, but failed; this means that ED will write its own rules. For all the developments that will happen in 2014, I’ll defer you to Ben Miller’s great work covering the topic.

1. PIRS roars to the public’s attention, and colleges are not happy. As regular readers of this blog know, I’m the methodologist for Washington Monthly’s annual college rankings. Yet I was completely floored when President Obama announced the impending development of a college ratings system for the 2014-15 academic year. (The official title—Postsecondary Institution Rating System or PIRS—just got released yesterday.) Thankfully, I was able to recover quickly enough to go on MSNBC the next night to talk about the proposal.

The Department of Education has done a lot of listening on the college ratings proposal, and the vast majority of the feedback in the higher education community appears to be negative. A recently released poll of college presidents highlights the opposition amid concerns of the ratings favoring highly selective institutions. (Yet the only measure that a majority of college presidents supported using was graduation rates—a measure strongly tied to selectivity.) This recent conference panel also shows some of the issues facing the ratings.

While the long-term goal is to tie ratings to financial aid by 2018 or so, I don’t see this as being likely to happen given its requirement of Congressional approval. However, the ratings could potentially help students even if institutions don’t like the bright lights of accountability. Let’s just say that the discussion around the release of the first ratings this summer should be spicy.

I’ll post a not-top-ten list of higher education policy issues later this week. Send me your suggestions for that piece, and let me know what you think of this list!

Don’t Dismiss Performance Based Funding Research

Performance-based funding (PBF), in which at least a small portion of state higher education appropriations are tied to outcomes, is a hot political topic in many states. According to the National Conference of State Legislatures and work by Janice Friedel and others, 22 states have PBF in place, seven more are transitioning to PBF, and ten more have discussed a switch.

The theory of PBF is simple: if colleges are incentivized to focus on improving student retention and graduation rates, they will redirect effort and funds from other areas to do so. PBF should work if two conditions hold:

(1) Colleges must currently be using their resources in ways that do not strongly correlate with student success, a point of contention with many institutions. If colleges are already operating in a way that maximizes student success, then PBF will not have an impact. PBF could also have negative effects if colleges end up using resources less effectively than they currently are.

(2) The expected funding tied to performance must be larger than the expected cost of changing institutional practices. Most state PBF systems currently tie small amounts of state appropriations to outcomes, which could result in the cost of making changes smaller than the benefits. Colleges also need to be convinced that PBF systems will be around for the long run instead of until the next governor ends the plan or state budget crises cut any funds for PBF. Otherwise, they may choose to wait out the current PBF system and not make any changes. Research by Kevin Dougherty and colleagues through the Community College Research Center highlights the unstable nature of many PBF systems.

For these reasons, the expected impacts of state PBF plans on student outcomes may not be positive. A recent WISCAPE policy brief by David Tandberg, an assistant professor at Florida State University, and Nicholas Hillman, an assistant professor at the University of Wisconsin-Madison, examines whether PBF plans appear to affect the number of associate’s and bachelor’s degrees awarded by institutions in affected states. Their primary findings are that although some states had significantly significant gains in degrees awarded (four at the four-year level and four at the two-year level), other states had significant declines (four at the four-year level and five at the two-year level). Moreover, PBF was most effective in inducing additional degree completions in states with long-running programs.

The general consensus in the research community is that more work needs to be done to understand the effects of state performance-based funding policies on student outcomes. PBF policies differ considerably by state, and it is too early to evaluate the impact of policies on states that have recently adopted the systems.

For these reasons, I was particularly excited to read the Inside Higher Ed piece by Nancy Shulock and Martha Snyder entitled, “Don’t Dismiss Performance Funding,” in response to Tandberg and Hillman’s policy brief. Shulock and Snyder are well-known individuals in the policy community and work for groups with significant PBF experience. However, their one-sided look at the research and cavalier assumptions about the authors’ motives upset me to the point that writing this response became necessary.

First of all, ad hominem attacks about the motives of well-respected researchers should never be a part of a published piece, regardless of the audience. Shulock and Snyder’s reference to the authors’ “surprising lack of curiosity about their own findings” is both an unfair personal attack and untrue. Tandberg and Hillman not only talk about the eight states with some positive impacts, they also discuss the nine states with negative impacts and a larger number of states with no statistically significant effects. Yet Shulock and Snyder do not bother mentioning the states with negative effects in their piece.

Shulock and Snyder are quite willing to attack Tandberg and Hillman for a perceived lack of complexity in their statistical model, particularly regarding their lack of controls for “realism and complexities.” In the academic community, criticisms like this are usually followed up with suggestions on how to improve the model given available data. Yet they fail to do so.

It is also unusual to see a short policy brief like this receive such a great degree of criticism, particularly when the findings are null, the methodology is not under serious question, and the authors are assistant professors. As a new assistant professor myself, I hope that this sort of criticism does not deter untenured faculty and graduate students from pursuing research in policy-relevant fields.

I teach higher education finance to graduate students, and one of the topics this semester was performance-based funding and accountability policy. If Shulock and Snyder submitted their essay for my class, I would ask for a series of revisions before the end of the semester. They need to provide empirical evidence in support of their position and to accurately describe the work done by Tandberg and Hillman. They deserve to have their research fairly characterized in the public sphere.