Analyzing UW-Madison’s Accountability Report

Recent legislative changes required the University of Wisconsin-Madison to submit an annual accountability report summarizing the university’s accomplishments over the previous year. While the UW System and UW-Madison already do a commendable job of making basic performance data public, this year’s accountability report nicely summarizes the performance data. A few highlights are below:

–The retention and graduation rates (for first-time, full-time students) are very high, as they should be given students’ academic and financial resources. Nearly 94% of students returned for a second year and 83% graduated within six years using the most recent data available. The retention and graduation rates are lower for targeted minority students (91% and 69%, respectively), but the gap is not nearly as large at UW-Madison as at many other universities.

–Just over half (52%) of all undergraduate students filed the FAFSA in 2011. Of these students, the median family income was just over $99,000. Given that most students who do not file the FAFSA and enroll in a selective college come from high-income families, the median family income of UW-Madison undergraduates is likely well in excess of $100,000 per year. This report does not include retention and graduation rates by Pell Grant receipt, but other UW-Madison data reports do.

–Roughly eight in ten students reported being able to enroll in desired classes most or all of the time (using data from the National Survey of Student Engagement). This is an improvement of roughly ten percentage points in the past five years, but more still needs to be done.

–ECON 101 (principles of microeconomics) was taken by 2,831 students in fall 2010 or spring 2011. That number makes me glad that I am no longer a TA for that course!

–UW-Madison claims an impact of $12.4 billion on the Wisconsin economy and creates or supports over 128,000 jobs. I am skeptical of those numbers, but the impact is clearly large. (But the question remains—what can we be doing better with our available funds?)

The accountability report for the rest of the UW System is available here.

Analyzing Wisconsin’s Workforce Development Report

Some people in Wisconsin, particularly in the business community, feel that the state’s secondary and postsecondary education systems do not efficiently prepare students for the types of jobs available in the local labor market. To address these concerns, Tim Sullivan, special consultant to Governor Walker on Economic, Workforce, and Education Development, compiled a report suggesting changes to the state’s educational systems and policies with the goal of improving Wisconsin’s workforce development programs.

I am pleased to see a clear plan on how the Walker administration may wish to move forward with economic development and education policy. Many of the suggestions are reasonable and can be implemented in a tough fiscal climate, but I do encourage careful evaluation of the policies and the consideration of unintended consequences. As an economist of education, I am focusing on the pieces of the report directly pertaining to Wisconsin’s postsecondary education systems instead of the recommendations regarding tax policy, unemployment insurance, and immigration reform. While these policies may tangentially affect higher education, they are not my area of expertise. They are also much less likely to be adopted than the core educational recommendations.

The first policy recommendation that I was expecting to see was that Wisconsin’s data from K-12 and postsecondary education should be linked with unemployment insurance data to examine the labor market outcomes of students by secondary and postsecondary institutions and programs. This has been done in other states, most notably Florida, with at least a modest amount of success. However, this long-overdue policy change was not a part of the report. The call for increased usage of the Labor Market Information software does provide useful information to students and institutions, but is less effective from a policy perspective.

The report is right on to note that Wisconsin has a large number of entities offering workforce development services. To the extent that it is feasible, this should be consolidated into one office. (I would also suggest that the Council for Workforce Readiness and a College and Workforce Readiness Council, both chaired by Mr. Sullivan, be consolidated.)The recommendation that there be a common core of transferrable courses across WTCS and UW System should also be implemented, as it has the potential to give students peace of mind and potentially reduce costs.

I am optimistic about the recommendations regarding evidence-based budgeting and performance-based funding. Both of these proposals have the potential to use the Wisconsin Idea, drawing upon the expertise of the higher education community to provide digestible data to decision makers in the state. These also can drive a cost-effectiveness agenda, which is essential to keeping public support for higher education funding. As someone who does research on cost-adjusted performance measures in higher education, I am glad to see the attention paid to some of the drawbacks of performance-based funding; however, I urge policymakers to consider ways in which colleges can “game the system” to increase graduation and job placement rates by accepting only the best students. Funding colleges in part based on outcomes fits in well with the call for stackable credentials, which provide good measurement points.

There are several recommendations in the report that are of concern to me, as they have the potential to reallocate state resources in less than optimal ways. The first area of concern is the call for UW System to guarantee that students can graduate in four years if certain conditions are met by the student (such as taking a sufficient number of classes per semester). This type of program can work well in certain situations; for example, many students at UW-Madison cannot get into intermediate microeconomics—a prerequisite for courses in several majors—due to not enough faculty members being assigned to teach the course. If this recommendation is to become policy, I strongly encourage the creation of a set of clear administrative rules so students and colleges can operate with a degree of certainty. The University of Minnesota’s four-year graduation agreement is a good starting point.

The focus on four-year graduation rates is not appropriate for all students or institutions. If a student comes to college without any prior credits, enrolls full-time (12 credits per semester), and works during the summer without taking any classes, he or she will take five years to accumulate 120 credits. However, if students enter with prior credits, graduating in four years while maintaining a good GPA is much more feasible. Policymakers should focus on making sure students who begin the semester enrolled full-time complete at least 12 credits per semester and make satisfactory progress toward a degree.

I am skeptical that making students with a bachelor’s degree pay a higher tuition rate to attend a WTCS institution will be cost-effective in the long run. Although there are currently no available data to examine this question, I would expect the majority of these individuals to have been in the workforce for at least a decade and have lost their job prior to returning to college. It is likely the case that paying a higher tuition rate would induce some of these students not to enroll and earn a substantially lower income. It may be a better investment for the state to foot more of the bill if the additional tax revenue exceeds the tuition subsidy. I would encourage a thorough analysis of these students and their circumstances before making a decision.

The recommendation that the WHEG be made available to part-time students in WTCS will at best do little to help increase the skill sets of Wisconsin workers. The WHEG currently has two separate pots of money—one for UW System students and one for WTCS students. The WTCS pool of money is consistently exhausted well before the start of the semester, which is when part-time students are more likely to enroll. This means that no money will be available to fund the program expansion unless it is taken from full-time WTCS or UW System students. These students do still receive Pell Grants, which are often sufficient to cover the cost of tuition for the neediest individuals. For these reasons, extending the WHEG to part-time students, many of whom are working adults, is unlikely to increase enrollment and completion rates.

In order to provide the WHEG to more students in WTCS, whether part-time of full-time, I would suggest scaling back or ending the Wisconsin Postsecondary Education Credit. This nonrefundable tax credit to the employer is less likely to be as effective as a voucher directly given to the student because not all businesses are willing to heavily invest in an employee who may not stay with the company for long enough to recoup the investment. Additionally, the nonrefundable nature of the credit means that only certain businesses are willing to make the investment—those which are highly profitable and/or have relatively small amounts of depreciation or other deductible expenses. I would also recommend dropping the distinction between UW System and WTCS students in the WHEG funding pool; this also streamlines the administrative process.

The report highlights the sizable minority of students with less than a bachelor’s degree who make more money than those with a bachelor’s degree. However, it is unclear whether we can identify these students and guide them toward the path which is more likely to have a higher expected salary. The Academic and Career Plan may be able to help with this somewhat, but this also requires K-12 teachers and guidance counselors to understand a student’s strengths and weaknesses. We do not have any empirical research as to the effectiveness of the ACP program, but it could certainly work well under the right circumstances.

I am not concerned that certain WTCS campuses spend money on GED preparation and liberal arts/transfer courses. Someone has to spend the money on GED preparation, and although the argument could certainly be made that high schools should pay for the coursework, it is easier from the viewpoints of technical expertise and administrative burden to offer the courses through the technical colleges. While the UW Colleges offer the majority of liberal arts/transfer courses at the two-year level, the three WTCS campuses which have liberal arts articulation agreements (Madison, Milwaukee, and Nicolet) are not near a UW College which offers the same services. This means that services are not being duplicated to as great of an extent as feared.

This report is far from perfect, but it does raise important questions about Wisconsin’s workforce development programs and proposes some feasible, implementable policies to address the concerns. I hope that the K-12 and higher education communities and the Walker administration can work together to improve the plan and find at least some common ground.

Estimating College Effectiveness: The 2012 Washington Monthly College Rankings

One of my primary research interests is examining whether and how we can estimate the extent to which colleges are doing a good job educating their students. Estimating institutional effectiveness in K-12 education is difficult, even with the ready availability of standardized tests across multiple grades. But in higher education, we do not even have these imperfect measures across a wide range of institutions.

Students, their families, and policymakers have been left to grasp at straws as they try to make important financial and educational decisions. The most visible set of college performance measures is the wide array of college rankings, with the ubiquitous U.S. News and World Report college rankings being the most influential in the rankings market.  However, these rankings serve much more as a measure of a college’s resources and selectivity instead of whether students actually benefit.

I have been working with Doug Harris, who is now an associate professor of economics at Tulane University, on developing a model for estimating a college’s value-added to students. Unfortunately, due to data limitations, we can only estimate value added with respect to graduation rates—the only outcome available for nearly all colleges. To do this, we estimate a college’s predicted graduation rate given certain student and fixed institutional characteristics and then compare that to the actual graduation rate.

This measure of college value-added gives us an idea of whether a college graduates as many students as would be expected, but this does not address whether colleges are educating students in an efficient manner. To estimate a college’s cost-effectiveness for students and their families, we divide the above value-added measure by the net price of attendance (defined as the cost of attendance minus all grant and scholarship aid). Colleges can then be ranked on this cost-effectiveness measure, which yields much different results than the U.S. News rankings.

This research has attracted attention in policy circles, which has been a great experience for me. I thought that at least some people would be interested in this way of looking at college performance, but I was quite flattered to get an e-mail from Washington Monthly magazine asking if I would be their methodologist for the 2012 college rankings. I have always appreciated their rankings, as the focus is much more on estimating educational effectiveness and public service instead of the quality of the incoming student body.

The 2012 college rankings are now available online and include a somewhat simplified version of our cost-adjusted value-added measure. This measure consists of one-half of the social mobility score, which is one-third of the overall rankings (research and service are the other two components). The college guide includes an article I wrote with Rachel Fishman of the New America Foundation looking at how some colleges provide students with good “bang for the buck.”

I am excited to see some of my research put into practice, and hopefully at least some people will find it useful. I welcome any questions or comments about the rankings and methodology!

Hello world!

Hello! My name is Robert Kelchen and I am a doctoral candidate in the Department of Educational Policy Studies at the University of Wisconsin-Madison. I am interested in higher education policy, particularly cost-effectiveness, financial aid, and data concerns. I plan to update this blog on a fairly regular basis as I go on the job market this year and beyond, so stay tuned for updates. And I always appreciate possible suggestions for blog topics!

–Robert