More on Rate My Professors and the Worst Universities List

It turns out that writing on the issue of whether Rate My Professors should be used to rank colleges is a popular topic. My previous blog post on the topic, in which I discuss why the website shouldn’t be used as a measure of teaching quality, was by far the most-viewed post that I’ve ever written and got picked up by other media outlets. I’m briefly returning to the topic to acknowledge a wonderful (albeit late) statement released by the Center for College Affordability and Productivity, the data source which compiled the Rate My Professors (RMP) data for Forbes.

The CCAP’s statement notes that the RMP data should only be considered as a measure of student satisfaction and not a measure of teaching quality. This is a much more reasonable interpretation, given the documented correlation between official course evaluations and RMP data—it’s also no secret that certain disciplines receive lower student evaluations regardless of teaching quality. The previous CBS MoneyWatch list should be interpreted as a list of schools with the least satisfied students before controlling for academic rigor or major fields, but that doesn’t make for as spicy of a headline.

Kudos to the CCAP for calling out CBS regarding its misinterpretation of the RMP data. Although I think that it is useful for colleges to document student satisfaction, this measure should not be interpreted as a measure of instructional quality—let alone student learning.

How Not to Rate the Worst Professors

I was surprised to come across an article from Yahoo! Finance claiming knowledge of the “25 Universities with the Worst Professors.” (Maybe I shouldn’t have been surprised, but that is another discussion for another day.) The top 25 list includes many technology and engineering-oriented institutions, as well as liberal arts colleges. I am particularly amused by the inclusion of my alma mater (Truman State University) at number 21, as well as my new institution starting next fall (Seton Hall University) at number 16. Additionally, 11 of the 25 universities are located in the Midwest, with none in the South.

This unusual distribution immediately led me to examine the methodology of the list, which comes from Forbes and CCAP’s annual college rankings. The worst professors list is based on Rate My Professor, a website which allows students to rate their instructors on a variety of characteristics. For the rankings, a mix of the helpfulness and clarity measures is used in conjunction with partially controlling for a professor’s “easiness.”

I understand their rationale for using Rate My Professor, as it’s the only widespread source of information about faculty teaching performance. I’m not opposed to using Rate My Professor as part of this measure, but controlling for grades received and the course’s home discipline is essential. At many universities, science and engineering courses have much lower average grades, which may influence students’ perceptions of the professor. The same is true at certain liberal arts colleges.

The course’s home discipline is currently in the Rate My Professor data, and I recommend that Forbes and CCAP weight results by discipline in order to more accurately make comparisons across institutions. I would also push them to aggregate a representative sample of comments for each institution, so students can learn more about what students think beyond a Likert score.

Student course evaluations are not going away (much to the chagrin of some faculty members), and they may be used in institutional accountability systems as well as a very small part of the tenure and promotion process. But like many of the larger college rankings, Forbes/CCAP’s work results in at best an incomplete and at worst a biased comparison of colleges. (And I promise that I will work hard on my helpfulness and clarity measures next fall!)

All Quiet on the Blogging Front

This blog has been fairly quiet through the month of April, a notable difference from my goal of writing about two posts per week. While I greatly enjoy being able to write my thoughts on timely issues in the higher education world, there are times when my day job doesn’t readily allow for time necessary to think through and write a post—let alone keep up with the news. But I do want to take a few minutes to share the reasons why I’ve been so busy, as well as why May will likely be a fairly slow month on this blog.

First of all, I’m preparing to defend my dissertation (three essays on higher education policy) toward the end of next week. The last few weeks have been fairly frantic as I’ve made substantial changes to two chapters before I sent them to my committee last week. Although there will certainly be a lot of changes required after my defense, it feels great to be ready to defend. I will be happy to share the dissertation chapters with anyone who is interested after final revisions have been made.

At the end of this week, I am flying to California to give a presentation at the annual Education Writers’ Association seminar at Stanford University. I was asked to give a talk on my research in the area of input-adjusted metrics in measuring institutional effectiveness, and particularly how adjusting for cost changes the ordering of institutions. This talk will be in front of a large group of journalists who cover education on a regular basis, which is a neat opportunity.

Finally, on the teaching front, I am giving my final lecture of the semester tomorrow on accountability and performance measures to a mixed undergraduate/grad student class on debates in higher education policy. I’ve really enjoyed giving several previous lectures, and this one has particular meaning to me as it is something that is both very policy-relevant and fun to teach.

I hope to get a post or two up sometime in the next two weeks, so please send along any ideas that you would like for me to explore in future posts. Until then, it’s back to the fun world of cleaning and coding administrative datasets!

Should Students Take a Common Summer MOOC?

For many years, a substantial number of colleges have asked their incoming students to all read the same book as a part of student orientation. For example, the University of Wisconsin-Madison’s Go Big Read program asked students last year to read “Radioactive: A Tale of Love and Fallout” by Lauren Redniss. And when I was a freshman in college at Truman State University, my common read was “The Souls of Black Folk” by W.E.B. Du Bois.

While I enjoyed my common reading experience, I have to wonder if alternative methods could be used to heighten student engagement. The great amount of recent discussion about MOOCs (massively open online courses) leads me to think that a small number of innovative colleges should assign their students a common MOOC instead of a common book. While a MOOC may be more work than reading a book, it has the potential to better prepare students for their future college experiences.

A common MOOC should have the same properties of a successful common read. It must be accessible to the typical student, yet be challenging enough to stimulate discussion and get students acclimated to college-level coursework. It should also reach students across a large number of majors and interests. While colleges may want to develop their own MOOC for this purpose, here are a few courses which could stimulate interesting discussion:

Generating the Wealth of Nations

Maps and the Geospatial Revolution

TechniCity” (how cities are changing)

While I hold no great hopes that MOOCs will completely transform higher education, I think the technology can help at least some students. And a common summer MOOC may be one way to do so, assuming issues of Internet accessibility can be addressed.

On MOOCs and Money

Massively open online courses (MOOCs) have become the newest fashionable trend in higher education. These courses, which are open to anyone and can cover anything from songwriting to combinatorial game theory (which sounds both fun and exceedingly challenging), have begun to gain recognition among policymakers and the public alike. The American Council on Education announced that five MOOCs from Coursera would be recommended for college credit, hastening the move into the technology.

The University of Wisconsin-Madison announced last night that its faculty would offer four MOOCs in conjunction with Coursera as a part of the university’s Educational Innovations program. While the MOOCs (in video games and learning, higher education globalization, evolution, and economics/finance) would not currently be offered for credit, the potential certainly exists for credit in the future. These courses could be a part of the Flexible Option program through the University of Wisconsin System, which has gained additional support in Governor Walker’s new budget.

With all of the potential promise of MOOCS to help students get access to higher education, there are still many concerns to be addressed. Coursera recently had to call off a MOOC on the fundamentals of online education, as the technology wasn’t ready for 40,000 students. Issues of access are also important with MOOCs, as they appeal to students who prefer more independent learning and are ready to handle that sort of delivery concern. We also know little about whether MOOCs are effective in promoting student learning, both compared to an in-person class or even to nothing at all.

From a university’s perspective, MOOCs present both promises and pitfalls. If a university can develop a successful MOOC (in the sense of gaining public support), it is a potential way to increase funding through either state appropriations (as could be the case in Wisconsin) or donations from satisfied students. If a few entry-level courses (such as UC-Irvine’s offerings of algebra and pre-calculus) were to be offered for credit, it could serve as a potential way for students to select colleges or familiarize themselves with higher-level coursework.

Current university students (and faculty) should be concerned about where the faculty time for developing these MOOCs comes from. Many faculty at research universities (the ones who are likely to develop MOOCs) are teaching one or two courses per semester in an environment where teaching is valued less than research. These courses would have to be developed on a professor’s own time or count as part of the service component—it is essential that teaching loads not be reduced for developing MOOCs unless the university is somehow compensated. An option for compensation is to have foundations help fund initial course development in the form of faculty buyouts.

I am glad that the University of Wisconsin-Madison is starting small with MOOCs, as these courses have potential to help improve student learning on the margin at this point in time. If a college can develop a MOOC for an entry-level class that can be cost-effective, I’ll be a lot more optimistic. But for right now, it’s a neat way to see new research in specialized fields and could certainly be a way for advanced undergraduate students to take an “elective” course in their field of study. I am waiting for more research before fully jumping on the MOOC bandwagon.

Back in the Classroom Again

A lot of things have happened since the spring of 2008—I’ve earned a master’s degree in economics and nearly completed a PhD in education policy, have spent thousands of hours staring at the black and then white backgrounds of Stata, and have been fortunate enough to work with many brilliant scholars and researchers on important policy issues. But I haven’t been in front of a classroom of students since May of 2008, when I completed a year of being a teaching assistant for principles of microeconomics classes. (In the meantime, I have continued to work with undergraduate and graduate students on a one-on-one or small group basis.)

This spring, I have the opportunity to be a teaching assistant for Sara Goldrick-Rab’s class on issues and debates in higher education policy. In this class, I will be giving at least one of the weekly lectures in addition to meeting with individual students while gaining just a small amount of familiarity with the departmental copy machine. This class also gives me the opportunity to think more about possible course preparations for my (hopefully) impending career as a faculty member and how I would advise undergraduate and graduate students with an interest in education.

My teaching philosophy is fairly straightforward, with a goal of helping students get the “so what” of the course material. For the majority of students who will not go on to careers in my fields of interest (higher education policy and challenges in conducting quantitative research in this area), the primary goal of my teaching should be to emphasize why it is important to understand the topics at hand rather than becoming experts in all of the literature and related terminology. Students can become experts in repeating the key points of the day’s readings (I’ve been guilty of that in the past as well), but this doesn’t help them in the long run.

Hopefully, I will be teaching a class or two of my own this fall as I set off on my own academic career. But as I start my twelfth and final semester of graduate school, the opportunity to get back in the teaching mindset among a group of stellar students is quite welcome.