Using Student Evaluations in SoTL Research

Anyone who has taught in higher ed knows that the end of the semester is often a mad dash to the finish line. There are emails from concerned students, exam instructions to give out, and final projects to grade. In the wake of all of this excitement, it can be easy to treat Student Evaluations of Teaching (SETs) as an afterthought at best, a nuisance at worst. After all, student evaluations, which vary throughout programs and colleges, are sometime flawed instruments that lead faculty to resist their use. For example, this 2017 study published in Studies in Educational Evaluation affirmed what most instructors have been saying for years: a high student rating does not equate to high student learning. Nevertheless, administrators still see and often misuse SETs as a key indicator of teacher effectiveness. As a result, teachers, especially adjunct and ‘tenure exempt’ instructors, are partially dependent on good evaluations for renewal of contracts, merit raises, hiring, and other professional advancement.

Given faculty resistance and the questionable value of SETS, why bring them up in a discussion of  the Scholarship of Teaching and Learning (SoTL)? While SETs may be flawed, my goal is to argue the merits of SETs when applied appropriately to SoTL research. Evaluations can be very helpful in SoTL because they are collected and calculated anonymously, thus faculty often do not have to get approval for human subjects research (however, it is always a good idea to check with your institution first). Aside from the convenience factor, SETs can be great for answering certain SoTL research questions.

What types of research questions can be addressed using student evaluations of teaching?

If your research question is something like,  “What did students learn?” “How much did X lesson improve students understanding of Y learning objective?” or anything else relating to demonstrable student learning, stop reading, close this tab, and go about your day. Although there have been studies that show a moderate positive correlation between SETs and student learning; correlation is not causation. We cannot use high student evaluations as evidence that students learned a lot from our classes. Bummer.

However, if your research question is something like, “What did students think about X instructional method?”, “How much time did students think they spent preparing for class?”, “How much did students think they learned?” or anything else about student perceptions, SETs may be able to help you out. SETs can help you determine how your students felt about the course as a whole, on a personal level.

The term “as a whole” is important here because SETs ask students to think about the course from beginning to end. This makes things a bit more difficult if you want information on a course component, such as a new policy, assignment, or teaching method that you tried. Many institutions give faculty the option of adding supplementary questions, which can be helpful in this regard.

A 2011 study from Mt. Royal University in Canada offers several examples of the effective use of SETs in SoTL. Many involve an iterative course design process where SETs are used to compare one semester to another. In one example, an instructor noticed low scores for “The instructor effectively relates the subject matter to areas of study beyond the course.” This caused her to make changes to her class while continuing to monitor her SET score for that question. In this case the question was not about whether students learned material that was relevant to other areas of life, but whether they felt that the material was relevant to other areas. An important distinction.

How can SoTL scholars use SET to strengthen assessment findings?

If you are already developing a SoTL project that incorporates assessment of student learning, consider using SETs to supplement findings by addressing  student perceptions. For example, SoTL research may demonstrate that a certain active learning teaching method increases student learning, but had an adverse effect on SETs. If your readers are among the many instructors whose SET scores are necessary to professional advancement, they need to know this potential outcome.

Likewise, SETs can be used to increase the faculty’s effectiveness. For example, one instructor from the Mt. Royal report tracked SET answers to “The instructor communicates course content in ways that help me learn” alongside evidence to measure student learning. Over the next few semesters, she continued to make improvements to her assignment instructions that addressed problems with students feeling confused about her new pedagogies.  

How to make the most of SETs in your SoTL research? 

If you are considering incorporating SETs results, rely on best practices for administering these evaluations to ensure useful student responses. Here are some tips to keep in mind:

• Give students time in the beginning of class when they don’t have pending deadlines so they can concentrate on their comments and responses—even if your school uses online evaluation tools.
• Inform students ahead of time that you will be doing the evaluatios on a certain day. This will give them time to think about what they want to say.
• Take advantage of opportunities to “write your own” question(s) on standard evaluations. Preview the questions with some peers or even some students in order to check for validity- is the question asking what you want it to know?
• If you have permissions, create your own SET, in addition to the official one. This can be helpful if your institution takes a while to get results to you.
• Consider using a midterm SET to check in on student experiences. Add to your research or use it to refine ongoing experiments.
• Reach out to your institution’s Center for Teaching and Learning. It is possible that they will have helpful resources and supplemental SETs that can better serve students and your research goals.

In closing, before you write off SETs as one more thing to do at the end of the semester, consider how they might contribute to your SoTL research.  As long as faculty recognize their limitations and the types of insight they may provide, SETS can be a useful tool to address research questions.