Research delves into issues when trying to gauge instructional and design quality in MOOCs for credit.
[Editor’s note: We originally reported that QM was under iNACOL, though it is a separate set of standards.]
As more institutions consider offering MOOCs for credit, often the MOOCs provided by third-party platforms, researchers say it’s imperative to gauge instructional and design effectiveness…but how, and with what quality standards?
These are the main questions posited by Patrick Lowenthal, assistant professor at the Educational Technology College of Education at Boise State University; and Dr. Charles Hodges, associate professor of Leadership, Technology & Human Development at Georgia Southern University, in their research study on trying to measure the quality of MOOCs.
The researchers explain that as institutions look to third-party (Coursera, edX, Udacity, etc.) provided MOOCs for credit, it’s important to investigate whether these MOOCs meet certain standards of quality. The problem is, very little research has been done to investigate the instructional quality of MOOCs.
“Due to this problem, we decided to investigate the design of MOOCs as determined by certain, accepted online course quality frameworks,” write the authors in their study. “We began this study with an assumption that MOOCs, just like formal online courses, are not inherently good or bad. Further, there are likely some things that members of the academy can learn (both good and bad) from analyzing MOOCs.”
Where to Start?
Though there are many quality assurance programs for online courses, such as California State University Chico’s, or the Online Learning Consortium’s (OLC) Five Pillars, Lowenthal and Hodges decided to use Quality Matters (QM), since it is “one of the most popular and widely used quality assurance frameworks in the U.S…there are currently more than 800 QM subscribers.”
The study notes that QM is a peer review and faculty development process that is centered on eight general standards (each of the standards has a number of related and more specific sub-standards):
- Course overview and introduction
- Learning objectives
- Assessments and measurement
- Instructional materials
- Learner interaction and engagement
- Course technology
- Learner support
- Accessibility
In order to pass QM’s standards, a course must receive an 85 percent and meet all of the essential standards. Yet, none of the six MOOCs chosen (2 from Coursera, 2 from edX and 2 from Udacity) by the researchers passed the initial review. However, one Coursera MOOC and one edX MOOC scored very well overall in terms of total points. The Udacity courses (both self-paced and offered year-round) performed the worst.
“The fact that all six MOOCs did not pass a QM review does not suggest that all of these MOOCs were poorly designed,” emphasize the researchers. “In fact, all six MOOCs met…specific review standards.”
For example, Lowenthal and Hodges found that all six MOOCs studied introduced students to the purpose and structure of the course; had the instructor provide an appropriate and available self-introduction; provided assessments that measure the stated learning objectives and are consistent with course activities and resources; provide students with multiple opportunities to measure their own learning progress; employed accessible technologies and provided guidance on how to obtain accommodation; provided course tools and media to support student engagement and guide the student to become an active learner; etc. (For a complete list of all the standards and sub-categories met, read the full study).
(Next page: MOOC quality standards failed; challenges to MOOC quality standards)
However, all six MOOCs also failed to meet the following standards:
- The module/unit learning objectives describe outcomes that are measurable and consistent with the course-level objectives.
- Course instructions articulate or link to the instruction’s accessibility policies and services.
- Course instructions articulate or link to an explanation of how the institution’s academic support services and resources can help students succeed in the course and how students can access the services.
- Course instructions articulate or link to an explanation of how the instruction’s student support services can help students succeed and how students can access the services.
“It is not surprising that none of these MOOCs performed well [on Standard 7: Learner Support],” say the researchers, as this standard focuses on students attending a course for college credit. “MOOCs, for the most part, are not designed as for-credit college courses. Rather, they are designed as professional development exercises.”
Yet, the researchers contend that many MOOCs could easily be updated to address Standard 7.
But should QM act as the measurement standard for credit-bearing MOOCs?
The Challenges with MOOC Quality Standards
Lowenthal and Hodges emphasize that though QM is a comprehensive and well-adopted quality measurement for online learning, MOOCs are a breed of their own, and come with a set of unique challenges in measuring their instructional and design quality.
First, the researchers point out that MOOCs are not known for their high-quality learning activities and learner interactions, so technically a course could meet QM standards, but still be a relatively boring course. “Thus, perhaps more differentiation needs to be made between a quality course and a truly high quality or exceptional course.”
Second, not all MOOCs provided by third-party platforms are designed to offer the same type of learning experience (self-paced vs. facilitated).
Third and final, MOOCs often have a “similar instructional approach that relies heavily on professionally-produced videos, readings and quizzes, all with minimal instructor contact.” Though production value helps, the researchers say that there might be limitations to the QM rubric if such simply designed courses can pass a review and be deemed “quality” courses. The QM rubric might focus too much on the basics, like learning objectives, and not enough on instructional approaches for “active engagement, communication, and collaboration.”
Lowenthal and Hodges emphasize that though the study sampled a very small number of courses, and therefore, the results of the study should be viewed in this context, it’s also important to remember that this data can serve to “advance our discussions about MOOCs beyond mere hyberbole,” they conclude. “For instance, can MOOCs meet the same design requirements as other for-credit online courses? If not, where do they fall short? Questions such as these are important as institutions struggle with offering credit for completing MOOCs.”
The researchers also hope this study has the possibility of informing and further evolving online quality assurance systems, like QM, by analyzing MOOCs—a non-traditional format of online learning.
For much more detailed information, including methodology, ratings on QM for each course studied, the choice to exclude cMOOCs and more, read “In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs).”
- 25 education trends for 2018 - January 1, 2018
- IT #1: 6 essential technologies on the higher ed horizon - December 27, 2017
- #3: 3 big ways today’s college students are different from just a decade ago - December 27, 2017
Comments are closed.