The MOOC metrics we should be tracking


Understanding student demographic factors might help us teach a new generation of learners how to succeed in MOOCs

mooc-experimentAn important technique for understanding what might really be happening in the world is to avoid being bowled over by stunning numbers.

For instance, when Massive Open Online Courses (or MOOCs – free classes offered by some of the world’s best-known colleges and universities) first made news, stories celebrating them as the solution to the planet’s educational ills focused on huge enrollments in the tens or even hundreds of thousands.

But soon critics of this new learning method started taking aim at a different remarkable figure: drop-out rates that typically topped 90 percent.

The problem with both huge enrollment and high drop-out figures is that they are based on the same inappropriate metaphor. For both assume that someone who hits the Enroll button on the Coursera or edX sign-up page is the equivalent of a student at a residential university committing to taking a course for the semester.

But MOOCs are offering a valuable product (a college course) for free. And, unlike a traditional college class, there is no consequence for signing up with no intention to complete (or even show up for) a class. This means a better metaphor for MOOC “enrollments” and “drop-outs” might be sign-ups and usage rates for any online service that attracts people willing to fill out a simple form to get something for nothing.

Which numbers are actually representative?

So if the most frequently touted figures related to MOOCs might not accurately reflect their success or failure, what numbers should we be using to evaluate their effectiveness?

Fortunately, the data generated by thousands of students taking hundreds of MOOCs over the last two years has given us a better understanding of what people do inside a massive online course, other than signing up and either finishing it (by earning a certificate of completion) or leaving before the end.

(Next page: Improving retention rates)

Studies released by the University of North Carolina, the University of Edinburgh, Harvard and MIT document a wide range of student behaviors within a MOOC. For instance, while 40-50 percent of enrollees might never do anything but sign up (which likely makes their activity the equivalent of browsing through a course catalog, rather than committing to take a class), a large percentage of those who do not earn a certificate of completion watch most or all lecture videos. And like auditors in a traditional college class, these students are getting the educational value they want from a MOOC.

Research also shows that among students who demonstrate engagement with course material (by completing the first assignment; for example) graduation rates can top 40-50 percent. And demographic data from millions of existing MOOC enrollees also shows that the majority of those participating in MOOCs are not college-age students but older learners who already have a BA or an advanced degree.

Results from this research are already finding their way into the development process of new MOOC classes. For example, usage patterns have demonstrated to MOOC developers the importance of starting a course with an overview lecture that gives students a better understanding of what they are committing to, or revised enrollment procedures that allow enrollees to specify from the outset if they are auditing or planning to take the course to completion.

Improving retention rates

More fine-grained data (such as completion rates for individual lecture videos) has also led course developers to break lectures into ever-shorter increments in order to increase retention. And given that student response data and scores are available for every question and assignment given within a MOOC, statistical information is available to help improve the types of assessments that typically contribute to grading (still considered one of the weakest elements in most massive courses).

Student demographic data are also contributing to the kinds of courses chosen to be given the MOOC treatment. For example, decisions to release courses for professionals like in-service teachers or public health professionals are based on the assumption that college-level classes provide important social benefit even when taken by those who already have a diploma. And another important number to watch is the total number of available MOOCs since, even with new players entering the space, a comprehensive catalog of MOOC classes still does not compare in scale or scope of what is offered by a moderately-sized residential college.

Finally, those 5-10 percent who complete their MOOCs are worth more careful anthropological study. For these are students who tend to do far more work than is necessary to receive a passing grade, meaning they are eager to learn, self-motivated, and able to succeed in an online learning environment.

If we can better understand the skills and psychological factors that contribute to their success, that might help us teach a new generation of learners how to succeed in MOOCs or whatever EdTech solution might ultimately provide an alternative to a higher-education system pricing itself out of reach for too many.

Sources:

“Data about the Metadata MOOC, Part 1: Student Activity,” Jeffrey Pomerantz, July 10, 2014, http://jeffrey.pomerantz.name/2013/11/data-about-the-metadata-mooc-part-1/

“MOOCs @ Edinburgh 2013 – Report #1,” Edingburgh Research Archive, July 10, 2014, https://www.era.lib.ed.ac.uk/bitstream/1842/6683/1/Edinburgh_MOOCs_Report2013_no1.pdf

“HarvardX Working Papers,” Harvard University, July 10, 2014, http://harvardx.harvard.edu/harvardx-working-papers

“MITx Working Papers,” MIT Office of Digital Learning, July 14, 2014, http://odl.mit.edu/mitx-working-papers/

Jonathan Haber is a writer/researcher who has worked in the fields of professional assessment, employment, curriculum development and educational technology. This article first appeared in World Wide Learn.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.

Oops! We could not locate your form.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.