Researchers say formative assessment plays a critical role in student retention.

formative-assessment-MOOCAnalyzing over 4,000 student comments and reviews of the 7 most active and highly-rated MOOCs, researchers say formative and peer assessment strategies in relation to MOOCs directly affect student engagement.

According to researchers at the Open University of Catalonia (OUC), Spain, research on student engagement in MOOCs exists for non-didactic factors (profiles, demographics, institutional reputation) and didactic factors (course structure and content, workload and duration, type of exams and assessments), but little research has been conducted to determine whether different types of assessment (formative, non-formative, peer and self-assessment) or different student feedback approaches actually affect student engagement.

With the aim of providing specific research-based recommendations on formative assessment and feedback practices that can advance student activity, “we analyzed some significant research papers on formative assessments and feedback methods applicable to face-to-face teaching environments that advance student engagement, and concluded with related requirements and conditions that can be applied also to MOOCs,” explained Nikolaos Floratos, PhD researcher at OUC and lead author of the report.

Researchers also analyzed 4050 comments and reviews of the 7 most active and highly-rated MOOCs (6 from Coursera and 1 from edX) provided by students who are in the final stages of completing those courses via an online review platform for MOOCs, called CourseTalk.

Based on this content analysis, Floratos and his team formulated 14 recommendations that support what the researchers say is a new conceptual and theoretical framework analysis on formative assessment in relation to MOOC engagement.

(Next page: The new theoretical framework)

Based on numerous, current research studies on best practices for formative assessment and feedback (which are listed in detail in the report), Floratos and his team argue that formative and related assessment methods can support student engagement in MOOCs by following their specific framework of requirements.

A sample of the 17 requirements includes:

  • The course should fulfill the need of autonomy and consequently behavioral engagement that is addressed through self-assessment practices (i.e. a student to assess their own work and assignments).
  • The course should fulfill the need of relatedness and consequently affective engagement that is achieved through peer assessment practices (i.e. a student to assess the work and assignment of other students/peers)
  • Students should know in advance what good performance is and be able to compare assessments.
  • Assessment tasks should orientate students to allocate appropriate amounts of time and effort to the most important aspects of the course.

(A full list of the 17 requirements part of the framework can be found in the report.)

Using comments accrued from CourseTalk, the OUC researchers collected responses on courses offered for free, offered by universities, that were top-rated (5/5 stars), and that received more than 100 reviews.

Of the seven courses used in the analysis (which are listed in the report), the main formative assessment and feedback methods applied in those courses focused on peer-assessments and feedback through discussion forums with the participation of the instructors.

Of the 4050 reviews collected, 418 responses specifically commented on feedback and assessment methods. Of the 418 responses, 237 comments and reviews provided specific details regarding practices.

Using the 237 comments and reviews, 11 recommendations/findings on feedback and formative assessments practices for higher student engagement in MOOCs were created by the OUC researchers; these recommendations also link to the requirements framework.

(Next page: The 11 recommendations on formative assessment for MOOC engagement)

(These recommendations are ranked from the highest to lowest significance by considering their weighted average. For information on how the researchers calculated the weighted average, read the full report.)

  1. Assignments should be based on practical problems with clear expected output that give the sense of completeness to the students, and are not that easy to address but are sufficiently challenging and interesting.
  2. Although MOOCs can automatically assess quizzes, self- and peer assessments must deal with more comprehensive assignments. However, self- and peer assessment should be formative as well (i.e. specifically intended to generate feedback on performance and improve learning rather than just providing a mark).
  3. A well-structured course syllabus with information on what assignments are required per week/module by each participant, by when, as well as its related training content, was appreciated.
  4. Discussion forums are appreciated and are a very good way for students to receive support, sympathy, formative feedback and clarification as well as share ideas on their work as long as there is an effective mechanism on managing and accessing the discussion threads.
  5. Assessment models, such as self-assessments and peer-assessments can be relatively complex in MOOCs and a comprehensive rubric should be provided in all MOOCS that involve peer assessments.
  6. Peer assessment was more welcome in those MOOCs where there was no mark given or alternatively feedback was given on the quality of the related assignment.
  7. Peer assessment in MOOCs does not always deliver accurate results and in many cases it creates frustrations or negative feelings since it doesn’t provide any feedback on the work assessed or there are concerns on the competencies of the peer assessor.
  8. Immediate feedback mainly from automated and simple, but well structured, quizzes is appreciated by students since it confirms their understanding about what they have learned. The appreciation is even higher in cases that some further feedback on the given answer or on the correct answer is provided.
  9. The posting of comments by human experts, for instance instructors’ assistants that intervene and provide comments and views in discussion forums or even in students’ work, is appreciated.
  10. Peer assessment can be supplemented with related forum discussions for interaction and the possibility to offer a communication channel for clarifications.
  11. Self- and peer assessments can be obligatory (or penalties can be applied) and should be a priority for peers to examine the work of others and provide constructive feedback (furthermore, students should be able to modify their own related assignment work even after its submission based on feedback received).

“…we have concluded some very interesting findings,” remarked Floratos. “For example, MOOCs can really benefit and encourage students’ activity and consequently their engagement, when assignments are based on practical problems with clear expected output that give the sense of completeness to the students and are not that easy to address but are sufficiently challenging and interesting.”

The researchers also highlighted that MOOC students appreciate when self- and peer assessments deal with more comprehensive assignments that are formative as well (i.e. specifically intended to generate feedback on performance and improve and accelerate learning rather than just providing a mark).

“On the other hand, some specific practices that are applied in MOOCs do not seem to be that important for the students,” Floratos noted; “for example, use of the same peers in all peer-assessment exercises, the provision of feedback on assignments before the next assignment, or no restrictions on the number of quiz attempts.”

The researchers concluded by emphasizing that the results of the study had a limited dataset, which was based only on MOOC participants that provided feedback through the Coursetalk platform. Further research could focus on comments and reviews collected directly from the MOOC participants in regards to formative and feedback assessment practices and how they can affect their engagement in the course.

For more information on the study, which was presented at the 2015 Open Education Consortium Global Conference and peer-reviewed in Open Praxis, read the full report here. For a slide presentation on the report, click here.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.

Oops! We could not locate your form.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.