MIT innovation could humanize online education programs

A new crowd-sourcing system allows students to receive more timely feedback in online education programs.

Critics of online education programs commonly refer to the perceived disconnect these create between students and instructors, but a new crowd-sourcing system that offers students personal feedback could help alleviate skeptics’ concerns.

The system is called Caesar, and it was developed by MIT Professor Rob Miller and two of his graduate students, Mason Tang and Elena Tatarchenko, through the MIT Computer Science & Artificial Intelligence Lab (CSAIL).

Caesar is able to split students’ submitted assignments in a programming course into prioritized chunks through a process called the “code selector.” Caesar then enacts the “task selector,” which sends those chunks that require review out to various MIT teaching assistants, course alumni, and computer science students.

The final step, called the “reviewing interface,” allows multiple reviewers to offer students feedback on their assignments. Students generally receive feedback in less than three days—something that Miller highlights is much quicker than with traditional grading processes.

(Next page: How Caesar works, and how it soon could impact MOOCs)

Miller says he first developed the idea for Caesar in 2010, and that it was partially inspired by a crowd-sourcing system that CSAIL had recently developed called Soylent, which draws on crowds of people to help with editing tasks. Soylent and Caesar embody similar characteristics: Both take a large program, chop it into pieces, and distribute those pieces to a crowd for review and discussion.

“The on-campus problem we were trying to solve was slow feedback,” said Miller. “The students in our programming courses write a lot of code, and it takes a long time for a small staff of human graders to read and grade. The old way, it might take a few weeks to get feedback about what they’d written, and in that time they’ve written more programs—often repeating the same mistakes over and over.”

Because Caesar’s “reviewing interface” aspect is speedy, students are able to read their feedback prior to completing their next assignment—something that Miller believes will result in a better understanding of class material, and possibly higher grades. Also, by connecting students with multiple resources, students gain invaluable insight they might not have otherwise.

“Caesar substantially reduces the time it takes to get feedback,” said Miller. “Students typically hand in their programs on a Thursday, and by Monday they have comments about it from the crowd of reviewers.”

Miller believes Caesar can resolve similar problems in the larger context of online education programs, including massive open online courses (MOOCs).

“Online, the problem of scale is even greater—too many students, not enough staff,” he said. “The usual approaches to the scale problem in massive online education are purely automatic, which have a lot of limitations in the kinds of feedback they can give and the kinds of work they can evaluate. Caesar is [ultimately] trying to add a human dimension to the feedback.”

Miller has selected reputable sources that offer helpful feedback to students in his programming course.

“What we are trying to do is to learn how to use a crowd of people with mixed expertise in an intelligent way,” he said—“one that helps students and the ‘crowd’ expand their knowledge and improve on their expertise.”

Students enrolled in Miller’s online course, “Elements of Software Construction,” can benefit from the assortment of feedback that Caesar offers, because the nature of the course is so technical.

“Every time you interact with a new person, it’s an opportunity to learn, whether it’s a student demonstrating a new technique to a teaching assistant, or an alum providing a student with a valuable piece of industry advice,” he said.

Currently, Miller’s pool of resources is limited to members of the MIT community. He hopes to continue to expand this network in the future. He also has implemented tools for students and reviewers to interact further after feedback has been offered.

“Like Facebook and other social networks, Caesar provides opportunities for other kinds of interaction among its users,” the CSAIL release explains. “Reviewers can agree or disagree with fellow reviewers’ comments via and ‘upvote’ or ‘downvote,’ a process similar to the ‘like’ feature on Facebook, and can also leave comments for both students and other reviewers.”

(Next page: How Caesar is being expanded into other courses)

Based on its success, the Caesar program is already being expanded into other MIT programming courses, and the technology might be adopted by edX, the MOOC platform founded by MIT and Harvard University last spring.

[field name= iframe2]


Miller understands that, although students certainly can benefit from the accessibility that online learning programs offers, human interaction and feedback remain essential in the larger educational scope.

“Ultimately, I believe that crowd sourcing is going to develop hand-in-hand with automatic mechanisms for making online education work, because there are so many aspects of learning that require a human touch,” he said.

He’s also used Caesar to review code in multiple languages like Java and Python. He believes the larger idea of crowd sourcing can, and will, become accessible to other types of courses in the future.

“The essential idea of crowd-sourced review—dividing student work into smaller pieces and distributing those pieces to a mixed crowd of reviewers who comment on and discuss them—is likely to be applicable to many kinds of courses, including liberal arts, business, and social sciences, not just technical [courses],” said Miller. “We’ve used these ideas in other domains in the past, such as document editing in the Soylent system, so I think they would transfer well.”

CSAIL has long been a significant contributor to the technological world and has had a hand in multiple advances in the past 50 years. Miller hopes that Caesar will become CSAIL’s next touted revolution.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.

Oops! We could not locate your form.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.