The paths chosen all have fatal flaws
At some point many institutions trade their present assessment circumstances for one of the options below:
- Each instructor uses their own rubric for their assignments.
- A committee meets to hammer out a common rubric for a given assignment type.
- The institution chooses someone else’s ‘rubrics with a pedigree’, deemed acceptable by virtue of the authoring organization.
The first two options are problematic. Almost without exception the language in these self-built rubrics is subjective, meaning that scores depend upon what the assessor thinks is important and how he or she interprets the work. The result is invalid data.
The third option is one that many institutions are sprinting to, even though the long-term efficacy has remained unexamined.
A recent example is the use of the American Association of Colleges & Universities (AAC&U) VALUE Rubrics. Between 2007 and 2010, AAC&U developed a set of criteria called Valid Assessment of Learning in Undergraduate Education (VALUE). The rubrics cover 16 dimensions of learning identified by university faculty as desirable skills for undergraduates. Several of the individual rubrics have been widely used by colleges and universities. Case studies describe their successful implementation and they have generated some valuable conversations around assessment and learning.
On the one hand, the AAC&U has sparked a process to move the higher education community forward in understanding and implementing standards-based competency assessment. On the other hand, there are unavoidable validity issues with their use. The VALUE Rubrics’ one claim to validity is face validity – they appear to do what they are supposed to do. The 16 VALUE Rubrics are meant to measure skills that are critical to career-readiness, and they appear to do that, at least to the institutions that have chosen to use them.
Unfortunately, face validity is not legitimate evidence of valid and reliable measure.
Fortunately, there is a new choice emerging that may lead to better assessment practice. Validity theory tells us that the language in the criteria used to evaluate student work must directly reflect the intent of a comprehensive set of standards. It became apparent years ago, as we studied the assessment systems of hundreds of universities, that there was a fundamental need for such a comprehensive standard to serve as an anchor for the whole process of generating robust scoring criteria, unencumbered by language already owned by stakeholders.
This set of comprehensive outcomes covers a learning progression from novice to expert and can be used to document developing skills, knowledge and dispositions over a student’s entire educational career from General Education through specialization, even if he or she changes schools. It is possible to measure all other outcomes as required for accreditation, using only one set of rubrics.
Admittedly, not many people have taken the fourth path yet. It’s new. But try it. The other three have proven unreliable–of that we are certain.
Geoff Irvine is CEO of Chalk & Wire.
- 25 education trends for 2018 - January 1, 2018
- IT #1: 6 essential technologies on the higher ed horizon - December 27, 2017
- #3: 3 big ways today’s college students are different from just a decade ago - December 27, 2017
Comments are closed.