Why a fourth path could finally provide legitimate evidence of valid and reliable measure.
Among high impact educational practices there are few more controversial than assessment.
The leading reason cited by reviewing agencies for not handing out the revered 10-Year Reaffirmation accreditation is that the assessment of outcomes and learning is not adequately addressed across an institution. As a result, five years has become the norm. For some institutions, reports are even required annually to show progress in data-driven improvement efforts.
The typical assessment approach involves 1) working on the General Education program to ensure it covers the appropriate outcomes and relates well to the institution’s educational mission—usually amounting to 8-12 big statements that virtually no one could argue against; 2) mapping the big outcomes to the key assignments in a broad range of courses that students will take; and 3) beginning the assessment process by using some type of assessment system and running reports as the data rolls in.
For most, this whole process does not go well despite good intentions. Assessment inter-rater reliability is consistently low. Why does this happen?
The all-inclusive outcome is usually too big
The problems begin with Outcome statements, which tend to read something like this:
“The student demonstrates communication competency in writing and speaking standard English, in critical reading and listening, and in using information and research resources effectively.”
Read that aloud a few times and ask yourself how one would collect data to demonstrate learning progress over time related to this broadly stated goal.
It’s daunting, because any valid statement of progress over time requires consistently measured criteria across multiple faculty, who are teaching in different programs and disciplines, collected from the moment the student crosses the threshold until he or she graduates. Stated as is, the outcome actually encompasses multiple competencies: writing, speaking, comprehending written and oral texts, and effective use of sources.
What are the chances of all faculty agreeing on the quality of student work related to an outcome statement that mixes several skills in so many contexts and over so many years? The odds are not good.
(Next page: Paths chosen and their fatal flaws)