standardized-assessments

5 ways to un-suck higher ed standardized assessments


Higher education standardized assessments are coming—but is there a way to turn them from the dark side?

As colleges and universities are increasingly required to “prove” efficacy of teaching and learning, many conversations—especially at the federal level—are circling around developing standardized assessments for higher education.

Naturally, postsecondary stakeholders and faculty worry that these assessments could have a negative impact, and shudder at the prospect of metrics mirroring those of K-12’s. But is it all doom-and-gloom in the standardized assessment realm, or can a postsecondary-specific design work to higher education’s advantage?

Take Our Quick Poll!

In this month’s Symposium, Dr. Fredrik deBoer, a Purdue University scholar and academic researcher, emphasizes that the only way to accurately and fairly assess postsecondary learning outcomes is to:

2. Adjust for Ability Effects.

“We know for a fact that the incoming populations of different colleges are deeply unequal in prerequisite ability, writes deBoer. “The most obvious and strongest reason for this is the very college admissions process itself…Differences in incoming ability effects are troubling, as they potentially represent serious confounds in our effort to sort out how much students are learning at different institutions. This problem is compounded by the fact that the biggest criterion for selecting a college, for the average student, is not its perceived quality but its geography, with most college students choosing to attend schools close to home.

There are several ways to address these issues. First, score results can be normed against incoming SAT scores, an imperfect but powerful means to sort students into ranks of incoming ability. Scores on tests of higher education learning tend to be highly correlated with SAT and ACT results. We can quantitatively adjust scores on the latter to help control for ability effects. Second, test-retest systems, where students are tested in freshman and senior year, can help to determine how much growth has occurred, and can give us scores that are based not on where students end up but on how much their scores have improved during the course of their education. Sometimes, these efforts can take advantage of complex Value Added Models, though such procedures are controversial.”

2. Understand the Testing Industry is Big Business.

“Whether assessments should be developed ‘in-house’ or should be provided by testing corporations and nonprofits is one of the perpetual controversies in the assessment literature,” emphasizes deBoer. “There are clear advantages to developing assessments internally. For one, internally-developed assessments can better adapt to the kinds of institution-specific complexity that I discussed previously. Internally-developed assessments also can better involve faculty, helping them to feel like stakeholders in the process, and in doing so, easing tensions that often result from assessment efforts. Internally-developed assessments also have the advantage of keeping funding within the university community, often resulting in money for graduate assistants and other staff.

But there are major hurdles to developing assessments internally. They represent a significant investment of time, manpower, energy, and money. Also, in many cases, state administrators and accreditors will likely insist on the use of standardized instruments developed externally.

What everyone involved in the assessment process must understand is that the testing industry is just that, an industry, made up of institutions that are primarily motivated by the drive for profits. Those involved in assessment must bear in mind that, when organizations attempt to sell them tests, they are receiving a marketing pitch like any other. Skepticism of the claims of the institutions that develop tests is perfectly warranted.”

[Read deBoer’s full essay with more thoughts on this subject here]

(Next page: More suggestions on how to un-suck higher ed’s standardized assessments)

In his essay, the President of the Council for Aid to Education (CAE), Roger Benjamin, discusses how a standardized assessments initiative could provide a great tool for decision-makers, as long as:

3. Validity and Reliability are Ensured.

“Measurement scientists have developed criteria to evaluate the validity and reliability of assessment protocols,” explains Benjamin in his essay. “Validity refers to the extent to which the test measures the knowledge, skills, and abilities it was designed to measure. Reliability is the degree of consistency of student’s (or institution’s) scores across a test’s questions, the consistency of scores across different assessors, and whether the tests are given to students under the same conditions and over the same time period.  Standardized assessments that provide statistical evidence of reliability on these criteria are preferred. Moreover, measurement scientists are insistent about recognizing this point for any tests that have stakes attached to them.

Boards of trustees and administrators want to know how well their institution is doing compared to institutions that are similar in student characteristics, financial support, size and other characteristics. Therefore, it is essential that test data for these purposes be based on the transparent criteria measurement scientists have developed for standardized tests.”

4. Researchers and Practitioners Work Together.

“…faculty must be partners with their measurement scientist colleagues in providing content for the design of standardized test items, evaluation of the standardized test results, and the development of formative test items that are aligned with the standardized tests,” writes Benjamin.

“Measurement scientists, the statistical-based tools they use, and the test analyses they produce are often challenged by faculty. Why? Faculty are housed within departments that are granted relative autonomy by the university to recommend what to teach, who to teach it, and how students should be assessed. Education assessment test results and analyses are typically isolated, one-off research activities that are not related to either faculty engaged in teaching or to researchers in other fields relevant to improving student learning. Independent experts, no matter how talented, are not considered to have the standing necessary to contribute to department affairs. However, measurement science, including its education assessment sub-groups, is a branch of statistics that has been in good standing in the Academy for hundreds of years.”

5. Everyone Works Toward an Integrated, Interdisciplinary Approach.

“The premises of the value system of science, peer review, transparency, and the ability to replicate results are familiar to faculty and administrators,” Benjamin notes. “Most faculty should and will accept assessment-related work based on these core principles. When paired with a coherent and compelling use-inspired basic research strategy, it is possible to imagine a more integrated, interdisciplinary approach to the challenges that higher education faces.

Already, an initial effort has been launched to include all subjects within standardized assessments. The Gates Foundation’s Measuring College Learning (MCL) project is a collaboration of six national disciplinary associations to define the core learning outcomes of their fields. Prospects for success of this endeavor are good. If this group of six associations succeeds in creating attractive, reliable and valid tests, other disciplines will follow. It will be important to develop standardized tests for the arts and sciences that form the basis for general education curriculum; other professional schools and applied subjects should and will follow.”

[Read Benjamin’s full essay with more thoughts on this subject here]

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.

Oops! We could not locate your form.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.