Why it’s not enough to measure STEM outcomes

New data reveals it’s just as important to measure behavior and engagement in different STEM fields as it is to measure outcomes.

STEM-engagement-genderSTEM is often lumped into a broad category of learning with specific fields overlooked in general research. But new research into behavioral differences within specific subjects reveals that not all STEM classes are created gender equal—and some colleges and universities are better at supporting female engagement than others.

“We went into this research with no preconceived notion of how the dynamics within a biology class or computer science class should be,” explained Jessica Gilmartin, head of business and lead researcher at Piazza—a STEM platform designed to connect teachers and students to those in the STEM field. “But we did know there’s a dearth in data when it comes to measuring behaviors within STEM. There’s lots of information on outcomes, and that’s also needed, but we need data on student/teacher interaction within class; such as asking questions, posting on class forums, interacting with other students, et cetera.”

Piazza’s research focused on 976,000 college and graduate STEM student enrollments at traditional college and graduate programs within the U.S. and Canada (1.126 million total student enrollments, including humanities and social sciences). Researchers tracked students over nearly four terms: Spring 2012, Fall 2013, Spring 2013, and Fall 2014 through November 10th. Data collected involved more than 2.3 million questions asked and answers given, 2.1 million of which involved STEM and computer science coursework.

According to Gilmartin, the results showed that gender plays a massive role in determining participation within STEM fields.

“It’s what we call the confidence gap,” she noted. “Participation in class, such as answering questions, tends to be higher for men in many subjects. But when answering questions online in an anonymous format, women tend to participate more than men.”

What was also interesting, noted researchers at Piazza, was that participation and engagement, though higher for men in many subjects, either reach an almost-equilibrium between men and women in some subjects, or in some fields women’s engagement was higher than men’s.

“One example is astronomy,” said Gilmartin: “women answered 39 percent more questions than men, and they actually answered the questions at a lower anonymous rate than men (2 percentage points). In this case, women answered questions 17 percent of the time anonymously (so 17/100 times women chose to answer, they did so anonymously), whereas men answered questions 19 percent of the time anonymously. This would be interpreted that women in astronomy are more confident than men. In robotics, by contrast, women answered 41 percent fewer questions, at a much higher anonymous rate (10 percentage points) than men. So, the confidence gap is much bigger there.”

According to Gilmartin, it’s important to measure behavioral differences like these because the research also revealed another sobering piece of the STEM engagement puzzle: If there’s a confidence gap in a class or field, the gap never closes; in fact, it widens.

“We thought that when the data showed there was a large discrepancy in engagement that the gap would eventually close as students moved forward in their field. But over time the gap actually widened!”

(Next page: How to increase STEM confidence; the best schools for STEM engagement)

Outside of comparing astronomy to robotics, here’s how other STEM subjects rated on the gender engagement scale, according to Piazza’s data [Ordered from lowest confidence gap for women to highest confidence gap for women. As a general rule, the more negative the number, the greater the confidence gap between women and men. When it’s positive, it means that women are more confident in that subject or field.]

Astronomy/astrophysics: Female/male answers: 39 percent; Female/male answers anonymous gap: 2 percent. Gilmartin says this is one of the few STEM subjects where women answer more than men.

Accounting/finance: 18 percent; -1 percent

Economics: 11 percent; -1 percent

Business: 8 percent; 10 percent

Psychology: -3 percent; -6 percent

Mathematics: -8 percent; -8 percent

Electrical/computer engineering: -15 percent; -14 percent

Biology: -17 percent; -2 percent

Physics: -22 percent; -18 percent

Aerospace/mechanical engineering: -25 percent; -14 percent

Statistics/probability: -30 percent; -11 percent

Chemical engineering: -34 percent; -6 percent

Civil/environmental engineering: -35 percent; -10 percent

Computer Science: -37 percent; -13 percent

Bioengineering: -39 percent; -14 percent

Robotics: -41 percent; -10 percent. Robotics has the biggest confidence gap between men and women in answering questions, says Gilmartin.

“We need to look at data like this because it’s important to understand that the culture of a class or field is just as important as what’s being taught,” she explained. “Not only is the implementation of an online component of the class critical for students who may not feel as confident participating in class, but curriculum design and engagement tools are also an integral part of attracting women to diverse STEM fields.”

Gilmartin continued, adding that “there needs to be so much more than just a lecture, since that type of “traditional” learning environment is not supportive of students who may feel intimidated during class.

Piazza’s data also looked at which colleges and universities currently do a great job of engaging women in STEM classes, as well as which ones could use improvement:

  • One of the few schools where women answer more questions than men in both STEM and computer science is Penn State.
  • At Duke University, women tend to answer more questions than the average in both STEM and CS. In CS, they actually answer questions at a lower anonymous rate than men. In STEM, they answer questions at a much higher anonymous rate than men.
  • At MIT, women have an average confidence level in CS, but are more confident than the average in STEM.
  • However, women are “much less confident” in both STEM and CS at Carnegie Mellon University and Cornell University.
  • And at Princeton University and Columbia University, women are “way less confident than the average” in STEM.

For more information on Piazza’s research, as well as to request data sets, contact here.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.

Sign up for our newsletter

Newsletter: Innovations in K12 Education
By submitting your information, you agree to our Terms & Conditions and Privacy Policy.