Maybe Not So 'High Impact'?

“High-impact” educational practices widely promoted and adopted to improve learning by college and university students and increase graduation rates have not led to those expected outcomes, according to new research in The Journal of Higher Education.

The study (abstract available here) found the effectiveness of 10 such practices -- first-year seminars, writing-intensive courses and collaborative assignments, among others -- recommended by the Association of American Colleges and Universities questionable and worthy of re-examination, at least as a tool to promote completion.

The research is based on data from 101 institutions that participated in the study by Sarah Randall Johnson, associate director of institutional research at Harvard Business School, and Frances King Stage, professor of higher education at New York University. Some of the institutions make extensive use of the practices, others minimal use and others no use at all.

Their study examined whether four-year public colleges that adopted the high-impact practices had higher four- and six-year graduation rates than institutions that did not adopt them. The study found that the graduation rates at colleges that incorporated all the practices were not higher than those that used few if any of the practices.

Over all, they concluded, “While some research has linked individual practices to engagement and learning outcomes, findings from this study question whether those benefits can be directly linked to timely college completion. Results also indicated that the current consensus about benefits of institutional adoption of high-impact practices may be misinformed.”

The researchers said examining the connection between the recommended practices and institutional outcomes was important because of the widespread use of the practices “at the expense of other possible offerings.”

Officials of AAC&U pointed to their own research on the effectiveness of the practices in question. The association, which represents 1,400 member institutions, did not take issue with the findings of the study or address its specific points.

A spokeswoman for the association said Wednesday that it needs more time to review the study and “address its theoretical basis and conclusions.”

"As the study notes, the intentionality with which institutions plan and add HIPs to their curricula is critically important. AAC&U promotes the use of HIPs as defined by their positive effect on student achievement and outcomes," the organization said in an emailed statement. "AAC&U strongly advises that institutional efforts to implement and/or increase HIPs should always be tied to clearly defined student learning outcomes set by the institution. These outcomes measurements may include (but are not limited to) higher learning gains, higher GPAs, higher persistence, retention and/or graduation rates, and higher levels of student engagement. We encourage institutions to analyze data on educational practices before calling them high impact, and we support ongoing research that examines their quality and effectiveness in helping students develop proficiency in defined learning outcomes."

George Kuh, an emeritus professor of higher education at Indiana University, whose research on high-impact practices served as the basis for the AAC&U’s promotion of the measures, said he had not yet read the study and declined to comment on it until he has reviewed it.

Kuh founded Indiana’s Center for Postsecondary Research and the National Survey of Student Engagement, or NSSE, which collects information annually at hundreds of four-year colleges about participation by first-year students and seniors in learning and personal development programs, and provides a measurement of how they benefit from college.

College completion rates became a subject of public interest in the past decade as concerns grew about skyrocketing tuition and a lack of accountability by the institutions. State and federal lawmakers, guided by budget constraints and taxpayer demands, began calling for colleges to increase their graduation rates. Further, many educators have said that the quality of a college education isn’t just about courses completed, but about the kinds of experiences in which students participate.

The high-impact practices were originally defined by the AAC&U in 2008 as being “particularly effective in cultivating student learning and preparing students for future careers,” the study’s authors wrote.

“One important finding from this study is that the quantity of practices offered on campus, measured by our composite score, was not related to graduation rates,” they wrote. “Institutions planning to add high-impact practices to their curricula should make intentional decisions about which practices fit well on their campus and would be most beneficial to their students instead of focusing on quantity of offerings.”

The researchers state that few studies have examined all 10 practices together, their relationships to one another and their relationships with college outcomes. They say continued research, with larger sample sizes, should be done “to sufficiently inform the higher education community of their benefits.”

They wrote, “This study revealed a missing link between academically engaging activities and college completion at large public institutions. Research is needed on college completion and administrative practices, such as advising models, campus resources, and required activities that are outside the curriculum, which could provide evidence of intermediary variables between academic engagement and timely graduation."

The researchers readily acknowledge the limitations of the study.

“The quantitative analyses performed have not determined causation between variables that were examined by this research, nor can they be used to make inferences about practices and outcomes at all colleges and universities,” they wrote.

Still, they contend that “Our findings are important for both researchers and practitioners in the higher education community because advocacy for these practices is widespread, they can be costly to implement, and our knowledge about their relationships to institutional and student outcomes is limited.”

A Call for More 'Unpacking'

Bridget Burns is executive director of the University Innovation Alliance, a group of research universities seeking to improve their student completion rates. She said via email that the researchers’ findings do not negate the intangible benefits of high-impact educational practices. “High-impact practices aren’t pitched as a silver bullet to increase graduation rates, instead, they are recommended as a means to cultivate deeper learning experiences,” she said.

Still, she said she appreciates the researchers’ focus on the topic. “We need far more literature contributing to greater understanding of student success in higher ed,” she said. “As a sector, we need help unpacking and explaining the various potential solutions implemented at campuses across the country.”