Since 2011, two national studies of successful learning outcome improvement through educational assessment have highlighted our efforts at Augustana College. First, the National Institute of Learning Outcomes Assessment (NILOA) published a report detailing the ways that a small group of uniquely successful institutions developed and maintain a positive culture of assessment and improvement. Second, the National Survey of Student Engagement (NSSE) conducted an in-depth study of eight institutions, chosen from an original pool of 534 colleges and universities that had made significant gains on various NSSE benchmark scores, to identify some of the organizational values and practices that allow these institutions to make such clearly demonstrable improvements in their educational environments.
The data point that most clearly jumped out to both research teams involved the degree to which our first-year scores on the NSSE Academic Challenge benchmark increased between 2003 and 2009. This benchmark scale asked a series of questions about the amount of time and effort students must put into their coursework to meet academic expectations and has been a staple of NSSE and the Wabash National Study. As many of you know, we can pin our own improved Academic Challenge scores to the overhaul of our general education and LSFY programs about seven years ago, when a preponderance of earlier data simply didn’t comport with the kind of institution we wanted to be. And even though we continue to note, discuss, and tweak perceived weaknesses that have emerged since implementing AGES, we shouldn’t let these more recently identified concerns detract from the fact that our earlier efforts were thoroughly successful in improving the educational quality of Augustana’s first year experience.
Yet the evidence of an improved educational environment (as represented by an increase in the academic challenge experienced by our students) did not seem to extend beyond the first year. In our 2009 NSSE report, despite a significant difference in first-year academic challenge scores between Augustana and a group of 30 similar residential liberal arts colleges, our fourth-year academic challenge scores remained no different than other institutions. Many of us were troubled by the possibility that the distinction in academic quality that we might have established in the first year could have eroded entirely by the end of the fourth year. Although senior inquiry was intended to help us increase our level of academic challenge in the fourth year, the 2009 NSSE report did not reflect any impact of that effort (likely because SI was not fully implemented until 2010 or 2011). So when we received our Wabash National Study four-year summary report a few weeks ago, I specifically wanted to examine our seniors’ overall score to the Academic Challenge scale to see if we’d made any progress on this rather important measure of educational quality.
(At this point, the empathetic side of my brain/soul/elven spirit/gaseous particles has guilted me into offering a pre-emptive apology. I am going to talk about some numbers without giving you all the detailed context behind those numbers. If you want more context, you know where to find me. Otherwise, try to hang in there and trust that the changes these numbers represent are substantial and worth discussing.)
The Wabash National Study evidence suggests that, once again, our efforts to respond to assessment data with changes that will improve Augustana’s educational quality seem to have born fruit. Between 2009 and 2012, our seniors’ Academic Challenge score jumped from 62.6 to 64.3 – a statistically significant increase. Moreover, the difference between our mean score and the average Academic Challenge score of the 32 similar institutions that participated in the Wabash National Study (61.0) was statistically significant – suggesting that something we are doing during the fourth year distinguishes the academic quality that we provide from those institutions. For my own information and confidence in this conclusion, I also looked at the 2012 NSSE annual report just to see if these Wabash Study numbers differed in any meaningful way from the much larger sample of institutions that participated in NSSE. Again, our Academic Challenge scores placed us above the NSSE average of similar liberal arts institutions (62.5) and well above the overall NSSE average (58.4).
All of this evidence seems to point toward a familiar and heartening – if not downright exciting – conclusion. Our efforts to improve the educational quality of an Augustana experience are working (or as the famous line goes, “I love it when a plan comes together!” . . . yes, I just quoted Hannibal Smith from the 1980s TV show “The A-Team” in a blog about institutional research. I’m fired up – deal with it.). The academic challenge our students’ experience in their fourth year appears to have increased. And while we don’t have comparative data on the degree to which this effort has increased our students’ learning outcome gains (because we don’t have identical pretest-posttest outcomes data from 2009), it is clear from the Wabash National Study data that our 2012 Wabash Study participants repeatedly made larger learning outcome gains than students at the 32 similar institutions participating in same study.
Later this year we will receive the full Wabash study dataset that will allow us to examine the responses to each individual question in this scale. I am looking forward to digging deeper into that data. But for the time being, I think we deserve to take a moment and congratulate ourselves as a community of educators dedicated to the success of our students. Although we continually hear critics of higher education lament that institutions refuse to collect the kind of data necessary to meaningfully assess themselves, or that faculty perpetually resist making the kind of changes that might substantively improve an institution’s educational quality, we now have multiple sources of evidence to demonstrate that, while we might not be without reproach, we have living, breathing evidence of our successful efforts to improve the Augustana education.
Are we there yet? No. Will we ever be there? Of course, not. But are we genuinely walking the walk of an institution committed to its students and its educational mission? Absolutely.
Make it a good day,
Mark
62.6 to 64.3