We probably all have a good example of a time when we decided to make a change – maybe drastic, maybe minimal – only to realize later the full ramifications of that change (“Yikes! Now I remember why I grew a beard.”). This is the problem with change – our own little lives aren’t as discretely organized as we’d like to think, and there are always unintended consequences and surprise effects.
When Augustana decided to move from measuring itself based on the quality of what we have (incoming student profile, endowment, number of faculty, etc.) to assessing our effectiveness based on what we do (student learning and development, educational improvement and efficiency, etc.), I don’t think we fully realized the ramifications of this shift. Although there are numerous ways in which this shift is impacting our work, I’d like to talk specifically about the implications of this shift in terms of institutional data collection and reporting.
First, let’s get two terms clarified. When I say “outcomes” I mean the learning that results from educating. When I say “experiences” I mean the experiences that students have during the course of their college career. They could be simply described by their participation in a particular activity (e.g., a philosophy major) or they could be more ambiguously described as the quality of a student’s interaction with faculty. Either way, the idea is – and has always been – that student experiences should lead to gains on educational outcomes.
I remember an early meeting during my first few months at Augustana College where one senior administrator turned to me and said, “We need outcomes. What have you got?” At many institutions, the answer would be something like, “I’ll get back to you in four years,” because that is how long it takes to gather dependable data. Just surveying students at any given point only tells you where they are at that point – it doesn’t tell you how much they’ve changed as a result of our efforts. Although we have some outcome data from a several studies that we happened to join, we still have to gather outcome data on everything that we need to measure – and that will take time.
But the other problem is one of design. Ideally, you choose what you want to measure, and then you start measuring it. In our case, although we have measured some outcomes, we don’t have measures on other outcomes that are equally important. And there isn’t a very strong centering framework for what we have measured, what we have not, and why. This is why we are having the conversation about identifying college-wide outcomes. The results of that conversation will tell us exactly what to measure.
The second issue is in some ways almost more important for our own purposes. We need to know what we should do to improve student learning – not just whether our students are learning (or not). As we should know by now, learning doesn’t happen by magic. There are specific experiences that accelerate learning, and certain experiences that grind it to a halt. Once we’ve identified the outcomes that define Augustana, then we can track the experiences that precede them. It is amazing how many times we have found that, despite the substantial amount of data we have on our students, the precise data on a specific experience is nowhere to be found because we never knew we were going to need it. This is the primary reason for the changes I made in the senior survey this year.
This move from measuring what we have to assessing what we do is not a simple one and it doesn’t happen overnight. And that is just the data collection side of the shop. Just wait until I start talking about what we do with the data once we get it! (Cue evil laughter soundtrack!)
Make it a good day!
Mark