Good morning everyone!

Welcome to campus – no matter if you can’t remember being anywhere else in late August or if you are the first person in your family to start your fall on a college campus! No matter how you got here, how long you’ve been here, or how soon you’ll be diving into your next great adventure, I’m really glad each of you are here right now.

Somehow you’ve stumbled onto a blog called “Delicious Ambiguity” written by me, Mark Salisbury. (Ok, so I emailed you the link and you clicked on it thinking it might be important). I’m the Director of Institutional Research and Assessment at Augustana College or, as some students have taken to calling me (a supreme compliment, IMHO) the Chief Nerd. I started writing this blog in 2011 as a column in the Faculty Newsletter. The goal then was to share snippets of Augustana data with everyone and hopefully encourage each of us to take a moment to ponder the implications of that data. Most of the time, it’s been statistical data (hence the name Chief Nerd), but sometimes it’s data that comes from interviews or focus groups. No matter the source, I try to explore data points that can help all of us – faculty, staff, and students alike – maximize our experience at Augustana. In case you’re wondering, if you ever think to yourself, “Why doesn’t Mark write about that?” send me an email or comment at the bottom of a blog post. If we’ve got the relevant data, I’ll try to write about it.

With every new group of students, be they traditional freshmen or non-traditional transfers, we gather a set of data points that help us better understand the breadth and depth of the diversity contained within that group. Tracking these data points is one way to remind all of us that cultivating a diverse and vibrant community is about exponentially more than just tracking skin color or biological sex.

Today I’d like to share two data tidbits from our incoming class that seem worth pondering.

First, 31.8% of our new students indicate that neither of their parents earned a four-year college degree. Certainly a substantial proportion of these students come from families where they are the first to go to any kind of college. Equally important, this is not a new phenomenon; this proportion has stayed near 30% since we began asking this question of incoming students in 2012 and, as best we can tell, Augustana has already enrolled a substantial proportion of “first generation” college students. While we can certainly parse the nuances of this student category, our reality remains that many students may not grasp the unstated but oft-assumed implications of our liberal arts college culture, both in terms of the intentions behind various policies or the behaviors that many of us enact every day without a second thought. Moreover, many of these students likely harbor an additional layer of internal anxiety about whether or not they truly “belong” in college at all, let alone a private institution like Augustana.

Second, Augustana’s growing enthusiasm for interfaith understanding in recent years couldn’t have come at a better time. Our incoming class is peppered with students from every kind of western and non-western faith. We have new students who self-identify as Muslim, Buddhist, Hindu, Jewish, Mormon, Greek Orthodox, Catholic, Episcopal, Lutheran, Presbyterian, Methodist, Baptist, Pentecostal, non-denomination, and a whopping 6% of students who categorized themselves as “other.” Oh, and to top it off, 16.7% of our incoming students identify as “no religious background” or “atheist.” I don’t know if this is a one-year phenomenon or if we’ve crossed a tipping point of some sort, but this year that group of students is larger than our incoming proportion of Lutheran students (14.4%).

These two data points hold important implications for the assumptions we make about individual students. All of us probably have some growing to do as we think about the way that we interact with each student. I certainly do. I’ve already made the mistake of assuming that someone I had just met came to Augustana from another country. Based on this faulty assumption, I made a comment that I wish I could take back because it might have been interpreted to reiterate the sense that I am a part of the “natural” in-group and they are still a member of a “probationary” out-group. I owe that person an apology, one that I intend to deliver soon.

I don’t say any of that to hold myself up as some grand example, but rather to suggest that adapting to this increasingly prevalent and multifaceted diversity is a process during which we are each likely to stumble. But in stumbling, depending on how we respond to it, we might just be able to communicate more clearly that we genuinely want to make Augustana a welcoming and inclusive place for everyone – no matter where they are from, who they are, or what they want to become.

Make it a good day,

Mark

Whether we like it or not, the ability to convey, interpret, and evaluate data affects every part of our personal and professional lives. So it’s not a surprise to find quantitative literacy among Augustana’s nine student learning outcomes. Yet, of all those outcomes, quantitative literacy may be the most difficult to pin down. First of all, this concept is relatively new when compared to other learning outcomes like intercultural competence or critical thinking. Second, there isn’t nearly the range measurement mechanisms – surveys or otherwise – that capture this concept effectively. And third, quantitative literacy is the kind of skill that is particularly susceptible to social desirability bias (i.e., the tendency to believe that you are better at a desirable intellectual skill than you actually are).

Despite the obstacles I noted above, the Assessment for Improvement Committee (AIC) felt like this was an outcome ripe for the assessing. First, we’ve never really measured quantitative literacy among Augustana students before (it wasn’t addressed in the Wabash National Study when we participated between 2008 and 2012). Second, it isn’t clear that we know how each student develops this skill, as we have defined it in our own college documents, beyond what a student might learn in a “Q” course required by the core curriculum. As a result, it’s entirely possible that we have established a learning outcome for all students that our required curriculum isn’t designed to achieve. Uh oh.

In all fairness, we do have one bit of data – imperfect as it is. A few years ago, we borrowed an idea from the National Survey of Student Engagement (NSSE) and inserted a question into our senior survey that asked students to respond to the statement, “I am confident in my ability to interpret numerical and statistical quantities,” giving them five response options that ranged from “strongly disagree” to “strongly agree.”

Since we began asking this question, about 75% of seniors have indicated that they “agree” or “strongly agree” with that statement. Unfortunately, our confidence in that number began to wain as we looked more closely at those responses. For that number to be credible, we would expect to see that students from majors that have no quantitative focus were less confident in their quantitative abilities than students from majors that employed extensive quantitative methods. However, we found the opposite to often be the case. It turned out that students who had learned something about how complicated quantitative methods can be were less confident in their quantitative literacy skills than those students who had no exposure to such complexities, almost as if knowing more about the nuances and trade-offs that can make statistics such a maddeningly imperfect exercise had a humbling effect. In the end it appeared that in the case of quantitative literacy, ignorance might indeed be bliss (a funny story about naming another bias called the Dunning-Kruger Effect).

So last year the AIC decided to conduct a more rigorous study of our students’ quantitative literacy skills. To make this happen, we first had to build an assessment instrument that matched our definition of quantitative literacy. Kimberly Dyer, our measurement ninja, spent weeks pouring over the research on quantitative literacy and the survey instruments that had already been created to find something that fit our definition of this learning outcome. Finally, she ended up combining the best of several surveys to build something that matched our conception of quantitative literacy and included questions that addressed interpreting data, understanding visual presentations of data, calculating simple equations (remember story problems from grade school?), applying findings from data, and evaluating the assumptions underlying a quantitative claim. We then solicited faculty volunteers who would be willing to take time out of their upper-level classes to give their students this survey. In the end, we were able to get surveys from about 100 students.

As you might suspect, the results of this assessment project provided a bit more sobering picture of our students quantitative literacy skills. These are the proportions of questions within each of the aforementioned quantitative literacy categories that students who had completed at least one Q course got right.

• Interpreting data  –  41%
• Understanding visual presentations of data  –  41%
• calculating simple equations  –  45%
• applying findings from data  –  52%
• evaluating assumptions underlying a quantitative claim  –  51%

Interestingly, students who had completed two Q classes didn’t fare any better.  It wasn’t until students had taken 3 or more Q classes that the proportion of correct answers improved significantly.

• Interpreting data  –  58%
• Understanding visual presentations of data  –  59%
• calculating simple equations  –  57%
• applying findings from data  –  65%
• evaluating assumptions underlying a quantitative claim  –  59%

There are all kinds of reasons that we should interpret these results with some caution – a relatively small sample of student participants, the difficulty of the questions in the survey, or the uneven distribution of the student participants across majors (the proportion of STEM and social science majors that took this survey was higher than the proportion of STEM and social science majors overall). But interpreting with caution doesn’t mean that we discount these results entirely. In fact, since prior research on students’ self-reporting of learning outcomes attainment indicates that students often overestimate their abilities on complex skills and dispositions, the 75% of students who agree or strongly agree is probably substantially higher than the proportion of graduates who are actually quantitatively literate. Furthermore, since the proportion of students who took this survey was skewed toward majors where quantitative literacy is a more prominent part of that major, these findings are more likely to overestimate the average student’s quantitative literacy than underestimate it. Triangulating these data with prior research suggests that our second set of findings might paint a more accurate picture of our graduates.

So how should we respond to these findings? To start, we probably ought to address the fact that there isn’t a clear pathway between what students are generally expected to learn in a “Q” course and what the college outcome spells out as our definition of quantitative literacy. That gap alone creates the condition in which we leave students’ likelihood of meeting our definition of quantitative literacy up to chance. So our first question might be to explore how we might ensure that all students get the chance to achieve this outcome; especially those students who major in disciplines that don’t normally include quantitative literacy skills.

The range of quantitative literacy, or illiteracy as the case might be, is a gnarly problem. It’s not something that we can dump onto an individual experience and expect that box to be checked. It’s hard work, but if we are serious about the learning outcomes that we’ve set for our students and ourselves, then we can’t be satisfied with leaving this outcome to chance.

Make it a good day,

Mark

It’s probably a bit of a reach to claim that the Institutional Effectiveness and Mission Fulfillment report (begrudgingly called the IEMF) is the cutting edge of data reporting, but it is true that this annual report is something that a lot of people work pretty hard on for several months at the end of each academic year. Unlike the college’s dashboard – a single page of data points that is supposed to cut the quantitative quick, the IEMF is a motherload of data and a treasure trove of information about Augustana College.

In past years we have posted the IEMF on the Institutional Research web page and hoped that people would look at it because, you know . . . nerd click-bait! Not since the first year that we produced this report have we hosted a public gathering to invite comment from anyone who might have an observation about the data and how it is conveyed. One thing I will not soon forget from that meeting was the degree to which data becomes political as soon as it becomes public, and therefore how important it is to convey precisely and anticipate how data presentations might be interpreted from different points of view.

With that in mind, I want to share with you the 2016 version of the IEMF. It is organized into nine sections that each cover different aspects of what and how we do what we do. For example, in the section titled Persistence, Graduation, and Attrition (p. 1) you might be interested in the distribution of reasons that students give for withdrawing and how those reasons might have changed over the last three years. Or, in the section titled Our Practices (p. 20) you might be interested in the rising costs to recruit a single student over the last three years. There are a lot of tidbits throughout the document that provide a glimpse into Augustana College – areas of strength, opportunities for growth, and how we compare to similar liberal liberal arts colleges around the country.

Click on the link below and swim in a river of data to your heart’s content.

2016_IEMF_Report

Certainly, the IEMF isn’t a perfect snapshot. Even though it has improved considerably from it’s first iteration several years ago, there are plenty of places where we wish our data were a little better or a little more precisely able to show who we are and what we do. Most importantly, this document isn’t intended to be a braggart’s bible. On the contrary, the IEMF is designed to be an honest presentation of Augustana College and of us. We aren’t perfect. And we know that. But we are trying to be as good as we can be with the resources we have. And in more than a few instances, we are doing pretty well.

Before I forget, a special and sincere “thank you” goes out to everyone who played a role in hunting down this data and putting the document together: Kimberly Dyer, Keri Rursch, Cindy Schroeder, Quan Vi, Erin Digney, Angie Williams, Katey Bignall, Kelly Hall, Randy Roy, Lisa Sears, Matt Walsh, Sheri Curran, Robert Scott, Jeff Thompson, Dom Sullivan, Katrina Friedrich, Bonnie Hewitt, Scott Dean, Shawn Beattie, and Kent Barnds.

So have a look. If you have any questions or critiques or suggestions, please send them to me. I’m genuinely looking for ways to improve this document.

For starters . . . anyone got any catchy ideas for a better name?

Make it a good day,

Mark

The more we dig into our retention data, the more interesting it gets. Earlier this term, I shared with you some of our findings regarding term-to-term retention rates. These data seem to suggest that we are slowly improving our within-year retention rates.

As always, the overall numbers only tell us so much. To make the most of the data we collect, we need to dig deeper and look at within-year retention rates for subpopulations of students that have historically left at a higher rate than their peers. Interestingly, this data might also tell us something about when these students are most vulnerable to departing and, as a result, when we might increase our focus on supporting their success.

The table below presents 2014-15 within-year retention rates of the five subpopulations of students that significantly deviated from the overall term-to-term retention rates. The percentages that are more than one point below the overall number are in red.

The first thing I’d like to highlight is a pair of subpopulations that aren’t on this list. Analyses of older data would no doubt highlight the lagging retention rates of students who came to Augustana with lower ACT scores or who applied test-optional (i.e., without submitting a standardized test score). However, in the 2014-15 cohort these subpopulations retained from fall to winter (96.9% and 97.9%, respectively) and from winter to spring (96.8% and 97.9%, respectively) at rates similar to the overall population. The winter-to-spring numbers are particularly encouraging because that is when first-year students can be suspended for academic performance. Although it would be premature to declare that this improvement results directly from our increased student support efforts, these numbers suggest that we may indeed be on the right track.

In looking at the table above, the highlighted demographic groups are probably not a surprise to those who are familiar with retention research. However, this table gives us  a glimpse into when certain groups are more vulnerable to departure. For example, male students’ retention rates are consistently lower than the campus average. By contrast, multicultural students were retained at a higher rate from fall to winter. But from winter to spring, our early success evaporated completely. Winter term might also play a role for non IL/IA residents who retain at rates similar to their peers from fall to winter but from winter to spring depart at a higher rate than the rest of the cohort.

Since this is only one year of data, I wouldn’t suggest making any emphatic claims based on it. But I do think that these findings should challenge us to think more deeply about the kind of support different types of student might need and when they might benefit most from it.

Make it a good day,

Mark

Last week I shared a comparison of the overall major advising data from seniors in 2014 and 2015. Although not all of the differences between the two years of data met the threshold for statistical significance, taken together it seemed pretty likely that these improved numbers weren’t just a function of chance. As you might expect by now, another aspect of this finding piqued my curiosity. Is this change a result of a relatively small campus-wide improvement or are the increases in the overall numbers a result of a particular department’s efforts to improve?

Since the distribution of our seniors’ major choices leans heavily toward a few departments (about half of our students major in Biology, Business, Psychology, or Education), it didn’t take too long to isolate the source of our jump in major advising scores. Advising scores in Business, Psychology, and Education didn’t change much between 2014 and 2015. But in Biology? Something pretty impressive happened.

Below is a comparison of the increases on each advising question overall and the increases on each advising question for Biology and Pre-Med majors.  In particular, notice the column marked “Diff.”

It’s pretty hard to miss the size of the increased scores for Biology and Pre-Med majors between 2014 and 2015. In every case, these increased scores are three or four times larger than the increases in overall scores.  In a word: Impressive!

So what happened?

Advising is a longstanding challenge for Biology and Pre-Med faculty. For decades this department has struggled to adequately advise a seemingly endless flow of majors. Last spring, Biology and Pre-Med graduated almost 150 students and at the beginning of the 2014-15 academic year there were 373 declared majors in either program. Moreover, that number probably underestimates the actual number of majors they have to work with since many students declare their major after the 10th day of the term (when this data snapshot was archived).

Yet the faculty in the Biology and Pre-Med department decided to tackle this challenge anyway. Despite the overwhelming numbers, maybe there was a way to get a little bit better by making even more of the limited time each adviser spent with each student. Each faculty adviser examined senior survey data from their own advisees and picked their own point of emphasis for the next year. Several of the Biology and Pre-Med faculty shared with me the kinds of things that they identified for themselves. Without fail, each faculty member decided to make sure that they talked about CORE in every meeting, be it the resources available in CORE for post-graduate preparation or just the value of making a visit to the CORE office and establishing a relationship. Several others talked about making sure that they pressed their advisees to describe the connections between the classes they were taking and the co-curricular activities in which they were involved, pushing their students to be intentional with everything they chose to do in college. Finally, more than one person noted that even though advising had always been important to them, they realized how easy it was to let one or more of the the usual faculty stresses color their mood during advising meetings, (e.g., succumbing to the stress of an upcoming meeting or a prior conversation). They found ways to get themselves into a frame of mind that improved the quality of their interaction with students.

None of these changes seem all that significant by themselves.  Yet together, it appears that the collective effort of the Biology and Pre-Med faculty – even in the face of a continued heavy stream of students, made a powerful difference in the way that students’ rated their advising experience in the major.

Improvement isn’t as daunting as it might sometimes seem. In many cases, it just takes an emphasis on identifying small changes and implementing them relentlessly. So three cheers for Biology and Pre-Med. You’ve demonstrated that even under pretty tough circumstances, we can improve something by focusing on it and making it happen.

Make it a good day,

Mark

With all of the focus on reaccreditation, budget concerns, employee engagement, and the consideration of a different academic calendar, it seems like we’ve spent a lot of time dwelling on things that aren’t going well or aren’t quite good enough. However, in the midst of these conversations I think we might do ourselves some good to remember that we are in the midst of doing some things very well. So before we plunge ourselves into another brooding conversation about calendar, workload disparity, or budget issues, I thought we could all use a step back from the precipice and a solid pat on the back.

You’d have to have been trapped under something thick and heavy to have missed all of the talk in recent years about the need to improve advising. We’ve added positions, increased the depth and breadth of training, and aspired to adopt an almost idyllic conception of deeply holistic advising. This has stretched many of us outside of our comfort zones and required that we apply a much more intentional framework to something that “in the old days” was supposed to be a relaxing and more open-ended conversation between scholar and student.

With this in mind, I thought it might be fun to start the spring term by sharing a comparison of 2014 and 2015 senior survey advising data.

Our senior survey asks seven questions about major advising. These questions are embedded into a section focused on our seniors’ experience in their major so that we can be sure that the student’s responses refer to their advising experience in each of their majors (especially since so many students have more than one major and therefore more than one major adviser). The first six questions focus on aspects of an intentional and developmental advising experience. The last question provides us with a way to put those efforts into the nitty gritty context of efficiency. In an ideal world, our student responses would show a trend toward higher scores on the first six questions, while average scores for the seventh question would remain relatively flat or even declining somewhat.

Here is a list of the senior survey advising questions and the corresponding response options.

• My major adviser genuinely seemed to care about my development as a whole person. (strongly disagree, disagree, neutral, agree, strongly agree)
• My major adviser helped me select courses that best met my educational and person goals. (strongly disagree, disagree, neutral, agree, strongly agree)
• How often did your major adviser ask you about your career goals? (never, rarely, sometimes, often, very often)
• My major adviser connected me with other campus resources and opportunities (OSL, CORE, the Counseling Center, etc.) that helped me succeed in college. (strongly disagree, disagree, neutral, agree, strongly agree)
• How often did your major adviser ask you to think about the connections between your academic plans, co-curricular activities, and your career or post-graduate plans? (never, rarely, sometimes, often, very often)
• My major adviser helped me plan to make the most of my college career. (strongly disagree, disagree, neutral, agree, strongly agree)
• About how often did you talk with your primary major adviser? (never, less than once per term, 1-2 times per term, 2-3 times per term, we communicated regularly through each term)

A comparison of overall numbers from seniors graduating in 2014 and 2015 seems to suggest a reason for optimism.

As you can tell, the change between 2014 and 2015 on each of these items aligns with what we would hope to see. We appear to be improving the quality of the student advising experience without taking more time to do so. Certainly this doesn’t mean that every single case reflects this overall picture, but taken together this data seems to suggest that our efforts to improve are working.

I suspect that more than a few of you are wondering whether or not these changes are statistically significant. Without throwing another table of data at you, here is what I found. The change in “how often advisers asked students to think about the links between curricular, co-curricular, and post-grad plans” (.16) solidly crossed the threshold of statistical significance. The change in “genuinely cared about my development” (.11) was not statistically significant. The change in each of the other five items (from .12 to .15) turned out to be “marginally significant,” meaning, in essence, that the difference between the two average scores is worth noting even if it doesn’t meet the gold standard for statistical significant.

The reason I would argue that these changes, when taken together, are worth noting is a function of looking at all of these changes together. The probability that all seven of these items would move in our intended direction randomly is less than 1% (.0078 to be exact). In other words, it’s likely that something is going on that would push all of these items in the directions we had hoped. Given the scope of our advising emphasis recently, these findings seem to me to suggest that we are indeed on the right track.

I know that there are plenty of reasons to pull the “correlation doesn’t equal causation” handbrake. But I’m not arguing that this data is inescapable proof. Rather, I’m arguing that these findings make a pretty strong case for the possibility that our efforts are producing results.

So before we get ourselves tied into knots about hard questions and tough choices over the next 10 weeks, maybe take a moment to remember that we can tackle issues that might at first seem overwhelming. It might not be easy, but where is the fun in “easy”?

Make it a good day,

Mark

One well-known feature of our trimester calendar is the crazy crush of busyness during the last few weeks of the term. You can see it in the eyes of almost every student as they trudge up and down the quad (and if you look closely you can see it in many instructors’ eyes too). Although it’s not fun, I’m not sure that experiencing this kind of crunch of deadlines is such a bad thing. After all, our students will more than likely be required to successfully juggle multiple time-sensitive pressures often after they graduate. So the issue may not be whether or how we might dial back the seemingly inevitable onslaught of curricular deadlines. Instead, our greatest contribution to our students might be to:

1. help them learn how to successfully navigate these experiences, and
2. ensure that our instructional interactions allow them to focus their energies on increasingly efficient navigating strategies instead of adding extraneous noise, unnecessarily vague direction, or downright confusion into the mix.

Last week, I shared findings that explored the things students can do to help them develop better time-management skills. This week I thought it might be useful to focus on the potential impact of instructional behaviors on this equation. Just like last week, these findings are the result of teamwork in the IR office. Katrina, one of my three student workers, and I worked together to develop the analysis and run the statistical tests; I’m responsible for writing up what we found.

In our mid-year freshman survey (data collected during the last third of fall term), we ask first-year students to tell us how often they struggle to balance their academics with their extra-curricular activities. They can choose from five response options that range from “most or all of the time” to “never.” Most of the students select “sometimes,” the middle response (i.e., a three), while the second most popular response selected is “rarely” (AKA a four).

Like last week’s analysis, we wanted to hold constant some of the typical potentially confounding variables – demographic traits and pre-college academic preparation – so that we could hone in on instructional behaviors that appear influential for all types of students. (This turned out to be particularly important for reasons that I’ll discuss later.) After a series of preliminary tests, we added three items to our analysis to see if they produced statistically significant results.

• I had access to my grades or other feedback early enough in the term to adjust my study habits or seek additional academic help as necessary.
• My LSFY/Honors instructor helped me develop at least one specific way to be a more successful college student.
• My first year adviser helped me understand my Student Readiness Survey (SRS) results.

The first two items offer five response options that range from “strongly disagree” to “strongly agree.” The question about the SRS includes a different set of four response options that seem more appropriate to the question: “We never talked about them (What is the SRS?),” “Only briefly,” “Yes, but they weren’t all that useful,” and “Yes, and they influenced how I approached the beginning of my freshman year.” We hypothesized that all three of these items might have positive effect on students’ frequency of struggling to balance academic and extra-curricular activities.

Our analysis confirmed our hypotheses in two out of the three cases (last week we were right once, wrong once, and neither once, so maybe we’ve improved our prognosticating skills a little bit since last week!). Even though one of the elements of the Student Readiness Survey focuses on academic habits, the first year adviser’s use of the Student Readiness Survey didn’t have any effect one way or the other. Maybe it was a bit aspirational to think that one conversation could impact something as potentially continuous or more likely intermittent as struggling to balance academic and co-curricular activities.

However, both early access to grades or other feedback and an LSFY/Honors instructor who helped the respondent develop at least one specific way to be a better student produced statistically significant positive effects. In other words, as students agreed more strongly that they had received access to information that helped them calibrate the effectiveness of their study habits they struggled less often to balance academics and co-curricular activities. Similarly, as students agreed more strongly that their LSFY/Honors instructor helped them become a better college student, the students indicated that they struggled less often in balancing academics and co-curricular activities. In both cases, these findings held even after accounting for differences in race/ethnicity, gender, socio-economic status, and pre-college academic preparation (i.e., ACT score).

Thus, it appears that instructional behaviors can play an important role in helping students develop the ability to effectively balance their curricular and co-curricular obligations. Instructors can certainly design their courses to provide substantive feedback early in the term instead of waiting until a mid-term exam or late term paper assignment. Likewise, LSFY/Honors instructors can insert experiences or assignments into their courses that, instead of merely focusing on the production of intellectual work, also focus on the process of more efficiently and effectively producing intellectual work.

Now, remember my foreshadowing about the importance of demographic traits in this analysis? Even in the final equation, being female or being white still produced statistically significant positive effects. In other words, even after accounting for all of the other variables in our equation, white students and female students struggled to balance less often than their non-white and male peers.

It makes intuitive sense that gender would appear significant in this equation. We know from all sorts of research on men that their time-management skills and general maturity is usually behind women at this age. This finding highlights the additional effort that we need to make to help male students grow. Finally, the finding about race struck me as particularly important. In the context of the other research we’ve published about a lower sense of belonging scores for non-white students, maybe the findings in the current study reflect another way in which the collateral damage that comes from feeling less like you belong on our campus hinders the success of minority students. We have all experienced the negative effects of disturbing distractions that hinder productivity. Maybe the persistent negative effect of being non-white is a function of just such a destructive distraction and is another confirmation of the corrosive effect of stereotype threat.

We all want our students to be active and involved on campus. We also want them to develop and maintain a healthy balance between their academic and co-curricular obligations. It seems to me that historically we have put the onus for maintaining this balance on our students, providing all sorts of guidance on time management, priority setting, and life organization. Maybe, just maybe, we could also contribute to their success by focusing more on what we do to set them up for success. And I don’t mean dialing back our expectations of them. Instead, we might add to their potential for growth if we help them obtain and use the right information at the right time in order to continually (re)calibrate their balancing efforts. In the midst of the seemingly accelerated trimester calendar, this might be even more important than usual.

Good luck with week 10, everyone. For all of you who by now have noted that I’ve pointed out the increased busyness that occurs at the end of the term and followed that with an unusually long blog post that has just soaked up that much more of your time . . . I’m truly sorry.  Ok, well kind of sorry.

Make it a good day,

Mark

A couple of months ago, I shared our findings from last year’s freshman data about the impact of study behaviors on GPA.  In essence, we found that studying in one’s dorm room appears to reduce GPA while studying during the day appears to increase it. One of my student workers, Katrina Friedrich, and I have been digging further into this set of data to identify significant predictors of another indicator of academic success: time management. It turns out that even after accounting for several other important student characteristics, studying during the day (again) makes a difference.

At the end of the first year, we ask freshman to respond to the statement, “During the year I got better at balancing my academics with my out-of-class activities.” Students can choose from a set of five “strongly agree” to “strongly disagree” response options. Since we also ask these students to evaluate their own academic habits in the Student Readiness Survey (a survey they take before they enroll at Augustana), as well as asking them near the end of the first term how often they think they struggle to balance academics and extra-curricular activities, we have a pretty good way of holding constant their pre-college and first-term assessment so that we can hone in on their sense of improvement over the course of the first year.

In addition to accounting for where our students might be on a spectrum of time management prior to college, our analysis also needs to account for variations in unscheduled time during the first year. Balancing curricular and co-curricular obligations is certainly impacted by the activities one chooses to join. For example, the pressure to balance multiple responsibilities would likely be higher for student-athletes because they have less discretionary time. The same is likely true for students who participate in music ensembles, for those who have committed themselves to be highly involved in a student organization or club, or even those students who work.

With both of these factors (early assessments of time management and potentially confounding first-year experiences) held in check, we tested three study behaviors to see if, and how, they might influence students’ sense of their own improvement in balancing curricular and co-curricular obligations:

• Using a planner
• Studying in one’s dorm room
• Studying during the day

For those of you scoring at home, studying during the day was the clear winner. The more students said they studied during the day, the higher they rated their improvement in balancing academics and co-curricular activities – regardless of their other obligations or pre-college time management skills. Although studying in the dorm negatively affected GPA in our prior study, it didn’t have any effect one way or the other this time around. I suppose that it’s good to know that studying in the dorm didn’t hurt time management, but that isn’t exactly a ringing endorsement for studying in one’s dorm room, either.

The finding regarding the use of a planner was perplexing. Although the effect barely missed the threshold for statistical significance (for all of you stats nerds, the p-value equalled .053 when it needed to be less than .05), the direction of the effect was negative. In other words, student’s increased use of a planner appears to inhibit their sense of improvement in balancing academics and co-curricular activities.

With my apologies to the stats gods, for the purposes of this discussion let’s round that .053 to a .05 and say that our finding was statistically significant (after all, we are still talking about a 94.7% likelihood that this finding is not a function of chance). Why might the increased use of a planner reduce one’s sense of improvement in balancing academics and co-curricular activities? Katrina suggests that many students don’t use their planner for anything beyond keeping a list of what they have to do, as opposed to allotting differing amounts of time to complete specific tasks, take care of various chores, etc. Or maybe students, particularly freshmen in this case, don’t really have the ability to estimate how long most academic tasks take. Some of you might have insights to share here, since I think we all probably tell students to use a planner, assuming that they will know how to use it.

In the end, we again found that making it a priority to study during the day is beneficial to student success, this time in terms of the development of a key skill: time management. In conjunction with the previous positive effects we’ve found that result from studying during the day, it seems that this should be one of the things we emphatically encourage our students to do.

But I’m curious to hear what you might think of the findings involving the use of a planner. It could be something messy in the data or the result of a variable that we haven’t accounted for yet. Even though it didn’t quite meet the significance threshold, the direction of the effect makes me wonder if this is another thing that we tell students to do while not realizing that they need much more information on the “how” and “why” in order to gain the benefit that we assume will come from it. Curious, indeed.

Make it a good day,

Mark

There is a lot going on this week, so I hope I can make this brief! (I know, I know . . .)

Last week I shared the second round of results from our study of Augustana students’ changing (or not changing) motivational orientations.  The first layer of results was a mixed bag . . . our students inclination toward extrinsic motivation decreased, but their inclination toward intrinsic motivation after four years remained unchanged.  The second layer of results found that, perhaps surprisingly to some, none of our measures of curricular experiences impacted our students’ inclination toward intrinsic motivation.  Instead, the primary driver of an increased intrinsic motivation orientation was our students’ perception of how much their out-of-class experiences had impacted their understanding of themselves.

But it was the other finding highlighted in my last post that I thought was particularly appropriate for the upcoming week. In addition to the impact of students’ out-of-class experiences, we found that the degree to which our students said that “Symposium Day activities influenced the way I now think about real world issues” significantly predicted (in a statistical sense) an increase in our students’ inclination toward intrinsic motivation.  In responding to my post one reader raised a good question, asking if we could discern whether these students’ engagement in Symposium Day was a reflection of their own dispositions or if this engagement in Symposium Day was a function of faculty and staff educational designs.  In essence, this question goes to the heart of our effort to tease out relationships in our data that we can rely on to design influential change.  And it gets at a pretty important issue for higher education folks: Can our educational efforts alter the course of our student’s learning or is their learning a function of the dispositions and abilities that they bring with them to college?

Of course, the question I just posed is really a false dichotomy.  It would be pretty hard to believe that any college’s educational efforts could completely overrun the dispositions that our students bring with them to college.  But we can test whether the things that we do light a spark that wouldn’t otherwise appear, functioning as an accelerator for those who are already heading in the right direction or as a pair of jumper cables for those whose engine just won’t seem to turn over.  Sparing the automotive metaphors, we can examine our data to find out if integrating learning experiences is actually improving our students’ growth.

So do we have any way to identify predictors of our students’ response to the statement “Symposium Day activities influenced the way I now think about real world issues”?  In fact, we do.  In our freshman year survey, we also ask our first-year students to respond (along five-response options ranging from strongly disagree to strongly agree) to the statement, “My instructors integrated themes or ideas from Symposium Day into their courses.”  Although this data comes from a different set of students than the senior survey data, I think its reasonable to suggest that whatever we find in our first-year data is applicable to all Augustana students.

It turns out that the degree to which first-year students say that their instructors integrated Symposium Day into their courses is a powerful predictor of the degree to which students say that Symposium Day activities influenced the way that they now think about real world issues, even after accounting for these students’ pre-college dispositions and traits.  While I wasn’t too surprised that this statistical relationship would appear, I was impressed by the size of the effect.

To put it simply, it seems pretty likely that when instructors integrate Symposium Day into their courses students come away from Symposium Day genuinely influenced by what they’ve heard.  And based on the analysis I shared last week, it may also be true that integrating courses with Symposium Day indirectly impacts our students’ development of a stronger inclination toward intrinsic motivation.  Maybe some students arrive at a presentation only because they have been required to attend. But by the time they leave, it appears that the chances are good that they will come away with something that sticks.

Well, I’ll be … who’d a thunk it!  Wednesday of this week just happens to be Symposium Day.  It isn’t too late to find a way to link your course to something happening on that day. Integrating learning experiences makes a difference.  More and more, we have evidence to show it.

Make it a good day (today and Wednesday),

Mark

Welcome to 2016!  It’s great to see all of you scurrying around campus.

Earlier this fall, I shared the first set of findings from our four-year assessment of Augustana students’ motivational orientations. As you might remember, this is the first set of results from our institutional outcomes assessment protocol in which we rotate through all of the college-wide student learning outcomes so that each year we have a new set of freshman-to-senior results for a different learning outcome. Moreover, because of the student experience data we collect over four years, we also have ways to identify experiences that appear to influence student gains (or losses) on those outcomes.

Intrinsic motivation is a key element of our students’ intrapersonal development. We talk about this attribute most specifically when we refer to the importance of a liberal arts education cultivating our students as life-long learners. Interestingly, although we aspire to strengthen our students’ orientation toward intrinsic motivation, the results of our four-year assessment revealed a change between the average freshman score and the average senior score was not statistically significant. In other words, our students’ orientation toward intrinsic motivation didn’t change – even though other research on motivational orientations suggests that individuals become more inclined toward intrinsic motivation as they get older.

Even though the average scores didn’t move like we would have hoped, there were certainly students who showed statistically significant gains in their orientation toward intrinsic motivation. So what makes those students different from the rest? And more importantly, are there lessons that we can learn from those students’ experience that we could apply more broadly?

To find answers to those questions, we designed an analysis that would test the impact of a variety of Augustana experiences. We tested the impact of curricular experiences, advising experiences, pre-college demographics and values, and co-curricular experiences. Sure enough, students who showed significant positive growth in their inclination toward intrinsic motivation also had two experiences in common, experiences that we ought to consider cultivating more broadly as we continue to improve the quality of the Augustana experiences.

First, the nature of the students’ co-curricular experiences produced a robust and statistically significant positive effect on the inclination toward intrinsic motivation. Specifically, as students more strongly agreed with the statement, “My out-of-class experiences have helped me develop a deeper understanding of myself,” they exhibited stronger gains in their inclination toward intrinsic motivation. Importantly, our findings suggest that mere participation in co-curricular experiences wasn’t enough. Instead, the effect came from the students’ perceived impact of those co-curricular experiences.

Second, students’ engagement in Symposium Day also produced a statistically significant, albeit smaller, effect. As students’ more strongly agreed with the statement, “Symposium Day activities influenced the way I now think about real world issues,” they made larger gains in their inclination toward intrinsic motivation. This is an item where the average score has increased each year since Symposium Day was introduced, even though senior survey results indicate that there is still substantial room for improvement (42.4% of 2015 seniors disagreed or strongly disagreed with this statement).

Somewhat surprisingly, neither the nature of the students’ classroom experience or advising experiences generated any effect on an orientation toward intrinsic motivation. Likewise, we accounted for sex, race, and socioeconomic status in our analysis and none of those variables produced a statistically significant effect.

So what might we make of these findings? First (as if we needed more evidence at this point), out-of-class experiences matter, a lot. And it’s not about quantity; it’s about quality. This is exactly the philosophy that undergirds the entire integration emphasis in the Augustana 2020 strategic plan. Students need to engage in experiences that help them grow in important ways. That kind of development doesn’t happen automatically. And every experience doesn’t necessarily produce the same type of growth, or any growth at all. This finding seems to re-emphasize the value of designing co-curricular experiences so that key teachable moments are most likely to occur, then prodding students to reflect on those moments with an eye toward how their own responses might teach them something about themselves.

Second, we’ve begun to notice some anecdotal suggestions of the educational value of Symposium Day and this finding presents further evidence that Symposium Day can, and is, impacting the growth of our students in important ways. One key take-away from this study reiterates that participation is necessary but not sufficient. The impact of Symposium Day in this study appears to come from the degree to which students felt that the experience shaped the way that they think about real world events. In other words, the value of Symposium Day in the context of this study is in the applicability of the learning and the way in which the experience can inspire students to reflect on their perceptions of real world events in the context of their Symposium Day experience. This suggests that all of the ways in which faculty and staff can link the curricular or co-curricular work they are doing with students to elements of Symposium Day may well be producing more than a deeper understanding of content knowledge.

As we build toward the next Symposium Day on January 20th, I hope you will find more ways to connect your work with students to the events and presentations scheduled for that day. And as we continue to reassess and redesign our students’ out-of-class experiences to maximize their educational and developmental benefit, I hope you will look for ways to link these experiences to our students’ understanding of themselves.

Make it a good (albeit cold) day,

Mark