It’s great to see the campus bustling again.  If you’ve been away during the two-week break, welcome back!  And if you stuck around to keep the place intact, thanks a ton!

Just in case you’re under the impression that every nugget of data I write about comes pre-packaged with a statistically significant bow on top, today I’d like to share some data findings from our senior survey that aren’t so pretty. In this instance, I’ve focused on data from the nine questions that comprise the section called “Experiences in the Major.” For purposes of brevity, I’ve paraphrased each of the items in the table below, but if you want to see the full text of the question, here’s the link to the 2015-16 senior survey on the IR web page. The table below disaggregates the responses to each of these items by Hispanic, African-American, and Caucasian students. The response options are one through five, and range either from strongly disagree to strongly agree or from never to very often (noted with an *).

Item	Hispanic	African-American	Caucasian
Courses allowed me to explore my interests	3.86	3.82	4.09
Courses seemed to follow in a logical sequence	3.85	3.93	4.11
Senior inquiry brought out my best intellectual work	3.61	4.00	3.78
I received consistent feedback on my writing	3.72	4.14	3.96
Frequency of analyzing in class *	3.85	4.18	4.09
Frequency of applying in class *	3.87	4.14	4.15
Frequency of evaluating in class *	3.76	4.11	4.13
Faculty were accessible and responsive outside of class	4.10	4.21	4.37
Faculty knew how to prepare me for my post-grad plans	3.69	4.00	4.07

Clearly, there are some differences in average scores that jump out right away. The scores from Hispanic students are lowest among the three groups on all but one item. Sometimes there is little discernible difference between African-American and Caucasian students’ score while in other instances the gap between those two groups seems large enough to indicate something worth noting.

So what makes this data messy? After all, shouldn’t we jump to the conclusion that Hispanic students’ major experience needs substantial and urgent attention?

The problem, from the standpoint of quantitative analysis, is that none of the differences conveyed in the table meet the threshold for statistical significance. Typically, that means that we have to conclude that there are no differences between the three groups. But putting these findings in the context of the other things that we know already about differences in student experiences and success across these three groups (i.e., differences in sense of belonging, retention, and graduation) makes a quick dismissal of the findings much more difficult. And a deeper dive into the data both adds more useful insights to the mess.

The lack of statistical significance seems attributable to two factors. First, the number of students/majors in each category (570 responses from Caucasian students, 70 responses from Hispanic students, and 28 responses from African-American students) makes it a little hard to reach statistical significance. The interesting problem is that, in order to increase the number of Hispanic and Black students we would need to enroll more students in those groups, which might in part happen as a result of improving the quality of those students’ experience. But if we adhere to the statistical significance threshold, we would have to conclude that there is no difference between the three groups and would then be less likely to take the steps that might help us improve the experience, which would in turn improve the likelihood of enrolling more students in these two groups and ultimately get us to the place where a quantitative analysis would find statistical significance.

The other factor that seems to be getting in the way is that the standard deviations among Hispanic and African-American students is unusually large. In essence, this means that their responses (and therefore their experiences) are much more widely dispersed across the range of response options, while the responses from white students are more closely packed around the average score.

So we have a small number of non-white students relative to the number of white students and the range of experiences for Hispanic or African-American students seem unusually varied. Both of these finds make it even harder to conclude that “there’s nothing to see here.”

Just in case, I checked to see if the distribution of majors among each group differed. They did not. I also checked to see if there were any other strange differences between these student groups that might somehow affect these data. Although average incoming test score, the proportion of first-generation status, and the proportion of Pell Grant qualifiers differed, these differences weren’t stark enough to explain all of the variation in the table.

So the challenge I’m struggling with in this case of messy data is this:

We know that non-Caucasian students on average indicate a lower sense of belonging than their Caucasian peers. We know that our retention and graduation rates of non-white students are consistently lower than white students. We also know that absolute differences between two groups of .20-.30 are often statistically significant if the number of cases in each group is closer in size and if the standard deviation (aka dispersion) is in an expected range.

As a result, I can’t help thinking that just because a particular analytic finding doesn’t meet the threshold for statistical significance doesn’t necessarily mean that we should discard it outright. At the same time, I’m not comfortable arguing that these findings are rock solid.

In cases like these, one way to inform the inquiry is to look for other data sources with which we might triangulate our findings. So I ask all of you, do any of these findings match with anything you’ve observed or heard from students?

Make it a good day,

Mark