Sometimes social science researchers get too excited about testing new hypotheses and forget about the importance of retesting old ones. Although it’s understandable (why drive a used car when you could drive a new car?), this tendency is exceedingly detrimental to the body of knowledge we claim to know. Because no matter how perfect the study design or how fantastic the results, one set of findings just doesn’t mean that much – a reality that often gets lost in the hype.

In recent years, the tendency to overhype a single set of findings has become the subject of much hand-wringing. In 2010, the New Yorker published a longer piece about a phenomenon called the decline effect where efforts to replicate prior studies are increasingly producing comparatively smaller and sometime even insignificant results. Such results call into question the validity of many prior research findings. A 2013 article in the Economist outlined other research that produced similarly chilling reminders of the fallibility of science and scientists. Not to be outdone, this conundrum starts to get really weird when a 2015 replication study appearing to challenge the validity of 100 well-known psychology findings was taken apart by a 2016 study that critiqued many of the 2015 study’s replication designs and summary conclusions.

I say all this to set up what might otherwise seem like a pretty mundane data point about first-generation students. But first, what do we think we know about first-gen students?

According to the current body of research on first generation students, the existing evidence suggests that these students a more likely to lack basic knowledge about how college is supposed to work. In the absence of this knowledge, the fog is a little thicker, the path is less clear, and they are more susceptible to feeling lost and uncertain about their progress. All this sets up an increased vulnerability that heightens the potential for difficulty and early departure. Although we can see the gap in first-second year retention rates between first-gen students and their peers, differences in retention rates don’t necessarily confirm the more granular elements of prior findings about the first-gen experience.

To find that kind of granular confirmation, we need to identify specific items in the first year surveys that could suss out these differences, parse the array of data we gather from first year students by first generation status, and test for statistically significant differences.

One prime possibility is a survey item from the end of the first year that asks first-year students to respond (i.e., choosing from 5 options that range from strongly disagree to strongly agree) to the statement, “Reflecting on the past year, I can think of specific experiences or conversations that helped me clarify my life/career goals.” If the first-generation student experience involves a relatively higher frequency of feeling lost or unsure about how to connect all of one’s activities, classes, and experiences into a coherent narrative, then first generation student responses, on average, should end up lower (and statistically significantly lower) than the overall response.

It turns out that this gap in average responses is profound.  While the overall average score is 3.83 (which translates to just south of ‘agree’), the average score for first-gen students is 3.23 (just north of ‘neither agree nor disagree’), a gap that amounts to an “extremely” statistically significant difference (i.e., p<.001 for all you quant nerds out there). Since we can conclude from these two mean scores that the average response from non first-gen students is a good bit higher than 3.83, it’s even more clear that whatever is going on isn’t merely a function of chance.

It’s possible that this difference mirrors the degree to which first-generation students simply do not engage in as many potentially influential activities and experiences as other students. If this were the case, we’d likely see these differences emerge elsewhere in the data. However, every other measure of involvement and participation suggests that there are no differences in frequency of engagement between first-generation students and their peers.

So maybe this difference in recalling specific experiences of conversations that helped clarify life/career goals is exactly the kind of thing that we might expect based on our prior understanding of first-generation students’ experience. Maybe first-gen students are engaged in the same average number of experiences as other students, but they are less likely to recognize the potential value of these experiences. As a result, maybe not knowing to look for the potential value of an experience makes it less likely that these students would see a way to connect these experiences to a longer-term goal.

It seems that this finding fits with our prior understanding of first-generation students. It also has important implications for the way that we talk with first-gen students about what they are doing in college. More than simply suggesting what they might do, it appears that first-gen students might need even more explicit guidance about how to reflect on the impact of a given experience, how that reflective activity might help them decide what experiences to prioritize, and how to connect what they might have learned through one experience with the developmental purpose of a subsequent experience.

In future years it’s very likely that a healthy proportion (about a third) of our new students will continue to be first-generation students. Much of what they don’t know about college is stuff that they don’t know they need to know. So our job is not only to tell them what they could do, but to show them how to decide what to do and how to use what they learn through those experiences to guide their future choices.

Make it a good (snow) day,

Mark