It is becoming increasingly apparent in the world of quantitative research that producing a single statistically significant finding shouldn’t carry too much weight (whether or not the value of a single statistically significant finding should have ever been allowed to rise to such a level of deference in the first place is a thoroughly valid discussion for a more technical blog than mine here or here).  In recent years, scholars in medicine, psychology, and economics (among others) have increasingly failed in their attempts to reproduce the statistically significant findings of an earlier study, creating what has been labeled “the replication crisis” across a host of disciplines and splattering egg on the faces of many well-known scholars.

So in the interest of making sure that our own studies of Augustana student data don’t fall prey to such an embarrassing fate (although I love a vaudevillian cracked egg on a buffoon’s head as much as the next guy), I thought it would be worth digging into the archives to rerun a prior Delicious Ambiguity analysis and see if the findings can be replicated when applied to a different dataset.

In February 2014, I posted some analysis under the provocative heading, “What if early feedback made your students work harder?”  Looking back, I’m a little embarrassed by the causal language that I used in the headline (my apologies to the “correlation ≠ causation” gods, but a humble blogger needs some clickbait!).  We had introduced a new item into our first-year survey that asked students to indicate their level of agreement (or disagreement) with the statement, “I had access to my grades and other feedback early enough in the term to adjust my study habits or seek help as necessary.”  The response set included the usual suspects: five options ranging from strongly disagree to strongly agree.

While we found this item to significantly predict (in a statistical sense) several important aspects of positive student interactions with faculty, the primary focus of the February 2014 post turned to another potentially important finding. Even after accounting for several important pre-college demographic traits (race, sex, economic background, and pre-college academic performance) and dispositions (academic habits, academic confidence, and persistence and grit), the degree to which students agreed that they had access to grades and other feedback early in the term significantly predicted student’s response to this item: “How often did you work harder than you have in the past in order to meet your instructor’s expectations?”  In essence, it appeared that students who felt that they got more substantive feedback early in the term also tended to work harder to meet their instructor’s expectations more often.

Replication is risky business. Although the technical details that need to be reconstructed can make for a dizzying free-fall into the minutiae, committing to reproduce a prior study and report the results publicly sort of feels like playing Russian Roulette with my integrity.  Nonetheless, into the breach rides . . . me and my trusty data-wrangling steed.

Although it would have been nice if none of the elements of the analysis had changed, that turned out not to be the case – albeit for good reason.  We tend to review the usefulness of various survey items every couple of years just to make sure that we aren’t wasting everyone’s time by asking questions that really don’t tell us anything.  This turned out to be a possibility with the original wording of the item we were predicting (what stats nerds would call the dependent variable). When we put the statement, “How often did you work harder than you have in the past in order to meet your instructor’s expectations?” under the microscope, we saw what appeared to be some pockets of noise (stats nerd parlance for unexplainable chaos) across the array of responses. Upon further evaluation, we decided that maybe the wording of the question was a little soft.  After all, what college freshman would plausibly say “never” or “rarely” in response?  I think it’s safe to assume that most students would expect college to make them work harder than they had in the past (i.e., in high school) to meet the college instructor’s expectations. If we were a college where students regularly found the curricular work easier than they experienced in high school . . . we’d have much bigger problems.

Since the purpose of this item was to act as a reasonable proxy for an intrinsically driven effort to learn, in 2016 we altered the wording of this item to, “How often did you push yourself to work harder on an assignment even though the extra effort wouldn’t necessarily improve your grade?” and added it to the end-of-the-first-year survey.  Although this wording has proven to increase the validity of the item (subsequent analyses suggests that we’ve reduced some of the previous noise in the data), it’s important to note at the outset that this change in wording and relocation of the item to the end of the year alters the degree to which we can precisely reproduce our previous study.  On the other hand, if the degree to which students get early feedback (an item that is asked at the end of the fall term) significantly predicts the degree to which students push themselves to work harder on their homework regardless of their grade (now asked at the end of the spring term) in the current replication study, it strikes me that this finding might be even more important than the 2014 study.

Thankfully, all of the other variables in the 2016-17 data remained the same as the 2014-15 first-year data. So . . . what did we find?

I’ve provided the vital statistics in the table below.  In a word – bingo!  Even after taking into account sex, race/ethnicity, socioeconomic status (i.e., Pell grant status), pre-college academic performance (i.e., ACT score), academic habits, academic confidence, and persistence and grit, the degree to which students receive early feedback appears to significantly predict the frequency of pushing oneself to work harder on assignment regardless of whether or not the extra effort might improve one’s grade.

Variable	Standardized coefficient	Standard error	P-value
Sex (female = 1)	0.022	0.143	0.738
Race/ethnicity (white = 1)	0.001	0.169	0.089
Socioeconomic Status (Pell = 1)	-0.088	0.149	0.161
Pre-college academic performance	-0.048	0.010	0.455
Academic habits scale	0.317 ***	0.149	0.000
Academic Confidence scale	-0.065	0.167	0.374
Persistence and grit scale	0.215 **	0.165	0.010
Received early feedback	0.182 **	0.056	0.005

It is particularly intriguing to me that the statistically significant effect of receiving early feedback in the fall term appears when the outcome item is asked at the end of the spring term – a full six months later. Furthermore, it seems important that receiving early feedback produces a unique effect even in the presence of measures of academic habits (e.g., establishing a plan before starting a paper, starting homework early, etc.) and persistence and grit (e.g., continuing toward a goal despite experiencing disappointment, sticking with a plan to reach a goal over a longer period of time, etc.), both of which produce unique effects of their own.

The implications of these findings seem pretty important. In essence, no matter the student’s pre-college academic performance, the degree of positive academic habits, or the depth of persistence and grit when arriving at Augustana, receiving early feedback in the fall term appears to improve a student’s likelihood of working harder on their schoolwork no matter how that effort might impact their grade.

Whew!  I guess my integrity survives for another day.  More importantly (MUCH more importantly), it seems even more clear now after replicating the 2014-15 finding with 2016-17 data, that creating ways to provide early feedback to students so that they can recalibrate their study habits as necessary appears to be a critical element of effective course design.

Make it a good day,

Mark