Introduction
During the 2023-24 school year, approximately 11% of Michigan’s K-12 students took at least one virtual course. These 154,056 students comprised over 1,019,000 enrollments, 71% of which came from grades 9-12. In addition, 68% of Michigan school districts reported at least one virtual enrollment (Freidhoff et al., 2024). Despite the number of students opting for online learning, virtual pass rates remain lower than non-virtual ones (Freidhoff, 2015). For example, the overall pass rate for virtual courses during the 2023-24 school year was 63% compared to 74% for non-virtual courses (Freidhoff et al., 2024). Perhaps this difference can partially be attributed to the various challenges students face that are unique to online education (Johnson et al., 2023).
Indeed, the flexibility of learning “anytime, anywhere,” particularly in asynchronous courses, requires students to possess or acquire a strong foundation of self-regulation, metacognitive and time management skills (Johnson et al., 2023; Digital Learning Institute, nd) because students can submit assignments at any time (but typically guided by an end-of-term deadline) and in any order they like, as courses are typically open (as opposed to having release conditions). Certain Michigan Virtual courses, such as those for core subject areas and electives, fall into this category, meaning students may submit any assignment at any time and in any order during the course term. To help students experience success in their online courses, Michigan Virtual provides students with pacing guides against which to benchmark their progress. Pacing guides outline the order in which students should complete course content. In other words, pacing guides show students which assignments they should complete each week to stay on track within their course.
While there is a growing body of research pointing to the role that the timing of assignment submissions plays in students’ course performance (Carvalho et al., 2022; DeBruler, 2021; Dunlosky et al., 2013; Kim & Seo, 2015; Lim, 2016; Michigan Virtual Learning Research Institute, 2019), limited research exists exploring how the order of students’ assignment submissions is associated with course outcomes. Research that has examined the order of students’ assignment submissions has focused on massive open enrollment courses or university students and, thus, is difficult to generalize to K-12 populations (e.g., Perna et al., 2014; Lim, 2016b). However, Cuccolo & DeBruler (2024) found that submitting assignments out of order was common among students enrolled in Michigan Virtual STEM courses. Submitting assignments out of order seemed to be the rule and not the exception, with about 93% of students submitting at least one assignment out of order. Despite being common, this behavior was not beneficial for students, as students who submitted at least one assignment out of order had final course scores that were 9.5 points lower than students who completely adhered to course pacing guides. In Michigan Virtual STEM courses, the order of students’ assignment submissions is associated with their final course scores (Cuccolo & DeBruler, 2024), suggesting that assignment sequencing may be an understudied aspect of pacing that has important implications for student outcomes.
Cuccolo & DeBruler’s original study focused solely on STEM courses, as the highly scaffolded nature of these courses was an ideal choice for examining the impact of pacing guide deviations. Because the amount of scaffolding, number and type of assignments, and course structure (linear vs. non-linear) likely vary by content area and individual course, it stands to reason that the extent to which student assignment submission patterns relate to final course scores may differ between subject areas. Preliminary analyses of assignment sequencing patterns in five core subject areas (English Language and Literature, Life and Physical Sciences, Mathematics, Social Sciences and History, and World Languages) revealed that moving out of alignment with course pacing guides was an especially common behavior for students enrolled in World Language courses. As such, the current study aimed to obtain a deeper understanding of the prevalence and impact of this behavior in these World Language courses.
Methods
Data & Sample Overview
A subset of highly enrolled Michigan Virtual World Languages courses was selected to facilitate generalizability about students’ sequencing behavior within these courses. The table in Appendix A shows the enrollment information used in the selection process and the number of enrollments included in the current study. A Michigan Virtual Technology Integration team member provided enrollment data from the selected courses for Spring 2024 (the most recently completed semester at the time the data was requested and pulled).
Analysis
Several variables were created to facilitate the analyses and answer the aforementioned research questions. First, because course pacing guides structure assignments sequentially (i.e., Assignments 1, 2, 3, 4, etc.), each student’s assignment submission was benchmarked against the one immediately preceding it—this ‘User-Driven’ variable coded in-sequence assignments with a 0 and out-of-sequence assignments with a 1. The total number of assignments submitted out of order was then calculated for each student.
Next, to better contextualize students’ deviation from course pacing guides, the number of assignments submitted out of order was divided by the total number of assignments the student submitted and multiplied by 100 (‘Percentage of Assignments Submitted Out of Order’). For example, if a student submitted 50 assignments, 5 of which were out of order, the percentage out of order would be 10 percent.
The ‘Magnitude’ variable represents the difference between the intended submission order of consecutively submitted assignments and was used to understand the degree to which students submitted assignments out of alignment with the course pacing guides. For example, if a student submitted assignment 5 and then 17, the magnitude value would be 12. Each student’s respective magnitude values for all submitted assignments were averaged (‘Average Magnitude’) to obtain a value that captured, on average, the extent to which they deviated from course pacing guide expectations.
Finally, based on how students’ assignment submission patterns were categorized, students were assigned to one of two user sequence groups. Students who submitted at least one assignment out of order were assigned to the out-of-sequence group. In contrast, if students submitted all their assignments in order (aligned to pacing guide expectations), they were assigned to the in-sequence group. Appendix A contains the names and definitions of key variables referenced in the report and examples when applicable. Table 1 below provides an example of the data layout for readers.
| Assignment Name | User-Driven | Pacing Guide | Magnitude |
|---|---|---|---|
| 11.1 Ein Quiz | 1 | 47 | 25 |
| 11.2 Ein Quiz | 0 | 48 | 0 |
| 5.3 Das Essen vorbereiten Arbeitsblatt | 1 | 23 | 25 |
| 6.3 Mein Selbstbericht | 1 | 27 | 4 |
| 6.1 Mahlzeit Diskussion | 1 | 25 | 2 |
| 4.3 Eine Aufnahme: Videodiskussion | 1 | 19 | 6 |
Results
Sample Description
After removing duplicate enrollments (students who were enrolled in more than one of the courses selected for the study) and outliers (students who had completed fewer than 50% of their assignments to ensure reliability, given analyses focused on understanding how assignment submission patterns related to course performance), the final sample consisted of 1,873 students. On average, students in the sample had completed about 3.29 (SD = 2.99) online courses. During the semester in which the data were collected (Spring 2024), students carried a Michigan Virtual (MV) online course load of about one course (SD = 0.73). Most students (60.86%, n = 1140) attended schools where the Non-White School Population was 25% or less. Additionally, about 39.94% of students came from Mid-Low Poverty schools, whereas only 4.91% attended high-poverty schools.
When examining the sample by school locale classification, approximately 25.79% (n = 483) of students were from large suburban areas, followed by 13.61% (n = 255) from rural fringe areas. Regarding entity type, most students (86.92%, n = 1628) came from LEA (Local Education Agency) Schools. American Sign Language 1B had the highest student enrollment of the courses included in the study. Please review the table in Appendix B for a breakdown of the number of students in each course.
Research Questions
What does students’ assignment sequencing look like in World Language courses?
What percentage of students go out of sequence? Is going out of sequence a common behavior?
The majority of students sampled went out of sequence at least once (n = 1,817, 97.01%). This left only 56 students (2.99%) who completed their World Language course fully aligned with their course pacing guides. As such, going out of sequence appeared to be the norm, implying that students in World Language courses are more likely to deviate from their course pacing guides than adhere to them.
What are the average and median number of assignments submitted out of sequence?
The total number of assignments submitted out of order ranged from zero (students fully adhered to course pacing guides) to 90. The average number of assignments submitted out of order was approximately 31 (SD = 19.90). The median (a metric referring to the middle value of a data set when organized in ascending order) number of assignments submitted out of order was 29, meaning half of the students sampled submitted fewer than 29 assignments out of order. In contrast, half submitted more than 29 assignments out of alignment with course pacing guides.
What is the average and median percentage of completed assignments submitted out of sequence?
To put the number of assignments students submitted out of sequence in the context of the number of available course assignments, the percentage of assignments submitted out of sequence was examined. This represents the number of assignments submitted out of sequence divided by the total number of assignments completed and multiplied by 100. While the percentage of assignments submitted out of sequence ranged from zero to 97.70%, the average was 44.70%. The median was slightly higher, with half of the students submitting more than 47% of assignments out of sequence and half submitting fewer than that.
What is the average and median magnitude of assignments submitted out of sequence?
In addition to examining the number of assignments submitted out of sequence, the extent to which assignments were out of alignment with course pacing guides was analyzed. On average, students were about three and a half assignments “off” from the intended pacing guide order (SD = 2.96). This may look like, for example, a student submitting assignment eight when the course pacing guide recommended submitting assignment eleven. While some students moved through the course completely aligned with pacing guide expectations, others were as much as approximately 15 assignments “off.” Please refer to Table 2 for descriptive information about key study variables.
| Variable | Average (SD) | Minimum | Median | Maximum |
|---|---|---|---|---|
| Final Course Scores | 82.64 (14.59) | 23.81 | 87.73 | 100.00 |
| Dropped Courses | 0.04 (0.26) | 0.00 | 0.00 | 6.00 |
| Completed Courses | 3.29 (2.99) | 1.00 | 3.00 | 36.00 |
| Current Class Load (Enrollment Load) | 1.17 (0.73) | 1.00 | 1.00 | 8.00 |
| Average Magnitude | 3.50 (2.96) | 0.00 | 2.67 | 14.57 |
| Percentage of Assignments Completed | 95.09 (8.49) | 55.22 | 100.00 | 100.00 |
| Percentage of Assignments Submitted Out of Order | 44.70 (25.24) | 0.00 | 47.27 | 97.70 |
| Total # of Assignments Submitted Out of Order | 31.09 (1.91) | 0.00 | 29.00 | 90.00 |
What does the relationship between students’ assignment sequencing and course performance look like in World Language courses?
Correlations helped describe the relationship between students’ assignment sequencing (percentage of assignments submitted out of order, average magnitude) and students’ course performance (final course scores). Correlations show the changes in one variable relative to changes in the other.
The negative correlation observed between final course scores and the percentage of assignments submitted out of order means that as one variable decreased, the other increased. For example, as students’ final course scores decreased, the percentage of assignments submitted out of order increased. Likewise, final course scores increased as the percentage of assignments submitted out of order decreased. A similar relationship was observed between the extent to which assignments were submitted out of order (average magnitude) and final course scores. As students’ magnitudes increased, scores decreased; as magnitude values decreased, final scores increased. A correlation matrix is provided in Appendix C for those interested in examining the specific correlation coefficients.
In short, sequencing variables and final course scores moved in opposite directions. An important caveat to these findings is that correlations only describe the relationship between variables and do not isolate the cause of the relationship (i.e., it cannot be said which variable causes the observed relationship).
What does student performance look like at each quartile of the percentage of assignments submitted out of order?
To understand how out-of-sequence movement related to course performance, students’ average final course scores at each quartile1 of the percentage of assignments submitted out of sequence were examined.
As illustrated in Table 3, students in the 1st quartile—those who completed the smallest percentage of assignments out of order—achieved the highest average final course scores (M = 88.3). In contrast, students in the 4th quartile, who submitted the largest percentage of assignments out of sequence, earned the lowest average scores (M = 78.7). This 9.6-point difference between the highest and lowest quartiles equals roughly one letter grade. Notably, the most significant drop in scores occurred between the 1st and 2nd quartiles, where average scores declined by 5.8 points, suggesting that even moderate increases in out-of-order submission behavior may be linked to a meaningful decline in academic performance.
What does student performance look like at each quartile of the average magnitude of assignments submitted out of order?
A similar trend emerged when analyzing the magnitude of out-of-order assignment submissions. Students in the 1st quartile—those with the lowest magnitude values—achieved the highest average final course scores (M = 87.3). In contrast, those in the 4th quartile, representing students with the highest magnitude values, had the lowest average scores (M = 78.2). This 9.1-point gap between the first and fourth quartiles is nearly equivalent to a full letter grade. The most pronounced drop occurred between the 2nd and 3rd quartiles, where average final course scores declined by 5.1 points, indicating that even modest increases in the degree of out-of-order submission may be associated with noticeable decreases in academic performance.
Reviewing Table 3 can provide additional information about the average final course scores at each quartile of the respective sequencing variables. A similar pattern was observed when looking at median final course scores at each quartile of the percentage of assignments submitted out of order and average magnitude; the reader can review these trends in Table 4.
| Quartiles | 1st Bottom 25% | 2nd 50% | 3rd 75% | 4th Top 25% |
|---|---|---|---|---|
| Average Final Course Scores | ||||
| Percentage of Assignments Completed Out of Order | 88.3 | 82.5 | 81.0 | 78.7 |
| Difference from Previous Quartile | n/a | -5.8 | -1.5 | -2.3 |
| Total Difference (Q1 – Q4) | -9.6 |
| Quartiles | 1st Bottom 25% | 2nd 50% | 3rd 75% | 4th Top 25% |
|---|---|---|---|---|
| Average Final Course Scores | ||||
| Average Magnitude | 87.3 | 85.1 | 80.0 | 78.2 |
| Difference from Previous Quartile | n/a | -2.2 | -5.1 | -1.8 |
| Total Difference (Q1 – Q4) | -9.1 |
| Quartiles | 1st Bottom 25% | 2nd 50% | 3rd 75% | 4th Top 25% |
|---|---|---|---|---|
| Median Final Course Scores | ||||
| Percentage of Assignments Completed Out of Order | 92.4 | 88.3 | 86.1 | 82.3 |
| Average Magnitude | 92.1 | 90.7 | 84.7 | 81.4 |
Discussion
The current study highlights the prevalence of submitting assignments out of alignment with course pacing guides in Michigan Virtual self-paced World Language courses. Nearly all students in the sample (97%) deviated from course pacing guides at least once, and on average, submitted approximately 44% of their completed course assignments out of order. These findings exceed those reported in prior studies. For example, Cuccolo & DeBruler (2024) found that 93% of students in Michigan Virtual online STEM courses submitted at least one assignment out of order, with an average of approximately 38% of assignments submitted out of sequence. Additionally, a brief visual inspection of the World Language course data supports anecdotal evidence from instructors: a substantial number of students fail to submit assignments requiring video recordings or scheduled meetings with instructors, assignment types that are especially critical in these particular courses for formative skill assessment and feedback. Taken together, these findings suggest that students in World Language courses may be particularly prone to deviating from the recommended assignments sequence, underscoring the importance of monitoring students’ assignment submission patterns in this context.
Consistent with Cuccolo & DeBruler’s (2024) findings, this study also observed a negative relationship between moving out of alignment with course pacing guides and final course scores. Specifically, final course scores steadily declined as students submitted a greater percentage of assignments out of order, and strayed further from the intended assignment sequence (i.e., higher magnitude). The most significant decline in scores occurred between the 1st and 2nd quartiles for the percentage of assignments submitted out of order, and between the 2nd and 3rd quartiles for average magnitude. Practically speaking, this suggests critical thresholds: a noticeable decline in student grades may be seen as they begin to submit more than about a quarter of assignments out of order, or when they are more than one assignment “off” from pacing guide recommendations. As such, these thresholds may serve as useful benchmarks and noteworthy intervention points for teachers and mentors.
Overall, the findings suggest that both the frequency and degree of out-of-order assignment completion are meaningfully associated with student performance. Students who submitted fewer assignments out of order and stayed closer to the intended sequence (i.e., lower magnitude) earned notably higher final course scores. As students increased the proportion or the extent to which they deviated from the expected assignment order, their average course scores declined by nearly a full letter grade between the lowest and highest quartiles for both predictors. These patterns highlight the potential academic consequences of straying from the designed learning sequence in online courses. As such, teachers and mentors should be mindful of intervening early when monitoring student performance.
While the current study’s findings provide actionable insights into how assignment submission patterns are associated with course scores, it is essential to note that this study was correlational by design. In other words, the causal agent in the observed relationship between assignment sequencing and final course score is unclear. Other unstudied factors may contribute to the observed relationship. For example, moving out of alignment with course pacing guides may be part of a broader pattern of student behaviors related to performance. For instance, self-regulatory and metacognitive skills have been positively associated with student achievement (Xu et al., 2023). These skills encompass students’ ability to engage with a task cognitively, reflect and evaluate their learning, manage resources, and direct their efforts (Xu et al., 2023). Self-regulatory and metacognitive skills are increasingly important in an online learning environment, where students have increased autonomy over when, where, and how they engage with course content (Xu et al., 2023). Without a strong foundation, students may struggle to engage deeply with assignments and feedback, scaffold knowledge, and manage their time appropriately. Thus, self-regulated learning could be effective in better understanding the relationship between assignment submission patterns and student performance.
To promote adherence to course pacing guides and foster effective course navigation, teachers and mentors should incorporate transparent and proactive communication practices into their routines. In particular, setting course expectations early by helping students understand course structure, workload, pacing, and tips for success can help students prepare for the demands of self-paced online learning (Cuccolo & Green, 2024). Monitoring the gradebook and benchmarking student progress against course pacing guides is also recommended. The thresholds identified in this study (e.g., exceeding about a quarter of out-of-sequence submissions or being more than one assignment “off” from the pacing guide’s intended submission order) can serve as timely indicators for intervention. Personalized feedback remains a highly effective strategy for engaging students in online learning as it works double duty as both a relationship-building strategy and a way to monitor/motivate academic progress (DeBruler & Harrington, 2024). Regular communication between mentors, teachers, and their students helps bridge information gaps and close feedback loops. Additionally, mentors can also help contextualize student behavior for teachers (Cuccolo & DeBruler, 2023). While it may be unrealistic to believe students will complete all assignments in order when enrolled in a self-paced course, teachers and mentors can help encourage pacing behaviors that set students up for success.
Appendix
Appendix A. Study Glossary
| Variable | Definition | Example |
|---|---|---|
| User-Driven | A variable that indicates if a student’s assignment was submitted in or out of alignment with pacing guide expectations. If a student’s current assignment was one greater than the previously submitted assignment, it was considered in-sequence (and given a value of “0”); otherwise, it was considered out-of-sequence (and given a value of “1”). | If a student submits Assignment 5 and then Assignment 17, this would be labeled as an out-of-sequence assignment because 17 is not one greater than 5. |
| Total # of Assignments Submitted Out of Order | The total number of assignments a student submitted out of order. The total sum of the number of “1s” in the “UserDriven” column for a student. | For example, if this value is 18, the student submitted 18 assignments out of their intended pacing guide order. |
| Percentage of Assignments Submitted Out of Order | The total number of assignments the student submitted out of order divided by the total number of assignments the student completed, multiplied by 100. | If a student submitted 50 assignments, 5 of which were out of order, the percentage out of order would be 10%. |
| Magnitude | This represents the extent to which students deviated from the pacing guide—the difference between the intended pacing guide and the actual submission order of consecutive assignment submissions. | If a student submitted assignments 5 and then 17, the magnitude value would be 12. |
| Average Magnitude | Takes the average of all of a student’s ‘Magnitude’ values. | If a student had magnitude values of 12, 3, 8, and 2, then the average magnitude would be 6.25. |
| Percentage of Assignments Completed | The number of completed assignments divided by the total number of assignments in the course, multiplied by 100. | A value of 70 would indicate a student completed 70% of all available course assignments. |
| Dropped Courses | Number of courses the student dropped. | A value of two would mean the student has dropped two virtual courses. |
| Completed Courses | Number of courses the student completed. | A value of four would mean the student has completed four virtual courses. |
| Current Class Load | The number of classes the student was taking in the target semester. | A value of two would mean that the student took two virtual classes during the spring 2024 semester. |
Appendix B. Enrollment Information For The Current Study
| Course Name | Students per Course (n) | Students per Course % |
|---|---|---|
| American Sign Language 1B | 740 | 39.51% |
| American Sign Language 2B | 260 | 13.88% |
| Spanish 2B | 129 | 6.89% |
| American Sign Language 1A | 113 | 6.03% |
| Spanish 1B | 111 | 5.93% |
| French 1B | 87 | 4.64% |
| French 2B | 84 | 4.48% |
| German 1B | 75 | 4.00% |
| German 2B | 67 | 3.58% |
| Japanese 1B | 62 | 3.31% |
| Japanese 2B | 32 | 1.71% |
| Spanish 1A | 24 | 1.28% |
| Japanese 1A | 15 | 0.80% |
| American Sign Language 2A | 14 | 0.75% |
| German 1A | 14 | 0.75% |
| French 1A | 13 | 0.69% |
| Spanish 2A | 13 | 0.69% |
| French 2A | 9 | 0.48% |
| German 2A | 6 | 0.32% |
| Japanese 2A | 5 | 0.27% |
| Total | 1,873 | 100% |
Appendix C. Relationship Between Sequencing Variables And Final Course Scores
| Final Course Scores | Average Magnitude | Percentage Complete | Percentage Out of Order | |
|---|---|---|---|---|
| Final Course Scores | 1 | -0.20 | 0.60 | -0.18 |
| Average Magnitude | -0.20 | 1 | -0.19 | 0.72 |
| Percentage Complete | 0.60 | -0.19 | 1 | -0.20 |
| Percentage Out of Order | -0.18 | 0.72 | -0.20 | 1 |
References
Cuccolo, K., & Green, C. (2024). Starting Strong: Understanding Teacher-Student Communication in Online Courses. Michigan Virtual. https://michiganvirtual.org/research/publications/understanding-teacher-student-communication
Cuccolo, K. & DeBruler, K. (2023). Examining Mentors’ Navigation of Online Environments and Use of Student Support Practices. Michigan Virtual. https://michiganvirtual.org/research/publications/examining-mentors-navigation-of-online-environments
DeBruler, K. (2021). Research On K-12 Online Best Practices. Michigan Virtual.
DeBruler, K. & Harrington, C. (2024). Key Strategies for Supporting Disengaged and Struggling Students in Virtual Learning Environments. Michigan Virtual. https://michiganvirtual.org/research/publications/key-strategies-for-supporting-disengaged-and-struggling-students-in-virtual-learning-environments/
Digital Learning Institute. (n.d.). “What is Self Paced Learning? Definition, Benefits and Tips.” [Blog]. Retrieved from https://www.digitallearninginstitute.com/blog/what-is-self-paced-learning-definition-benefits-and-tips
Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58. https://doi.org/10.1177/1529100612453266
Freidhoff, J. R. (2015). Michigan’s K-12 virtual learning effectiveness report 2013-14. Lansing, MI: Michigan Virtual University. Retrieved from http://media.mivu.org/institute/pdf/er_2014.pdf
Freidhoff, J. R., DeBruler, K., Cuccolo, K., & Green, C. (2024). Michigan’s K-12 virtual learning effectiveness report 2022-23. Michigan Virtual. https://michiganvirtual.org/research/publications/michigans-k-12-virtual-learning-effectiveness-report-2022-23/
Johnson, C. C., Walton, J. B., Strickler, L., & Elliott, J. B. (2023). Online teaching in K-12 education in the United States: A systematic review. Review of Educational Research, 93(3), 353-411. https://doi.org/10.3102/00346543221105550
Kim, K. R., & Seo, E. H. (2015). The relationship between procrastination and academic performance: A meta-analysis. Personality and Individual Differences, 82, 26-33. https://doi.org/10.1016/j.paid.2015.02.038
Lim, J. M. (2016). Predicting successful completion using student delay indicators in undergraduate self-paced online courses. Distance Education, 37(3), 317-332. https://doi.org/10.1080/01587919.2016.1233050
Lim, J. M. (2016b). The relationship between successful completion and sequential movement in self-paced distance courses. International Review of Research in Open and Distributed Learning, 17(1), 159-179. https://doi.org/10.19173/irrodl.v17i1.2167
Michigan Virtual Learning Research Institute. (2019). Pacing Guide For Success In Online Mathematics Courses. https://michiganvirtual.org/blog/pacing-guide-for-success-in-online-mathematics-courses/
Perna, L. W., Ruby, A., Boruch, R. F., Wang, N., Scull, J., Ahmad, S., & Evans, C. (2014). Moving through MOOCs: Understanding the progression of users in massive open online courses. Educational Researcher, 43(9), 421-432. https://doi.org/10.3102/0013189X14562423
Xu, Z., Zhao, Y., Zhang, B., Liew, J., & Kogut, A. (2023). A meta-analysis of the efficacy of self-regulated learning interventions on academic achievement in online and blended environments in K-12 and higher education. Behaviour & Information Technology, 42(16), 2911-2931. https://doi.org/10.1080/0144929X.2022.2151935
