What do students learn by using GitHub?
Results from a classroom study of 8,000 teachers and students
We want to understand (and improve) technical education as a whole. Our hypothesis was that using GitHub in programming classes can shape students’ learning outcomes and classroom experiences.
First, we conducted a literature review of research relevant to using GitHub in the classroom, focusing on research that explored GitHub in a learning context and on student outcomes.
Conducting the literature review helped us identify key variables important to both students and educators in programming classes, and informed the design of our student learning outcomes survey research.
We’ll start with our background research on the subject of learning outcomes and GitHub, and then dig in deeper with more detailed explanations of the data you’ll find in the slides above.
Further, the literature review guided our predictions for how using GitHub in the classroom may affect student learning, which we used to design our recent study.
Literature Review
How using GitHub in the classroom shapes student learning outcomes
GitHub has become a key social platform for managing projects and facilitating collaborative development within the software development community. More recently, educators have begun to implement GitHub in the classroom. Although its utility is not limited to specific fields of study, GitHub may be especially valuable when used in computer science and software engineering courses. It facilitates learning via features that support the distribution of individual and group assignments, opportunities for collaboration, line-specific feedback, and others. Unsurprisingly, the general idea of using GitHub in the classroom has generated interest not only among educators, but also among researchers interested in the implications GitHub has for student learning outcomes. To understand how GitHub can boost students' learning success, we will review research that has specifically explored GitHub in a learning context and also draw on research focused on student outcomes more generally. Lastly, we highlight future avenues of research to further our understanding.
Review of past research
First, implementing GitHub in the classroom may create opportunities for students to be more involved and engaged. Research supports that using GitHub in the classroom boosts student engagement, the overall energy that a student invests in the academic experience (Gunnarsson et al. 2017). By facilitating student contributions and cross-team collaboration, the use of GitHub enables a participatory culture where students feel their contributions are impactful (Feliciano, Story, and Zagalsky 2016). Further, instructors are able to view a summary of student activity, and this may encourage students to make active contributions as their participation may be tied to grades. In one study, researchers interviewed instructors and found that some instructors use students' contributions on GitHub as materials for discussion in class, thus participation is incentivized because student contributions can direct class discussions (Zagalsky et al. 2015). Indeed, past research found that one main motivation instructors have for implementing GitHub in the classroom is to offer students the opportunity to directly influence the course and contribute to course materials (e.g., via pull requests) (Feliciano, Story, and Zagalsky 2016), creating novel ways for students to be involved in the classroom.
Using GitHub in the classroom is also valuable to students because it gives them the chance to learn how to use a popular industry tool. Research supports that one of the main motivations instructors have for implementing GitHub in the classroom is to help students develop familiarity with an important industry tool, thus making students more competitive on the job market (Feliciano, Story, and Zagalsky 2016). Further, if students are able to draw a direct connection between their classroom learning and future career, research suggests they may experience an increase in motivation (Kauffman and Husman 2004). Using GitHub in the classroom also provides students the chance to develop a portfolio of their work. In an interview study, students mentioned the importance of publicly presenting their work on GitHub and that it is common for employers to refer to GitHub during the hiring process (Feliciano, Story, and Zagalsky 2016). Research also supports that industry collaborations may encourage students to take assignments more seriously, invest greater effort, find assignments more meaningful, and be more motivated to perform well on assignments (Marcketti and Karpova 2014). Therefore, using a popular industry tool and collaborating on projects in a way that mimics work in industry, could offer similar benefits.
Implementing GitHub in the classroom can also help boost student performance. First, GitHub allows students to receive line-specific feedback from their instructors or peers via pull requests. Because detailed informational feedback tends to increase student self-efficacy and ability to monitor the effectiveness of their learning strategies (Butler and Winne 1995), relying on GitHub may foster more confident and successful students. In addition, student performance may benefit from learning from others when using GitHub. Past research found that some students ventured beyond requirements to explore not only the work of other students but also projects led by professionals in the field (Feliciano, Story, and Zagalsky 2016). Exposure to the work of peers and individuals in industry may undoubtedly provide new perspectives on ideas and learning opportunities to students. For example, in a few cases, students were able to identify and fix issues in their own code after examining code developed and shared by others on GitHub (Feliciano, Story, and Zagalsky 2016). Thus, via detailed feedback and exposure to the work of peers and industry professionals, students may be able to improve their own work and learning strategies.
In addition, using GitHub, students can build soft skills that will help them work with others in industry. By encouraging students to use GitHub for collaborative activities, such as peer reviews and group projects, instructors may facilitate students' development of communication, teamwork, and critical analysis skills (Falkner and Falkner 2012). Soft skills are important when working in industry, and computer science and software engineering education have begun to embrace a pedagogy that focuses on soft skills in addition to technical skills (Feliciano, Story, and Zagalsky 2016). In a study where instructors were interviewed, the instructors mentioned that using GitHub in their courses effectively introduced students to collaborative practices essential to software development (e.g., cross-team collaboration, peer review) (Feliciano, Story, and Zagalsky 2016). Further, a survey study found that, although students noted the higher learning curve associated with using GitHub for the first time, 15 out of 22 students preferred the collaborative platform over more traditional individual assignments (Kertész 2015). One of the key advantages students observed was that GitHub facilitated teamwork and collaboration, allowing students to connect with peers for help outside of the classroom and learn from each other's ideas (Kertész 2015). These collaborative experiences may be critical to students' development of soft skills that will make them more effective team members in industry.
Lastly, although past research has not directly addressed this specific idea, using GitHub in the classroom may strengthen the sense of belonging students feel within the classroom because it allows students to collaborate and contribute to each other's learning process. Research finds that, more collaborative projects, where students work closely with peers, can foster a stronger sense of belonging (Kember, Lee, and Li 2001). Importantly, students' sense of belonging can positively predict variables tied to academic success, such as intrinsic motivation and academic self-efficacy (Freeman, Anderman, and Jensen 2007). Self-efficacy, the degree to which individuals feel capable of accomplishing a task, is essential to student success because it can protect students' confidence and help them persist when they encounter setbacks (Pajares 1996). In addition, GitHub supports interactions with individuals in industry (Feliciano, Story, and Zagalsky 2016) and may promote a sense of belonging to the field. For instance, one instructor from a previous study reported that unregistered students and other GitHub users visited and "starred" their public repositories, establishing a connection between students and others in the field (Zagalsky et al. 2015). Notably, improving students' sense of belonging may be especially beneficial for minority-group students. For instance, first-generation college students tend to be less engaged and have lower retention rates compared with non-first-generation college students (Soria and Stebleton 2012), and studies suggest that improving students' sense of belonging can have a protective effect on retention and increase achievement motivation (Walton et al. 2012; O'Keeffe 2013).
Design of learning outcomes study
Our future research will deviate from previous studies in several ways. Notably, to date, studies on the use of GitHub in the classroom have relied on data from case studies, interviews, and surveys with low sample sizes. In addition, they typically focus on educators and their perceptions of student learning. Although these approaches have highlighted important issues, trends, and ideas, the studies are susceptible to bias and results may not be generalizable. Further, past studies have heavily relied on qualitative approaches, and there is little insight to the size of different effects. In the future, we hope to conduct a study relying on a representative sample large enough for more reliable statistical analyses and focus more on the students' perception.
In addition, our future research will make it possible to draw comparisons between or within subjects. Because, to our knowledge, previous studies on the use of GitHub in the classroom did not include a "control" group or collect within-subjects data, it is difficult to gauge the effect GitHub has on learning outcomes and other variables. In order to highlight the benefits of using GitHub, it is important to design a study where comparisons are possible and learning outcomes are quantified. Lastly, we hope to extend our understanding of how GitHub shapes student learning outcomes by measuring relevant outcome variables that have not been studied in previous research.
GitHub Classroom Report 2018
Introduction and method (slides 1-10)
We surveyed 8,000 students and teachers who had (and had not) used GitHub in the classroom. The aim was to examine how using GitHub in the classroom shapes students' learning experiences. Based on the trends we uncovered during the literature review, we focused on key domains of interest to educators and researchers: preparation for the future, classroom experiences, developing a sense of belonging, feedback, and the experience of learning to use GitHub. Next, we will dive into the details of the study, including the study's overall design, method, and key results.
Learning outcomes: preparation for the future (slides 11-22)
Students and Teachers felt students learned more about teamwork and collaboration, popular industry tools(s), and project management in GitHub classrooms as opposed to non GitHub classrooms this past semester.
Students who used GitHub in the classroom felt they learned more about teamwork and collaboration than students who did not use GitHub in the classroom. 32% of students who used GitHub in the classroom and 17% of students who didn't use GitHub in the classroom felt they had learned "very much" about teamwork and collaboration.
Students who used GitHub in the classroom felt they learned more about popular industry tools than students who did not use GitHub in the classroom. 29% of students who used GitHub in the classroom and 14% of students who didn't use GitHub in the classroom felt they had learned "very much" about popular industry tools.
Students who used GitHub in the classroom felt they learned more about project management than students who did not use GitHub in the classroom. 25% of students who used GitHub in the classroom and 12% of students who didn't use GitHub in the classroom felt they had learned "very much" about project management.
Both students and teachers of GitHub (versus non GitHub) classrooms felt their course prepared students for being a part of the developer community and developing a portfolio to a greater extent. Students in GitHub classrooms also felt their course prepared them for future internship/career and taking more advanced courses to a greater extent than students in non GitHub classrooms. Statistical analyses indicated there was no difference (ns) between how much teachers of GitHub (versus non GitHub) classrooms felt their course prepared students for future internship/career or taking more advanced courses.
Students who used GitHub in the classroom felt more prepared for future internships or career than students who did not use GitHub in the classroom. 33% of students who used GitHub in the classroom and 19% of students who didn't use GitHub in the classroom felt their course prepared them for future internships or career "very much."
Students who used GitHub in the classroom felt more prepared for being part of the developer community than students who did not use GitHub in the classroom. 31% of students who used GitHub in the classroom and 14% of students who didn't use GitHub in the classroom felt their course prepared them for being a part of the developer community "very much."
Students who used GitHub in the classroom felt more prepared for developing a portfolio than students who did not use GitHub in the classroom. 30% of students who used GitHub in the classroom and 15% of students who didn't use GitHub in the classroom felt their course prepared them for developing a portfolio of their own work "very much."
Students who used GitHub in the classroom felt more prepared for taking more advanced courses than students who did not use GitHub in the classroom. 36% of students who used GitHub in the classroom and 32% of students who didn't use GitHub in the classroom felt their course prepared them for taking more advanced courses "very much."
Responses from students and teachers suggest the number of GitHub features used in the classroom predicted greater positive learning outcomes for students. Using more GitHub features in the classroom seemed most important for student learning outcomes related to teamwork and collaborations, as well as project management.
Learning outcomes: classroom experience (slides 23-37)
Overall, differences between GitHub and non GitHub classrooms were mostly small, but likely not due to chance. Taken together, consistent trends suggest slight benefits to students' classroom experience in GitHub classrooms versus non GitHub classrooms.
In general, students and teachers who used GitHub in the classroom felt GitHub improved rather than worsened students' learning experience. 81% of students and 85% of teachers felt that using GitHub in the classroom improved students' learning experience. 8% of students and 4% of teachers felt that using GitHub in the classroom worsened students' learning experience.
Students in GitHub classrooms felt a slightly higher level of engagement, participation, and attendance than students in non GitHub classrooms. Teachers reported that, on average, students' level of engagement, participation, and attendance were about the same between GitHub and non GitHub classrooms. For teachers, statistical analyses indicated there were no differences across items (ns).
Students who used GitHub in the classroom felt a slightly higher level of engagement than students who did not use GitHub in the classroom. 83% of students who used GitHub in the classroom felt a "slightly high" or "very high" level of engagement in the classroom. 79% of students who did not use GitHub in the classroom felt a "slightly high" or "very high" level of engagement in the classroom.
Students who used GitHub in the classroom felt a slightly higher level of participation than students who did not use GitHub in the classroom. 81% of students who used GitHub in the classroom felt a "slightly high" or "very high" level of engagement in the classroom. 77% of students who did not use GitHub in the classroom felt a "slightly high" or "very high" level of participation in the classroom.
Students who used GitHub in the classroom felt a slightly higher level of attendance than students who did not use GitHub in the classroom. 78% of students who used GitHub in the classroom felt a "slightly high" or "very high" level of engagement in the classroom. 74% of students who did not use GitHub in the classroom felt a "slightly high" or "very high" level of attendance in the classroom.
Students in GitHub classrooms liked their course slightly more and put slightly more effort into their course this past semester than students in non GitHub classrooms.
Students in GitHub classrooms (versus non GitHub classrooms) were slightly more likely to recommend their course to other students and to take a similar course in the future.
Students who used GitHub in the classroom were slightly more likely to recommend their course to other students than students who did not use GitHub in the classroom. 73% of students who used GitHub in the classroom agreed or strongly agreed that they enjoyed their course very much. 62% of students who did not use GitHub in the classroom agreed or strongly agreed that they would recommend their course to other students.
Students who used GitHub in the classroom were slightly more likely to take a similar course in the future. 70% of students who used GitHub in the classroom agreed or strongly agreed that they would take a similar course in the future. 61% of students who did not use GitHub in the classroom agreed or strongly agreed that they would take a similar course in the future.
Students in GitHub classrooms felt slightly more competent than students in non GitHub classrooms. In addition, students in GitHub classrooms were slightly more satisfied with their performance and felt slightly more like they had the skills necessary to do well in their course this past semester.
Students in GitHub classrooms felt slightly more competent than students in non GitHub classrooms. 73% of students who used GitHub in the classroom felt it was at least slightly true that they feel pretty confident. 66% of students who did not use GitHub in the classroom felt it was at least slightly true that they feel pretty confident.
Students in GitHub classrooms felt slightly more satisfied with their performance than students in non GitHub classrooms. 76% of students who used GitHub in the classroom felt it was at least slightly true that they are satisfied with their performance in their course. 71% of students who did not use GitHub in the classroom felt it was at least slightly true that they are satisfied with their performance in their course.
Students in GitHub classrooms felt slightly more like they have the skills necessary to do well in their course than students in non GitHub classrooms. 83% of students who used GitHub in the classroom felt it was at least slightly true that they have the skills necessary to do well in their course. 79% of students who did not use GitHub in the classroom felt it was at least slightly true that they have the skills necessary to do well in their course.
Sense of belonging (slides 38-49)
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their class this past semester and in the field more generally.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their class this past semester. Out of the students who responded to all three statements measuring students' sense of belonging in class, 51% of students who used GitHub in the classroom and 37% of students who did not use GitHub in the classroom agreed with all statements.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their class this past semester. 63% of students who used GitHub in the classroom and 49% of students who did not use GitHub in the classroom agreed or strongly agreed that their contributions were valued.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their class this past semester. 71% of students who used GitHub in the classroom and 65% of students who did not use GitHub in the classroom agreed or strongly agreed they feel comfortable in their course.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their class this past semester. 72% of students who used GitHub in the classroom and 65% of students who did not use GitHub in the classroom agreed or strongly agreed people in their course accept them.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their field. Out of the students who responded to all three statements measuring students' sense of belonging in the field, 48% of students who used GItHub in the classroom and 37% of students who did not use GitHub in the classroom agreed with all statements.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their field. 60% of students who used GitHub in the classroom and 48% of students who did not use GitHub in the classroom agreed or strongly agreed that they see themselves as part of the developer community.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their field. 56% of students who used GitHub in the classroom and 44% of students who did not use GitHub in the classroom agreed or strongly agreed they feel like a member of the developer community.
Students in GitHub classrooms (versus non GitHub classrooms) felt a greater sense of belonging in their field. 58% of students who used GitHub in the classroom and 45% of students who did not use GitHub in the classroom agreed or strongly agreed they feel a sense of belonging to the developer community.
Overall, respondents experienced a high sense of belonging. Students who were less familiar with GitHub than their peers at the beginning of the semester felt a slightly lower sense of belonging both in class and in the field than students who felt either as familiar or more familiar with GitHub than peers at the beginning of the semester.
Feedback (slides 50-58)
Students who received (versus did not receive) feedback from their teachers via GitHub felt they used their teacher's feedback more effectively, found their teacher's feedback more helpful, and felt their teacher had a better understanding of their needs as a student. This suggests using GitHub to provide feedback may impact how students apply teacher feedback. However, using (versus not using) GitHub to provide feedback did not affect students' understanding of feedback or the extent to which students pay attention to teacher feedback, and statistical analyses reveal no significant difference between those who did (versus did not) receive feedback from their teacher via GitHub (ns). Teachers did not feel that using (versus not using) GitHub to provide feedback made a difference on students' perception of feedback.
Students who received (versus did not receive) feedback from their teachers via GitHub felt they used their teacher's feedback more effectively. 74% of students who used GitHub in the classroom and 67% of students who did not use GitHub in the classroom agreed or strongly agreed they use their instructors' feedback effectively.
Students who received (versus did not receive) feedback from their teachers via GitHub found their instructors' feedback more helpful. 75% of students who used GitHub in the classroom and 65% of students who did not use GitHub in the classroom agreed or strongly agreed they find their instructors' feedback helpful.
Students who received (versus did not receive) feedback from their teachers via GitHub felt their teachers better understand had a better understanding of their needs as a student . 68% of students who used GitHub in the classroom and 60% of students who did not use GitHub in the classroom agreed or strongly agreed their instructors understand their needs as students.
Students who received (versus did not receive) feedback from their peers via GitHub felt they used their peers' feedback more effectively, and found their peers' feedback more helpful. However, using (versus not using) GitHub to provide feedback did not affect students' understanding of peer feedback or the extent to which students pay attention to peer feedback, and statistical analyses reveal no significant difference between those who did (versus did not) receive feedback from their peers via GitHub (ns).
Students who received (versus did not receive) feedback from their peers via GitHub felt they used their peers' feedback more effectively. 73% of students who used GitHub in the classroom and 63% of students who did not use GitHub in the classroom agreed or strongly agreed they use their peers' feedback effectively.
Students who received (versus did not receive) feedback from their peers via GitHub found their peers' feedback more helpful. 75% of students who used GitHub in the classroom and 67% of students who did not use GitHub in the classroom agreed or strongly agreed they find their peers' feedback helpful.
Learning GitHub (slides 59-64)
In general, both students and teachers felt familiarity with GitHub increased over the course of a semester. Whereas 44% of students felt very familiar or extremely familiar with GitHub at the beginning of the semester, 75% of students felt very familiar or extremely familiar with GitHub by the end of the semester. On the teacher's side, 4% felt students were very familiar or extremely familiar with GitHub at the beginning of the semester, and 30% felt students were very familiar or extremely familiar with GitHub by the end of the semester. The discrepancy between student and teacher responses may be attributed to the student respondents, on average, being more familiar with GitHub than peers at the beginning of the semester.
In general, over half of student respondents found it easy to slightly easy to learn GitHub, but about 15% found learning GitHub to be on the difficult side. In contrast, about 40% of teacher respondents thought learning GitHub was on the difficult side for the average student in their course. Again, the difference in patterns may reflect student respondents' greater familiarity with GitHub relative to peers at the beginning of the semester.
Students who were first-time users of GitHub felt it was slightly more difficult learning GitHub than students who were not first-time users. Whereas 14% of non first-timer users felt it was at least slightly difficult to learn how to use GitHub, 23% of first-time users felt it was at least slightly difficult.
Roughly half the students felt they received just the right amount of support from teachers and peers, and roughly half the teachers felt students in their class received just the right amount of support from teachers and peers as they were learning GitHub. However, 25% of students felt they received less support than they needed from teachers, and 32% of students felt they received less support than they needed from peers. In contrast, 12% of teachers felt students received less support than they needed from teachers, and 28% of teachers felt students received less support than they needed from peers.
Conclusion
Overall, we found that students benefited from using GitHub in the classroom in a few important ways. Together, the literature review and the learning outcomes survey research boost our understanding of how students may be affected by the implementation of GitHub in the classroom. Importantly, our research also demonstrates that how GitHub is used in the classroom matters. Specifically, students may benefit from using more GitHub features and receiving feedback via GitHub. Future research will continue to focus on students' learning outcomes and assess ways in which we can better support student users.
Sharing data for further research
In the spirit of furthering research on technical education, we're sharing the anonymized data sets we collected. Those files are under an MIT license, along with for a key for understanding our analysis, in GitHub Education’s repository.
Bibliography
Butler, Deborah L., and Philip H. Winne, Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, (1995): 245-281, https://doi.org/10.3102/00346543065003245.
Falkner, Katrina, and Nickolas J.G. Falkner, Supporting and structuring "contributing student pedagogy" in computer science curricula. Computer Science Education, 22, (2012): 413-443, https://doi.org/10.1080/08993408.2012.727713.
Freeman, Tierra M., Lynley H. Anderman, and Jane M. Jensen. Sense of belonging in college freshmen at the classroom and campus levels. Journal of Experimental Education. 75, (2007): 203-220, https://www.jstor.org/stable/20157456.
Gunnarsson, Sara, Peter Larsson, Sara Månsson, Erik Mårtensson, and Jonathan Sönnerup, Enhancing student engagement using GitHub as an educational tool. Introduction to Teaching and Learning in Higher Education, (2017): 1-14.
Kauffman, Douglas F. and Jenefer Husman, Effects of time perspective on student motivation: Introduction to a Special Issue. Educational Psychology Review, 16, (2004): 1-7, https://www.jstor.org/stable/23363944.
Kember, David, Kenneth Lee, and Natalia Li, Cultivating a sense of belonging in part-time students. International Journal of Lifelong Education. 20, (2001): 326-341.
Kertész, Csaba-Zoltán, Using GitHub in the classroom - a collaborative learning experience. International Symposium for Design and Technology in Electronic Packaging, Brașov, Romania, (2015): 381-386.
Marcketti, Sara B., and Elena E. Karpova, Getting ready for the real world: Student perspectives on bringing industry collaboration into the classroom. Journal of Family and Consumer Sciences. 106, (2014): 27-31.
O'Keeffe, Patrick, A sense of belonging: Improving student retention. College Student Journal. 47, (2013): 605-613.
Pajares, Frank, Self-efficacy beliefs in academic settings. Review of Educational Research. 66, (1996): 543-578, https://doi.org/10.3102/00346543066004543.
Soria, Krista M., and Michael J. Stebleton, M.J. First-generation students' academic engagement and retention. Teaching in Higher Education. 17, (2012): 673-685.
Walton, Gregory M., Geoffrey L. Cohen, David Cwir, and Steven J. Spencer, Mere belonging: The power of social connections. Journal of Personality and Social Psychology, 102, (2012): 513-532, https://doi.org/10.1037/a0025731.
Zagalsky, Alexey, Joseph Feliciano, Margaret-Anne Storey, Yiyun Zhao, and Weiliang Wang, The emergence of github as a collaborative platform for education. In Proc. of the 18th \*ACM Conference on Computer Supported Cooperative Work & Social Computing, (2015): 1906-1917, http://dx.doi.org/10.1145/2675133.2675284.