Automatically recovering students’ missing trace links between commits and user stories

Abstract

Trace links between commits and user stories can be used in educational software engineering projects to track progress and determine the students’ contribution to projects’ requirements. Thus, traceability can be helpful in courses for grade assessment, and project monitoring and improvement. Currently developers, including students in courses, manually link commits and issues using version control systems (e.g., Git) and issue tracking systems (e.g., Jira). However, manual trace links are often incomplete. In our study, we found that only 43% of the commits are linked to stories in the analyzed project. Therefore, there is a need to automatically or semi-automatically create trace links. This study aims to automatically recover trace links between commits and user stories requirements in an undergraduate student project with twenty students and four teams. We used unstructured data from messages, code and files of commits and stories to gather textual similarity measures. We evaluated the effectiveness of information retrieval (Vector space model, Latent semantic indexing and BM25) and machine learning (Random forests, Decision trees and Neural networks) techniques in recovering missing links using textual and process data. Machine learning models outperformed information retrieval models in precision, recall, and F-measure. Machine learning models were able to effectively recover missing trace links with an average of 93% precision and 94% recall, showing the applicability of the approach.