This paper describes an approach integrating cognitive engineering with computational social science in the context of open source software (OSS) development. Through an analysis of large-scale collaborations in a complex operational setting, we study how expertise and task complexity predict changes in productivity when knowledge loss occurs. Using team data from thousands of software files, we model the effects of expertise, complexity, and knowledge loss on productivity. On its own, knowledge loss had a negative effect on productivity, but this effect was reversed when knowledge loss was combined with high complexity and high numbers of newcomers, suggesting that experts are better able to utilize crowdsourced work. We identify opportunities for research to inform prediction of outcomes in OSS projects based on team and task characteristics and demonstrate the value of integrating cognitive engineering with computational social science to study collaborative work in sociotechnical systems.
Get full access to this article
View all access options for this article.
References
1.
ArgoteL.AvenB. L.KushJ. (2018). The effects of communication networks and turnover on transactive memory and group performance. Organization Science, 29(2), 191–206.
2.
BaayenR. H.DavidsonD. J.BatesD. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412.
3.
BatraD.DavisJ. G. (1992). Conceptual data modelling in database design: Similarities and differences between expert and novice designers. International Journal of Man-Machine Studies, 37, 83–101.
4.
BatesD.MächlerM.BolkerB.WalkerS. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48.
5.
ChevalierA.IvoryM. Y. (2003). Web site designs: Influences of designer’s expertise and design constraints. International Journal of Human-Computer Studies, 58, 57–87.
6.
ChoiH.-S.ThompsonL. (2005). Old wine in a new bottle: Impact of membership change on group creativity. Organizational Behavior and Human Decision Processes, 98(2), 121–132.
7.
CurtisB.KrasnerH.IscoeN. (1988). A field study of the software design process for large systems. CACM, 31, 1268–1287.
DiesteO.ArandaA. M.UyaguariF.TurhanB.TosunA.FucciD.… JuristoN. (2017). Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study. Empirical Software Engineering, 22(5), 2457–2542.
10.
DorfmanR. (1979). A formula for the Gini coefficient. Review of Economics and Statistics, 61, 146–156.
11.
EspinosaJ. A.SlaughterS. A.KrautR. E.HerbslebJ. D. (2007a). Familiarity, complexity, and team performance in geographically distributed software development. Organization Science, 18(4), 613-630.
12.
EspinosaJ. A.SlaughterS. A.KrautR. E.HerbslebJ. D. (2007b). Team knowledge and coordination in geographically distributed software development. Journal of Management Information Systems, 24, 135–169.
13.
FioreS. M.RosenM. A.Smith-JentschK. A.SalasE.LetskyM.WarnerN. (2010). Toward an understanding of macrocognition in teams: predicting processes in complex collaborative contexts. Human Factors, 52(2), 203–224.
14.
FoucaultM.PalyartM.BlancX.MurphyG. C.FalleriJ.-R. (2015). Impact of developer turnover on quality in open-source software. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2015 (pp. 829–841). Bergamo, Italy: ACM Press.
15.
GogginsS. P.MascaroC.ValettoG. (2013). Group informatics: A methodological approach and ontology for sociotechnical group research. Journal of the Association for Information Science and Technology, 64(3), 516–539.
16.
HausknechtJ. P.HolwerdaJ. A. (2013). When does employee turnover matter? Dynamic member configurations, productive capacity, and collective performance. Organization Science, 24(1), 210–225.
17.
HoffmanR. R.FioreS. M. (2007). Perceptual (re)learning: A leverage point for human-centered computing. IEEE Intelligent Systems, 22, 3, 79–83.
18.
HoffmanR. R.WardP.FeltovichP. J.DiBelloL.FioreS. M.AndrewsD. H. (2013). Accelerated Expertise: Training for High Proficiency in a Complex World. East Sussex, UK: Psychology Press, Taylor & Francis Group Ltd.
HuangS.CummingsJ. N. (2011). When critical knowledge is most critical: centralization in knowledge-intensive teams. Small Group Research, 42, 669–699.
22.
JarczykO.JaroszewiczS.WierzbickiA.PawlakK.Jankowski-LorekM. (2018). Surgical teams on GitHub: Modeling performance of GitHub project development processes. Information and Software Technology, 100, 32–46.
23.
JoblinM.ApelS.MauererW. (2017). Evolutionary trends of developer coordination: a network approach. Empirical Software Engineering, 22(4), 2050–2094.
24.
KelleyR.CaplanJ. (1993). How Bell Labs creates star performers. Harvard Business Review, 71, 128–139.
25.
KilamoT.LeppänenM.MikkonenT. (2015). The social developer: now, then, and tomorrow. Proceedings of the 7th International Workshop on Social Software Engineering (SSE’15), 41–48.
26.
KleinG. (2008). Naturalistic decision making. Human Factors, 50(3), 456-460.
LazerD.PentlandA.AdamicL.AralS.BarabasiA. L.BrewerD.…AlstyneM. V. (2009). Computational social science. Science, 323(5915), 721-723.
29.
NassifM.RobillardM. P. (2017). Revisiting turnover-induced knowledge loss in software projects. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 261–272). Shanghai, China: IEEE.
30.
NewtonO. B.FioreS. M.SongJ. (2018). Developing theory and methods to understand and improve collaboration in open source software development on GitHub. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62(1), 1118–1122.
31.
PawadeD.DaveD. J.KamathA. (2016). Exploring software complexity metric from procedure oriented to object oriented. 2016 6th International Conference - Cloud System and Big Data Engineering (Confluence), 630–634.
32.
R Core Team. (2018). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org
33.
RiedlT. R.WeitzenfeldJ. S.FreemanJ. T.KleinG. A.MusaJ. (1991). What we have learned about software engineering expertise. Proceedings of the Fifth Software Engineering Institute Conference on Software Engineering Education (pp. 261–270). New York: Springer.
34.
RigbyP. C.ZhuY. C.DonadelliS. M.MockusA. (2016). Quantifying and mitigating turnover-induced knowledge loss: case studies of Chrome and a project at Avaya. In Proceedings of the 38th International Conference on Software Engineering - ICSE ’16 (pp. 1006–1016). Austin, Texas: ACM Press.
35.
SimonH. A. (1973). The structure of ill-structured problems. Artificial Intelligence, 4, 181–204.
36.
SonnentagS. (1995). Excellent software professionals: Experience, work activities, and perceptions by peers. Behaviour & Information Technology, 14, 289–299.
37.
SonnentagS. (1998). Expertise in professional software design: A process study. Journal of Applied Psychology, 83, 703–715.
38.
SpeierC. (2006). The influence of information presentation formats on complex task decision-making performance. International Journal of Human-Computer Studies, 64(11), 1115-1131.
39.
TopiH.ValacichJ.S.HofferJ.A. (2005). The effects of task complexity and time availability limitations on human performance in database query tasks. International Journal of Human-Computer Studies, 62(3), 349-379.
40.
VasilescuB.PosnettD.RayB.van den BrandM. G. J.SerebrenikA.DevanbuP.FilkovV. (2015). Gender and tenure diversity in GitHub teams. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15 (pp. 3789–3798). Seoul, Republic of Korea: ACM Press.
41.
WinterB. (2013). Linear models and linear mixed effects models in R with linguistic applications. arXiv:1308.5499.
42.
WoodR. E. (1986). Task complexity: definition of the construct. Organizational Behavior and Human Decision Processes, 37, 60-82.
43.
YuY.YinG.WangH.WangT. (2014). Exploring the patterns of social behavior in GitHub. In Proceedings of the 1st International Workshop on Crowd-based Software Development Methods and Technologies (pp. 31–36). New York, NY: ACM Press.
