{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,30]],"date-time":"2026-01-30T05:49:52Z","timestamp":1769752192749,"version":"3.49.0"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,12,31]],"date-time":"2020-12-31T00:00:00Z","timestamp":1609372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"crossref","award":["01IS19059B"],"award-info":[{"award-number":["01IS19059B"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"crossref"}]},{"name":"DFG","award":["SI 2045\/2-2, SI 2171\/2, SI 2171\/3, AP 206\/6"],"award-info":[{"award-number":["SI 2045\/2-2, SI 2171\/2, SI 2171\/3, AP 206\/6"]}]},{"name":"Centre Digitisation.Bavaria","award":["PICCARD"],"award-info":[{"award-number":["PICCARD"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2021,1,31]]},"abstract":"<jats:p>The human factor is prevalent in empirical software engineering research. However, human studies often do not use the full potential of analysis methods by combining analysis of individual tasks and participants with an analysis that aggregates results over tasks and\/or participants. This may hide interesting insights of tasks and participants and may lead to false conclusions by overrating or underrating single-task or participant performance. We show that studying multiple levels of aggregation of individual tasks and participants allows researchers to have both insights from individual variations as well as generalized, reliable conclusions based on aggregated data. Our literature survey revealed that most human studies perform either a fully aggregated analysis or an analysis of individual tasks. To show that there is important, non-trivial variation when including human participants, we reanalyze 12 published empirical studies, thereby changing the conclusions or making them more nuanced. Moreover, we demonstrate the effects of different aggregation levels by answering a novel research question on published sets of fMRI data. We show that when more data are aggregated, the results become more accurate. This proposed technique can help researchers to find a sweet spot in the tradeoff between cost of a study and reliability of conclusions.<\/jats:p>","DOI":"10.1145\/3406544","type":"journal-article","created":{"date-parts":[[2020,12,31]],"date-time":"2020-12-31T18:48:06Z","timestamp":1609440486000},"page":"1-40","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Mastering Variation in Human Studies"],"prefix":"10.1145","volume":"30","author":[{"given":"Janet","family":"Siegmund","sequence":"first","affiliation":[{"name":"Chemnitz University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norman","family":"Peitek","sequence":"additional","affiliation":[{"name":"Leibniz Institute for Neurobiology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3687-2233","authenticated-orcid":false,"given":"Sven","family":"Apel","sequence":"additional","affiliation":[{"name":"Saarland University, Saarland Informatics Campus"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Norbert","family":"Siegmund","sequence":"additional","affiliation":[{"name":"Leipzig University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,12,31]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9301-4"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/2780084.2780169"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","volume-title":"The New Statistical Analysis of Data","author":"Anderson Theodore","DOI":"10.1007\/978-1-4612-4000-6"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.59"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007515423169"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_1_9_1","unstructured":"Rolf Brickenkamp Lothar Schmidt-Atzert and Detlev Liepmann. 2010. Test d2-Revision: Aufmerksamkeits-und Konzentrationstest. Hogrefe G\u00f6ttingen. Rolf Brickenkamp Lothar Schmidt-Atzert and Detlev Liepmann. 2010. Test d2-Revision: Aufmerksamkeits-und Konzentrationstest. Hogrefe G\u00f6ttingen."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2048066.2048117"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11682-018-9885-1"},{"key":"e_1_2_1_12_1","volume-title":"Factor Analysis","author":"Cattell Raymond"},{"key":"e_1_2_1_13_1","volume-title":"Applied Multiple Regression: Correlation Analysis for the Behavioral Sciences (2 ed.)","author":"Cohen Jacob"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00994018"},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","volume-title":"Constructing the Subject","author":"Danziger Kurt","DOI":"10.1017\/CBO9780511524059"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491455"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining (KDD\u201997)","author":"Domingos Pedro","year":"1997"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the International Symposium Software Reliability Engineering (ISSRE\u201916)","author":"Duraes J."},{"key":"e_1_2_1_19_1","unstructured":"Rick Hoyle (Ed.). 2012. Handbook of Structural Equation Modeling. The Guilford Press. Rick Hoyle (Ed.). 2012. Handbook of Structural Equation Modeling. The Guilford Press."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1602413113"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568299"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9208-x"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2012.6240511"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.24"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227192"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X005010003"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(92)90056-P"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/990680.990686"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(02)00049-6"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"M. Grabisch J. L. Marichal R. Mesiar and E. Pap. 2009. Aggregation Functions. Cambridge University Press. M. Grabisch J. L. Marichal R. Mesiar and E. Pap. 2009. Aggregation Functions. Cambridge University Press.","DOI":"10.1017\/CBO9781139644150"},{"key":"e_1_2_1_31_1","volume-title":"Bagging equalizes influence. Mach. Learn.g 55, 3 (June","author":"Grandvalet Yves","year":"2004"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9289-1"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9623-8"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00053"},{"key":"e_1_2_1_35_1","volume-title":"Functional Magnetic Resonance Imaging","author":"Huettel Scott"},{"key":"e_1_2_1_36_1","unstructured":"Yoshiharu Ikutani Takatomi Kubo Satoshi Nishida Hideaki Hata Kenichi Matsumoto Kazushi Ikeda and Shinji Nishimoto. 2020. Expert programmers have fine-tuned cortical representations of source code. Retrieved from https:\/\/www.biorxiv.org\/content\/early\/2020\/01\/29\/2020.01.28.923953. Yoshiharu Ikutani Takatomi Kubo Satoshi Nishida Hideaki Hata Kenichi Matsumoto Kazushi Ikeda and Shinji Nishimoto. 2020. Expert programmers have fine-tuned cortical representations of source code. Retrieved from https:\/\/www.biorxiv.org\/content\/early\/2020\/01\/29\/2020.01.28.923953."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9605-x"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33014039"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9277-5"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1049\/ic.2011.0006"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9279-3"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9395-3"},{"key":"e_1_2_1_44_1","unstructured":"Alex Krizhevsky Ilya Sutskever and Geoffrey Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. Curran Associates 1097--1105. Alex Krizhevsky Ilya Sutskever and Geoffrey Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. Curran Associates 1097--1105."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380348"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9521-5"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the Asia-Pacific Software Engineering Conference (APSEC\u201914)","author":"Larsson H."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz470"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the International Conference on Software Engineering (ICSE\u201913)","author":"Lee Y."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786871"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.43"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368115"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9314-z"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1162\/089892903321593117"},{"key":"e_1_2_1_55_1","volume-title":"Testtheorie und Fragebogenkonstruktion","author":"Moosbrugger Helfried"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference (GECCO\u201916)","author":"Olson Randal S."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786807"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3239235.3240495"},{"key":"e_1_2_1_59_1","volume-title":"What\u2019s wrong with bonferroni adjustments. BMJ 316, 7139","author":"Perneger Thomas","year":"1998"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180161"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491448"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568252"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9286-4"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106268"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.24"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106251"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISESE.2002.1166921"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.97"},{"key":"e_1_2_1_69_1","unstructured":"Jean Talairach and Pierre Tournoux. 1988. Co-Planar Stereotaxic Atlas of the Human Brain. Thieme. Jean Talairach and Pierre Tournoux. 1988. Co-Planar Stereotaxic Atlas of the Human Brain. Thieme."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/0164-1212(94)00111-Y"},{"key":"e_1_2_1_71_1","volume-title":"Measures of Complexity","author":"Vapnik Vladimir"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.2002.11045721"},{"key":"e_1_2_1_73_1","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1080\/00031305.1982.10482778","article-title":"Simpson\u2019s paradox in real life","volume":"36","author":"Wagner Clifford","year":"1982","journal-title":"Amer. Stat."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180237"},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the International Conference on Software Engineering (ICSE\u201912)","author":"Yskout K."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.49"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.48.1.26"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3406544","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3406544","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:35Z","timestamp":1750197695000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3406544"}},"subtitle":["The Role of Aggregation"],"short-title":[],"issued":{"date-parts":[[2020,12,31]]},"references-count":76,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,1,31]]}},"alternative-id":["10.1145\/3406544"],"URL":"https:\/\/doi.org\/10.1145\/3406544","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,31]]},"assertion":[{"value":"2019-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}