{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T05:03:29Z","timestamp":1773032609573,"version":"3.50.1"},"reference-count":68,"publisher":"Institution of Engineering and Technology (IET)","issue":"1","license":[{"start":{"date-parts":[[2025,12,26]],"date-time":"2025-12-26T00:00:00Z","timestamp":1766707200000},"content-version":"vor","delay-in-days":359,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/501100008784","name":"Universidad de las Fuerzas Armadas ESPE","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100008784","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["ietresearch.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["IET Software"],"published-print":{"date-parts":[[2025,1]]},"abstract":"<jats:sec>\n                    <jats:title>Context<\/jats:title>\n                    <jats:p>Replication plays a critical role in building cumulative scientific knowledge. However, in the context of Empirical Software Engineering (ESE), replication efforts face persistent difficulties, both technical and methodological, which hinder reproducibility and generalizability. We explored whether the problems were due to formal issues, such as under\u2010specification or miscommunication, or intrinsic reasons, i.e., the existing replication procedures may not meet the researchers\u2019 needs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To understand how ESE researchers conduct experiments in real\u2010world settings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Method<\/jats:title>\n                    <jats:p>We conducted an ethnographic study with an experimental software engineering group, using interviews, observations, and analysis of internal documentation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We have created conceptual and process models representing experimentation, replication, and synthesis in the target research group. These models align with mainstream procedures at a high level but often break down in practice during specialized tasks such as coordination or documentation. The experimental process observed in the group differs from common assumptions and textbook descriptions in terms of (1) the number and diversity of activities involved, (2) the presence of differentiated roles, (3) the granularity of conceptual elements, and (4) the varying perspectives across subareas or families of experiments.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The discrepancies between actual laboratory practices and idealized process models may hinder knowledge transfer and complicate replication. We plan to extend this research by involving additional research groups, for instance, through surveys, to examine the generalizability of our findings.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1049\/sfw2\/7082969","type":"journal-article","created":{"date-parts":[[2025,12,26]],"date-time":"2025-12-26T14:05:32Z","timestamp":1766757932000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Investigation of the Activities Performed by Experimental Researchers in a Software Engineering Lab Using an Ethnographic Methodology"],"prefix":"10.1049","volume":"2025","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7129-9335","authenticated-orcid":false,"given":"Efra\u00edn R.","family":"Fonseca C.","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9252-0470","authenticated-orcid":false,"given":"Marta","family":"L\u00f3pez-Fern\u00e1ndez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3060-7853","authenticated-orcid":false,"given":"Oscar","family":"Dieste","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2465-7141","authenticated-orcid":false,"given":"Natalia","family":"Juristo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"265","published-online":{"date-parts":[[2025,12,26]]},"reference":[{"key":"e_1_2_16_1_2","doi-asserted-by":"publisher","DOI":"10.1177\/2515245918810225"},{"key":"e_1_2_16_2_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612465253"},{"key":"e_1_2_16_3_2","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_2_16_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.01.006"},{"key":"e_1_2_16_5_2","doi-asserted-by":"crossref","unstructured":"BezerraR. M. M. da SilvaF. Q. B. SantanaA. M. MagalhaesC. V. C. andSantosR. E. S. Replication of Empirical Studies in Software Engineering: An Update of A Systematic Mapping Study 2015 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2015 Beijing China IEEE 1\u20134.","DOI":"10.1109\/ESEM.2015.7321213"},{"key":"e_1_2_16_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.03.065"},{"key":"e_1_2_16_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2004.08.005"},{"key":"e_1_2_16_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.02.001"},{"key":"e_1_2_16_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_2_16_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.02.003"},{"key":"e_1_2_16_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.106870"},{"key":"e_1_2_16_12_2","unstructured":"CarverJ. C. Towards Reporting Guidelines for Experimental Replications: A Proposal Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) 2010 Cape Town South Africa ACM."},{"key":"e_1_2_16_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_8"},{"key":"e_1_2_16_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2519887"},{"key":"e_1_2_16_15_2","doi-asserted-by":"crossref","unstructured":"ZhangH. HuangX. ZhouX. HuangH. andBabarM. A. Ethnographic Research in Software Engineering: A Critical Review and Checklist 2019 Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC-FSE) 2019 Tallinn Estonia ACM 659\u2013670.","DOI":"10.1145\/3338906.3338976"},{"key":"e_1_2_16_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10069-3"},{"key":"e_1_2_16_17_2","doi-asserted-by":"publisher","DOI":"10.1177\/1461444820975723"},{"key":"e_1_2_16_18_2","doi-asserted-by":"crossref","unstructured":"de Souza.SantosR. E.andRalphP. Practices to Improve Teamwork in Software Development During the Covid-19 Pandemic: An Ethnographic Study 2022 International Conference on Cooperative and Human Aspects of Software Engineering (CHASE) 2022 Pittsburgh Pennsylvania United States ACM 81\u201385.","DOI":"10.1145\/3528579.3529174"},{"key":"e_1_2_16_19_2","doi-asserted-by":"crossref","unstructured":"WangZ. LiuB. ZhanZ. ZhangH. andLiG. An Ethnographic Study on the CI of a Large Scale Project 2024 International Conference on Software Engineering: Software Engineering in Practice (SEIP) 2024 Lisboa Portugal ACM 287\u2013297.","DOI":"10.1145\/3639477.3639750"},{"key":"e_1_2_16_20_2","doi-asserted-by":"publisher","DOI":"10.4135\/9781848607958"},{"key":"e_1_2_16_21_2","doi-asserted-by":"crossref","unstructured":"YadavS. SudmanM. S. I. DubeyP. K. SrinivasR. V. SrisainathR. andDeviV. C. Development of an GA-RBF Based Model for Penetration of Electric Vehicles and its Projections 2023 International Conference on Self Sustainable Artificial Intelligence Systems (ICSSAS) 2023 Tamilnadu India IEEE 1\u20136.","DOI":"10.1109\/ICSSAS57918.2023.10331883"},{"key":"e_1_2_16_22_2","doi-asserted-by":"crossref","unstructured":"SjobergD. I. K. DybaT. andJorgensenM. The Future of Empirical Methods in Software Engineering Research 2007 Future of Software Engineering (FOSE) 2007 Washington DC USA IEEE 358\u2013378.","DOI":"10.1109\/FOSE.2007.30"},{"key":"e_1_2_16_23_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781118181034"},{"key":"e_1_2_16_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_2_16_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-005-1290-x"},{"key":"e_1_2_16_26_2","doi-asserted-by":"publisher","DOI":"10.1525\/jer.2007.2.1.3"},{"key":"e_1_2_16_27_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2169"},{"key":"e_1_2_16_28_2","doi-asserted-by":"crossref","unstructured":"StrandbergP. E. Ethical Interviews In Software Engineering 2019 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2019 Porto de Galinhas Recife Brazil IEEE 1\u201311.","DOI":"10.1109\/ESEM.2019.8870192"},{"key":"e_1_2_16_29_2","doi-asserted-by":"crossref","unstructured":"YangN. CuijpersP. SchiffelersR. LukkienJ. andSerebrenikA. An Interview Study of How Developers use Execution Logs in Embedded Software Engineering 2021 IEEE\/ACM. 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) 2021 Montr\u00e9al Canada IEEE 61\u201370.","DOI":"10.1109\/ICSE-SEIP52600.2021.00015"},{"key":"e_1_2_16_30_2","doi-asserted-by":"crossref","unstructured":"ModiB. A. CainA. Wood.BradleyG. andRenzellaJ. Using Focus to Personalise Learning and Feedback in Software Engineering Education 2023 International Conference on Software Engineering: Software Engineering Education and Training (SEET) 2023 Melbourne Australia IEEE 296\u2013301.","DOI":"10.1109\/ICSE-SEET58685.2023.00033"},{"key":"e_1_2_16_31_2","doi-asserted-by":"crossref","unstructured":"RunesonP.andOlssonT. Challenges and Opportunities in Open Data Collaboration\u2013A Focus Group Study 2020 Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2020 Portoroz Slovenia IEEE 205\u2013212.","DOI":"10.1109\/SEAA51224.2020.00044"},{"key":"e_1_2_16_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-4625-2"},{"key":"e_1_2_16_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3304-4"},{"key":"e_1_2_16_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1986.6312975"},{"key":"e_1_2_16_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_2_16_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.799939"},{"key":"e_1_2_16_37_2","doi-asserted-by":"publisher","DOI":"10.1201\/b19467"},{"key":"e_1_2_16_38_2","volume-title":"The Tacit Dimension\u2019","author":"Polanyi M.","year":"1966"},{"key":"e_1_2_16_39_2","article-title":"Learn by Doing as a Didactic Strategy Through the Codecademy Web Portal in Pedagogical Practice to Promote Creative Thinking and Significant Learning in Students","volume":"1408","author":"Mendoza F. G.","year":"2019","journal-title":"Journal of Physics: Conference Series"},{"key":"e_1_2_16_40_2","doi-asserted-by":"crossref","unstructured":"GallardoE. G. E. Using Configuration Management and Product Line Software Paradigms to Support the Experimentation Process in Software Engineering 2012 Sixth International Conference on Research Challenges in Information Science (RCIS) 2012 Valencia Spain IEEE 1\u20136.","DOI":"10.1109\/RCIS.2012.6240454"},{"key":"e_1_2_16_41_2","doi-asserted-by":"crossref","unstructured":"VegasS. JuristoN. MorenoA. SolariM. andLetelierP. Analysis of the Influence of Communication Between Researchers on Experiment Replication 2006 International Symposium on Empirical Software Engineering (ISESE) 2006 Rio de Janeiro Brazil ACM 28\u201337.","DOI":"10.1145\/1159733.1159741"},{"key":"e_1_2_16_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.04.004"},{"key":"e_1_2_16_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2467378"},{"key":"e_1_2_16_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25231-0_2"},{"key":"e_1_2_16_45_2","volume-title":"Verbal Protocols of Reading: The Nature of Constructively Responsive Reading\u2019","author":"Pressley M.","year":"1995"},{"key":"e_1_2_16_46_2","doi-asserted-by":"publisher","DOI":"10.4337\/9781786439659.00033"},{"key":"e_1_2_16_47_2","unstructured":"The International Organization for Standardization ISO\/IEC\/IEEE 12207 Systems and Software Engineering - Software Life Cycle Processes 2017 https:\/\/www.iso.org\/standard\/63712.html."},{"key":"e_1_2_16_48_2","unstructured":"DalkeyN. C. Delphi 1967 RAND Corporation Santa Monica California USA: https:\/\/apps.dtic.mil\/dtic\/tr\/fulltext\/u2\/660554.pdf."},{"key":"e_1_2_16_49_2","volume-title":"Research Design: Qualitative, Quantitative, and Mixed Methods Approaches","author":"Creswell J. W.","year":"2014"},{"key":"e_1_2_16_50_2","volume-title":"Design and Analysis of Experiments\u2019","author":"Montgomery D. C.","year":"2017"},{"key":"e_1_2_16_51_2","doi-asserted-by":"crossref","unstructured":"RozierK. Y.andRozierE. W. Reproducibility Correctness and Buildability: The Three Principles for Ethical Public Dissemination of Computer Science and Engineering Research 2014 IEEE International Symposium on Ethics in Science Technology and Engineering 2014 Chicago Illinois USA IEEE 1\u201313.","DOI":"10.1109\/ETHICS.2014.6893384"},{"key":"e_1_2_16_52_2","article-title":"Experimentation in Software Engineering: A New Framework","volume":"4","author":"Mohamed W.","year":"1970","journal-title":"WIT Transactions on Information and Communication Technologies"},{"key":"e_1_2_16_53_2","doi-asserted-by":"publisher","DOI":"10.1023\/B:EMSE.0000013516.80487.33"},{"key":"e_1_2_16_54_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781003014584-19"},{"key":"e_1_2_16_55_2","doi-asserted-by":"crossref","unstructured":"FerreiraW. BaldassarreM. T. SoaresS. CartaxoB. andVisaggioG. A Comparative Study of Model-Driven Approaches for Scoping and Planning Experiments 2017 International Conference on Evaluation and Assessment in Software Engineering (EASE) 2017 Karlskrona Sweden ACM 78\u201387.","DOI":"10.1145\/3084226.3084258"},{"key":"e_1_2_16_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.97"},{"key":"e_1_2_16_57_2","volume-title":"Case Study Research Design and Methods\u2019","author":"Yin R. K.","year":"2009"},{"key":"e_1_2_16_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.celrep.2014.02.021"},{"key":"e_1_2_16_59_2","doi-asserted-by":"crossref","unstructured":"ShullF. BasiliV. andCarverJ. et al.Replicating Software Engineering Experiments: Addressing the Tacit Knowledge Problem 2002 International Symposium on Empirical Software Engineering (ISESE) 2002 Nara Japan IEEE 7\u201316.","DOI":"10.1109\/ISESE.2002.1166920"},{"key":"e_1_2_16_60_2","article-title":"Experimental Error: What Happens in the Lab Stays in the Lab","author":"Ruben A.","year":"2011","journal-title":"American Association for the Advancement of Science (AAAS)"},{"key":"e_1_2_16_61_2","article-title":"Beyond Lab Folklore and Mythology","author":"Loukides M.","year":"2015","journal-title":"O\u2019Reilly"},{"key":"e_1_2_16_62_2","unstructured":"The International Organization for Standardization ISO\/IEC 20000-1: 2018 Information Technology - Service Management - Part 1: Service Management System Requirements 2018 https:\/\/www.iso.org\/standard\/70636.html."},{"key":"e_1_2_16_63_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1486"},{"key":"e_1_2_16_64_2","doi-asserted-by":"publisher","DOI":"10.1111\/spc3.12110"},{"key":"e_1_2_16_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-349-13693-3_4"},{"key":"e_1_2_16_66_2","doi-asserted-by":"publisher","DOI":"10.1053\/bean.2002.0258"},{"key":"e_1_2_16_67_2","unstructured":"European Commission Researchcomp: The European Competence Framework for Researchers 2022 accessed: 2024-02-18. Available from:https:\/\/research-and-innovation.ec.europa.eu\/document\/download\/7da29338-37bf-4d51-b5eb-a1571b84c7ad_en?filename=ec_rtd_research-competence-presentation.pdf."},{"key":"e_1_2_16_68_2","unstructured":"Fonseca C.E. R. DiesteO. andJuristoN. A City Upon a Hill: Casting Light on a Real Experimental Process 2021 https:\/\/arxiv.org\/abs\/2108.12800 preprint available at."}],"container-title":["IET Software"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/sfw2\/7082969","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/full-xml\/10.1049\/sfw2\/7082969","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/sfw2\/7082969","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T04:12:38Z","timestamp":1773029558000},"score":1,"resource":{"primary":{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/10.1049\/sfw2\/7082969"}},"subtitle":[],"editor":[{"given":"Nadeem","family":"Sarwar","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2025,1]]},"references-count":68,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1]]}},"alternative-id":["10.1049\/sfw2\/7082969"],"URL":"https:\/\/doi.org\/10.1049\/sfw2\/7082969","archive":["Portico"],"relation":{},"ISSN":["1751-8806","1751-8814"],"issn-type":[{"value":"1751-8806","type":"print"},{"value":"1751-8814","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1]]},"assertion":[{"value":"2023-10-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"7082969"}}