{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T04:06:35Z","timestamp":1773115595409,"version":"3.50.1"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T00:00:00Z","timestamp":1688515200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T00:00:00Z","timestamp":1688515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100009123","name":"Norges Teknisk-Naturvitenskapelige Universitet","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009123","id-type":"DOI","asserted-by":"publisher"}]},{"name":"NTNU Norwegian University of Science and Technology"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Qual User Exp"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The concept of conducting ecologically valid user studies is gaining traction in the field of Quality of Experience (QoE). However, despite previous research exploring this concept, the increasing volume of studies has made it challenging to obtain a comprehensive overview of existing guidelines and the key aspects to consider when designing ecologically valid studies. Therefor this paper aims to provide a systematic review of research articles published between 2011 and 2021 that offer insight into conducting ecologically valid user studies. From an initial count of 782 retrieved studies, a final count of 12 studies met the predefined criteria and were included in the final review. The systematic review resulted in the extraction of 55 guidelines that provide guidance towards conducting ecologically valid user studies. These guidelines have been grouped within 8 categories (<jats:italic>Environment, Technology, Content, Participant Recruitment, User Behavior, Study Design, Task<\/jats:italic> and <jats:italic>data collection<\/jats:italic>) overarching the three main dimensions (Setting, Users and Research Methodology). Furthermore, the review discusses: the flip side of ecological validity, the implications for QoE research, as well as provides a basic visualisation model for assessing the ecological validity of a study. In conclusion, the current review indicates that future research should address more in detail how and when research approaches characterized by high ecological validity (and correspondingly, low internal validity) and those characterized by low ecological validity (and normally high internal validity) can best complement each other in order to better understand the key factors influencing QoE for various types of applications, user segments, settings. Further, we argue that more transparency around the (sub)dimensions of ecological validity with respect to a particular study or set of studies is necessary.<\/jats:p>","DOI":"10.1007\/s41233-023-00059-2","type":"journal-article","created":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T11:02:31Z","timestamp":1688554951000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Towards enhancing ecological validity in user studies: a systematic review of guidelines and implications for QoE research"],"prefix":"10.1007","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6286-9789","authenticated-orcid":false,"given":"Sruti","family":"Subramanian","sequence":"first","affiliation":[]},{"given":"Katrien","family":"De Moor","sequence":"additional","affiliation":[]},{"given":"Markus","family":"Fiedler","sequence":"additional","affiliation":[]},{"given":"Kamil","family":"Koniuch","sequence":"additional","affiliation":[]},{"given":"Lucjan","family":"Janowski","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,7,5]]},"reference":[{"key":"59_CR1","doi-asserted-by":"crossref","unstructured":"Raake A, Egger S (2014) Quality and quality of experience. In: Quality of experience, pp 11\u201333. Springer, Berlin","DOI":"10.1007\/978-3-319-02681-7_2"},{"issue":"8","key":"59_CR2","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1080\/10447318.2015.1064664","volume":"31","author":"M Hassenzahl","year":"2015","unstructured":"Hassenzahl M, Wiklund-Engblom A, Bengs A, H\u00e4gglund S, Diefenbach S (2015) Experience-oriented and product-oriented evaluation: psychological need fulfillment, positive affect, and product perception. Int J Human Comput Interaction 31(8):530\u2013544","journal-title":"Int J Human Comput Interaction"},{"key":"59_CR3","volume-title":"Technology as experience","author":"J McCarthy","year":"2007","unstructured":"McCarthy J, Wright P (2007) Technology as experience. MIT press, London"},{"key":"59_CR4","doi-asserted-by":"crossref","unstructured":"Reiter U, Brunnstr\u00f6m K, De Moor K, Larabi M-C, Pereira M, Pinheiro A, You J, Zgank A (2014) Factors influencing quality of experience. In: Quality of experience, pp 55\u201372. Springer, Berlin","DOI":"10.1007\/978-3-319-02681-7_4"},{"key":"59_CR5","doi-asserted-by":"publisher","unstructured":"Barakovi\u0107\u00a0Husi\u0107 J, Barakovi\u0107 S, Cero E, Slamnik N, O\u0107uz M, Dedovi\u0107 A, Zup\u010di\u0107 O (2020) Quality of experience for unified communications: a survey. Int J Netw Manage 30(3):2083. https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/nem.2083. https:\/\/doi.org\/10.1002\/nem.2083. e2083 NEM-18-0124","DOI":"10.1002\/nem.2083"},{"key":"59_CR6","doi-asserted-by":"crossref","unstructured":"Wechsung I, De\u00a0Moor K (2014) Quality of experience versus user experience. In: Quality of experience, pp 35\u201354. Springer, Berlin","DOI":"10.1007\/978-3-319-02681-7_3"},{"issue":"1","key":"59_CR7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s41233-022-00050-3","volume":"7","author":"I Viola","year":"2022","unstructured":"Viola I, Subramanyam S, Li J, Cesar P (2022) On the impact of vr assessment on the quality of experience of highly realistic digital humans. Quality User Exp 7(1):1\u201332","journal-title":"Quality User Exp"},{"key":"59_CR8","doi-asserted-by":"crossref","unstructured":"Hodzic K, Cosovic M, Mrdovic S, Quinlan JJ, Raca D (2022) Realistic video sequences for subjective qoe analysis. arXiv preprint arXiv:2204.06829","DOI":"10.1145\/3524273.3532894"},{"key":"59_CR9","volume-title":"Real world research","author":"C Robson","year":"2016","unstructured":"Robson C, McCartan K (2016) Real world research. Wiley Global Education, Chichester"},{"key":"59_CR10","doi-asserted-by":"crossref","unstructured":"Dole L, Ju W (2019) Face and ecological validity in simulations: lessons from search-and-rescue hri. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1\u20138","DOI":"10.1145\/3290605.3300681"},{"key":"59_CR11","doi-asserted-by":"crossref","unstructured":"Borchert K, Schwind A, Hirth M, Hosfeld T (2019) In vivo or in vitro? Influence of the study design on crowdsourced video QoE. In: 2019 11th International conference on quality of multimedia experience, QoMEX 2019. Institute of Electrical and Electronics Engineers Inc.","DOI":"10.1109\/QoMEX.2019.8743317"},{"key":"59_CR12","doi-asserted-by":"crossref","unstructured":"Masi AD, Wac K (2019) Predicting quality of experience of popular mobile applications from a living lab study. In: 2019 11th International conference on quality of multimedia experience, QoMEX 2019","DOI":"10.1109\/QoMEX.2019.8743306"},{"key":"59_CR13","doi-asserted-by":"crossref","unstructured":"Robitza W, Garcia MN, Raake A (2015) At home in the lab: assessing audiovisual quality of HTTP-based adaptive streaming with an immersive test paradigm. In: 2015 Seventh international workshop on quality of multimedia experience (QoMEX), pp 1\u20136. IEEE. http:\/\/ieeexplore.ieee.org\/document\/7148122\/","DOI":"10.1109\/QoMEX.2015.7148122"},{"key":"59_CR14","doi-asserted-by":"publisher","unstructured":"P\u00e9rez P, Gonz\u00e1lez-Sosa E, Kachach R, Pereira F, Villegas \u00c1 (2021) Ecological validity through gamification: an experiment with a mixed reality escape room. In: 2021 IEEE international conference on artificial intelligence and virtual reality (AIVR), pp 179\u2013183. https:\/\/doi.org\/10.1109\/AIVR52153.2021.00040","DOI":"10.1109\/AIVR52153.2021.00040"},{"key":"59_CR15","doi-asserted-by":"publisher","unstructured":"Dole L, Ju W (2019) Face and ecological validity in simulations:lessons from search-and-rescue hri. In: Proceedings of the 2019 CHI conference on human factors in computing systems. CHI \u201919, pp 1\u20138. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3290605.3300681","DOI":"10.1145\/3290605.3300681"},{"key":"59_CR16","doi-asserted-by":"publisher","unstructured":"Deniaud C, Mestre D, Honnet V, Jeanne B (2014) The concept of \"presence\" used as a measure for ecological validity in driving simulators. In: Proceedings of the 2014 European conference on cognitive ergonomics. ECCE \u201914. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/2637248.2637270","DOI":"10.1145\/2637248.2637270"},{"key":"59_CR17","doi-asserted-by":"publisher","unstructured":"Vona F, Pieri L, Patti A, Tafaro S, Saccoccio S, Garzotto F, Romano D (2022) Explore 360$$^{\\circ }$$ vr to improve the ecological validity of screening tests on cognitive functions. In: Proceedings of the 2022 international conference on advanced visual interfaces. AVI 2022. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3531073.3531171","DOI":"10.1145\/3531073.3531171"},{"issue":"4","key":"59_CR18","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1207\/S15327078IN0204_03","volume":"2","author":"DJ Lewkowicz","year":"2001","unstructured":"Lewkowicz DJ (2001) The concept of ecological validity: What are its limitations and is it bad to be invalid? Infancy 2(4):437\u2013450","journal-title":"Infancy"},{"key":"59_CR19","volume-title":"Quasi-experimentation","author":"DT Campbell","year":"1979","unstructured":"Campbell DT, Cook TD (1979) Quasi-experimentation. Houghton Mifflin, Boston"},{"key":"59_CR20","doi-asserted-by":"crossref","unstructured":"M\u00f6ller S, Raake A (2014) Quality of experience: advanced concepts, applications and methods. Springer, Berlin","DOI":"10.1007\/978-3-319-02681-7"},{"issue":"1","key":"59_CR21","doi-asserted-by":"publisher","first-page":"30","DOI":"10.4230\/DagMan.7.1.30","volume":"7","author":"M Fiedler","year":"2018","unstructured":"Fiedler M, M\u00f6ller S, Reichl P, Xie M (2018) QoE Vadisl (Dagstuhl Perspectives Workshop 16472). Dagstuhl Manifestos 7(1):30\u201351. https:\/\/doi.org\/10.4230\/DagMan.7.1.30","journal-title":"Dagstuhl Manifestos"},{"issue":"1","key":"59_CR22","doi-asserted-by":"publisher","first-page":"57","DOI":"10.4230\/DagRep.5.1.57","volume":"5","author":"K De Moor","year":"2015","unstructured":"De Moor K, Fiedler M, Reichl P, Varela M (2015) Quality of experience: from assessment to application (Dagstuhl Seminar 15022). Dagstuhl Rep 5(1):57\u201395. https:\/\/doi.org\/10.4230\/DagRep.5.1.57","journal-title":"Dagstuhl Rep"},{"key":"59_CR23","doi-asserted-by":"crossref","unstructured":"Robitza W, Garcia MN, Raake A (2015) At home in the lab: Assessing audiovisual quality of http-based adaptive streaming with an immersive test paradigm. In: 2015 Seventh international workshop on quality of multimedia experience (QoMEX), pp 1\u20136. IEEE","DOI":"10.1109\/QoMEX.2015.7148122"},{"issue":"3","key":"59_CR24","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1007\/s11036-010-0223-0","volume":"15","author":"K De Moor","year":"2010","unstructured":"De Moor K, Ketyko I, Joseph W, Deryckere T, De Marez L, Martens L, Verleye G (2010) Proposed framework for evaluating quality of experience in a mobile, testbed-oriented living lab setting. Mobile Netw Appl 15(3):378\u2013391","journal-title":"Mobile Netw Appl"},{"issue":"4","key":"59_CR25","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1109\/TBC.2010.2067710","volume":"56","author":"N Staelens","year":"2010","unstructured":"Staelens N, Moens S, Van den Broeck W, Marien I, Vermeulen B, Lambert P, Van de Walle R, Demeester P (2010) Assessing quality of experience of iptv and video on demand services in real-life environments. IEEE Trans Broadcast 56(4):458\u2013466","journal-title":"IEEE Trans Broadcast"},{"key":"59_CR26","doi-asserted-by":"crossref","unstructured":"Van\u00a0den Broeck W, Jacobs A, Staelens N (2012) Integrating the everyday-life context in subjective video quality experiments. In: 2012 Fourth international workshop on quality of multimedia experience, pp 19\u201324. IEEE","DOI":"10.1109\/QoMEX.2012.6263848"},{"key":"59_CR27","doi-asserted-by":"publisher","unstructured":"Schmidt S, Naderi B, Sabet SS, Zadtootaghaj S, M\u00f6ller S (2020) Assessing interactive gaming quality of experience using a crowdsourcing approach. In: 2020 Twelfth international conference on quality of multimedia experience (QoMEX), pp 1\u20136. https:\/\/doi.org\/10.1109\/QoMEX48832.2020.9123122","DOI":"10.1109\/QoMEX48832.2020.9123122"},{"key":"59_CR28","doi-asserted-by":"publisher","unstructured":"Seufert A, Wamser F, Yarish D, Macdonald H, Ho\u00dffeld T (2021) Qoe models in the wild: Comparing video qoe models using a crowdsourced data set. In: 2021 13th International Conference on Quality of Multimedia Experience (QoMEX), pp. 55\u201360. https:\/\/doi.org\/10.1109\/QoMEX51781.2021.9465422","DOI":"10.1109\/QoMEX51781.2021.9465422"},{"key":"59_CR29","doi-asserted-by":"publisher","unstructured":"Ho\u00dffeld T, Keimel C (2014) In: M\u00f6ller, S., Raake, A. (eds.) Crowdsourcing in QoE evaluation, pp 315\u2013327. Springer, Berlin. https:\/\/doi.org\/10.1007\/978-3-319-02681-7_21","DOI":"10.1007\/978-3-319-02681-7_21"},{"issue":"2","key":"59_CR30","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1109\/TMM.2013.2291663","volume":"16","author":"T Ho\u00dffeld","year":"2013","unstructured":"Ho\u00dffeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran-Gia P (2013) Best practices for qoe crowdtesting: Qoe assessment with crowdsourcing. IEEE Trans Multimedia 16(2):541\u2013558","journal-title":"IEEE Trans Multimedia"},{"issue":"1","key":"59_CR31","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1177\/0739456X17723971","volume":"39","author":"Y Xiao","year":"2019","unstructured":"Xiao Y, Watson M (2019) Guidance on conducting a systematic literature review. J Plan Educ Res 39(1):93\u2013112","journal-title":"J Plan Educ Res"},{"key":"59_CR32","doi-asserted-by":"crossref","unstructured":"Okoli C (2015) A guide to conducting a standalone systematic literature review. Commun Assoc Inf Syst, 37","DOI":"10.17705\/1CAIS.03743"},{"key":"59_CR33","unstructured":"Fink A (2019) Conducting research literature reviews: from the internet to paper. Sage publications"},{"key":"59_CR34","doi-asserted-by":"crossref","unstructured":"M\u00e4ki T, Varela M, Laulajainen J-P (2013) Estimating the effect of context on qoe of audiovisual services: Laboratory vs. public places. In: 2013 International conference on smart communications in network technologies (SaCoNeT), vol 3, pp 1\u20136. IEEE","DOI":"10.1109\/SaCoNeT.2013.6654556"},{"key":"59_CR35","doi-asserted-by":"crossref","unstructured":"Lew L, Nguyen T, Messing S, Westwood S (2011) Of course i wouldn\u2019t do that in real life: advancing the arguments for increasing realism in hci experiments. In: CHI\u201911 extended abstracts on human factors in computing systems, pp 419\u2013428","DOI":"10.1145\/1979742.1979621"},{"issue":"4","key":"59_CR36","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1109\/TBC.2014.2359255","volume":"60","author":"N Staelens","year":"2014","unstructured":"Staelens N, De Meulenaere J, Claeys M, Van Wallendael G, Van den Broeck W, De Cock J, Van de Walle R, Demeester P, De Turck F (2014) Subjective quality assessment of longer duration video sequences delivered over http adaptive streaming to tablet devices. IEEE Trans Broadcast 60(4):707\u2013714","journal-title":"IEEE Trans Broadcast"},{"key":"59_CR37","doi-asserted-by":"crossref","unstructured":"Maguire J, Renaud K (2015) Alternative authentication in the wild. In: 2015 Workshop on socio-technical aspects in security and trust, pp 32\u201339. IEEE","DOI":"10.1109\/STAST.2015.13"},{"key":"59_CR38","doi-asserted-by":"crossref","unstructured":"Robitza W, Kara PA, Martini MG, Raake A (2016) On the experimental biases in user behavior and qoe assessment in the lab. In: 2016 IEEE globecom workshops (GC Wkshps), pp 1\u20136. IEEE","DOI":"10.1109\/GLOCOMW.2016.7848978"},{"key":"59_CR39","doi-asserted-by":"crossref","unstructured":"Robitza W, Raake A (2016) (re-) actions speak louder than words? a novel test method for tracking user behavior in web video services. In: 2016 Eighth international conference on quality of multimedia experience (QoMEX), pp 1\u20136. IEEE","DOI":"10.1109\/QoMEX.2016.7498926"},{"key":"59_CR40","doi-asserted-by":"crossref","unstructured":"Mottelson A, Hornb\u00e6k K (2017) Virtual reality studies outside the laboratory. In: Proceedings of the 23rd Acm symposium on virtual reality software and technology, pp 1\u201310","DOI":"10.1145\/3139131.3139141"},{"issue":"3","key":"59_CR41","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1109\/TOH.2019.2930608","volume":"12","author":"JR Blum","year":"2019","unstructured":"Blum JR, Fortin PE, Al Taha F, Alirezaee P, Demers M, Weill-Duflos A, Cooperstock JR (2019) Getting your hands dirty outside the lab: a practical primer for conducting wearable vibrotactile haptics research. IEEE Trans Haptics 12(3):232\u2013246","journal-title":"IEEE Trans Haptics"},{"key":"59_CR42","doi-asserted-by":"crossref","unstructured":"Labont\u00e9-LeMoyne \u00c9, Courtemanche F, Fredette M, L\u00e9ger P-M (2018) How wild is too wild: Lessons learned and recommendations for ecological validity in physiological computing research. In: PhyCS, pp 123\u2013130","DOI":"10.5220\/0006962901230130"},{"key":"59_CR43","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103553","volume":"110","author":"N van Berkel","year":"2020","unstructured":"van Berkel N, Clarkson MJ, Xiao G, Dursun E, Allam M, Davidson BR, Blandford A (2020) Dimensions of ecological validity for usability evaluations in clinical settings. J Biomed Inf 110:103553","journal-title":"J Biomed Inf"},{"key":"59_CR44","doi-asserted-by":"crossref","unstructured":"Orne MT (2009) Demand characteristics and the concept of quasi-controls. Artifacts in behavioral research: robert Rosenthal and Ralph L. Rosnow\u2019s classic books 110:110\u2013137","DOI":"10.1093\/acprof:oso\/9780195385540.003.0005"},{"issue":"4","key":"59_CR45","doi-asserted-by":"publisher","first-page":"829","DOI":"10.2307\/2082710","volume":"88","author":"S Ansolabehere","year":"1994","unstructured":"Ansolabehere S, Iyengar S, Simon A, Valentino N (1994) Does attack advertising demobilize the electorate? Am Political Sci Rev 88(4):829\u2013838","journal-title":"Am Political Sci Rev"}],"container-title":["Quality and User Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-023-00059-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41233-023-00059-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-023-00059-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T10:05:42Z","timestamp":1734516342000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41233-023-00059-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,5]]},"references-count":45,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["59"],"URL":"https:\/\/doi.org\/10.1007\/s41233-023-00059-2","relation":{},"ISSN":["2366-0139","2366-0147"],"issn-type":[{"value":"2366-0139","type":"print"},{"value":"2366-0147","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,5]]},"assertion":[{"value":"31 January 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 July 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"6"}}