{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,7]],"date-time":"2026-02-07T19:00:25Z","timestamp":1770490825374,"version":"3.49.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,11,2]],"date-time":"2020-11-02T00:00:00Z","timestamp":1604275200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,11,2]],"date-time":"2020-11-02T00:00:00Z","timestamp":1604275200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["HO4770\/2-2"],"award-info":[{"award-number":["HO4770\/2-2"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008769","name":"Julius-Maximilians-Universit\u00e4t W\u00fcrzburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008769","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Qual User Exp"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Evaluating the Quality of Experience (QoE) of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, most of these crowdsourcing studies either use, what we refer to, as an <jats:italic>in\u00a0vivo<\/jats:italic> or an <jats:italic>in\u00a0vitro<\/jats:italic> interface design. <jats:italic>In vivo<\/jats:italic> design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. <jats:italic>In vitro<\/jats:italic> design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these interface designs vary widely, the results are often compared and generalized. In this work, we use a crowdsourcing study to investigate the influence of three interface design alternatives, an <jats:italic>in\u00a0vitro<\/jats:italic> and two <jats:italic>in\u00a0vivo<\/jats:italic> designs with different levels of interactiveness, on the perceived video QoE. Contrary to our expectations, the results indicate that there is no significant influence of the study\u2019s interface design in general on the video experience. Furthermore, we found that the <jats:italic>in\u00a0vivo<\/jats:italic> design does not reduce the test takers\u2019 attentiveness. However, we observed that participants who interacted with the test interface reported a higher video QoE than other groups.<\/jats:p>","DOI":"10.1007\/s41233-020-00041-2","type":"journal-article","created":{"date-parts":[[2020,11,2]],"date-time":"2020-11-02T13:04:28Z","timestamp":1604322268000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["In vitro vs in vivo: does the study\u2019s interface design influence crowdsourced video QoE?"],"prefix":"10.1007","volume":"6","author":[{"given":"Kathrin","family":"Borchert","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3329-9910","authenticated-orcid":false,"given":"Anika","family":"Seufert","sequence":"additional","affiliation":[]},{"given":"Edwin","family":"Gamboa","sequence":"additional","affiliation":[]},{"given":"Matthias","family":"Hirth","sequence":"additional","affiliation":[]},{"given":"Tobias","family":"Ho\u00dffeld","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,11,2]]},"reference":[{"key":"41_CR1","doi-asserted-by":"crossref","unstructured":"Borchert K, Schwind A, Hirth M, Ho\u00dffeld T (2019) In vivo or in vitro? influence of the study design on crowdsourced video QoE. In: 2019 Eleventh international conference on quality of multimedia experience (QoMEX), IEEE, pp 1\u20136","DOI":"10.1109\/QoMEX.2019.8743317"},{"key":"41_CR2","doi-asserted-by":"crossref","unstructured":"Van\u00a0den Broeck W, Jacobs A, Staelens N (2012) Integrating the everyday-life context in subjective video quality experiments. In: Workshop on quality of multimedia experience","DOI":"10.1109\/QoMEX.2012.6263848"},{"key":"41_CR3","unstructured":"Cisco: Cisco visual networking index: Global mobile data traffic forecast update, 2016\u20132021, (2017) San Jose. CA, USA"},{"issue":"1","key":"41_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3148148","volume":"51","author":"F Daniel","year":"2018","unstructured":"Daniel F, Kucherbaev P, Cappiello C, Benatallah B, Allahbakhsh M (2018) Quality control in crowdsourcing: a survey of quality attributes, assessment techniques, and assurance actions. ACM Comput Surv 51(1):1\u201340","journal-title":"ACM Comput Surv"},{"key":"41_CR5","doi-asserted-by":"crossref","unstructured":"Difallah D, Filatova E, Ipeirotis P (2018) Demographics and dynamics of mechanical Turk workers. In: Proceedings of the eleventh ACM international conference on web search and data mining, pp 135\u2013143","DOI":"10.1145\/3159652.3159661"},{"key":"41_CR6","doi-asserted-by":"crossref","unstructured":"Egger-Lampl S, Redi J, Ho\u00dffeld T, Hirth M, M\u00f6ller S, Naderi B, Keimel C, Saupe D (2017) Crowdsourcing quality of experience experiments. In: Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Springer","DOI":"10.1007\/978-3-319-66435-4_7"},{"issue":"2","key":"41_CR7","doi-asserted-by":"publisher","first-page":"189","DOI":"10.1177\/0165551512437638","volume":"38","author":"E Estell\u00e9s-Arolas","year":"2012","unstructured":"Estell\u00e9s-Arolas E, Gonz\u00e1lez-Ladr\u00f3n-De-Guevara F (2012) Towards an integrated crowdsourcing definition. J Inf Sci 38(2):189\u2013200","journal-title":"J Inf Sci"},{"key":"41_CR8","doi-asserted-by":"crossref","unstructured":"Finnerty A, Kucherbaev P, Tranquillini S, Convertino G (2013) Keep it simple: Reward and task design in crowdsourcing. In: Proceedings of the biannual conference of the Italian chapter of SIGCHI, ACM, p\u00a014","DOI":"10.1145\/2499149.2499168"},{"key":"41_CR9","doi-asserted-by":"crossref","unstructured":"Gardlo B, Egger S, Seufert M, Schatz R (2014) Crowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing. In: Proceedings of the international conference on communications","DOI":"10.1109\/ICC.2014.6883463"},{"key":"41_CR10","doi-asserted-by":"crossref","unstructured":"Guse D, Egger S, Raake A, M\u00f6ller S (2014) Web-QoE under real-world distractions: two test cases. In: 2014 sixth international workshop on quality of multimedia experience (QoMEX), IEEE, pp 220\u2013225","DOI":"10.1109\/QoMEX.2014.6982322"},{"key":"41_CR11","doi-asserted-by":"crossref","unstructured":"Hirth M, Borchert K, De\u00a0Moor K, Borst V, Ho\u00dffeld T (2020) Personal task design preferences of crowdworkers. In: 2020 Twelfth international conference on quality of multimedia experience (QoMEX). IEEE","DOI":"10.1109\/QoMEX48832.2020.9123094"},{"key":"41_CR12","doi-asserted-by":"crossref","unstructured":"Hirth M, Ho\u00dffeld T, Tran-Gia P (2011) Anatomy of a crowdsourcing platform\u2014using the example of microworkers.com. In: Proceedings of the conference on innovative mobile and internet services in ubiquitous computing","DOI":"10.1109\/IMIS.2011.89"},{"issue":"11\u201312","key":"41_CR13","doi-asserted-by":"publisher","first-page":"2918","DOI":"10.1016\/j.mcm.2012.01.006","volume":"57","author":"M Hirth","year":"2013","unstructured":"Hirth M, Ho\u00dffeld T, Tran-Gia P (2013) Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms. Math Comput Model 57(11\u201312):2918\u20132932","journal-title":"Math Comput Model"},{"key":"41_CR14","doi-asserted-by":"crossref","unstructured":"Hirth M, Scheuring S, Ho\u00dffeld T, Schwartz C, Tran-Gia P (2014) Predicting result quality in crowdsourcing using application layer monitoring. In: 2014 IEEE fifth international conference on communications and electronics (ICCE), IEEE, pp 510\u2013515","DOI":"10.1109\/CCE.2014.6916756"},{"issue":"2","key":"41_CR15","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1109\/TMM.2013.2291663","volume":"16","author":"T Hossfeld","year":"2014","unstructured":"Hossfeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran-Gia P (2014) Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Trans Multimedia 16(2):541\u2013558","journal-title":"IEEE Trans Multimedia"},{"key":"41_CR16","unstructured":"ITU-T\u00a0Recommendation P (2008) Subjective video quality assessment methods for multimedia applications. International telecommunication union"},{"key":"41_CR17","doi-asserted-by":"crossref","unstructured":"Jumisko-Pyykk\u00f6 S, Hannuksela MM (2008) Does context matter in quality evaluation of mobile television? In: Proceedings of the 10th international conference on Human computer interaction with mobile devices and services, ACM, pp 63\u201372","DOI":"10.1145\/1409240.1409248"},{"key":"41_CR18","doi-asserted-by":"crossref","unstructured":"Kazai G, Kamps J, Milic-Frayling N (2011) Worker types and personality traits in crowdsourcing relevance labels. In: Proceedings of the conference on information and knowledge management","DOI":"10.1145\/2063576.2063860"},{"key":"41_CR19","doi-asserted-by":"crossref","unstructured":"Keimel C, Habigt J, Horch C, Diepold K (2012) Qualitycrowd\u2014a framework for crowd-basedquality evaluation. In: Proceedings of the picture coding symposium","DOI":"10.1109\/PCS.2012.6213338"},{"key":"41_CR20","doi-asserted-by":"crossref","unstructured":"Ketyk\u00f3 I, De\u00a0Moor K, De\u00a0Pessemier T, Verdejo AJ, Vanhecke K, Joseph W, Martens L, De\u00a0Marez L (2010) QoE measurement of mobile YouTube video streaming. In: Proceedings of the 3rd workshop on mobile video delivery, MoViD \u201910, pp 27\u201332","DOI":"10.1145\/1878022.1878030"},{"issue":"2","key":"41_CR21","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1016\/j.bushor.2016.10.001","volume":"60","author":"JH Kietzmann","year":"2017","unstructured":"Kietzmann JH (2017) Crowdsourcing: a revised definition and introduction to new research. Bus Horizons 60(2):151\u2013153","journal-title":"Bus Horizons"},{"key":"41_CR22","unstructured":"Le\u00a0Callet P, M\u00f6ller S, Perkis A (2012) (eds.): Qualinet White Paper on Definitions of Quality of Experience. European network on quality of experience in multimedia systems and services (COST Action IC 1003), Lausanne, Switzerland"},{"key":"41_CR23","doi-asserted-by":"crossref","unstructured":"Martin D, Carpendale S, Gupta N, Ho\u00dffeld T, Naderi B, Redi J, Siahaan E, Wechsung I (2017) Understanding the crowd: ethical and practical matters in the academic use of crowdsourcing. In: Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Springer","DOI":"10.1007\/978-3-319-66435-4_3"},{"issue":"4","key":"41_CR24","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1145\/2740070.2631433","volume":"44","author":"H Nam","year":"2014","unstructured":"Nam H, Kim KH, Calin D, Schulzrinne H (2014) YouSlow: a performance analysis tool for adaptive bitrate video streaming. SIGCOMM Comput Commun Rev 44(4):111\u2013112","journal-title":"SIGCOMM Comput Commun Rev"},{"key":"41_CR25","doi-asserted-by":"crossref","unstructured":"Nam H, Kim KH, Schulzrinne H (2016) QoE matters more than QoS: Why people stop watching cat videos. In: Proceedings of the international conference on computer communications","DOI":"10.1109\/INFOCOM.2016.7524426"},{"issue":"1","key":"41_CR26","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1109\/COMST.2014.2360940","volume":"17","author":"M Seufert","year":"2015","unstructured":"Seufert M, Egger S, Slanina M, Zinner T, Hossfeld T, Tran-Gia P (2015) A survey on quality of experience of HTTP adaptive streaming. IEEE Commun Surv Tutorials 17(1):469\u2013492","journal-title":"IEEE Commun Surv Tutorials"},{"key":"41_CR27","doi-asserted-by":"crossref","unstructured":"Seufert M, Ho\u00dffeld T, Sieber C (2015) Impact of intermediate layer on quality of experience of HTTP adaptive streaming. In: Proceedings of the conference on network and service management, pp 256\u2013260","DOI":"10.1109\/CNSM.2015.7367367"},{"key":"41_CR28","doi-asserted-by":"crossref","unstructured":"Seufert M, Zach O, Slanina M, Tran-Gia P (2017) Unperturbed video streaming QoE under web page related context factors. In: Proceedings of the conference on quality of multimedia experience","DOI":"10.1109\/QoMEX.2017.7965675"},{"issue":"1","key":"41_CR29","doi-asserted-by":"publisher","first-page":"18","DOI":"10.18352\/lq.9948","volume":"25","author":"E Simperl","year":"2015","unstructured":"Simperl E (2015) How to use crowdsourcing effectively: Guidelines and examples. Liber Quart,  25(1):18-39","journal-title":"Liber Quart"},{"issue":"4","key":"41_CR30","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1109\/TBC.2010.2067710","volume":"56","author":"N Staelens","year":"2010","unstructured":"Staelens N, Moens S, Van den Broeck W, Marien I, Vermeulen B, Lambert P, Van de Walle R, Demeester P (2010) Assessing quality of experience of IPTV and video on demand services in real-life environments. IEEE Trans Broadcast 56(4):458\u2013466","journal-title":"IEEE Trans Broadcast"},{"key":"41_CR31","doi-asserted-by":"crossref","unstructured":"Wamser F, Seufert M, Casas P, Irmer R, Tran-Gia P, Schatz R (2015) YoMoApp: A tool for analyzing qoe of YouTube HTTP adaptive streaming in mobile networks. In: European conference on networks and communications. IEEE","DOI":"10.1109\/EuCNC.2015.7194076"},{"key":"41_CR32","doi-asserted-by":"crossref","unstructured":"Xue J, Chen CW (2012) A study on perception of mobile video with surrounding contextual influences. In: 2012 Fourth international workshop on quality of multimedia experience, pp 248\u2013253","DOI":"10.1109\/QoMEX.2012.6263869"},{"key":"41_CR33","doi-asserted-by":"crossref","unstructured":"Zach O, Seufert M, Hirth M, Slanina M, Tran-Gia P (2017) On use of crowdsourcing for H. 264\/AVC and H. 265\/HEVC video quality evaluation. In: 2017 27th international conference radioelektronika (RADIOELEKTRONIKA), IEEE, pp 1\u20136","DOI":"10.1109\/RADIOELEK.2017.7937581"},{"key":"41_CR34","doi-asserted-by":"publisher","first-page":"412","DOI":"10.1016\/j.chb.2015.02.054","volume":"49","author":"Y Zhu","year":"2015","unstructured":"Zhu Y, Heynderickx I, Redi JA (2015) Understanding the role of social context and user factors in video quality of experience. Comput Human Behav 49:412\u2013426","journal-title":"Comput Human Behav"},{"key":"41_CR35","doi-asserted-by":"crossref","unstructured":"Zinner T, Hirth M, Fischer V, Hohlfeld O (2016) ERWIN-Enabling the reproducible investigation of waiting times for arbitrary workflows. In: Proceedings of the international conference on quality of multimedia experience","DOI":"10.1109\/QoMEX.2016.7498938"}],"container-title":["Quality and User Experience"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-020-00041-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41233-020-00041-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41233-020-00041-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,18]],"date-time":"2021-11-18T13:20:07Z","timestamp":1637241607000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41233-020-00041-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,2]]},"references-count":35,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["41"],"URL":"https:\/\/doi.org\/10.1007\/s41233-020-00041-2","relation":{},"ISSN":["2366-0139","2366-0147"],"issn-type":[{"value":"2366-0139","type":"print"},{"value":"2366-0147","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,2]]},"assertion":[{"value":"1 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 November 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors state that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"1"}}