{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:17:52Z","timestamp":1750220272286,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,6,14]],"date-time":"2022-06-14T00:00:00Z","timestamp":1655164800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,14]]},"DOI":"10.1145\/3524273.3532887","type":"proceedings-article","created":{"date-parts":[[2022,8,5]],"date-time":"2022-08-05T22:23:21Z","timestamp":1659738201000},"page":"203-209","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Huldra"],"prefix":"10.1145","author":[{"given":"Malek","family":"Hammou","sequence":"first","affiliation":[{"name":"SimulaMet, Norway"}]},{"given":"Cise","family":"Midoglu","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway"}]},{"given":"Steven A.","family":"Hicks","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway"}]},{"given":"Andrea","family":"Stor\u00e5s","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway and Oslo Metropolitan University, Norway"}]},{"given":"Saeed Shafiee","family":"Sabet","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway"}]},{"given":"Inga","family":"Str\u00fcmke","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway"}]},{"given":"Michael A.","family":"Riegler","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway and UIT The Arctic University of Norway, Norway"}]},{"given":"P\u00e5l","family":"Halvorsen","sequence":"additional","affiliation":[{"name":"SimulaMet, Norway and Oslo Metropolitan University, Norway"}]}],"member":"320","published-online":{"date-parts":[[2022,8,5]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2021. HOST-XAI. https:\/\/host-xai.herokuapp.com.  2021. HOST-XAI. https:\/\/host-xai.herokuapp.com."},{"key":"e_1_3_2_1_2_1","unstructured":"2021. HOST-XAI Tutorial. https:\/\/www.youtube.com\/watch?v=JJ8uc5gReko\/.  2021. HOST-XAI Tutorial. https:\/\/www.youtube.com\/watch?v=JJ8uc5gReko\/."},{"key":"e_1_3_2_1_3_1","unstructured":"2021. Huldra. https:\/\/github.com\/simula\/huldra.  2021. Huldra. https:\/\/github.com\/simula\/huldra."},{"key":"e_1_3_2_1_4_1","unstructured":"2021. Templates - Microworkers - work & earn or offer a micro job. https:\/\/www.microworkers.com\/.  2021. Templates - Microworkers - work & earn or offer a micro job. https:\/\/www.microworkers.com\/."},{"key":"e_1_3_2_1_5_1","unstructured":"2022. Cloud Computing Services - Google Cloud. https:\/\/cloud.google.com\/.  2022. Cloud Computing Services - Google Cloud. https:\/\/cloud.google.com\/."},{"key":"e_1_3_2_1_6_1","unstructured":"2022. Firebase. https:\/\/firebase.google.com\/.  2022. Firebase. https:\/\/firebase.google.com\/."},{"key":"e_1_3_2_1_7_1","unstructured":"2022. Free online meeting scheduling tool - Doodle. https:\/\/doodle.com\/.  2022. Free online meeting scheduling tool - Doodle. https:\/\/doodle.com\/."},{"key":"e_1_3_2_1_8_1","unstructured":"2022. Google Forms. https:\/\/docs.google.com\/forms.  2022. Google Forms. https:\/\/docs.google.com\/forms."},{"key":"e_1_3_2_1_9_1","unstructured":"2022. HOST-ATS. https:\/\/host-ats.herokuapp.com.  2022. HOST-ATS. https:\/\/host-ats.herokuapp.com."},{"key":"e_1_3_2_1_10_1","unstructured":"2022. React - A JavaScript library for building user interfaces. https:\/\/reactjs.org\/.  2022. React - A JavaScript library for building user interfaces. https:\/\/reactjs.org\/."},{"key":"e_1_3_2_1_11_1","unstructured":"2022. SurveyMonkey - Free online survey software and questionnaire tool. https:\/\/www.surveymonkey.com\/.  2022. SurveyMonkey - Free online survey software and questionnaire tool. https:\/\/www.surveymonkey.com\/."},{"key":"e_1_3_2_1_12_1","unstructured":"Amazon. 2021. Amazon Mechanical Turk. https:\/\/www.mturk.com  Amazon. 2021. Amazon Mechanical Turk. https:\/\/www.mturk.com"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-73374-6_7"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2015.160"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2204.00617"},{"key":"e_1_3_2_1_16_1","unstructured":"Tobias Ho\u00dffeld Matthias Hirth Judith Redi Filippo Mazza Pavel Korshunov Babak Naderi Michael Seufert Bruno Gardlo Sebastian Egger and Christian Keimel. 2014. Best Practices and Recommendations for Crowdsourced QoE-Lessons learned from the Qualinet Task Force\" Crowdsourcing\". (2014).  Tobias Ho\u00dffeld Matthias Hirth Judith Redi Filippo Mazza Pavel Korshunov Babak Naderi Michael Seufert Bruno Gardlo Sebastian Egger and Christian Keimel. 2014. Best Practices and Recommendations for Crowdsourced QoE-Lessons learned from the Qualinet Task Force\" Crowdsourcing\". (2014)."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3524273.3532908"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3524273.3528182"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ETFA.1993.396416"},{"volume-title":"Subjective evaluation of speech quality with a crowdsourcing approach","author":"ITU-T","key":"e_1_3_2_1_20_1","unstructured":"ITU-T Recommendation P.808. 2018. Subjective evaluation of speech quality with a crowdsourcing approach . International Telecommunication Union , Geneva . ITU-T Recommendation P.808. 2018. Subjective evaluation of speech quality with a crowdsourcing approach. International Telecommunication Union, Geneva."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/PCS.2012.6213338"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbv021"},{"key":"e_1_3_2_1_23_1","unstructured":"Sebastian Kraft and Udo Z\u00f6lzer. 2014. BeaqleJS: HTML5 and JavaScript based Framework for the Subjective Evaluation of Audio Quality.  Sebastian Kraft and Udo Z\u00f6lzer. 2014. BeaqleJS: HTML5 and JavaScript based Framework for the Subjective Evaluation of Audio Quality."},{"key":"e_1_3_2_1_24_1","unstructured":"Cise Midoglu Steven A. Hicks Vajira Thambawita Tomas Kupka and P\u00e5l Halvorsen. 2022. MMSys'22 Grand Challenge on AI-based Video Production for Soccer. arXiv:2202.01031 [cs.CV]  Cise Midoglu Steven A. Hicks Vajira Thambawita Tomas Kupka and P\u00e5l Halvorsen. 2022. MMSys'22 Grand Challenge on AI-based Video Production for Soccer. arXiv:2202.01031 [cs.CV]"},{"key":"e_1_3_2_1_25_1","volume-title":"The evaluation of artificial intelligence systems in medicine. Computer methods and programs in biomedicine 22, 1","author":"Miller Perry L","year":"1986","unstructured":"Perry L Miller . 1986. The evaluation of artificial intelligence systems in medicine. Computer methods and programs in biomedicine 22, 1 ( 1986 ), 3--11. Perry L Miller. 1986. The evaluation of artificial intelligence systems in medicine. Computer methods and programs in biomedicine 22, 1 (1986), 3--11."},{"key":"e_1_3_2_1_26_1","volume-title":"CROWDMOS: An Approach for Crowdsourcing Mean Opinion Score Studies. In ICASSP (icassp ed.)","author":"Ribeiro Flavio Protasio","year":"2011","unstructured":"Flavio Protasio Ribeiro , Dinei Florencio , Cha Zhang , and Mike Seltzer . 2011 . CROWDMOS: An Approach for Crowdsourcing Mean Opinion Score Studies. In ICASSP (icassp ed.) . IEEE. https:\/\/www.microsoft.com\/en-us\/research\/publication\/crowdmos-an-approach-for-crowdsourcing-mean-opinion-score-studies\/ Flavio Protasio Ribeiro, Dinei Florencio, Cha Zhang, and Mike Seltzer. 2011. CROWDMOS: An Approach for Crowdsourcing Mean Opinion Score Studies. In ICASSP (icassp ed.). IEEE. https:\/\/www.microsoft.com\/en-us\/research\/publication\/crowdmos-an-approach-for-crowdsourcing-mean-opinion-score-studies\/"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CBMI.2016.7500256"},{"volume-title":"QoMEX 2016: 8th International Conference on Quality of Multimedia Experience.","author":"Saupe Dietmar","key":"e_1_3_2_1_28_1","unstructured":"Dietmar Saupe , Franz Hahn , Vlad Hosu , Igor Zingman , Masud Rana , and Shujun Li. 2016. Crowd workers proven useful: A comparative study of subjective video quality assessment . In QoMEX 2016: 8th International Conference on Quality of Multimedia Experience. Dietmar Saupe, Franz Hahn, Vlad Hosu, Igor Zingman, Masud Rana, and Shujun Li. 2016. Crowd workers proven useful: A comparative study of subjective video quality assessment. In QoMEX 2016: 8th International Conference on Quality of Multimedia Experience."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/QoMEX48832.2020.9123122"},{"volume-title":"And How ?. In Knowledge Based Systems in Medicine: Methods, Applications and Evaluation, Jan L","author":"Wyatt Jeremy","key":"e_1_3_2_1_30_1","unstructured":"Jeremy Wyatt and David Spiegelhalter . 1991. Evaluating Medical Expert Systems: What To Test , And How ?. In Knowledge Based Systems in Medicine: Methods, Applications and Evaluation, Jan L . Talmon and John Fox (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg , 274--290. Jeremy Wyatt and David Spiegelhalter. 1991. Evaluating Medical Expert Systems: What To Test, And How ?. In Knowledge Based Systems in Medicine: Methods, Applications and Evaluation, Jan L. Talmon and John Fox (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 274--290."}],"event":{"name":"MMSys '22: 13th ACM Multimedia Systems Conference","sponsor":["SIGMM ACM Special Interest Group on Multimedia","SIGCOMM ACM Special Interest Group on Data Communication","SIGMOBILE ACM Special Interest Group on Mobility of Systems, Users, Data and Computing"],"location":"Athlone Ireland","acronym":"MMSys '22"},"container-title":["Proceedings of the 13th ACM Multimedia Systems Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3524273.3532887","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3524273.3532887","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:06Z","timestamp":1750188666000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3524273.3532887"}},"subtitle":["a framework for collecting crowdsourced feedback on multimedia assets"],"short-title":[],"issued":{"date-parts":[[2022,6,14]]},"references-count":30,"alternative-id":["10.1145\/3524273.3532887","10.1145\/3524273"],"URL":"https:\/\/doi.org\/10.1145\/3524273.3532887","relation":{},"subject":[],"published":{"date-parts":[[2022,6,14]]},"assertion":[{"value":"2022-08-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}