{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T19:08:58Z","timestamp":1776366538135,"version":"3.51.2"},"publisher-location":"New York, NY, USA","reference-count":25,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,13]]},"DOI":"10.1145\/3308560.3317084","type":"proceedings-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T12:17:59Z","timestamp":1557749879000},"page":"1138-1143","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Intra- and Inter-rater Agreement in a Subjective Speech Quality Assessment Task in Crowdsourcing"],"prefix":"10.1145","author":[{"given":"Rafael","family":"Zequeira Jim\u00e9nez","sequence":"first","affiliation":[{"name":"TU Berlin"}]},{"given":"Anna","family":"Llagostera","sequence":"additional","affiliation":[{"name":"Rohde & Schwarz SwissQual AG,"}]},{"given":"Babak","family":"Naderi","sequence":"additional","affiliation":[{"name":"TU Berlin,"}]},{"given":"Sebastian","family":"M\u00f6ller","sequence":"additional","affiliation":[{"name":"TU Berlin,"}]},{"given":"Jens","family":"Berger","sequence":"additional","affiliation":[{"name":"Rohde & Schwarz SwissQual AG,"}]}],"member":"320","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"d.}. Mandatory speech CODEC speech processing functions","author":"GPP","unstructured":"3 GPP T\u00a0S 26.070. {n. d.}. Mandatory speech CODEC speech processing functions ; AMR speech Codec; General de scription. 3GPP T\u00a0S 26.070. {n. d.}. Mandatory speech CODEC speech processing functions; AMR speech Codec; General description."},{"key":"e_1_3_2_1_2_1","volume-title":"d.}. Speech codec speech processing functions","author":"GPP","unstructured":"3 GPP T\u00a0S 26.171. {n. d.}. Speech codec speech processing functions ; Adaptive Multi-Rate - Wideband (AMR-WB) speech codec; General de scription. 3GPP T\u00a0S 26.171. {n. d.}. Speech codec speech processing functions; Adaptive Multi-Rate - Wideband (AMR-WB) speech codec; General description."},{"key":"e_1_3_2_1_3_1","volume-title":"d.}. Codec for Enhanced Voice Services (EVS)","author":"GPP","unstructured":"3 GPP T\u00a0S 26.441. {n. d.}. Codec for Enhanced Voice Services (EVS) ; General overview. 3GPP T\u00a0S 26.441. {n. d.}. Codec for Enhanced Voice Services (EVS); General overview."},{"key":"e_1_3_2_1_4_1","volume-title":"A subjective ACR LOT testing super-wideband speech coding in real field measurements and prediction by P.863. ITU-T Contribution SG12-C.286","author":"Berger Jens","unstructured":"Jens Berger and Anna Llagostera . 2018. A subjective ACR LOT testing super-wideband speech coding in real field measurements and prediction by P.863. ITU-T Contribution SG12-C.286 . International Telecommunication Union , CH- Geneva . 1\u201311 pages. Jens Berger and Anna Llagostera. 2018. A subjective ACR LOT testing super-wideband speech coding in real field measurements and prediction by P.863. ITU-T Contribution SG12-C.286. International Telecommunication Union, CH-Geneva. 1\u201311 pages."},{"key":"e_1_3_2_1_5_1","volume-title":"Applied nonparametric statistics","author":"Daniel W","unstructured":"Wayne\u00a0 W Daniel . 1980. Applied nonparametric statistics ( 2 nd ed.). Boston, MA : Cengage Learning . Wayne\u00a0W Daniel. 1980. Applied nonparametric statistics (2nd ed.). Boston, MA: Cengage Learning.","edition":"2"},{"key":"e_1_3_2_1_6_1","volume-title":"Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology 8, 1","author":"Hallgren A.","year":"2012","unstructured":"Kevin\u00a0 A. Hallgren . 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology 8, 1 ( 2012 ), 23. Kevin\u00a0A. Hallgren. 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology 8, 1 (2012), 23."},{"key":"e_1_3_2_1_7_1","unstructured":"Tobias Ho\u00dffeld Matthias Hirth Judith Redi Filippo Mazza Pavel Korshunov Babak Naderi Michael Seufert Bruno Gardlo Sebastian Egger and Christian Keimel. 2014. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force \u201dCrowdsourcing\u201d. https:\/\/hal.archives-ouvertes.fr\/hal-01078761  Tobias Ho\u00dffeld Matthias Hirth Judith Redi Filippo Mazza Pavel Korshunov Babak Naderi Michael Seufert Bruno Gardlo Sebastian Egger and Christian Keimel. 2014. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force \u201dCrowdsourcing\u201d. https:\/\/hal.archives-ouvertes.fr\/hal-01078761"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/QoMEX.2011.6065690"},{"key":"e_1_3_2_1_9_1","volume-title":"Test signals for use in telephonometry","author":"ITU-T","unstructured":"ITU-T Recommandation P.501. 2017. Test signals for use in telephonometry . International Telecommunication Union , Geneva . ITU-T Recommandation P.501. 2017. Test signals for use in telephonometry. International Telecommunication Union, Geneva."},{"key":"e_1_3_2_1_10_1","volume-title":"Methods for subjective determination of transmission quality","author":"ITU-T","unstructured":"ITU-T Recommandation P.800. 1996. Methods for subjective determination of transmission quality . International Telecommunication Union , Geneva . ITU-T Recommandation P.800. 1996. Methods for subjective determination of transmission quality. International Telecommunication Union, Geneva."},{"key":"e_1_3_2_1_11_1","volume-title":"Mean Opinion Score (MOS) Terminology","author":"ITU-T","unstructured":"ITU-T Recommandation P.800.1. 2016. Mean Opinion Score (MOS) Terminology . International Telecommunication Union , Geneva . ITU-T Recommandation P.800.1. 2016. Mean Opinion Score (MOS) Terminology. International Telecommunication Union, Geneva."},{"key":"e_1_3_2_1_12_1","volume-title":"Subjective evaluation of speech quality with a crowdsourcing approach","author":"ITU-T","unstructured":"ITU-T Recommandation P.808. 2018. Subjective evaluation of speech quality with a crowdsourcing approach . International Telecommunication Union , Geneva . ITU-T Recommandation P.808. 2018. Subjective evaluation of speech quality with a crowdsourcing approach. International Telecommunication Union, Geneva."},{"key":"e_1_3_2_1_13_1","volume-title":"Rank Correlation Methods","author":"Kendall Maurice\u00a0George","unstructured":"Maurice\u00a0George Kendall . 1970. Rank Correlation Methods ( 4 th ed.). Charles Griffin . Maurice\u00a0George Kendall. 1970. Rank Correlation Methods(4th ed.). Charles Griffin.","edition":"4"},{"key":"e_1_3_2_1_14_1","volume-title":"Computing Intraclass Correlations (ICC) as Estimates of Interrater Reliability in SPSS. The Winnower","author":"Landers Richard","year":"2015","unstructured":"Richard Landers . 2015. Computing Intraclass Correlations (ICC) as Estimates of Interrater Reliability in SPSS. The Winnower ( 2015 ). Richard Landers. 2015. Computing Intraclass Correlations (ICC) as Estimates of Interrater Reliability in SPSS. The Winnower (2015)."},{"key":"e_1_3_2_1_15_1","volume-title":"Nonparametric and distribution-free methods for the social sciences","author":"Marascuilo A","unstructured":"Leonard\u00a0 A Marascuilo and Maryellen McSweeney . 1977. Nonparametric and distribution-free methods for the social sciences . Belmont, CA : Wadsworth Publishing Company . Leonard\u00a0A Marascuilo and Maryellen McSweeney. 1977. Nonparametric and distribution-free methods for the social sciences. Belmont, CA: Wadsworth Publishing Company."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.3102\/10769986005003269"},{"key":"e_1_3_2_1_17_1","volume-title":"Evaluation of the Draft of P.CROWD Recommendation. ITU-T Contribution SG12-C.204","author":"Naderi Babak","year":"2018","unstructured":"Babak Naderi , Sebastian M\u00f6ller , and Rafael Zequeira Jim\u00e9nez . 2018. Evaluation of the Draft of P.CROWD Recommendation. ITU-T Contribution SG12-C.204 . International Telecommunication Union , CH- Geneva . 1\u20138 pages. https:\/\/www.qu.tu-berlin.de\/fileadmin\/fg41\/publications\/naderi_ 2018 _evaluation-of-the-draft-of-p.crowd-recommendation.pdf Babak Naderi, Sebastian M\u00f6ller, and Rafael Zequeira Jim\u00e9nez. 2018. Evaluation of the Draft of P.CROWD Recommendation. ITU-T Contribution SG12-C.204. International Telecommunication Union, CH-Geneva. 1\u20138 pages. https:\/\/www.qu.tu-berlin.de\/fileadmin\/fg41\/publications\/naderi_2018_evaluation-of-the-draft-of-p.crowd-recommendation.pdf"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Babak Naderi Tim Polzehl Ina Wechsung Friedemann K\u00f6ster and Sebastian M\u00f6ller. 2015. Effect of Trapping Questions on the Reliability of Speech Quality Judgments in a Crowdsourcing Paradigm. In Interspeech. ISCA 2799\u20132803.  Babak Naderi Tim Polzehl Ina Wechsung Friedemann K\u00f6ster and Sebastian M\u00f6ller. 2015. Effect of Trapping Questions on the Reliability of Speech Quality Judgments in a Crowdsourcing Paradigm. In Interspeech. ISCA 2799\u20132803.","DOI":"10.21437\/Interspeech.2015-589"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Tim Polzehl Babak Naderi Friedemann K\u00f6ster and Sebastian M\u00f6ller. 2015. Robustness in speech quality assessment and temporal training expiry in mobile crowdsourcing environments. In INTERSPEECH. 2794\u20132798.  Tim Polzehl Babak Naderi Friedemann K\u00f6ster and Sebastian M\u00f6ller. 2015. Robustness in speech quality assessment and temporal training expiry in mobile crowdsourcing environments. In INTERSPEECH. 2794\u20132798.","DOI":"10.21437\/Interspeech.2015-588"},{"key":"e_1_3_2_1_20_1","volume-title":"Speech Quality of VoIP: Assessment and Prediction","author":"Raake Alexander","unstructured":"Alexander Raake . 2007. Speech Quality of VoIP: Assessment and Prediction . John Wiley & Sons . Alexander Raake. 2007. Speech Quality of VoIP: Assessment and Prediction. John Wiley & Sons."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2810188.2810194"},{"key":"e_1_3_2_1_22_1","volume-title":"CROWDMOS: An Approach for Crowdsourcing Mean Opinion Score Studies. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2416\u20132419","author":"Ribeiro P","year":"2011","unstructured":"Flavio\u00a0 P Ribeiro , Dinei A\u00a0F Flor\u00eancio , Cha Zhang , and Michael\u00a0 L Seltzer . 2011 . CROWDMOS: An Approach for Crowdsourcing Mean Opinion Score Studies. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2416\u20132419 . Flavio\u00a0P Ribeiro, Dinei A\u00a0F Flor\u00eancio, Cha Zhang, and Michael\u00a0L Seltzer. 2011. CROWDMOS: An Approach for Crowdsourcing Mean Opinion Score Studies. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2416\u20132419."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.86.2.420"},{"key":"e_1_3_2_1_24_1","volume-title":"MANOVA, and HLM. In L. G. Grimm & P","author":"Weinfurt P","unstructured":"Kevin\u00a0 P Weinfurt . {n. d.}. Repeated measures analysis: ANOVA , MANOVA, and HLM. In L. G. Grimm & P . R. Yarnold (Eds.), Reading and understanding MORE multivariate statistics. 317\u2013361 pages. Kevin\u00a0P Weinfurt. {n. d.}. Repeated measures analysis: ANOVA, MANOVA, and HLM. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding MORE multivariate statistics. 317\u2013361 pages."},{"key":"e_1_3_2_1_25_1","volume-title":"Influence of Number of Stimuli for Subjective Speech Quality Assessment in Crowdsourcing. In 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX). 1\u20136.","author":"Jim\u00e9nez Rafael Zequeira","year":"2018","unstructured":"Rafael Zequeira Jim\u00e9nez , Laura Fern\u00e1ndez Gallardo , and Sebastian M\u00f6ller . 2018 . Influence of Number of Stimuli for Subjective Speech Quality Assessment in Crowdsourcing. In 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX). 1\u20136. Rafael Zequeira Jim\u00e9nez, Laura Fern\u00e1ndez Gallardo, and Sebastian M\u00f6ller. 2018. Influence of Number of Stimuli for Subjective Speech Quality Assessment in Crowdsourcing. In 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX). 1\u20136."}],"event":{"name":"WWW '19: The Web Conference","location":"San Francisco USA","acronym":"WWW '19","sponsor":["IW3C2 International World Wide Web Conference Committee"]},"container-title":["Companion Proceedings of The 2019 World Wide Web Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308560.3317084","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3308560.3317084","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:35Z","timestamp":1750204415000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308560.3317084"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":25,"alternative-id":["10.1145\/3308560.3317084","10.1145\/3308560"],"URL":"https:\/\/doi.org\/10.1145\/3308560.3317084","relation":{},"subject":[],"published":{"date-parts":[[2019,5,13]]},"assertion":[{"value":"2019-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}