{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T19:46:46Z","timestamp":1776109606375,"version":"3.50.1"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2019,11,7]],"date-time":"2019-11-07T00:00:00Z","timestamp":1573084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000024","name":"Canadian Institutes of Health Research","doi-asserted-by":"publisher","award":["CPG-140200"],"award-info":[{"award-number":["CPG-140200"]}],"id":[{"id":"10.13039\/501100000024","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002790","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["CHRP 478468-15"],"award-info":[{"award-number":["CHRP 478468-15"]}],"id":[{"id":"10.13039\/501100002790","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2019,11,7]]},"abstract":"<jats:p>Expert disagreement is pervasive in clinical decision making and collective adjudication is a useful approach for resolving divergent assessments. Prior work shows that expert disagreement can arise due to diverse factors including expert background, the quality and presentation of data, and guideline clarity. In this work, we study how these factors predict initial discrepancies in the context of medical time series analysis, examining why certain disagreements persist after adjudication, and how adjudication impacts clinical decisions. Results from a case study with 36 experts and 4,543 adjudicated cases in a sleep stage classification task show that these factors contribute to both initial disagreement and resolvability, each in their own unique way. We provide evidence suggesting that structured adjudication can lead to significant revisions in treatment-relevant clinical parameters. Our work demonstrates how structured adjudication can support consensus and facilitate a deep understanding of expert disagreement in medical data analysis.<\/jats:p>","DOI":"10.1145\/3359178","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T13:40:21Z","timestamp":1573220421000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":42,"title":["Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication"],"prefix":"10.1145","volume":"3","author":[{"given":"Mike","family":"Schaekermann","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Graeme","family":"Beaton","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Minahz","family":"Habib","sequence":"additional","affiliation":[{"name":"University of Toronto, Toronto, ON, Canada"}]},{"given":"Andrew","family":"Lim","sequence":"additional","affiliation":[{"name":"Sunnybrook Health Sciences Centre, Toronto, ON, Canada"}]},{"given":"Kate","family":"Larson","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Edith","family":"Law","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]}],"member":"320","published-online":{"date-parts":[[2019,11,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531653"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.15346\/hc.v1i1.3"},{"key":"e_1_2_1_3_1","volume-title":"Halford","author":"Bagheri Elham","year":"2017","unstructured":"Elham Bagheri , Justin Dauwels , Brian C. Dean , Chad G. Waters , M. Brandon Westover , and Jonathan J . Halford . 2017 . Interictal epileptiform discharge characteristics underlying expert interrater agreement. Clinical Neurophysiology , Vol. 128 , 10 (10 2017), 1994--2005. https:\/\/doi.org\/10.1016\/j.clinph.2017.06.252 Elham Bagheri, Justin Dauwels, Brian C. Dean, Chad G. Waters, M. Brandon Westover, and Jonathan J. Halford. 2017. Interictal epileptiform discharge characteristics underlying expert interrater agreement. Clinical Neurophysiology, Vol. 128, 10 (10 2017), 1994--2005. https:\/\/doi.org\/10.1016\/j.clinph.2017.06.252"},{"key":"e_1_2_1_4_1","volume-title":"Clinical Medicine","volume":"10","author":"Baker A.","year":"2010","unstructured":"A. Baker , K. Young , J. Potter , and I. Madan . 2010. A review of grading systems for evidence-based guidelines produced by medical specialties . Clinical Medicine , Vol. 10 , 4 (8 2010 ), 358--363. https:\/\/doi.org\/10.7861\/clinmedicine.10--4--358 A. Baker, K. Young, J. Potter, and I. Madan. 2010. A review of grading systems for evidence-based guidelines produced by medical specialties. Clinical Medicine, Vol. 10, 4 (8 2010), 358--363. https:\/\/doi.org\/10.7861\/clinmedicine.10--4--358"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Erin P. Balogh Bryan T. Miller and John R. Ball (Eds.). 2015. Improving Diagnosis in Health Care. National Academies Press Washington D.C. https:\/\/doi.org\/10.17226\/21794  Erin P. Balogh Bryan T. Miller and John R. Ball (Eds.). 2015. Improving Diagnosis in Health Care. National Academies Press Washington D.C. https:\/\/doi.org\/10.17226\/21794","DOI":"10.17226\/21794"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1155\/2011\/406391"},{"key":"e_1_2_1_7_1","volume-title":"Bates","author":"Barnett Michael L.","year":"2019","unstructured":"Michael L. Barnett , Dhruv Boddupalli , Shantanu Nundy , and David W . Bates . 2019 . Comparative Accuracy of Diagnosis by Collective Intelligence of Multiple Physicians vs Individual Physicians. JAMA Network Open , Vol. 2 , 3 (3 2019), e190096. https:\/\/doi.org\/10.1001\/jamanetworkopen.2019.0096 Michael L. Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W. Bates. 2019. Comparative Accuracy of Diagnosis by Collective Intelligence of Multiple Physicians vs Individual Physicians. JAMA Network Open, Vol. 2, 3 (3 2019), e190096. https:\/\/doi.org\/10.1001\/jamanetworkopen.2019.0096"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:ARTI.0000046007.11806.9a"},{"key":"e_1_2_1_9_1","volume-title":"Computational Models of Argument - Proceedings of COMMA 2014","author":"Budzynska Katarzyna","year":"2014","unstructured":"Katarzyna Budzynska , Mathilde Janier , Juyeon Kang , Chris Reed , Patrick Saint-Dizier , Manfred Stede , and Olena Yaskorska . 2014 . Towards Argument Mining from Dialogue . In Computational Models of Argument - Proceedings of COMMA 2014 , Atholl Palace Hotel, Scottish Highlands, UK, September 9--12 , 2014. 185--196. https:\/\/doi.org\/10.3233\/978--1--61499--436--7--185 Katarzyna Budzynska, Mathilde Janier, Juyeon Kang, Chris Reed, Patrick Saint-Dizier, Manfred Stede, and Olena Yaskorska. 2014. Towards Argument Mining from Dialogue. In Computational Models of Argument - Proceedings of COMMA 2014, Atholl Palace Hotel, Scottish Highlands, UK, September 9--12, 2014. 185--196. https:\/\/doi.org\/10.3233\/978--1--61499--436--7--185"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-1601"},{"key":"e_1_2_1_12_1","volume-title":"Cicero. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press","author":"Chen Quanze","unstructured":"Quanze Chen , Jonathan Bragg , Lydia B. Chilton , and Daniel S. Weld . 2019 . Cicero. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press , New York, New York, USA, 1--14. https:\/\/doi.org\/10.1145\/3290605.3300761 Quanze Chen, Jonathan Bragg, Lydia B. Chilton, and Daniel S. Weld. 2019. Cicero. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press, New York, New York, USA, 1--14. https:\/\/doi.org\/10.1145\/3290605.3300761"},{"key":"e_1_2_1_13_1","article-title":"Towards an argument interchange format","volume":"21","author":"Carlos Ches","year":"2006","unstructured":"Carlos Ches n evar, Jarred McGinnis , Sanjay Modgil , Iyad Rahwan , Chris Reed , Guillermo Simari , Matthew South , Gerard Vreeswijk , and Steven Willmott . 2006 . Towards an argument interchange format . The Knowledge Engineering Review , Vol. 21 , 04 (12 2006), 293. https:\/\/doi.org\/10.1017\/S0269888906001044 Carlos Ches n evar, Jarred McGinnis, Sanjay Modgil, Iyad Rahwan, Chris Reed, Guillermo Simari, Matthew South, Gerard Vreeswijk, and Steven Willmott. 2006. Towards an argument interchange format. The Knowledge Engineering Review, Vol. 21, 04 (12 2006), 293. https:\/\/doi.org\/10.1017\/S0269888906001044","journal-title":"The Knowledge Engineering Review"},{"key":"e_1_2_1_14_1","first-page":"1","article-title":"Analyzing the Structure of Argumentative","volume":"13","author":"Cohen Robin","year":"1987","unstructured":"Robin Cohen . 1987 . Analyzing the Structure of Argumentative Discourse. Comput. Linguist. , Vol. 13 , 1 -- 2 (1 1987), 11--24. http:\/\/dl.acm.org\/citation.cfm?id=26386.26388 Robin Cohen. 1987. Analyzing the Structure of Argumentative Discourse. Comput. Linguist., Vol. 13, 1--2 (1 1987), 11--24. http:\/\/dl.acm.org\/citation.cfm?id=26386.26388","journal-title":"Discourse. Comput. Linguist."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/3306127.3331890"},{"key":"e_1_2_1_16_1","volume-title":"Management Science","volume":"9","author":"Dalkey Norman","year":"1963","unstructured":"Norman Dalkey and Olaf Helmer . 1963 . An Experimental Application of the DELPHI Method to the Use of Experts . Management Science , Vol. 9 , 3 (4 1963), 458--467. https:\/\/doi.org\/10.1287\/mnsc.9.3.458 Norman Dalkey and Olaf Helmer. 1963. An Experimental Application of the DELPHI Method to the Use of Experts. Management Science, Vol. 9, 3 (4 1963), 458--467. https:\/\/doi.org\/10.1287\/mnsc.9.3.458"},{"key":"e_1_2_1_17_1","volume-title":"Online deliberation design. Democracy in motion: Evaluation the practice and impact of deliberative civic engagement","author":"Davies Todd","year":"2012","unstructured":"Todd Davies and Reid Chandler . 2012. Online deliberation design. Democracy in motion: Evaluation the practice and impact of deliberative civic engagement ( 2012 ), 103--131. Todd Davies and Reid Chandler. 2012. Online deliberation design. Democracy in motion: Evaluation the practice and impact of deliberative civic engagement (2012), 103--131."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP) .","author":"Drapeau Ryan","unstructured":"Ryan Drapeau , Lydia B. Chilton , Jonathan Bragg , and Daniel S. Weld . 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy . In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP) . Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP) ."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3152889"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/2584544"},{"key":"e_1_2_1_21_1","volume-title":"Patrick MM Bossuyt, and Mariska MG Leeflang.","author":"Gopalakrishna Gowri","year":"2013","unstructured":"Gowri Gopalakrishna , Miranda W Langendam , Rob JPM Scholten , Patrick MM Bossuyt, and Mariska MG Leeflang. 2013 . Guidelines for guideline developers: a systematic review of grading systems for medical tests. Implementation Science , Vol. 8 , 1 (12 2013), 78. https:\/\/doi.org\/10.1186\/1748--5908--8--78 Gowri Gopalakrishna, Miranda W Langendam, Rob JPM Scholten, Patrick MM Bossuyt, and Mariska MG Leeflang. 2013. Guidelines for guideline developers: a systematic review of grading systems for medical tests. Implementation Science, Vol. 8, 1 (12 2013), 78. https:\/\/doi.org\/10.1186\/1748--5908--8--78"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820071"},{"key":"e_1_2_1_23_1","volume-title":"AAAI Conference on Artificial Intelligence. https:\/\/arxiv.org\/pdf\/1703","author":"Guan Melody","year":"2018","unstructured":"Melody Guan , Varun Gulshan , Andrew Dai , and Geoffrey Hinton . 2018 . Who said what: Modeling individual labelers improves classification . In AAAI Conference on Artificial Intelligence. https:\/\/arxiv.org\/pdf\/1703 .08774.pdf Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. 2018. Who said what: Modeling individual labelers improves classification. In AAAI Conference on Artificial Intelligence. https:\/\/arxiv.org\/pdf\/1703.08774.pdf"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025781"},{"key":"e_1_2_1_25_1","volume-title":"Stan Sclaroff, Margrit Betke, and Kristen Grauman.","author":"Gurari Danna","year":"2017","unstructured":"Danna Gurari , Kun He , Bo Xiong , Jianming Zhang , Mehrnoosh Sameki , Suyog Dutt Jain , Stan Sclaroff, Margrit Betke, and Kristen Grauman. 2017 . Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation( s). (4 2017). http:\/\/arxiv.org\/abs\/1705.00366 Danna Gurari, Kun He, Bo Xiong, Jianming Zhang, Mehrnoosh Sameki, Suyog Dutt Jain, Stan Sclaroff, Margrit Betke, and Kristen Grauman. 2017. Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation(s). (4 2017). http:\/\/arxiv.org\/abs\/1705.00366"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1061\/(ASCE)0887-3801(1995)9:4(244)"},{"key":"e_1_2_1_27_1","volume-title":"Andrew L Cheeson Jr., and Stuart F Quan","author":"Iber Conrad","year":"2007","unstructured":"Conrad Iber , Sonia Ancoli-Israel , Andrew L Cheeson Jr., and Stuart F Quan . 2007 . The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine . Conrad Iber, Sonia Ancoli-Israel, Andrew L Cheeson Jr., and Stuart F Quan. 2007. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_29_1","volume-title":"Webster","author":"Krause Jonathan","year":"2018","unstructured":"Jonathan Krause , Varun Gulshan , Ehsan Rahimy , Peter Karth , Kasumi Widner , Greg S. Corrado , Lily Peng , and Dale R . Webster . 2018 . Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology ( 3 2018). https:\/\/doi.org\/10.1016\/j.ophtha.2018.01.034 Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S. Corrado, Lily Peng, and Dale R. Webster. 2018. Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology (3 2018). https:\/\/doi.org\/10.1016\/j.ophtha.2018.01.034"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W15-0516"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 6th International Conference on Computational Models of Argument (COMMA 2016","volume":"0","author":"Lawrence John","year":"2016","unstructured":"John Lawrence and Chris Reed . 2016 . Argument Mining using Argumentation Scheme Structures . Proceedings of the 6th International Conference on Computational Models of Argument (COMMA 2016 ), Vol. 0 (2016), 379 -- 390. https:\/\/doi.org\/10.3233\/978--1--61499--686--6--379 John Lawrence and Chris Reed. 2016. Argument Mining using Argumentation Scheme Structures. Proceedings of the 6th International Conference on Computational Models of Argument (COMMA 2016), Vol. 0 (2016), 379 -- 390. https:\/\/doi.org\/10.3233\/978--1--61499--686--6--379"},{"key":"e_1_2_1_32_1","volume-title":"WingIt: Efficient Refinement of Unclear Task Instructions. In The Sixth AAAI Conference on Human Computation and Crowdsourcing. 108--116","author":"Chaithanya Manam V K","year":"2018","unstructured":"V K Chaithanya Manam and Alexander J Quinn . 2018 . WingIt: Efficient Refinement of Unclear Task Instructions. In The Sixth AAAI Conference on Human Computation and Crowdsourcing. 108--116 . https:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP18\/paper\/view\/17931 V K Chaithanya Manam and Alexander J Quinn. 2018. WingIt: Efficient Refinement of Unclear Task Instructions. In The Sixth AAAI Conference on Human Computation and Crowdsourcing. 108--116. https:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP18\/paper\/view\/17931"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/135467896394500"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1002\/acp.3213"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS '13)","author":"Parsons Simon","year":"2013","unstructured":"Simon Parsons , Elizabeth Sklar , Jordan Salvit , Holly Wall , and Zimi Li . 2013 . ArgTrust: Decision Making with Information from Sources of Varying Trustworthiness . In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS '13) . International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1395--1396. http:\/\/dl.acm.org\/citation.cfm?id=2484920.2485242 Simon Parsons, Elizabeth Sklar, Jordan Salvit, Holly Wall, and Zimi Li. 2013. ArgTrust: Decision Making with Information from Sources of Varying Trustworthiness. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS '13). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1395--1396. http:\/\/dl.acm.org\/citation.cfm?id=2484920.2485242"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.0000000000004373"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.5664\/jcsm.2352"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1093\/brain\/awz030"},{"key":"e_1_2_1_39_1","volume-title":"2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 179--188","author":"Stefan","year":"2018","unstructured":"Stefan R\"abiger, Gizem Gezici , Y\u00fccel Saygin , and Myra Spiliopoulou . 2018 . Predicting worker disagreement for more effective crowd labeling . In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 179--188 . Stefan R\"abiger, Gizem Gezici, Y\u00fccel Saygin, and Myra Spiliopoulou. 2018. Predicting worker disagreement for more effective crowd labeling. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 179--188."},{"key":"e_1_2_1_40_1","volume-title":"Direct Uncertainty Prediction for Medical Second Opinions. (7","author":"Raghu Maithra","year":"2018","unstructured":"Maithra Raghu , Katy Blumer , Rory Sayres , Ziad Obermeyer , Robert Kleinberg , Sendhil Mullainathan , and Jon Kleinberg . 2018. Direct Uncertainty Prediction for Medical Second Opinions. (7 2018 ). http:\/\/arxiv.org\/abs\/1807.01771 Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Robert Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. 2018. Direct Uncertainty Prediction for Medical Second Opinions. (7 2018). http:\/\/arxiv.org\/abs\/1807.01771"},{"key":"e_1_2_1_41_1","volume-title":"Ng","author":"Rajpurkar Pranav","year":"2017","unstructured":"Pranav Rajpurkar , Awni Y. Hannun , Masoumeh Haghpanahi , Codie Bourn , and Andrew Y . Ng . 2017 . Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks . (7 2017). http:\/\/arxiv.org\/abs\/1707.01836 Pranav Rajpurkar, Awni Y. Hannun, Masoumeh Haghpanahi, Codie Bourn, and Andrew Y. Ng. 2017. Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks. (7 2017). http:\/\/arxiv.org\/abs\/1707.01836"},{"key":"e_1_2_1_42_1","volume-title":"Argumentation Library","volume":"9","author":"Reed Chris","year":"2004","unstructured":"Chris Reed and Timothy Norman . 2004 . Argumentation Machines . Argumentation Library , Vol. 9 . Springer Netherlands, Dordrecht. https:\/\/doi.org\/10.1007\/978--94-017-0431--1 Chris Reed and Timothy Norman. 2004. Argumentation Machines. Argumentation Library, Vol. 9. Springer Netherlands, Dordrecht. https:\/\/doi.org\/10.1007\/978--94-017-0431--1"},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Chris Reed and Doug Walton. 2005. Towards a Formal and Implemented Model of Argumentation Schemes in Agent Communication. 19--30. https:\/\/doi.org\/10.1007\/978--3--540--32261-0_2  Chris Reed and Doug Walton. 2005. Towards a Formal and Implemented Model of Argumentation Schemes in Agent Communication. 19--30. https:\/\/doi.org\/10.1007\/978--3--540--32261-0_2","DOI":"10.1007\/978-3-540-32261-0_2"},{"key":"e_1_2_1_44_1","volume-title":"Rosenberg and Steven van Hout","author":"Richard","year":"2013","unstructured":"Richard S. Rosenberg and Steven van Hout . 2013 . The American Academy of Sleep Medicine Inter-scorer Reliability Program : Sleep Stage Scoring. Journal of Clinical Sleep Medicine (1 2013). https:\/\/doi.org\/10.5664\/jcsm.2350 Richard S. Rosenberg and Steven van Hout. 2013. The American Academy of Sleep Medicine Inter-scorer Reliability Program: Sleep Stage Scoring. Journal of Clinical Sleep Medicine (1 2013). https:\/\/doi.org\/10.5664\/jcsm.2350"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317085"},{"key":"e_1_2_1_46_1","volume-title":"8th Workshop on Interactive Systems in Healthcare (WISH) at CHI","author":"Schaekermann Mike","year":"2019","unstructured":"Mike Schaekermann , Graeme Beaton , Minahz Habib , Andrew Lim , Kate Larson , and Edith Law . 2019 . crowdEEG: A Platform for Structured Consensus Formation in Medical Time Series Analysis . In 8th Workshop on Interactive Systems in Healthcare (WISH) at CHI 2019. Glasgow, UK. Mike Schaekermann, Graeme Beaton, Minahz Habib, Andrew Lim, Kate Larson, and Edith Law. 2019. crowdEEG: A Platform for Structured Consensus Formation in Medical Time Series Analysis. In 8th Workshop on Interactive Systems in Healthcare (WISH) at CHI 2019. Glasgow, UK."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_2_1_48_1","volume-title":"Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP","author":"Schaekermann Mike","year":"2018","unstructured":"Mike Schaekermann , Edith Law , Kate Larson , and Andrew Lim . 2018 b. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland. Mike Schaekermann, Edith Law, Kate Larson, and Andrew Lim. 2018b. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland."},{"key":"e_1_2_1_49_1","volume-title":"Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI","author":"Schaekermann Mike","year":"2016","unstructured":"Mike Schaekermann , Edith Law , Alex C Williams , and William Callaghan . 2016 . Resolvable vs . Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI 2016. San Jose, CA. Mike Schaekermann, Edith Law, Alex C Williams, and William Callaghan. 2016. Resolvable vs. Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI 2016. San Jose, CA."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2041-6962.2006.tb00028.x"},{"key":"e_1_2_1_51_1","volume-title":"Establishing medical reality","author":"Solomon Miriam","unstructured":"Miriam Solomon . 2007. The social epistemology of NIH consensus conferences . In Establishing medical reality . Springer , 167--177. Miriam Solomon. 2007. The social epistemology of NIH consensus conferences. In Establishing medical reality. Springer, 167--177."},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"D Walton C Reed and F Macagno. 2008. Argumentation Schemes. Cambridge University Press. https:\/\/books.google.ca\/books?id=qc3LCgAAQBAJ  D Walton C Reed and F Macagno. 2008. Argumentation Schemes. Cambridge University Press. https:\/\/books.google.ca\/books?id=qc3LCgAAQBAJ","DOI":"10.1017\/CBO9780511802034"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359178","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359178","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:53Z","timestamp":1750201973000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359178"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,7]]},"references-count":52,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2019,11,7]]}},"alternative-id":["10.1145\/3359178"],"URL":"https:\/\/doi.org\/10.1145\/3359178","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,7]]},"assertion":[{"value":"2019-11-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}