{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:19:26Z","timestamp":1776115166757,"version":"3.50.1"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>This paper describes the design and development of a preliminary qualitative coding tool as well as a method to improve the process of achieving inter-coder reliability (ICR) in small teams. Software applications that support qualitative coding do not sufficiently assist collaboration among coders and overlook some fundamental issues related to ICR. We propose a new dimension of collaborative coding called 'coders' certainty\" and demonstrate its ability to illustrate valuable code disagreements that are missing from existing approaches. Through a case study, we describe the utility of our tool, Code Wizard, and how it helped a group of researchers effectively collaborate to code naturalistic observation data. We report the valuable lessons we learned from the development of our tool and method: (1) identifying coders' certainty constitutes an important part of determining the quality of data analysis and facilitates identifying overlapping and ambiguous codes, (2) making the details of coding process visible helps streamline the coding process and leads to a sense of ownership of the research results, and (3) there is valuable information hidden in coding disagreements that can be leveraged for improving the process of data analysis.<\/jats:p>","DOI":"10.1145\/3274401","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Ease on Down the Code"],"prefix":"10.1145","volume":"2","author":[{"given":"Abbas","family":"Ganji","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Mania","family":"Orand","sequence":"additional","affiliation":[{"name":"Institute for Research in Fundamental Sciences (IPM) &amp; University of Washington, Seattle, WA, USA"}]},{"given":"David W.","family":"McDonald","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312458.2011.568376"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1369-7625.2012.00810.x"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1097\/01.NNR.0000313482.33917.7d"},{"key":"e_1_2_1_4_1","volume-title":"Qualitative content analysis in practice","author":"Schreier M.","unstructured":"M. Schreier. 2012. Qualitative content analysis in practice. Thousand Oaks, CA: Sage."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1177\/2158244014522633"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.320.7227.114"},{"issue":"3","key":"e_1_2_1_7_1","first-page":"411","article-title":"Reliability in Content Analysis","volume":"30","author":"Krippendorff Klaus","year":"2004","unstructured":"Klaus Krippendorff. 2004. Reliability in Content Analysis. Human Communication Research 30 (3): 411--33.","journal-title":"Human Communication Research"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/1468794115577012"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X04266540"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1027\/1015-5759.22.1.28"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-2958.2002.tb00826.x"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1080\/16066350500102237"},{"key":"e_1_2_1_13_1","volume-title":"Accessed April 19th","author":"Nielsen Peter","year":"2012","unstructured":"Peter Nielsen. 2012. Collaborative Coding of Qualitative Data (White Paper), www.researchgate.net\/publication\/261633642_Collaborative_Coding_of_Qualitative_Data_White_paper, Accessed April 19th, 2018."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.322.7294.1115"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_17_1","unstructured":"https:\/\/cat.texifter.com\/ Accessed April 19th 2018"},{"key":"e_1_2_1_18_1","unstructured":"https:\/\/atlasti.com\/ Accessed April 19th 2018"},{"key":"e_1_2_1_19_1","volume-title":"a study of coder training,\" Transl Behav Med","author":"Wood Caroline E","unstructured":"Caroline E Wood, Michelle Richardson, Marie Johnston, Charles Abraham, Jill Francis, Wendy Hardeman, Susan Michie, DPhil,, \"Applying the behaviour change technique (BCT) taxonomy v1: a study of coder training,\" Transl Behav Med, vol. 5, no. 2, pp. 134--148, Jun. 2015."},{"key":"e_1_2_1_20_1","unstructured":"http:\/\/www.qsrinternational.com\/nvivo\/nvivo-products Accessed April 19th 2018"},{"key":"e_1_2_1_21_1","unstructured":"http:\/\/www.dedoose.com\/ Accessed April 19th 2018"},{"key":"e_1_2_1_22_1","volume-title":"IEEE TVCG","author":"Blascheck T.","year":"2017","unstructured":"T. Blascheck, F. Beck, S. Baltes, T. Ertl, and D. Weiskopf. 2017. Visual analysis and coding of data-rich user behavior. IEEE TVCG, 2017."},{"key":"e_1_2_1_23_1","unstructured":"https:\/\/www.quirkos.com\/index.html Accessed April 19th 2018"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/PACIFICVIS.2017.8031598"},{"key":"e_1_2_1_25_1","unstructured":"https:\/\/www.fema.gov\/media-library-data\/1484078710188--2e6b753f3f9c6037dd22922cde32e3dd\/CR16_AAR_508.pdf Accessed April 19th 2018"},{"key":"e_1_2_1_26_1","unstructured":"https:\/\/www.fema.gov\/media-library-data\/1462203815175--6b989e683e8c2d34864f007fbde2c3fd\/CR2016Flyer.pdf Accessed April 19th 2018"},{"key":"e_1_2_1_27_1","unstructured":"https:\/\/www.fema.gov\/news-release\/2017\/06\/07\/emergency-managers-announce-improvements-after-cascadia-rising-exercise Accessed April 19th 2018"},{"key":"e_1_2_1_28_1","unstructured":"https:\/\/slack.com\/ Accessed April 19th 2018"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2893365"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647367"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_32_1","volume-title":"CHI","author":"Zade Himanshu","year":"2018","unstructured":"Himanshu Zade, Margaret Drouhard, Bonnie Chinh, Lu Gan, and Cecilia Aragon. n.d. 2018. Conceptualizing Disagreement in Qualitative Coding, 11. CHI 2018."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010690908200"},{"issue":"2","key":"e_1_2_1_34_1","first-page":"2009","article-title":"Making visible the coding process: Using qualitative data software in a post-structural study","volume":"19","author":"Ryan Mary","year":"2009","unstructured":"Mary Ryan. 2009. Making visible the coding process: Using qualitative data software in a post-structural study. Issues in Educational Research, Vol 19(2), 2009","journal-title":"Issues in Educational Research"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X02014002005"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274401","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274401","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:35Z","timestamp":1750207475000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274401"}},"subtitle":["Complex Collaborative Qualitative Coding Simplified with 'Code Wizard'"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":35,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274401"],"URL":"https:\/\/doi.org\/10.1145\/3274401","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}