{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T16:53:05Z","timestamp":1779382385804,"version":"3.53.1"},"reference-count":112,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>While artificial intelligence (AI) holds promise for supporting healthcare providers and improving the accuracy of medical diagnoses, a lack of transparency in the composition of datasets exposes AI models to the possibility of unintentional and avoidable mistakes. In particular, public and private image datasets of dermatological conditions rarely include information on skin color. As a start towards increasing transparency, AI researchers have appropriated the use of the Fitzpatrick skin type (FST) from a measure of patient photosensitivity to a measure for estimating skin tone in algorithmic audits of computer vision applications including facial recognition and dermatology diagnosis. In order to understand the variability of estimated FST annotations on images, we compare several FST annotation methods on a diverse set of 460 images of skin conditions from both textbooks and online dermatology atlases. These methods include expert annotation by board-certified dermatologists, algorithmic annotation via the Individual Typology Angle algorithm, which is then converted to estimated FST (ITA-FST), and two crowd-sourced, dynamic consensus protocols for annotating estimated FSTs. We find the inter-rater reliability between three board-certified dermatologists is comparable to the inter-rater reliability between the board-certified dermatologists and either of the crowdsourcing methods. In contrast, we find that the ITA-FST method produces annotations that are significantly less correlated with the experts' annotations than the experts' annotations are correlated with each other. These results demonstrate that algorithms based on ITA-FST are not reliable for annotating large-scale image datasets, but human-centered, crowd-based protocols can reliably add skin type transparency to dermatology datasets. Furthermore, we introduce the concept of dynamic consensus protocols with tunable parameters including expert review that increase the visibility of crowdwork and provide guidance for future crowdsourced annotations of large image datasets.<\/jats:p>","DOI":"10.1145\/3555634","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:59:06Z","timestamp":1668207546000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":52,"title":["Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithm"],"prefix":"10.1145","volume":"6","author":[{"given":"Matthew","family":"Groh","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caleb","family":"Harris","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roxana","family":"Daneshjou","sequence":"additional","affiliation":[{"name":"Stanford, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omar","family":"Badri","sequence":"additional","affiliation":[{"name":"Northeast Dermatology Associates, Boston, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arash","family":"Koochek","sequence":"additional","affiliation":[{"name":"Banner Health, Phoenix, AZ, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, Barcelona Spain, 252--260","author":"Abebe Rediet","unstructured":"Rediet Abebe , Solon Barocas , Jon Kleinberg , Karen Levy , Manish Raghavan , and David G. Robinson . 2020. Roles for computing in social change . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, Barcelona Spain, 252--260 . https:\/\/doi.org\/10.1145\/3351095.3372871 10.1145\/3351095.3372871 Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G. Robinson. 2020. Roles for computing in social change. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. ACM, Barcelona Spain, 252--260. https:\/\/doi.org\/10.1145\/3351095.3372871"},{"key":"e_1_2_1_2_1","article-title":"Skin color in dermatology textbooks: An updated evaluation and analysis","author":"Adelekun Ademide","year":"2020","unstructured":"Ademide Adelekun , Ginikanwa Onyekaba , and Jules B. Lipoff . 2020 . Skin color in dermatology textbooks: An updated evaluation and analysis . Journal of the American Academy of Dermatology ( April 2020), S0190962220307003. https:\/\/doi.org\/10.1016\/j.jaad.2020.04.084 10.1016\/j.jaad.2020.04.084 Ademide Adelekun, Ginikanwa Onyekaba, and Jules B. Lipoff. 2020. Skin color in dermatology textbooks: An updated evaluation and analysis. Journal of the American Academy of Dermatology (April 2020), S0190962220307003. https:\/\/doi.org\/10.1016\/j.jaad.2020.04.084","journal-title":"Journal of the American Academy of Dermatology"},{"key":"e_1_2_1_3_1","unstructured":"Jehad Amin AlKattash. [n. d.]. DermaAmin. https:\/\/www.dermaamin.com\/site\/ ( [n. d.]).  Jehad Amin AlKattash. [n. d.]. DermaAmin. https:\/\/www.dermaamin.com\/site\/ ( [n. d.])."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300760"},{"key":"e_1_2_1_5_1","article-title":"Representation of dark skin images of common dermatologic conditions in educational resources: a cross-sectional analysis","author":"Alvarado Savannah M.","year":"2020","unstructured":"Savannah M. Alvarado and Hao Feng . 2020 . Representation of dark skin images of common dermatologic conditions in educational resources: a cross-sectional analysis . Journal of the American Academy of Dermatology ( June 2020), S0190962220311385. https:\/\/doi.org\/10.1016\/j.jaad.2020.06.041 10.1016\/j.jaad.2020.06.041 Savannah M. Alvarado and Hao Feng. 2020. Representation of dark skin images of common dermatologic conditions in educational resources: a cross-sectional analysis. Journal of the American Academy of Dermatology (June 2020), S0190962220311385. https:\/\/doi.org\/10.1016\/j.jaad.2020.06.041","journal-title":"Journal of the American Academy of Dermatology"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531653"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372859"},{"key":"e_1_2_1_9_1","volume-title":"Selbst","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D . Selbst . 2016 . Big Data's Disparate Impact. SSRN Electronic Journal ( 2016). https:\/\/doi.org\/10.2139\/ssrn.2477899 10.2139\/ssrn.2477899 Solon Barocas and Andrew D. Selbst. 2016. Big Data's Disparate Impact. SSRN Electronic Journal (2016). https:\/\/doi.org\/10.2139\/ssrn.2477899"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376718"},{"key":"e_1_2_1_11_1","volume-title":"Science","volume":"366","author":"Benjamin Ruha","year":"2019","unstructured":"Ruha Benjamin . 2019 . Assessing risk, automating racism . Science , Vol. 366 , 6464 (2019), 421--422. Ruha Benjamin. 2019. Assessing risk, automating racism. Science, Vol. 366, 6464 (2019), 421--422."},{"key":"e_1_2_1_12_1","unstructured":"Jean L Bolognia Julie V Schaffer and Lorenzo Cerroni. 2018. Dermatolog'ia. Elsevier Health Sciences.  Jean L Bolognia Julie V Schaffer and Lorenzo Cerroni. 2018. Dermatolog'ia. Elsevier Health Sciences."},{"key":"e_1_2_1_13_1","volume-title":"Conference on fairness, accountability and transparency. PMLR, 77--91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. PMLR, 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479569"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press, Glasgow, Scotland Uk, 1--14","author":"Cai Carrie J.","unstructured":"Carrie J. Cai , Martin C. Stumpe , Michael Terry , Emily Reif , Narayan Hegde , Jason Hipp , Been Kim , Daniel Smilkov , Martin Wattenberg , Fernanda Viegas , and Greg S. Corrado . 2019a. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making . In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press, Glasgow, Scotland Uk, 1--14 . https:\/\/doi.org\/10.1145\/3290605.3300234 10.1145\/3290605.3300234 Carrie J. Cai, Martin C. Stumpe, Michael Terry, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, and Greg S. Corrado. 2019a. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press, Glasgow, Scotland Uk, 1--14. https:\/\/doi.org\/10.1145\/3290605.3300234"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359206"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411763.3443435"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-2494.1991.tb00561.x"},{"key":"e_1_2_1_19_1","volume-title":"Why is my classifier discriminatory? Advances in neural information processing systems","author":"Chen Irene","year":"2018","unstructured":"Irene Chen , Fredrik D Johansson , and David Sontag . 2018. Why is my classifier discriminatory? Advances in neural information processing systems , Vol. 31 ( 2018 ). Irene Chen, Fredrik D Johansson, and David Sontag. 2018. Why is my classifier discriminatory? Advances in neural information processing systems , Vol. 31 (2018)."},{"key":"e_1_2_1_20_1","volume-title":"A coefficient of agreement for nominal scales. Educational and psychological measurement","author":"Cohen Jacob","year":"1960","unstructured":"Jacob Cohen . 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement , Vol. 20 , 1 ( 1960 ), 37--46. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement , Vol. 20, 1 (1960), 37--46."},{"key":"e_1_2_1_21_1","volume-title":"H. Peter Soyer, Eric R. Tkaczyk, Philipp Tschandl, and Veronica Rotemberg.","author":"Daneshjou Roxana","year":"2021","unstructured":"Roxana Daneshjou , Catarina Barata , Brigid Betz-Stablein , M. Emre Celebi , Noel Codella , Marc Combalia , Pascale Guitera , David Gutman , Allan Halpern , Brian Helba , Harald Kittler , Kivanc Kose , Konstantinos Liopyris , Josep Malvehy , Han Seung Seog , H. Peter Soyer, Eric R. Tkaczyk, Philipp Tschandl, and Veronica Rotemberg. 2021 a. Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology : CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group . JAMA Dermatology (Dec. 2021). https:\/\/doi.org\/10.1001\/jamadermatol.2021.4915 10.1001\/jamadermatol.2021.4915 Roxana Daneshjou, Catarina Barata, Brigid Betz-Stablein, M. Emre Celebi, Noel Codella, Marc Combalia, Pascale Guitera, David Gutman, Allan Halpern, Brian Helba, Harald Kittler, Kivanc Kose, Konstantinos Liopyris, Josep Malvehy, Han Seung Seog, H. Peter Soyer, Eric R. Tkaczyk, Philipp Tschandl, and Veronica Rotemberg. 2021a. Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group. JAMA Dermatology (Dec. 2021). https:\/\/doi.org\/10.1001\/jamadermatol.2021.4915"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Roxana Daneshjou Mary Smith Mary Sun Veronica Rotemberg and James Zou. 2021b. Lack of Transparency and Potential Bias in Artificial Intelligence Data Sets and Algorithms: A Scoping Review. (2021) 8.  Roxana Daneshjou Mary Smith Mary Sun Veronica Rotemberg and James Zou. 2021b. Lack of Transparency and Potential Bias in Artificial Intelligence Data Sets and Algorithms: A Scoping Review. (2021) 8.","DOI":"10.1001\/jamadermatol.2021.3129"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Roxana Daneshjou Kailas Vodrahalli Weixin Liang Roberto A Novoa Melissa Jenkins Veronica Rotemberg Justin Ko Susan M Swetter Elizabeth E Bailey Olivier Gevaert etal 2022. Disparities in Dermatology AI Performance on a Diverse Curated Clinical Image Set. arXiv preprint arXiv:2203.08807 (2022).  Roxana Daneshjou Kailas Vodrahalli Weixin Liang Roberto A Novoa Melissa Jenkins Veronica Rotemberg Justin Ko Susan M Swetter Elizabeth E Bailey Olivier Gevaert et al. 2022. Disparities in Dermatology AI Performance on a Diverse Curated Clinical Image Set. arXiv preprint arXiv:2203.08807 (2022).","DOI":"10.1126\/sciadv.abq6147"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376638"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjd.12529"},{"key":"e_1_2_1_26_1","volume-title":"Reid","author":"Penna Nicol\u00e1s Della","year":"2012","unstructured":"Nicol\u00e1s Della Penna and Mark D . Reid . 2012 . Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard . arXiv:1204.3511 [cs] (April 2012). http:\/\/arxiv.org\/abs\/1204.3511 arXiv: 1204.3511. Nicol\u00e1s Della Penna and Mark D. Reid. 2012. Crowd & Prejudice: An Impossibility Theorem for Crowd Labelling without a Gold Standard. arXiv:1204.3511 [cs] (April 2012). http:\/\/arxiv.org\/abs\/1204.3511 arXiv: 1204.3511."},{"key":"e_1_2_1_27_1","article-title":"A Point-of-Care, Real-Time Artificial Intelligence System to Support Clinician Diagnosis of a Wide Range of Skin Diseases","author":"Dulmage Brittany","year":"2020","unstructured":"Brittany Dulmage , Kyle Tegtmeyer , Michael Z. Zhang , Maria Colavincenzo , and Shuai Xu . 2020 . A Point-of-Care, Real-Time Artificial Intelligence System to Support Clinician Diagnosis of a Wide Range of Skin Diseases . Journal of Investigative Dermatology ( Oct. 2020), S0022202X20321679. https:\/\/doi.org\/10.1016\/j.jid.2020.08.027 10.1016\/j.jid.2020.08.027 Brittany Dulmage, Kyle Tegtmeyer, Michael Z. Zhang, Maria Colavincenzo, and Shuai Xu. 2020. A Point-of-Care, Real-Time Artificial Intelligence System to Support Clinician Diagnosis of a Wide Range of Skin Diseases. Journal of Investigative Dermatology (Oct. 2020), S0022202X20321679. https:\/\/doi.org\/10.1016\/j.jid.2020.08.027","journal-title":"Journal of Investigative Dermatology"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaad.2005.10.068"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445188"},{"key":"e_1_2_1_30_1","volume-title":"Accuracy of self-report in assessing Fitzpatrick skin phototypes I through VI. JAMA dermatology","author":"Eilers Steven","year":"2013","unstructured":"Steven Eilers , Daniel Q Bach , Rikki Gaber , Hanz Blatt , Yanina Guevara , Katie Nitsche , Roopal V Kundu , and June K Robinson . 2013. Accuracy of self-report in assessing Fitzpatrick skin phototypes I through VI. JAMA dermatology , Vol. 149 , 11 ( 2013 ), 1289--1294. Steven Eilers, Daniel Q Bach, Rikki Gaber, Hanz Blatt, Yanina Guevara, Katie Nitsche, Roopal V Kundu, and June K Robinson. 2013. Accuracy of self-report in assessing Fitzpatrick skin phototypes I through VI. JAMA dermatology, Vol. 149, 11 (2013), 1289--1294."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature21056"},{"key":"e_1_2_1_32_1","first-page":"507","article-title":"Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population","volume":"10","author":"Fisher Ronald A","year":"1915","unstructured":"Ronald A Fisher . 1915 . Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population . Biometrika , Vol. 10 , 4 (1915), 507 -- 521 . Ronald A Fisher. 1915. Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, Vol. 10, 4 (1915), 507--521.","journal-title":"Biometrika"},{"key":"e_1_2_1_33_1","first-page":"1","article-title":"On the'probable error'of a coefficient of correlation deduced from a small sample","volume":"1","author":"Fisher Ronald A","year":"1921","unstructured":"Ronald A Fisher . 1921 . On the'probable error'of a coefficient of correlation deduced from a small sample . Metron , Vol. 1 (1921), 1 -- 32 . Ronald A Fisher. 1921. On the'probable error'of a coefficient of correlation deduced from a small sample. Metron , Vol. 1 (1921), 1--32.","journal-title":"Metron"},{"key":"e_1_2_1_34_1","volume-title":"The validity and practicality of sun-reactive skin types I through VI. Archives of dermatology","author":"Fitzpatrick Thomas B","year":"1988","unstructured":"Thomas B Fitzpatrick . 1988. The validity and practicality of sun-reactive skin types I through VI. Archives of dermatology , Vol. 124 , 6 ( 1988 ), 869--871. Thomas B Fitzpatrick. 1988. The validity and practicality of sun-reactive skin types I through VI. Archives of dermatology , Vol. 124, 6 (1988), 869--871."},{"key":"e_1_2_1_35_1","unstructured":"Samuel Freire da Silva. [n. d.]. Atlas Dermatologico. http:\/\/atlasdermatologico.com.br\/ ( [n. d.]).  Samuel Freire da Silva. [n. d.]. Atlas Dermatologico. http:\/\/atlasdermatologico.com.br\/ ( [n. d.])."},{"key":"e_1_2_1_36_1","volume-title":"Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digital Medicine","author":"Gaube Susanne","year":"2021","unstructured":"Susanne Gaube , Harini Suresh , Martina Raue , Alexander Merritt , Seth J. Berkowitz , Eva Lermer , Joseph F. Coughlin , John V. Guttag , Errol Colak , and Marzyeh Ghassemi . 2021. Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digital Medicine , Vol. 4 , 1 ( Dec. 2021 ), 31. https:\/\/doi.org\/10.1038\/s41746-021-00385--9 10.1038\/s41746-021-00385--9 Susanne Gaube, Harini Suresh, Martina Raue, Alexander Merritt, Seth J. Berkowitz, Eva Lermer, Joseph F. Coughlin, John V. Guttag, Errol Colak, and Marzyeh Ghassemi. 2021. Do as AI say: susceptibility in deployment of clinical decision-aids. npj Digital Medicine, Vol. 4, 1 (Dec. 2021), 31. https:\/\/doi.org\/10.1038\/s41746-021-00385--9"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_2_1_38_1","volume-title":"Practices of color classification. Mind, culture, and activity","author":"Goodwin Charles","year":"2000","unstructured":"Charles Goodwin . 2000. Practices of color classification. Mind, culture, and activity , Vol. 7 , 1--2 ( 2000 ), 19--36. Charles Goodwin. 2000. Practices of color classification. Mind, culture, and activity , Vol. 7, 1--2 (2000), 19--36."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359152"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479562"},{"key":"e_1_2_1_41_1","volume-title":"Rook's textbook of dermatology","author":"Griffiths Christopher","unstructured":"Christopher Griffiths , Jonathan Barker , Tanya O Bleiker , Robert Chalmers , and Daniel Creamer . 2016. Rook's textbook of dermatology . John Wiley & Sons . Christopher Griffiths, Jonathan Barker, Tanya O Bleiker, Robert Chalmers, and Daniel Creamer. 2016. Rook's textbook of dermatology. John Wiley & Sons."},{"key":"#cr-split#-e_1_2_1_42_1.1","unstructured":"Matthew Groh. 2022. Identifying the Context Shift between Test Benchmarks and Production Data. https:\/\/doi.org\/10.48550\/ARXIV.2207.01059 10.48550\/ARXIV.2207.01059"},{"key":"#cr-split#-e_1_2_1_42_1.2","unstructured":"Matthew Groh. 2022. Identifying the Context Shift between Test Benchmarks and Production Data. https:\/\/doi.org\/10.48550\/ARXIV.2207.01059"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2110013119"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW53098.2021.00201"},{"key":"e_1_2_1_45_1","volume-title":"Skin typing: Fitzpatrick grading and others. Clinics in dermatology","author":"Gupta Vishal","year":"2019","unstructured":"Vishal Gupta and Vinod Kumar Sharma . 2019. Skin typing: Fitzpatrick grading and others. Clinics in dermatology , Vol. 37 , 5 ( 2019 ), 430--436. Vishal Gupta and Vinod Kumar Sharma. 2019. Skin typing: Fitzpatrick grading and others. Clinics in dermatology , Vol. 37, 5 (2019), 430--436."},{"key":"e_1_2_1_46_1","unstructured":"Thomas Habif. 2010. Clinical dermatology: A color guide to diagnosis and therapy. (2010).  Thomas Habif. 2010. Clinical dermatology: A color guide to diagnosis and therapy. (2010)."},{"key":"e_1_2_1_47_1","volume-title":"Nature","volume":"586","author":"Haibe-Kains Benjamin","year":"2020","unstructured":"Benjamin Haibe-Kains , George Alexandru Adam , Ahmed Hosny , Farnoosh Khodakarami , Levi Waldron , Bo Wang , Chris McIntosh , Anna Goldenberg , Anshul Kundaje , Casey S Greene , 2020 . Transparency and reproducibility in artificial intelligence . Nature , Vol. 586 , 7829 (2020), E14--E16. Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Levi Waldron, Bo Wang, Chris McIntosh, Anna Goldenberg, Anshul Kundaje, Casey S Greene, et al. 2020. Transparency and reproducibility in artificial intelligence. Nature, Vol. 586, 7829 (2020), E14--E16."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jid.2020.01.019"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174023"},{"key":"e_1_2_1_50_1","unstructured":"Caner Hazirbas Joanna Bitton Brian Dolhansky Jacqueline Pan Albert Gordo and Cristian Canton Ferrer. [n. d.]. Towards measuring fairness in AI: the Casual Conversations dataset. ( [n. d.]) 9.  Caner Hazirbas Joanna Bitton Brian Dolhansky Jacqueline Pan Albert Gordo and Cristian Canton Ferrer. [n. d.]. Towards measuring fairness in AI: the Casual Conversations dataset. ( [n. d.]) 9."},{"key":"e_1_2_1_51_1","volume-title":"The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677","author":"Holland Sarah","year":"2018","unstructured":"Sarah Holland , Ahmed Hosny , Sarah Newman , Joshua Joseph , and Kasia Chmielinski . 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 ( 2018 ). Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 (2018)."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1953.tb00135.x"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_1_54_1","volume-title":"Dermatology atlas for skin of color","author":"Jackson-Richards Diane","unstructured":"Diane Jackson-Richards and Amit G Pandya . 2014. Dermatology atlas for skin of color . Springer . Diane Jackson-Richards and Amit G Pandya. 2014. Dermatology atlas for skin of color. Springer."},{"key":"e_1_2_1_55_1","volume-title":"How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry","author":"Jacobs Maia","year":"2021","unstructured":"Maia Jacobs , Melanie F Pradier , Thomas H McCoy , Roy H Perlis , Finale Doshi-Velez , and Krzysztof Z Gajos . 2021. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry , Vol. 11 , 1 ( 2021 ), 1--9. Maia Jacobs, Melanie F Pradier, Thomas H McCoy, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. 2021. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry , Vol. 11, 1 (2021), 1--9."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.7249"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jaad.2016.09.033"},{"key":"e_1_2_1_58_1","unstructured":"Sewon Kang. 2019. Fitzpatrick's Dermatology 2-Volume Set (Fitzpatricks.  Sewon Kang. 2019. Fitzpatrick's Dermatology 2-Volume Set (Fitzpatricks."},{"key":"e_1_2_1_59_1","volume-title":"Utility of sun-reactive skin typing and melanin index for discerning vitamin D deficiency. Pediatric research","author":"Khalid Arshad T","year":"2017","unstructured":"Arshad T Khalid , Charity G Moore , Christopher Hall , Flora Olabopo , Nigel L Rozario , Michael F Holick , Susan L Greenspan , and Kumaravel Rajakumar . 2017. Utility of sun-reactive skin typing and melanin index for discerning vitamin D deficiency. Pediatric research , Vol. 82 , 3 ( 2017 ), 444--451. Arshad T Khalid, Charity G Moore, Christopher Hall, Flora Olabopo, Nigel L Rozario, Michael F Holick, Susan L Greenspan, and Kumaravel Rajakumar. 2017. Utility of sun-reactive skin typing and melanin index for discerning vitamin D deficiency. Pediatric research, Vol. 82, 3 (2017), 444--451."},{"key":"e_1_2_1_60_1","volume-title":"Varshney","author":"Kinyanjui Newton M.","year":"2019","unstructured":"Newton M. Kinyanjui , Timothy Odonga , Celia Cintas , Noel C. F. Codella , Rameswar Panda , Prasanna Sattigeri , and Kush R . Varshney . 2019 . Estimating Skin Tone and Effects on Classification Performance in Dermatology Datasets . arXiv:1910.13268 [cs, stat] (Oct. 2019). http:\/\/arxiv.org\/abs\/1910.13268 arXiv: 1910.13268. Newton M. Kinyanjui, Timothy Odonga, Celia Cintas, Noel C. F. Codella, Rameswar Panda, Prasanna Sattigeri, and Kush R. Varshney. 2019. Estimating Skin Tone and Effects on Classification Performance in Dermatology Datasets. arXiv:1910.13268 [cs, stat] (Oct. 2019). http:\/\/arxiv.org\/abs\/1910.13268 arXiv: 1910.13268."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1912790117"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.2991\/iccasp-16.2017.51"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACVW54805.2022.00049"},{"key":"e_1_2_1_64_1","volume-title":"Clinical Photography in Skin of Color: Tips and Best Practices. The British Journal of Dermatology","author":"Lester JC","year":"2021","unstructured":"JC Lester , L Clark Jr , E Linos , and R Daneshjou . 2021. Clinical Photography in Skin of Color: Tips and Best Practices. The British Journal of Dermatology ( 2021 ). JC Lester, L Clark Jr, E Linos, and R Daneshjou. 2021. Clinical Photography in Skin of Color: Tips and Best Practices. The British Journal of Dermatology (2021)."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjd.19258"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479552"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0842-3"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2018.12.005"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.socscimed.2018.02.023"},{"key":"e_1_2_1_70_1","volume-title":"Shu-Fang Newman, Jerry Kim, et al.","author":"Lundberg Scott M","year":"2018","unstructured":"Scott M Lundberg , Bala Nair , Monica S Vavilala , Mayumi Horibe , Michael J Eisses , Trevor Adams , David E Liston , Daniel King-Wai Low , Shu-Fang Newman, Jerry Kim, et al. 2018 . Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering , Vol. 2 , 10 (2018), 749--760. Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Wai Low, Shu-Fang Newman, Jerry Kim, et al. 2018. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineering , Vol. 2, 10 (2018), 749--760."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415168"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3361118"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531663"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1799-6"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3492853"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415186"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445880"},{"key":"e_1_2_1_78_1","volume-title":"E-Book","author":"Micheletti Robert G","unstructured":"Robert G Micheletti , William D James , Dirk Elston , and Patrick J McMahon . 2022. Andrews' Diseases of the Skin Clinical Atlas , E-Book . Elsevier Health Sciences . Robert G Micheletti, William D James, Dirk Elston, and Patrick J McMahon. 2022. Andrews' Diseases of the Skin Clinical Atlas, E-Book. Elsevier Health Sciences."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1086\/682162"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1257\/pandp.20211078"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300356"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445402"},{"key":"e_1_2_1_84_1","unstructured":"Laura Nader. 1972. Up the anthropologist: Perspectives gained from studying up. (1972).  Laura Nader. 1972. Up the anthropologist: Perspectives gained from studying up. (1972)."},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aax2342"},{"key":"e_1_2_1_86_1","volume-title":"Standardized clinical photography considerations in patients across skin tones. British Journal of Dermatology","author":"Oh Yuna","year":"2021","unstructured":"Yuna Oh , A Markova , SJ Noor , and Veronica Rotemberg . 2021. Standardized clinical photography considerations in patients across skin tones. British Journal of Dermatology ( 2021 ). Yuna Oh, A Markova, SJ Noor, and Veronica Rotemberg. 2021. Standardized clinical photography considerations in patients across skin tones. British Journal of Dermatology (2021)."},{"key":"e_1_2_1_87_1","volume-title":"Equity in skin typing: why it's time to replace the Fitzpatrick scale. The British Journal of Dermatology","author":"Okoji UK","year":"2021","unstructured":"UK Okoji , SC Taylor , and JB Lipoff . 2021. Equity in skin typing: why it's time to replace the Fitzpatrick scale. The British Journal of Dermatology ( 2021 ). UK Okoji, SC Taylor, and JB Lipoff. 2021. Equity in skin typing: why it's time to replace the Fitzpatrick scale. The British Journal of Dermatology (2021)."},{"key":"e_1_2_1_88_1","volume-title":"A survey of crowdsourcing in medical image analysis. arXiv preprint arXiv:1902.09159","author":"\u00d8rting Silas","year":"2019","unstructured":"Silas \u00d8rting , Andrew Doyle , Arno van Hilten , Matthias Hirth , Oana Inel , Christopher R Madan , Panagiotis Mavridis , Helen Spiers , and Veronika Cheplygina . 2019. A survey of crowdsourcing in medical image analysis. arXiv preprint arXiv:1902.09159 ( 2019 ). Silas \u00d8rting, Andrew Doyle, Arno van Hilten, Matthias Hirth, Oana Inel, Christopher R Madan, Panagiotis Mavridis, Helen Spiers, and Veronika Cheplygina. 2019. A survey of crowdsourcing in medical image analysis. arXiv preprint arXiv:1902.09159 (2019)."},{"key":"e_1_2_1_89_1","volume-title":"Individual Typology Angle and Fitzpatrick Skin Phototypes are Not Equivalent in Photodermatology. Photochemistry and photobiology","author":"Osto Muhammad","year":"2022","unstructured":"Muhammad Osto , Iltefat H Hamzavi , Henry W Lim , and Indermeet Kohli . 2022. Individual Typology Angle and Fitzpatrick Skin Phototypes are Not Equivalent in Photodermatology. Photochemistry and photobiology , Vol. 98 , 1 ( 2022 ), 127--129. Muhammad Osto, Iltefat H Hamzavi, Henry W Lim, and Indermeet Kohli. 2022. Individual Typology Angle and Fitzpatrick Skin Phototypes are Not Equivalent in Photodermatology. Photochemistry and photobiology , Vol. 98, 1 (2022), 127--129."},{"key":"e_1_2_1_90_1","doi-asserted-by":"crossref","unstructured":"Bhavik N Patel Louis Rosenberg Gregg Willcox David Baltaxe Mimi Lyons Jeremy Irvin Pranav Rajpurkar Timothy Amrhein Rajan Gupta Safwan Halabi etal 2019. Human--machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ digital medicine Vol. 2 1 (2019) 1--10.  Bhavik N Patel Louis Rosenberg Gregg Willcox David Baltaxe Mimi Lyons Jeremy Irvin Pranav Rajpurkar Timothy Amrhein Rajan Gupta Safwan Halabi et al. 2019. Human--machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ digital medicine Vol. 2 1 (2019) 1--10.","DOI":"10.1038\/s41746-019-0189-7"},{"key":"e_1_2_1_91_1","doi-asserted-by":"crossref","unstructured":"Michael Phillips Helen Marsden Wayne Jaffe Rubeta N Matin Gorav N Wali Jack Greenhalgh Emily McGrath Rob James Evmorfia Ladoyanni Anthony Bewley etal 2019. Assessment of accuracy of an artificial intelligence algorithm to detect melanoma in images of skin lesions. JAMA network open Vol. 2 10 (2019) e1913436--e1913436.  Michael Phillips Helen Marsden Wayne Jaffe Rubeta N Matin Gorav N Wali Jack Greenhalgh Emily McGrath Rob James Evmorfia Ladoyanni Anthony Bewley et al. 2019. Assessment of accuracy of an artificial intelligence algorithm to detect melanoma in images of skin lesions. JAMA network open Vol. 2 10 (2019) e1913436--e1913436.","DOI":"10.1001\/jamanetworkopen.2019.13436"},{"key":"e_1_2_1_92_1","volume-title":"Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta psychologica","author":"Preston Carolyn C","year":"2000","unstructured":"Carolyn C Preston and Andrew M Colman . 2000. Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta psychologica , Vol. 104 , 1 ( 2000 ), 1--15. Carolyn C Preston and Andrew M Colman. 2000. Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences. Acta psychologica, Vol. 104, 1 (2000), 1--15."},{"key":"e_1_2_1_93_1","unstructured":"Maithra Raghu Katy Blumer Rory Sayres Ziad Obermeyer Robert Kleinberg Sendhil Mullainathan and Jon Kleinberg. [n. d.]. Direct Uncertainty Prediction for Medical Second Opinions. ( [n. d.]) 10.  Maithra Raghu Katy Blumer Rory Sayres Ziad Obermeyer Robert Kleinberg Sendhil Mullainathan and Jon Kleinberg. [n. d.]. Direct Uncertainty Prediction for Medical Second Opinions. ( [n. d.]) 10."},{"key":"e_1_2_1_94_1","volume-title":"Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225","author":"Rajpurkar Pranav","year":"2017","unstructured":"Pranav Rajpurkar , Jeremy Irvin , Kaylie Zhu , Brandon Yang , Hershel Mehta , Tony Duan , Daisy Ding , Aarti Bagul , Curtis Langlotz , Katie Shpanskaya , 2017 . Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017). Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. 2017. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)."},{"key":"e_1_2_1_95_1","volume-title":"Machine Predictions and Human Decisions with Variation in Payoffs and Skills. SSRN Electronic Journal","author":"Ribers Michael Allen","year":"2020","unstructured":"Michael Allen Ribers and Hannes Ullrich . 2020. Machine Predictions and Human Decisions with Variation in Payoffs and Skills. SSRN Electronic Journal ( 2020 ). https:\/\/doi.org\/10.2139\/ssrn.3726018 10.2139\/ssrn.3726018 Michael Allen Ribers and Hannes Ullrich. 2020. Machine Predictions and Human Decisions with Variation in Payoffs and Skills. SSRN Electronic Journal (2020). https:\/\/doi.org\/10.2139\/ssrn.3726018"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2021.102274"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijwd.2020.05.005"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445518"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476058"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392866"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415224"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-020-0942-0"},{"key":"e_1_2_1_103_1","volume-title":"The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data","author":"Tschandl Philipp","year":"2018","unstructured":"Philipp Tschandl , Cliff Rosendahl , and Harald Kittler . 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data , Vol. 5 , 1 ( 2018 ), 1--9. Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. 2018. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, Vol. 5, 1 (2018), 1--9."},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.8226"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1111\/jep.12747"},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476068"},{"key":"e_1_2_1_107_1","first-page":"77","article-title":"Racial limitations of fitzpatrick skin type","volume":"105","author":"Ware Olivia R","year":"2020","unstructured":"Olivia R Ware , Jessica E Dawson , Michi M Shinohara , and Susan C Taylor . 2020 . Racial limitations of fitzpatrick skin type . Cutis , Vol. 105 , 2 (2020), 77 -- 80 . Olivia R Ware, Jessica E Dawson, Michi M Shinohara, and Susan C Taylor. 2020. Racial limitations of fitzpatrick skin type. Cutis, Vol. 105, 2 (2020), 77--80.","journal-title":"Cutis"},{"key":"e_1_2_1_108_1","volume-title":"Alastair Denniston, Xiaoxuan Liu, and Rubeta Matin.","author":"Wen Davis","year":"2021","unstructured":"Davis Wen , Saad Khan , Antonio Xu , Hussein Ibrahim , Luke Smith , Jose Caballero , Luis Zepeda , Carlos de Blas Perez , Alastair Denniston, Xiaoxuan Liu, and Rubeta Matin. 2021 . Characteristics of publicly available skin cancer image datasets: a systematic review. Lancet Digital Health (2021). Davis Wen, Saad Khan, Antonio Xu, Hussein Ibrahim, Luke Smith, Jose Caballero, Luis Zepeda, Carlos de Blas Perez, Alastair Denniston, Xiaoxuan Liu, and Rubeta Matin. 2021. Characteristics of publicly available skin cancer image datasets: a systematic review. Lancet Digital Health (2021)."},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1001\/jamadermatol.2015.0351"},{"key":"e_1_2_1_110_1","unstructured":"K Wolff L Goldsmith S Katz B Gilchrest A Paller and D Lafell. 2008. Fitzpatricks Textbook of Dermatology in General Medicine.  K Wolff L Goldsmith S Katz B Gilchrest A Paller and D Lafell. 2008. Fitzpatricks Textbook of Dermatology in General Medicine."},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01312-x"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555634","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555634","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:18Z","timestamp":1750182558000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555634"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":112,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555634"],"URL":"https:\/\/doi.org\/10.1145\/3555634","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}