{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T15:51:08Z","timestamp":1774021868612,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":51,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,7,21]],"date-time":"2021-07-21T00:00:00Z","timestamp":1626825600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"EPSRC","award":["EP\/P024394\/1, EP\/R033501\/1"],"award-info":[{"award-number":["EP\/P024394\/1, EP\/R033501\/1"]}]},{"name":"Aviva"},{"DOI":"10.13039\/100004318","name":"Microsoft","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004318","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,21]]},"DOI":"10.1145\/3461702.3462572","type":"proceedings-article","created":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T01:21:32Z","timestamp":1627694492000},"page":"704-714","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Risk Identification Questionnaire for Detecting Unintended Bias in the Machine Learning Development Lifecycle"],"prefix":"10.1145","author":[{"given":"Michelle Seng Ah","family":"Lee","sequence":"first","affiliation":[{"name":"University of Cambridge, Cambridge, United Kingdom"}]},{"given":"Jatinder","family":"Singh","sequence":"additional","affiliation":[{"name":"University of Cambridge, Cambridge, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2021,7,30]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/2835587.2835589"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2019.2942287"},{"key":"e_1_3_2_2_3_1","volume-title":"A convex framework for fair regression. arXiv preprint arXiv:1706.02409","author":"Berk Richard","year":"2017","unstructured":"Richard Berk , Hoda Heidari , Shahin Jabbari , Matthew Joseph , Michael Kearns , Jamie Morgenstern , Seth Neel , and Aaron Roth . 2017. A convex framework for fair regression. arXiv preprint arXiv:1706.02409 ( 2017 ). Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2017. A convex framework for fair regression. arXiv preprint arXiv:1706.02409 (2017)."},{"key":"e_1_3_2_2_4_1","volume-title":"SUS: A quick and dirty usability scale. In Usability evaluation in industry","author":"Brooke John","year":"1996","unstructured":"John Brooke . 1996 . SUS: A quick and dirty usability scale. In Usability evaluation in industry . Taylor and Francis . John Brooke. 1996. SUS: A quick and dirty usability scale. In Usability evaluation in industry. Taylor and Francis."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445921"},{"key":"e_1_3_2_2_7_1","volume-title":"The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023","author":"Corbett-Davies Sam","year":"2018","unstructured":"Sam Corbett-Davies and Sharad Goel . 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023 ( 2018 ). Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023 (2018)."},{"key":"e_1_3_2_2_8_1","unstructured":"COSO. 2017. Committee of Sponsoring Organizations of the Treadway Commission Enterprise Risk Management Integrated Framework. https:\/\/www.coso.org\/Pages\/erm.aspx  COSO. 2017. Committee of Sponsoring Organizations of the Treadway Commission Enterprise Risk Management Integrated Framework. https:\/\/www.coso.org\/Pages\/erm.aspx"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/07366981.2012.682494"},{"key":"e_1_3_2_2_10_1","volume-title":"Cockpit checklists: Concepts, design, and use. Human factors","author":"Degani Asaf","year":"1993","unstructured":"Asaf Degani and Earl L Wiener . 1993. Cockpit checklists: Concepts, design, and use. Human factors , Vol. 35 , 2 ( 1993 ), 345--359. Asaf Degani and Earl L Wiener. 1993. Cockpit checklists: Concepts, design, and use. Human factors, Vol. 35, 2 (1993), 345--359."},{"key":"e_1_3_2_2_11_1","volume-title":"Deon: An ethics checklist for data scientists. DrivenData","year":"2019","unstructured":"DrivenData. 2019 . Deon: An ethics checklist for data scientists. DrivenData . http:\/\/deon.drivendata.org\/ DrivenData. 2019. Deon: An ethics checklist for data scientists. DrivenData. http:\/\/deon.drivendata.org\/"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_2_14_1","volume-title":"Checklist manifesto, the (HB)","author":"Gawande Atul","unstructured":"Atul Gawande . 2010. Checklist manifesto, the (HB) . Penguin Books India . Atul Gawande. 2010. Checklist manifesto, the (HB) .Penguin Books India."},{"key":"e_1_3_2_2_15_1","volume-title":"Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford.","author":"Gebru Timnit","year":"2018","unstructured":"Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018 . Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018). Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010 (2018)."},{"key":"e_1_3_2_2_16_1","volume-title":"Proceedings of the machine learning: the debates workshop .","author":"Green Ben","year":"2018","unstructured":"Ben Green and Lily Hu . 2018 . The myth in the methodology: Towards a recontextualization of fairness in machine learning . In Proceedings of the machine learning: the debates workshop . Ben Green and Lily Hu. 2018. The myth in the methodology: Towards a recontextualization of fairness in machine learning. In Proceedings of the machine learning: the debates workshop ."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186138"},{"key":"e_1_3_2_2_18_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315--3323."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372875"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.45"},{"key":"e_1_3_2_2_22_1","volume-title":"Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807","author":"Kleinberg Jon","year":"2016","unstructured":"Jon Kleinberg , Sendhil Mullainathan , and Manish Raghavan . 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 ( 2016 ). Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016)."},{"key":"e_1_3_2_2_23_1","volume-title":"Article arXiv:1703.06856 (March","author":"Kusner Matt J.","year":"2017","unstructured":"Matt J. Kusner , Joshua R. Loftus , Chris Russell , and Ricardo Silva . 2017. Counterfactual Fairness . arXiv e-prints , Article arXiv:1703.06856 (March 2017 ), arXiv:1703.06856 pages.arxiv: 1703.06856 [stat.ML] Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual Fairness. arXiv e-prints, Article arXiv:1703.06856 (March 2017), arXiv:1703.06856 pages.arxiv: 1703.06856 [stat.ML]"},{"key":"e_1_3_2_2_24_1","article-title":"Innovating with Confidence: Embedding AI Governance and Fairness in a Financial Services Risk Management Framework","volume":"34","author":"Lee Michelle","year":"2020","unstructured":"Michelle Lee , Luciano Floridi , and Alexander Denev . 2020 . Innovating with Confidence: Embedding AI Governance and Fairness in a Financial Services Risk Management Framework . Berkeley Technology Law Journal , Vol. 34 , 2 (2020). Michelle Lee, Luciano Floridi, and Alexander Denev. 2020. Innovating with Confidence: Embedding AI Governance and Fairness in a Financial Services Risk Management Framework. Berkeley Technology Law Journal, Vol. 34, 2 (2020).","journal-title":"Berkeley Technology Law Journal"},{"key":"e_1_3_2_2_25_1","volume-title":"Algorithmic fairness in mortgage lending: from absolute conditions to relational trade-offs. Minds and Machines","author":"Ah Lee Michelle Seng","year":"2020","unstructured":"Michelle Seng Ah Lee and Luciano Floridi . 2020. Algorithmic fairness in mortgage lending: from absolute conditions to relational trade-offs. Minds and Machines ( 2020 ). https:\/\/doi.org\/10.1007\/s11023-020-09529--4 Michelle Seng Ah Lee and Luciano Floridi. 2020. Algorithmic fairness in mortgage lending: from absolute conditions to relational trade-offs. Minds and Machines (2020). https:\/\/doi.org\/10.1007\/s11023-020-09529--4"},{"key":"e_1_3_2_2_26_1","volume-title":"Formalising trade-offs beyond algorithmic fairness: lessons from ethical philosophy and welfare economics. Available at SSRN","author":"Ah Lee Michelle Seng","year":"2021","unstructured":"Michelle Seng Ah Lee , Luciano Floridi , and Jatinder Singh . 2021. Formalising trade-offs beyond algorithmic fairness: lessons from ethical philosophy and welfare economics. Available at SSRN ( 2021 ). https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3679975 Michelle Seng Ah Lee, Luciano Floridi, and Jatinder Singh. 2021. Formalising trade-offs beyond algorithmic fairness: lessons from ethical philosophy and welfare economics. Available at SSRN (2021). https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3679975"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445261"},{"key":"e_1_3_2_2_28_1","volume-title":"Delayed impact of fair machine learning. arXiv preprint arXiv:1803.04383","author":"Liu Lydia T","year":"2018","unstructured":"Lydia T Liu , Sarah Dean , Esther Rolf , Max Simchowitz , and Moritz Hardt . 2018. Delayed impact of fair machine learning. arXiv preprint arXiv:1803.04383 ( 2018 ). Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed impact of fair machine learning. arXiv preprint arXiv:1803.04383 (2018)."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376445"},{"key":"e_1_3_2_2_30_1","volume-title":"A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635","author":"Mehrabi Ninareh","year":"2019","unstructured":"Ninareh Mehrabi , Fred Morstatter , Nripsuta Saxena , Kristina Lerman , and Aram Galstyan . 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 ( 2019 ). Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635 (2019)."},{"key":"e_1_3_2_2_31_1","unstructured":"Microsoft. 2019. Fairlearn. https:\/\/fairlearn.github.io\/  Microsoft. 2019. Fairlearn. https:\/\/fairlearn.github.io\/"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_3_2_2_33_1","volume-title":"New COLD Feature Based Handwriting Analysis for Enthnicity\/Nationality Identification. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE, 523--527","author":"Nag Sauradip","year":"2018","unstructured":"Sauradip Nag , Palaiahnakote Shivakumara , Yirui Wu , Umapada Pal , and Tong Lu . 2018 . New COLD Feature Based Handwriting Analysis for Enthnicity\/Nationality Identification. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE, 523--527 . Sauradip Nag, Palaiahnakote Shivakumara, Yirui Wu, Umapada Pal, and Tong Lu. 2018. New COLD Feature Based Handwriting Analysis for Enthnicity\/Nationality Identification. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE, 523--527."},{"key":"e_1_3_2_2_34_1","unstructured":"Arvind Narayanan. 2018. Tutorial: 21 Definitions of Fairness and their Politics. YouTube. https:\/\/www.youtube.com\/watch?v=jIXIuYdnyyk  Arvind Narayanan. 2018. Tutorial: 21 Definitions of Fairness and their Politics. YouTube. https:\/\/www.youtube.com\/watch?v=jIXIuYdnyyk"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2019.00013"},{"key":"e_1_3_2_2_36_1","unstructured":"Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680--5689.  Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680--5689."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.7326\/M18-1990"},{"key":"e_1_3_2_2_38_1","volume-title":"Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577","author":"Saleiro Pedro","year":"2018","unstructured":"Pedro Saleiro , Benedict Kuester , Loren Hinkson , Jesse London , Abby Stevens , Ari Anisfeld , Kit T Rodolfa , and Rayid Ghani . 2018 . Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577 (2018). Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. 2018. Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577 (2018)."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_2_2_40_1","first-page":"194","article-title":"Towards checklists to reduce common sources of problem-solving failure. Policing","volume":"6","author":"Sidebottom Aiden","year":"2012","unstructured":"Aiden Sidebottom , Nick Tilley , and John E Eck . 2012 . Towards checklists to reduce common sources of problem-solving failure. Policing : A Journal of Policy and Practice , Vol. 6 , 2 (2012), 194 -- 209 . Aiden Sidebottom, Nick Tilley, and John E Eck. 2012. Towards checklists to reduce common sources of problem-solving failure. Policing: A Journal of Policy and Practice, Vol. 6, 2 (2012), 194--209.","journal-title":"A Journal of Policy and Practice"},{"key":"e_1_3_2_2_41_1","volume-title":"A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002","author":"Suresh Harini","year":"2019","unstructured":"Harini Suresh and John V Guttag . 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 ( 2019 ). Harini Suresh and John V Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 (2019)."},{"key":"e_1_3_2_2_42_1","volume-title":"Managing risks in complex projects. Project management journal","author":"Thamhain Hans","year":"2013","unstructured":"Hans Thamhain . 2013. Managing risks in complex projects. Project management journal , Vol. 44 , 2 ( 2013 ), 20--35. Hans Thamhain. 2013. Managing risks in complex projects. Project management journal, Vol. 44, 2 (2013), 20--35."},{"key":"e_1_3_2_2_43_1","volume-title":"Surgical checklists: a systematic review of impacts and implementation. BMJ quality & safety","author":"Treadwell Jonathan R","year":"2014","unstructured":"Jonathan R Treadwell , Scott Lucas , and Amy Y Tsou . 2014. Surgical checklists: a systematic review of impacts and implementation. BMJ quality & safety , Vol. 23 , 4 ( 2014 ), 299--318. Jonathan R Treadwell, Scott Lucas, and Amy Y Tsou. 2014. Surgical checklists: a systematic review of impacts and implementation. BMJ quality & safety, Vol. 23, 4 (2014), 299--318."},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2019.2906038"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174014"},{"key":"e_1_3_2_2_46_1","volume-title":"Requirements for the design and implementation of checklists for surgical processes. Surgical endoscopy","author":"Verdaasdonk EGG","year":"2009","unstructured":"EGG Verdaasdonk , LPS Stassen , Prama P Widhiasmara , and Jenny Dankelman . 2009. Requirements for the design and implementation of checklists for surgical processes. Surgical endoscopy , Vol. 23 , 4 ( 2009 ), 715--726. EGG Verdaasdonk, LPS Stassen, Prama P Widhiasmara, and Jenny Dankelman. 2009. Requirements for the design and implementation of checklists for surgical processes. Surgical endoscopy, Vol. 23, 4 (2009), 715--726."},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_2_2_48_1","unstructured":"Matthijs Vincent. 2019. scikit-fairness. https:\/\/github.com\/koaning\/scikit-fairness  Matthijs Vincent. 2019. scikit-fairness. https:\/\/github.com\/koaning\/scikit-fairness"},{"key":"e_1_3_2_2_49_1","volume-title":"Proceedings of 6th IRNOP Project Research Conference","author":"Voetsch Robert James","year":"2004","unstructured":"Robert James Voetsch , Denis F Cioffi , and Frank T Anbari . 2004 . Project risk management practices and their association with reported project success . In Proceedings of 6th IRNOP Project Research Conference , Turku, Finland. Citeseer, 680--97. Robert James Voetsch, Denis F Cioffi, and Frank T Anbari. 2004. Project risk management practices and their association with reported project success. In Proceedings of 6th IRNOP Project Research Conference, Turku, Finland. Citeseer, 680--97."},{"key":"e_1_3_2_2_50_1","volume-title":"The what-if tool: Interactive probing of machine learning models","author":"Wexler James","year":"2019","unstructured":"James Wexler , Mahima Pushkarna , Tolga Bolukbasi , Martin Wattenberg , Fernanda Vi\u00e9gas , and Jimbo Wilson . 2019. The what-if tool: Interactive probing of machine learning models . IEEE transactions on visualization and computer graphics, Vol. 26 , 1 ( 2019 ), 56--65. James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Vi\u00e9gas, and Jimbo Wilson. 2019. The what-if tool: Interactive probing of machine learning models. IEEE transactions on visualization and computer graphics, Vol. 26, 1 (2019), 56--65."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2010.01470.x"}],"event":{"name":"AIES '21: AAAI\/ACM Conference on AI, Ethics, and Society","location":"Virtual Event USA","acronym":"AIES '21","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","AAAI"]},"container-title":["Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462572","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3461702.3462572","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:49:06Z","timestamp":1750193346000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3461702.3462572"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,21]]},"references-count":51,"alternative-id":["10.1145\/3461702.3462572","10.1145\/3461702"],"URL":"https:\/\/doi.org\/10.1145\/3461702.3462572","relation":{},"subject":[],"published":{"date-parts":[[2021,7,21]]},"assertion":[{"value":"2021-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}