{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T17:48:40Z","timestamp":1774460920441,"version":"3.50.1"},"reference-count":76,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,9,17]],"date-time":"2022-09-17T00:00:00Z","timestamp":1663372800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,17]],"date-time":"2022-09-17T00:00:00Z","timestamp":1663372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1934464"],"award-info":[{"award-number":["1934464"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1922658"],"award-info":[{"award-number":["1922658"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1916505"],"award-info":[{"award-number":["1916505"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"NYU Center for the Humanities Digital Humanities Seed Grant"},{"name":"NYU Center for the Humanities Digital Humanities Seed Grant"},{"DOI":"10.13039\/100018295","name":"Underwriters Laboratories","doi-asserted-by":"publisher","award":["CASMI seed grant"],"award-info":[{"award-number":["CASMI seed grant"]}],"id":[{"id":"10.13039\/100018295","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers\u2019 resumes or social media profiles. We interrogate the validity of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. Crucially, rather than challenging or affirming the assumptions made in psychometric testing \u2014 that personality is a meaningful and measurable construct, and that personality traits are indicative of future success on the job \u2014 we frame our audit methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves. Our main contribution is the development of a socio-technical framework for auditing the stability of algorithmic systems. This contribution is supplemented with an open-source software library that implements the technical components of the audit, and can be used to conduct similar stability audits of algorithmic systems. We instantiate our framework with the audit of two real-world personality prediction systems, namely, Humantic AI and Crystal. The application of our audit framework demonstrates that both these systems show substantial instability with respect to key facets of measurement, and hence cannot be considered valid testing instruments.<\/jats:p>","DOI":"10.1007\/s10618-022-00861-0","type":"journal-article","created":{"date-parts":[[2022,9,17]],"date-time":"2022-09-17T06:02:36Z","timestamp":1663394556000},"page":"2153-2193","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["An external stability audit framework to test the validity of personality prediction in AI hiring"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5284-2064","authenticated-orcid":false,"given":"Alene K.","family":"Rhea","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1151-2952","authenticated-orcid":false,"given":"Kelsey","family":"Markey","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6452-9032","authenticated-orcid":false,"given":"Lauren","family":"D\u2019Arinzo","sequence":"additional","affiliation":[]},{"given":"Hilke","family":"Schellmann","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1049-2267","authenticated-orcid":false,"given":"Mona","family":"Sloane","sequence":"additional","affiliation":[]},{"given":"Paul","family":"Squires","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4678-5929","authenticated-orcid":false,"given":"Falaah","family":"Arif Khan","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1587-0450","authenticated-orcid":false,"given":"Julia","family":"Stoyanovich","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,9,17]]},"reference":[{"issue":"2","key":"861_CR1","first-page":"80","volume":"34","author":"I Ajunwa","year":"2021","unstructured":"Ajunwa I (2021) An Auditing Imperative for Automated Hiring Systems. Harvard J Law & Tech 34(2):80","journal-title":"Harvard J Law & Tech"},{"issue":"1","key":"861_CR2","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1108\/IJCED-12-2016-0023","volume":"19","author":"L Araujo","year":"2017","unstructured":"Araujo L, Saltelli A, Schnepf SV (2017) Do PISA data justify PISA-based education policy? Inter J Comparative Educ Dev 19(1):20\u201334","journal-title":"Inter J Comparative Educ Dev"},{"issue":"CSCW1","key":"861_CR3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3449148","volume":"5","author":"J Bandy","year":"2021","unstructured":"Bandy J (2021) Problematic machine behavior: A systematic literature review of algorithm audits. Proc. ACM Hum. Comput. Interact. 5(CSCW1):1\u201334","journal-title":"Proc. ACM Hum. Comput. Interact."},{"issue":"104","key":"861_CR4","first-page":"671","volume":"671","author":"S Barocas","year":"2016","unstructured":"Barocas S, Selbst AD (2016) Big Data\u2019s Disparate Impact. Calif Law Rev 671(104):671\u2013732","journal-title":"Calif Law Rev"},{"issue":"3","key":"861_CR5","doi-asserted-by":"publisher","first-page":"17","DOI":"10.3917\/hori.005.0017","volume":"5","author":"M Bendick","year":"2007","unstructured":"Bendick M (2007) Situation Testing for Employment Discrimination in the United States of America. Horizons strategiques 5(3):17\u201339","journal-title":"Horizons strategiques"},{"issue":"1","key":"861_CR6","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","volume":"57","author":"Y Benjamini","year":"1995","unstructured":"Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc: Ser B (Methodol) 57(1):289\u2013300","journal-title":"J Roy Stat Soc: Ser B (Methodol)"},{"issue":"2","key":"861_CR7","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1007\/s13524-012-0151-7","volume":"50","author":"FD Blau","year":"2013","unstructured":"Blau FD, Brummund P, Liu AY-H (2013) Trends in Occupational Segregation by Gender 1970\u20132009: Adjusting for the Impact of Changes in the Occupational Coding System. Demography 50(2):471\u2013494","journal-title":"Demography"},{"key":"861_CR8","volume-title":"Help Wanted: An Exploration of Hiring Algorithms","author":"M Bogen","year":"2018","unstructured":"Bogen M, Rieke A (2018) Help Wanted: An Exploration of Hiring Algorithms. Equity and Bias, Technical report, Upturn"},{"issue":"1","key":"861_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/2053951720983865","volume":"8","author":"S Brown","year":"2021","unstructured":"Brown S, Davidovic J, Hasan A (2021) The algorithm audit: Scoring the algorithms that score us. Big Data Soc 8(1):1\u20138","journal-title":"Big Data Soc"},{"key":"861_CR10","unstructured":"California Civil Code (2018). Title 1.81.5. California Consumer Privacy Act of 2018"},{"issue":"2","key":"861_CR11","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1111\/j.1745-3984.1976.tb00003.x","volume":"13","author":"J Cardinet","year":"1976","unstructured":"Cardinet J, Tourneur Y, Allal L (1976) The Symmetry of Generalizability Theory: Applications to Educational Measurement. J Educ Meas 13(2):119\u2013135","journal-title":"J Educ Meas"},{"key":"861_CR12","doi-asserted-by":"crossref","unstructured":"Carmines E, Zeller R (1979) Reliability and Validity Assessment. SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States of America","DOI":"10.4135\/9781412985642"},{"issue":"3","key":"861_CR13","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1017\/iop.2016.6","volume":"9","author":"T Chamorro-Premuzic","year":"2016","unstructured":"Chamorro-Premuzic T, Winsborough D, Sherman RA, Hogan R (2016) New Talent Signals: Shiny New Objects or a Brave New World? Ind Organ Psychol 9(3):621\u2013640","journal-title":"Ind Organ Psychol"},{"key":"861_CR14","doi-asserted-by":"crossref","unstructured":"Chen L, Ma R, Hann\u00e1k A, Wilson C (2018) Investigating the Impact of Gender on Rank in Resume Search Engines. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp 1\u201314. ACM","DOI":"10.1145\/3173574.3174225"},{"key":"861_CR15","unstructured":"Dastin J (2018) Amazon scraps secret AI recruiting tool that showed bias against women. Reuters"},{"issue":"1","key":"861_CR16","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1515\/popets-2015-0007","volume":"2015","author":"A Datta","year":"2015","unstructured":"Datta A, Tschantz MC, Datta A (2015) Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. Proc Privacy Enhancing Technol 2015(1):92\u2013112","journal-title":"Proc Privacy Enhancing Technol"},{"key":"861_CR17","doi-asserted-by":"crossref","unstructured":"De-Arteaga M, Romanov A, Wallach HM, Chayes JT, Borgs C, Chouldechova A, Geyik SC, Kenthapadi K, Kalai AT (2019) Bias in bios: A case study of semantic representation bias in a high-stakes setting. In: danah boyd and Morgenstern, J.\u00a0H., editors, Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* 2019, pp 120\u2013128. ACM","DOI":"10.1145\/3287560.3287572"},{"key":"861_CR18","volume-title":"The Personality Brokers: The Strange History of Myers-Briggs and the Birth of Personality Testing","author":"M Emre","year":"2018","unstructured":"Emre M (2018) The Personality Brokers: The Strange History of Myers-Briggs and the Birth of Personality Testing, 1st edn. Doubleday, New York","edition":"1"},{"key":"861_CR19","unstructured":"Equal Employment Opportunity Commission (EEOC), Civil Service Commission, Department of Labor, and Department of Justice (1978). Uniform guidelines on employee selection procedures. Federal Register"},{"key":"861_CR20","unstructured":"European Commission (2021) Better regulation toolbox. https:\/\/ec.europa.eu\/info\/law\/law-making-process\/planning-and-proposing-law\/better-regulation-why-and-how\/better-regulation-guidelines-and-toolbox\/better-regulation-toolbox_en; Accessed on 07\/29\/2022"},{"key":"861_CR21","doi-asserted-by":"crossref","unstructured":"Fjeld J, Achten N, Hilligoss H, Nagy A, Srikumar M (2020) Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI. SSRN Electronic Journal","DOI":"10.2139\/ssrn.3518482"},{"key":"861_CR22","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1016\/j.ecolind.2016.04.014","volume":"69","author":"A Galli","year":"2016","unstructured":"Galli A, Giampietro M, Goldfinger S, Lazarus E, Lin D, Saltelli A, Wackernagel M, M\u00fcller F (2016) Questioning the Ecological Footprint. Ecol Ind 69:224\u2013232","journal-title":"Ecol Ind"},{"issue":"3","key":"861_CR23","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1023\/A:1022941331649","volume":"13","author":"LD Goodstein","year":"1999","unstructured":"Goodstein LD, Lanyon RI (1999) Applications of Personality Assessment to the Workplace: A Review. J Bus Psychol 13(3):32","journal-title":"J Bus Psychol"},{"key":"861_CR24","doi-asserted-by":"crossref","unstructured":"Guion RM, Gottier RF (1965) Validity Of Personality Measures In Personnel Selection. Personnel Psychology, 18(2):135\u2013164. _eprint: https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/j.1744-6570.1965.tb00273.x","DOI":"10.1111\/j.1744-6570.1965.tb00273.x"},{"issue":"4","key":"861_CR25","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1017\/iop.2015.40","volume":"8","author":"RA Guzzo","year":"2015","unstructured":"Guzzo RA, Fink AA, King E, Tonidandel S, Landis RS (2015) Big Data Recommendations for Industrial-Organizational Psychology. Ind Organ Psychol 8(4):491\u2013508","journal-title":"Ind Organ Psychol"},{"issue":"1","key":"861_CR26","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1007\/s11023-020-09517-8","volume":"30","author":"T Hagendorff","year":"2020","unstructured":"Hagendorff T (2020) The Ethics of AI Ethics: An Evaluation of Guidelines. Mind Mach 30(1):99\u2013120","journal-title":"Mind Mach"},{"key":"861_CR27","doi-asserted-by":"crossref","unstructured":"Hann\u00e1k A, Wagner C, Garcia D, Mislove A, Strohmaier M, Wilson C (2017) Bias in Online Freelance Marketplaces: Evidence from TaskRabbit and Fiverr. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pp 1914\u20131933, Portland Oregon USA. ACM","DOI":"10.1145\/2998181.2998327"},{"key":"861_CR28","doi-asserted-by":"crossref","unstructured":"Hegewisch A, Liepmann H, Hayes J, Hartmann H (2010) Separate and Not Equal? Gender Segregation in the Labor Market and the Gender Wage Gap. Technical report, Institute for Women\u2019s Policy Research","DOI":"10.1037\/e686432011-001"},{"issue":"5","key":"861_CR29","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1037\/0021-9010.75.5.581","volume":"75","author":"LM Hough","year":"1990","unstructured":"Hough LM, Eaton NK, Dunnette MD, Kamp JD, McCloy RA (1990) Criterion-related validities of personality constructs and the effect of response distortion on those validities. J Appl Psychol 75(5):581\u2013595","journal-title":"J Appl Psychol"},{"issue":"6","key":"861_CR30","doi-asserted-by":"publisher","first-page":"869","DOI":"10.1037\/0021-9010.85.6.869","volume":"85","author":"GM Hurtz","year":"2000","unstructured":"Hurtz GM, Donovan JJ (2000) Personality and job performance: The Big Five revisited. J Appl Psychol 85(6):869\u2013879","journal-title":"J Appl Psychol"},{"key":"861_CR31","volume-title":"Challenging Biased Hiring Algorithms. SSRN Scholarly Paper ID 3744248","author":"A Kelly-Lyth","year":"2020","unstructured":"Kelly-Lyth A (2020) Challenging Biased Hiring Algorithms. SSRN Scholarly Paper ID 3744248. Social Science Research Network, Rochester, NY"},{"key":"861_CR32","first-page":"81","volume":"58","author":"PT Kim","year":"2017","unstructured":"Kim PT (2017) Data-Driven Discrimination at Work. William & Mary Law 58:81","journal-title":"William & Mary Law"},{"issue":"3","key":"861_CR33","doi-asserted-by":"publisher","first-page":"795","DOI":"10.1007\/s40685-020-00134-w","volume":"13","author":"A K\u00f6chling","year":"2020","unstructured":"K\u00f6chling A, Wehner MC (2020) Discriminated by an algorithm: A systematic review of discrimination and fairness by algorithmic decision-making in the context of hr recruitment and hr development. Bus Res 13(3):795\u2013848","journal-title":"Bus Res"},{"key":"861_CR34","volume-title":"Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI","author":"A Koshiyama","year":"2021","unstructured":"Koshiyama A, Kazim E, Treleaven P, Rai P, Szpruch L, Pavey G, Ahamat G, Leutner F, Goebel R, Knight A, Adams J, Hitrova C, Barnett J, Nachev P, Barber D, Chamorro-Premuzic T, Klemmer K, Gregorovic M, Khan S, Lomas E (2021) Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI. ML and Associated Algorithms, Technical report"},{"key":"861_CR35","doi-asserted-by":"crossref","unstructured":"Langenkamp M, Costa A, Cheung C (2020) Hiring Fairly in the Age of Algorithms. arXiv:2004.07132 [cs]. arXiv: 2004.07132","DOI":"10.2139\/ssrn.3723046"},{"key":"861_CR36","doi-asserted-by":"crossref","unstructured":"Lo\u00a0Piano S, Robinson M (2019) Nutrition and public health economic evaluations under the lenses of post normal science | Elsevier Enhanced Reader. Futures, 112","DOI":"10.1016\/j.futures.2019.06.008"},{"issue":"2","key":"861_CR37","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1037\/hop0000081","volume":"21","author":"K Lussier","year":"2018","unstructured":"Lussier K (2018) Temperamental workers: Psychology, business, and the Humm-Wadsworth Temperament Scale in interwar America. Hist Psychol 21(2):79","journal-title":"Hist Psychol"},{"key":"861_CR38","unstructured":"Meinert D (2015) What Do Personality Tests Really Reveal? HR Magazine, SHRM. https:\/\/www.shrm.org\/hr-today\/news\/hr-magazine\/pages\/0615-personality-tests.aspx; accessed on 07\/29\/2022"},{"key":"861_CR39","doi-asserted-by":"crossref","unstructured":"Metcalf J, Moss E, Watkins EA, Singh R, Elish MC (2021) Algorithmic impact assessments and accountability: The co-construction of impacts. In: Elish MC, Isaac W, Zemel RS (eds) FAccT \u201921: 2021 ACM Conference on Fairness, Accountability, and Transparency. ACM, New York, NY, pp 735\u2013746","DOI":"10.1145\/3442188.3445935"},{"issue":"3","key":"861_CR40","doi-asserted-by":"publisher","first-page":"683","DOI":"10.1111\/j.1744-6570.2007.00089.x","volume":"60","author":"FP Morgeson","year":"2007","unstructured":"Morgeson FP, Campion MA, Dipboye RL, Hollenbeck JR, Murphy K, Schmitt N (2007) Reconsidering the use of personality tests in personnel selection contexts. Pers Psychol 60(3):683\u2013729","journal-title":"Pers Psychol"},{"issue":"3","key":"861_CR41","doi-asserted-by":"publisher","first-page":"352","DOI":"10.1080\/02701367.1993.10608821","volume":"64","author":"JR Morrow","year":"1993","unstructured":"Morrow JR, Jackson Aw (1993) How Significant is Your Reliability? Res Q Exerc Sport 64(3):352\u2013355","journal-title":"Res Q Exerc Sport"},{"key":"861_CR42","doi-asserted-by":"crossref","unstructured":"Mueller RO, Knapp TR (2018) Reliability and Validity. In: The Reviewer\u2019s Guide to Quantitative Methods in the Social Sciences. Routledge, 2 edition","DOI":"10.4324\/9781315755649-29"},{"issue":"4","key":"861_CR43","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1007\/s11948-021-00319-4","volume":"27","author":"J M\u00f6kander","year":"2021","unstructured":"M\u00f6kander J, Morley J, Taddeo M, Floridi L (2021) Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Sci Eng Ethics 27(4):44","journal-title":"Sci Eng Ethics"},{"key":"861_CR44","unstructured":"New York City Council (2021). Local Law 144 to amend the administrative code of the city of New York, in relation to automated employment decision tools"},{"key":"861_CR45","volume-title":"Psychometric Theory","author":"JC Nunnally","year":"1994","unstructured":"Nunnally JC, Bernstein IH (1994) Psychometric Theory, 3rd edn. McGraw Hill, New York, NY","edition":"3"},{"key":"861_CR46","unstructured":"Oala L, Fehr J, Gilli L, Balachandran P, Leite AW, Ram\u00edrez SC, Li DX, Nobis G, Alvarado EAM, Jaramillo-Gutierrez G, Matek C, Shroff A, Kherif F, Sanguinetti B, Wiegand T (2020) ML4H auditing: From paper to practice. In: Alsentzer E, McDermott MBA, Falck F, Sarkar SK, Roy S, Hyland SL, editors, Machine Learning for Health Workshop, ML4H@NeurIPS, volume 136 of Proceedings of Machine Learning Research, pages 280\u2013317. PMLR"},{"key":"861_CR47","unstructured":"ORCAA (2020) Description of Algorithmic Audit: Pre-built Assessments. Technical report"},{"key":"861_CR48","doi-asserted-by":"crossref","unstructured":"Raghavan M, Barocas S, Kleinberg J, Levy K (2020) Mitigating bias in algorithmic hiring: evaluating claims and practices. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp 469\u2013481. ACM","DOI":"10.1145\/3351095.3372828"},{"key":"861_CR49","doi-asserted-by":"crossref","unstructured":"Raji ID, Smart A, White RN, Mitchell M, Gebru T, Hutchinson B, Smith-Loud J, Theron D, Barnes P (2020) Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In: Hildebrandt M, Castillo C, Celis LE, Ruggieri S, Taylor L, Zanfir-Fortuna G (eds) FAT* \u201920: Conference on Fairness, Accountability, and Transparency. ACM, New York, NY, pp 33\u201344","DOI":"10.1145\/3351095.3372873"},{"key":"861_CR50","doi-asserted-by":"publisher","DOI":"10.1016\/j.envsoft.2020.104954","volume":"137","author":"S Razavi","year":"2021","unstructured":"Razavi S, Jakeman A, Saltelli A, Prieur C, Iooss B, Borgonovo E, Plischke E, Lo Piano S, Iwanaga T, Becker W, Tarantola S, Guillaume JHA, Jakeman J, Gupta H, Melillo N, Rabitti G, Chabridon V, Duan Q, Sun X, Smith S, Sheikholeslami R, Hosseini N, Asadzadeh M, Puy A, Kucherenko S, Maier HR (2021) The Future of Sensitivity Analysis: An essential discipline for systems modeling and policy support. Environ Model Software 137:104954","journal-title":"Environ Model Software"},{"key":"861_CR51","unstructured":"Riksrevisjonen (2020) Auditing machine learning algorithms: Report by the Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK. https:\/\/www.auditingalgorithms.net\/; Accessed on 07\/29\/2022"},{"issue":"CSCW","key":"861_CR52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3274417","volume":"2","author":"RE Robertson","year":"2018","unstructured":"Robertson RE, Jiang S, Joseph K, Friedland L, Lazer D, Wilson C (2018) Auditing Partisan Audience Bias within Google Search. Proc ACM Human-Comput Interaction 2(CSCW):1\u201322","journal-title":"Proc ACM Human-Comput Interaction"},{"issue":"7813","key":"861_CR53","doi-asserted-by":"publisher","first-page":"482","DOI":"10.1038\/d41586-020-01812-9","volume":"582","author":"A Saltelli","year":"2020","unstructured":"Saltelli A, Bammer G, Bruno I, Charters E, Di Fiore M, Didier E, Nelson Espeland W, Kay J, Lo Piano S, Mayo D, Pielke R Jr, Portaluri T, Porter TM, Puy A, Rafols I, Ravetz JR, Reinert E, Sarewitz D, Stark PB, Stirling A, van der Sluijs J, Vineis P (2020) Five ways to ensure that models serve society: a manifesto. Nature 582(7813):482\u2013484","journal-title":"Nature"},{"issue":"2","key":"861_CR54","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1007\/s41055-017-0020-6","volume":"1","author":"A Saltelli","year":"2017","unstructured":"Saltelli A, Lo Piano S (2017) Problematic Quantifications: a Critical Appraisal of Scenario Making for a Global Sustainable Food Production. Food Ethics 1(2):173\u2013179","journal-title":"Food Ethics"},{"key":"861_CR55","unstructured":"Sandvig C, Hamilton K, Karahalios K, Langbort C (2014) Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. In: Data and Discrimination: Converting Critical Concerns into Productive Inquiry, p\u00a023, Seattle, WA, USA"},{"key":"861_CR56","unstructured":"Schellmann H, Strong J, Siegel I (2021a) Hired by an algorithm, In: Machines We Trust podcast series, MIT Technology Review. issued: 2021-06-23"},{"key":"861_CR57","unstructured":"Schellmann H, Strong J, Siegel I (2021b) Want a job? The AI will see you now. In: Machines We Trust podcast series, MIT Technology Review. issued: 2021-07-07"},{"issue":"2","key":"861_CR58","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1037\/1082-989X.8.2.206","volume":"8","author":"FL Schmidt","year":"2003","unstructured":"Schmidt FL, Le H, Ilies R (2003) Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs. Psychol Methods 8(2):206","journal-title":"Psychol Methods"},{"key":"861_CR59","doi-asserted-by":"crossref","unstructured":"Schmitt N, Gooding RZ, Noe RA, Kirsch M (1984) Metaanalyses of Validity Studies Published Between 1964 and 1982 and the Investigation of Study Characteristics. Personnel Psychology, 37(3):407\u2013422. _eprint: https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/j.1744-6570.1984.tb00519.x","DOI":"10.1111\/j.1744-6570.1984.tb00519.x"},{"key":"861_CR60","unstructured":"Science Advice for Policy by European Academies (SAPEA) (2019) Making sense of science for policy under conditions of complexity and uncertainty. Science Advice for Policy by European Academies, DE"},{"issue":"1","key":"861_CR61","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1177\/009102600803700107","volume":"37","author":"WA Scroggins","year":"2008","unstructured":"Scroggins WA, Thomas SL, Morris JA (2008) Psychological Testing in Personnel Selection, Part I: A Century of Psychological Testing. Public Personnel Manag 37(1):99\u2013109","journal-title":"Public Personnel Manag"},{"key":"861_CR62","doi-asserted-by":"crossref","unstructured":"Sharma S, Henderson J, Ghosh J (2020) CERTIFAI: A Common Framework to Provide Explanations and Analyse the Fairness and Robustness of Black-box Models. In: Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, pp 166\u2013172. ACM","DOI":"10.1145\/3375627.3375812"},{"issue":"4","key":"861_CR63","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3419764","volume":"10","author":"B Shneiderman","year":"2020","unstructured":"Shneiderman B (2020) Bridging the Gap Between Ethics and Practice: Guidelines for Reliable, Safe, and Trustworthy Human-centered AI Systems. ACM Trans Interactive Intell Syst 10(4):1\u201331","journal-title":"ACM Trans Interactive Intell Syst"},{"key":"861_CR64","unstructured":"Sloane M (2021) The Algorithmic Auditing Trap. https:\/\/onezero.medium.com\/the-algorithmic-auditing-trap-9a6f2d4d461d; Accessed 07\/29\/2022"},{"issue":"2","key":"861_CR65","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2021.100425","volume":"3","author":"M Sloane","year":"2022","unstructured":"Sloane M, Moss E, Chowdhury R (2022) A silicon valley love triangle: Hiring algorithms, pseudo-science, and the quest for auditability. Patterns 3(2):100425","journal-title":"Patterns"},{"key":"861_CR66","doi-asserted-by":"crossref","unstructured":"Stark L, Hutson J (2021) Physiognomic Artificial Intelligence. SSRN Electronic Journal","DOI":"10.2139\/ssrn.3927300"},{"key":"861_CR67","unstructured":"Stoyanovich J (2021) Hiring and AI: Let Job Candidates Know Why They Were Rejected. The Wall Street Journal"},{"key":"861_CR68","doi-asserted-by":"crossref","unstructured":"S\u00fchr T, Hilgard S, Lakkaraju H (2021) Does Fair Ranking Improve Minority Outcomes? Understanding the Interplay of Human and Algorithmic Biases in Online Hiring. In: Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society, pp 989\u2013999. ACM","DOI":"10.1145\/3461702.3462602"},{"key":"861_CR69","unstructured":"The European Commission (2021) Proposal for a Regulation of the European Parliament and of the Council Laying down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts"},{"issue":"1\u20132","key":"861_CR70","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1007\/BF00993473","volume":"20","author":"P Turney","year":"1995","unstructured":"Turney P (1995) Technical note: Bias and the quantification of stability. Mach Learn 20(1\u20132):23\u201333","journal-title":"Mach Learn"},{"issue":"3","key":"861_CR71","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1093\/aje\/kwy250","volume":"188","author":"TJ VanderWeele","year":"2019","unstructured":"VanderWeele TJ, Mathur MB (2019) Some desirable properties of the Bonferroni correction: Is the Bonferroni correction really so bad? Am J Epidemiol 188(3):617\u2013618","journal-title":"Am J Epidemiol"},{"key":"861_CR72","unstructured":"Vecchione B, Levy K, Barocas S (2021) Algorithmic auditing and social justice: Lessons from the history of audit studies. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp 19:1\u201319:9. ACM"},{"key":"861_CR73","doi-asserted-by":"crossref","unstructured":"Venkatadri G, Andreou A, Liu Y, Mislove A, Gummadi KP, Loiseau P, Goga O (2018) Privacy Risks with Facebook\u2019s PII-Based Targeting: Auditing a Data Broker\u2019s Advertising Interface. In: 2018 IEEE Symposium on Security and Privacy (SP), pp 89\u2013107. ISSN: 2375-1207","DOI":"10.1109\/SP.2018.00014"},{"key":"861_CR74","unstructured":"Weber L, Dwoskin E (2014) Are Workplace Personality Tests Fair? Wall Street Journal"},{"key":"861_CR75","doi-asserted-by":"crossref","unstructured":"Wilson C, Ghosh A, Jiang S, Mislove A, Baker L, Szary J, Trindel K, Polli F (2021) Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp 666\u2013677. ACM","DOI":"10.1145\/3442188.3445928"},{"key":"861_CR76","unstructured":"Xue S, Yurochkin M, Sun Y (2020) Auditing ML Models for Individual Bias and Unfairness. In: International Conference on Artificial Intelligence and Statistics, pp 4552\u20134562. PMLR. ISSN: 2640-3498"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00861-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-022-00861-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-022-00861-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T01:34:51Z","timestamp":1728005691000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-022-00861-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,17]]},"references-count":76,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["861"],"URL":"https:\/\/doi.org\/10.1007\/s10618-022-00861-0","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"value":"1384-5810","type":"print"},{"value":"1573-756X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,17]]},"assertion":[{"value":"6 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 August 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"All procedures performed involving human participants were in accordance with the ethical standards of the NYU institutional review board and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Informed consent was obtained from all individual participants included in the study.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Participants granted informed consent to publish information not containing identifiers, including personality profile results and demographic data.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}