{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T02:21:29Z","timestamp":1768011689497,"version":"3.49.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"GROUP","license":[{"start":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T00:00:00Z","timestamp":1642118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100012774","name":"Innovation Fund Denmark","doi-asserted-by":"crossref","award":["7050- 00034A"],"award-info":[{"award-number":["7050- 00034A"]}],"id":[{"id":"10.13039\/100012774","id-type":"DOI","asserted-by":"crossref"}]},{"name":"The Independent Research Fund Denmark","award":["8091-00025b"],"award-info":[{"award-number":["8091-00025b"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,1,14]]},"abstract":"<jats:p>As more and more governments adopt algorithms to support bureaucratic decision-making processes, it becomes urgent to address issues of responsible use and accountability. We examine a contested public service algorithm used in Danish job placement for assessing an individual's risk of long-term unemployment. The study takes inspiration from cooperative audits and was carried out in dialogue with the Danish unemployment services agency. Our audit investigated the practical implementation of algorithms. We find (1) a divergence between the formal documentation and the model tuning code, (2) that the algorithmic model relies on subjectivity, namely the variable which focus on the individual's self-assessment of how long it will take before they get a job, (3) that the algorithm uses the variable \"origin\" to determine its predictions, and (4) that the documentation neglects to consider the implications of using variables indicating personal characteristics when predicting employment outcomes. We discuss the benefits and limitations of cooperative audits in a public sector context. We specifically focus on the importance of collaboration across different public actors when investigating the use of algorithms in the algorithmic society.<\/jats:p>","DOI":"10.1145\/3492827","type":"journal-article","created":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T21:26:16Z","timestamp":1642195576000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Auditing Risk Prediction of Long-Term Unemployment"],"prefix":"10.1145","volume":"6","author":[{"given":"Cathrine","family":"Seidelin","sequence":"first","affiliation":[{"name":"University of Copenhagen, Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Therese","family":"Moreau","sequence":"additional","affiliation":[{"name":"Zetland, Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Irina","family":"Shklovski","sequence":"additional","affiliation":[{"name":"University of Copenhagen, Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Naja","family":"Holten M\u00f8ller","sequence":"additional","affiliation":[{"name":"University of Copenhagen, Copenhagen, Denmark"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,1,14]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Austria: How Austerity Politics Are Made Effective. Frontiers in Big Data. 3","author":"Allhutter D.","year":"2020","unstructured":"Allhutter , D. , Cech , F. , Fischer , F. , Grill , G. and Mager , A . 2020 . Algorithmic Profiling of Job Seekers in Austria: How Austerity Politics Are Made Effective. Frontiers in Big Data. 3 , (2020), 5. DOI:https:\/\/doi.org\/10.3389\/fdata.2020.00005. 10.3389\/fdata.2020.00005 Allhutter, D., Cech, F., Fischer, F., Grill, G. and Mager, A. 2020. Algorithmic Profiling of Job Seekers in Austria: How Austerity Politics Are Made Effective. Frontiers in Big Data. 3, (2020), 5. DOI:https:\/\/doi.org\/10.3389\/fdata.2020.00005."},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1 (Apr.","author":"Ammitzb\u00f8ll Fl\u00fcgge A.","year":"2021","unstructured":"Ammitzb\u00f8ll Fl\u00fcgge , A. , Hildebrandt , T. and M\u00f8ller , N.H . 2021. Street-Level Algorithms and AI in Bureaucratic Decision-Making: A Caseworker Perspective . Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1 (Apr. 2021 ), 40:1--40:23. DOI:https:\/\/doi.org\/10.1145\/3449114. 10.1145\/3449114 Ammitzb\u00f8ll Fl\u00fcgge, A., Hildebrandt, T. and M\u00f8ller, N.H. 2021. Street-Level Algorithms and AI in Bureaucratic Decision-Making: A Caseworker Perspective. Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1 (Apr. 2021), 40:1--40:23. DOI:https:\/\/doi.org\/10.1145\/3449114."},{"key":"e_1_2_1_3_1","volume-title":"Jurist: Dataprofilering af langtidsledige med etnicitet er ulovlig. Version2.","author":"Andersen T.","year":"2019","unstructured":"Andersen , T. 2019 . Jurist: Dataprofilering af langtidsledige med etnicitet er ulovlig. Version2. Andersen, T. 2019. Jurist: Dataprofilering af langtidsledige med etnicitet er ulovlig. Version2."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449148"},{"key":"e_1_2_1_5_1","volume-title":"Race after technology: abolitionist tools for the new Jim code","author":"Benjamin R.","unstructured":"Benjamin , R. 2019. Race after technology: abolitionist tools for the new Jim code . Polity . Benjamin, R. 2019. Race after technology: abolitionist tools for the new Jim code. Polity."},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"Bright J. Ganesh B. Seidelin C. and Vogl T.M. 2019. Data Science for Local Government. Technical Report #ID 3370217. Social Science Research Network.  Bright J. Ganesh B. Seidelin C. and Vogl T.M. 2019. Data Science for Local Government. Technical Report #ID 3370217. Social Science Research Network.","DOI":"10.2139\/ssrn.3370217"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720983865"},{"key":"e_1_2_1_8_1","volume-title":"Atlas of Ai: power, politics, and the planetary costs of artificial intelligence","author":"Crawford K.","unstructured":"Crawford , K. 2021. Atlas of Ai: power, politics, and the planetary costs of artificial intelligence . Yale University Press . Crawford, K. 2021. Atlas of Ai: power, politics, and the planetary costs of artificial intelligence. Yale University Press."},{"key":"e_1_2_1_9_1","unstructured":"Danish Human Rights Institute 2020. Klage til Ligebehandlingsn\u00e6vnet.  Danish Human Rights Institute 2020. Klage til Ligebehandlingsn\u00e6vnet."},{"key":"e_1_2_1_10_1","unstructured":"Danske A-kasser 2019. Hvis sagsbehandleren stoler blindt p\u00e5 algoritmerne risikerer han at stemple de forkerte | Danske A-kasser.  Danske A-kasser 2019. Hvis sagsbehandleren stoler blindt p\u00e5 algoritmerne risikerer han at stemple de forkerte | Danske A-kasser."},{"key":"e_1_2_1_11_1","volume-title":"Meet the Researchers Fighting to Make Sure Artificial Intelligence Is a Force for Good. Time.","author":"De La Garza A.","year":"2019","unstructured":"De La Garza , A. 2019 . Meet the Researchers Fighting to Make Sure Artificial Intelligence Is a Force for Good. Time. De La Garza, A. 2019. Meet the Researchers Fighting to Make Sure Artificial Intelligence Is a Force for Good. Time."},{"key":"e_1_2_1_12_1","volume-title":"The stuff of bits: an essay on the materialities of information","author":"Dourish P.","unstructured":"Dourish , P. 2017. The stuff of bits: an essay on the materialities of information . The MIT Press . Dourish, P. 2017. The stuff of bits: an essay on the materialities of information. The MIT Press."},{"key":"e_1_2_1_13_1","volume-title":"Policing Predictive Policing","author":"Ferguson A.G.","year":"2017","unstructured":"Ferguson , A.G. 2017. Policing Predictive Policing . Washington University Law Review . 94, ( 2017 ), 82. Ferguson, A.G. 2017. Policing Predictive Policing. Washington University Law Review. 94, (2017), 82."},{"key":"e_1_2_1_14_1","unstructured":"H\u00f8jbjerg Jacobsen R. and Bendix Kleif H. 2016. Effekten af indsatser for langtidsledige og ledige i risiko for langtidsledighed: litteraturreview. KORA.  H\u00f8jbjerg Jacobsen R. and Bendix Kleif H. 2016. Effekten af indsatser for langtidsledige og ledige i risiko for langtidsledighed: litteraturreview. KORA."},{"key":"e_1_2_1_15_1","unstructured":"Holten M\u00f8ller N. Nielsen T.R. and Le Dantec C.A. 2021. Work of the Unemployed. An inquiry into individuals' experience of data usage in public services and possibilities for their agency. (2021).  Holten M\u00f8ller N. Nielsen T.R. and Le Dantec C.A. 2021. Work of the Unemployed. An inquiry into individuals' experience of data usage in public services and possibilities for their agency. (2021)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419249.3420149"},{"key":"e_1_2_1_17_1","unstructured":"Immigrants and Descendants: https:\/\/www.dst.dk\/da\/Statistik\/emner\/befolkning-og-valg\/indvandrere-og-efterkommere\/indvandrere-og-efterkommere. Accessed: 2021-08-09.  Immigrants and Descendants: https:\/\/www.dst.dk\/da\/Statistik\/emner\/befolkning-og-valg\/indvandrere-og-efterkommere\/indvandrere-og-efterkommere. Accessed: 2021-08-09."},{"key":"e_1_2_1_18_1","unstructured":"Kulager F. 2021. Kan algoritmer se ind i et barns fremtid? I Hj\u00f8rring og Silkeborg eksperimenterede man p\u00e5 udsatte b\u00f8rn. Zetland.  Kulager F. 2021. Kan algoritmer se ind i et barns fremtid? I Hj\u00f8rring og Silkeborg eksperimenterede man p\u00e5 udsatte b\u00f8rn. Zetland."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.giq.2008.12.009"},{"key":"e_1_2_1_20_1","unstructured":"Marie H. Blinde vinkler.  Marie H. Blinde vinkler."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1080\/01900692.2019.1575664"},{"key":"e_1_2_1_22_1","unstructured":"Mittelstadt B.D. Allo P. Taddeo M. Wachter S. and Floridi L. The ethics of algorithms: Mapping the debate. Big Data. 21.  Mittelstadt B.D. Allo P. Taddeo M. Wachter S. and Floridi L. The ethics of algorithms: Mapping the debate. Big Data. 21."},{"key":"e_1_2_1_23_1","unstructured":"Moreau T. and Kulager F. 2021. Vi har skilt jobcentrenes algoritme ad. Og kan se hvor anderledes indvandrere og efterkommere bliver behandlet. Zetland.  Moreau T. and Kulager F. 2021. Vi har skilt jobcentrenes algoritme ad. Og kan se hvor anderledes indvandrere og efterkommere bliver behandlet. Zetland."},{"key":"e_1_2_1_24_1","unstructured":"OECD Glossary of Statistical Terms - Long-term unemployment Definition: 2002. https:\/\/stats.oecd.org\/glossary\/detail.asp?ID=3586. Accessed: 2021-08-06.  OECD Glossary of Statistical Terms - Long-term unemployment Definition: 2002. https:\/\/stats.oecd.org\/glossary\/detail.asp?ID=3586. Accessed: 2021-08-06."},{"key":"e_1_2_1_25_1","volume-title":"New laws of robotics: defending human expertise in the age of AI","author":"Pasquale F.","unstructured":"Pasquale , F. 2020. New laws of robotics: defending human expertise in the age of AI . The Belknap Press of Harvard University Press . Pasquale, F. 2020. New laws of robotics: defending human expertise in the age of AI. The Belknap Press of Harvard University Press."},{"key":"e_1_2_1_26_1","volume-title":"The Black box society: the secret algorithms that control money and information","author":"Pasquale F.","unstructured":"Pasquale , F. 2015. The Black box society: the secret algorithms that control money and information . Harvard University Press . Pasquale, F. 2015. The Black box society: the secret algorithms that control money and information. Harvard University Press."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1 (Apr.","author":"Petersen A.C.M.","year":"2021","unstructured":"Petersen , A.C.M. , Christensen , L.R. , Harper , R. and Hildebrandt , T . 2021. \"We Would Never Write That Down\": Classifications of Unemployed and Data Challenges for AI . Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1 (Apr. 2021 ), 102:1--102:26. DOI:https:\/\/doi.org\/10.1145\/3449176. 10.1145\/3449176 Petersen, A.C.M., Christensen, L.R., Harper, R. and Hildebrandt, T. 2021. \"We Would Never Write That Down\": Classifications of Unemployed and Data Challenges for AI. Proceedings of the ACM on Human-Computer Interaction. 5, CSCW1 (Apr. 2021), 102:1--102:26. DOI:https:\/\/doi.org\/10.1145\/3449176."},{"key":"e_1_2_1_28_1","volume-title":"A., White, R.N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron","author":"Raji I.D.","year":"2020","unstructured":"Raji , I.D. , Smart , A., White, R.N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron , D. and Barnes, P. 2020 . Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing . arXiv:2001.00973 [cs]. (Jan. 2020). Raji, I.D., Smart, A., White, R.N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D. and Barnes, P. 2020. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. arXiv:2001.00973 [cs]. (Jan. 2020)."},{"key":"e_1_2_1_29_1","volume-title":"USA","author":"Sandvig C.","year":"2014","unstructured":"Sandvig , C. , Hamilton , K. , Karahalios , K. and Langbort , C . 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (Seattle, WA , USA , 2014 ), 4349--4357. Sandvig, C., Hamilton, K., Karahalios, K. and Langbort, C. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. (Seattle, WA, USA, 2014), 4349--4357."},{"key":"e_1_2_1_31_1","unstructured":"Sorgenfri Kj\u00e6r J. and Danielsen L. 2019. \u00bbDet er jo v\u00e6rre end sagen om Gladsaxe-modellen det her\u00ab: Algoritmer skal udpege langtidsledige. Politiken.  Sorgenfri Kj\u00e6r J. and Danielsen L. 2019. \u00bbDet er jo v\u00e6rre end sagen om Gladsaxe-modellen det her\u00ab: Algoritmer skal udpege langtidsledige. Politiken."},{"key":"e_1_2_1_32_1","unstructured":"Statistics Denmark 2018. Bilag 4 - Notat om ny landegruppering.  Statistics Denmark 2018. Bilag 4 - Notat om ny landegruppering."},{"key":"e_1_2_1_33_1","unstructured":"The Danish Agency for Labour Market and Recruitment 2019. 3.3 Modeldokumentation.  The Danish Agency for Labour Market and Recruitment 2019. 3.3 Modeldokumentation."},{"key":"e_1_2_1_34_1","unstructured":"The Danish Agency for Labour Market and Recruitment 2016. Effekten af indsatser for langtidsledige og ledige i risiko for langtidsledighed: litteraturreview. KORA.  The Danish Agency for Labour Market and Recruitment 2016. Effekten af indsatser for langtidsledige og ledige i risiko for langtidsledighed: litteraturreview. KORA."},{"key":"e_1_2_1_35_1","unstructured":"The Danish Agency for Labour Market and Recruitment 2019. Notat - Kronologisk oversigt over udviklingen af det digitale afklarings-og dialogv\u00e6rkt\u00f8j p\u00e5 Jobnet.  The Danish Agency for Labour Market and Recruitment 2019. Notat - Kronologisk oversigt over udviklingen af det digitale afklarings-og dialogv\u00e6rkt\u00f8j p\u00e5 Jobnet."},{"key":"e_1_2_1_36_1","unstructured":"The Danish Agency for Labour Market and Recruitment C.S. 2020. Beskrivelse af profilafklaringsv\u00e6rkt\u00f8jet til dagpengemodtagere.  The Danish Agency for Labour Market and Recruitment C.S. 2020. Beskrivelse af profilafklaringsv\u00e6rkt\u00f8jet til dagpengemodtagere."},{"key":"e_1_2_1_37_1","first-page":"59","article-title":"Improving Evaluability Through Self-Evaluation","volume":"16","author":"Usher C.L.","year":"1995","unstructured":"Usher , C.L. 1995 . Improving Evaluability Through Self-Evaluation . American Journal of Evaluation. 16 , 1 (1995), 59 -- 68 . Usher, C.L. 1995. Improving Evaluability Through Self-Evaluation. American Journal of Evaluation. 16, 1 (1995), 59--68.","journal-title":"American Journal of Evaluation."},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","unstructured":"Veale M. and Brass I. 2019. Administration by Algorithm? Public Management Meets Public Sector Machine Learning. Technical Report #ID 3375391. Social Science Research Network.  Veale M. and Brass I. 2019. Administration by Algorithm? Public Management Meets Public Sector Machine Learning. Technical Report #ID 3375391. Social Science Research Network.","DOI":"10.31235\/osf.io\/mwhnb"},{"key":"e_1_2_1_39_1","volume-title":"Algorithmic Bureaucracy. 20th Annual International Conference on Digital Government Research on - dg.o 2019 (Dubai, United Arab Emirates","author":"Vogl T.","year":"2019","unstructured":"Vogl , T. , Seidelin , C. , Ganesh , B. and Bright , J . 2019 . Algorithmic Bureaucracy. 20th Annual International Conference on Digital Government Research on - dg.o 2019 (Dubai, United Arab Emirates , 2019 ), 148--153. Vogl, T., Seidelin, C., Ganesh, B. and Bright, J. 2019. Algorithmic Bureaucracy. 20th Annual International Conference on Digital Government Research on - dg.o 2019 (Dubai, United Arab Emirates, 2019), 148--153."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1111\/puar.13286"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445928"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3492827","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3492827","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:48:26Z","timestamp":1750193306000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3492827"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,14]]},"references-count":40,"journal-issue":{"issue":"GROUP","published-print":{"date-parts":[[2022,1,14]]}},"alternative-id":["10.1145\/3492827"],"URL":"https:\/\/doi.org\/10.1145\/3492827","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,14]]},"assertion":[{"value":"2022-01-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}