{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T16:47:36Z","timestamp":1770742056304,"version":"3.49.0"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Digital Transformation and Innovation Center sponsored by PwC","award":["A026204"],"award-info":[{"award-number":["A026204"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>As algorithmic risk assessment instruments (RAIs) are increasingly adopted to assist decision makers, their predictive performance and potential to promote inequity have come under scrutiny. However, while most studies examine these tools in isolation, researchers have come to recognize that assessing their impact requires understanding the behavior of their human interactants. In this paper, building off of several recent crowdsourcing works focused on criminal justice, we conduct a vignette study in which laypersons are tasked with predicting future re-arrests. Our key findings are as follows: (1) Participants often predict that an offender will be rearrested even when they deem the likelihood of re-arrest to be well below 50%; (2) Participants do not anchor on the RAI's predictions; (3) The time spent on the survey varies widely across participants and most cases are assessed in less than 10 seconds; (4) Judicial decisions, unlike participants' predictions, depend in part on factors that are orthogonal to the likelihood of re-arrest. These results highlight the influence of several crucial but often overlooked design decisions and concerns around generalizability when constructing crowdsourcing studies to analyze the impacts of RAI<\/jats:p>","DOI":"10.1145\/3479572","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:46:19Z","timestamp":1634611579000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies"],"prefix":"10.1145","volume":"5","author":[{"given":"Riccardo","family":"Fogliato","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Alexandra","family":"Chouldechova","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Zachary","family":"Lipton","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Alex Albright. 2019. If You Give a Judge a Risk Score: Evidence from Kentucky Bail Decisions. Technical Report. Working paper."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3290419"},{"key":"e_1_2_2_3_1","volume-title":"Machine bias. ProPublica. See https:\/\/www. propublica. org\/article\/machine-bias-risk-assessments-in-criminal-sentencing","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica. See https:\/\/www. propublica. org\/article\/machine-bias-risk-assessments-in-criminal-sentencing (2016)."},{"key":"e_1_2_2_4_1","unstructured":"Dean Attali. 2020. shinyjs: Easily Improve the User Experience of Your Shiny Apps in Seconds. https:\/\/CRAN.R-project.org\/package=shinyjs R package version 1.1."},{"key":"e_1_2_2_5_1","volume-title":"Why Bail Reform is Safe and Effective: The Case of Cook County. Available at SSRN 3599410","author":"Austin James","year":"2020","unstructured":"James Austin and Wendy Ware. 2020. Why Bail Reform is Safe and Effective: The Case of Cook County. Available at SSRN 3599410 (2020)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445717"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11292-017-9286-2"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/0049124118782533"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3378393.3402507"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1093\/jnci\/87.10.720"},{"key":"e_1_2_2_11_1","volume-title":"Random forests. Machine learning","author":"Breiman Leo","year":"2001","unstructured":"Leo Breiman. 2001. Random forests. Machine learning, Vol. 45, 1 (2001), 5--32."},{"key":"e_1_2_2_12_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"5","author":"Zana Bucc","year":"2021","unstructured":"Zana Bucc inca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW1 (2021), 1--21."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1740-1461.2012.01254.x"},{"key":"e_1_2_2_14_1","unstructured":"Winston Chang Joe Cheng JJ Allaire Yihui Xie and Jonathan McPherson. 2020. shiny: Web Application Framework for R. https:\/\/CRAN.R-project.org\/package=shiny R package version 1.4.0.2."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_2_2_16_1","volume-title":"Conference on Fairness, Accountability and Transparency. 134--148","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. 134--148."},{"key":"e_1_2_2_17_1","volume-title":"Characterizing fairness over the set of good models under selective labels. arXiv preprint arXiv:2101.00352","author":"Coston Amanda","year":"2021","unstructured":"Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. 2021. Characterizing fairness over the set of good models under selective labels. arXiv preprint arXiv:2101.00352 (2021)."},{"key":"e_1_2_2_18_1","volume-title":"Designing effective web surveys","author":"Couper Mick P","unstructured":"Mick P Couper. 2008. Designing effective web surveys .Cambridge University Press."},{"key":"e_1_2_2_19_1","unstructured":"New Jersey Courts. 2018. Pretrial Release Recommendation Decision Making Framework (DMF). https:\/\/www.njcourts.gov\/courts\/assets\/criminal\/ decmakframwork.pdf?cacheID=nDrtJn4"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376638"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2010.06.003"},{"key":"e_1_2_2_22_1","volume-title":"The public safety assessment: A re-validation and assessment of predictive utility and differential prediction by race and gender in kentucky. Available at SSRN 3168452","author":"DeMichele Matthew","year":"2018","unstructured":"Matthew DeMichele, Peter Baumgartner, Michael Wenger, Kelle Barrick, Megan Comfort, and Shilpi Misra. 2018a. The public safety assessment: A re-validation and assessment of predictive utility and differential prediction by race and gender in kentucky. Available at SSRN 3168452 (2018)."},{"key":"e_1_2_2_23_1","volume-title":"The intuitive-override model: Nudging judges toward pretrial risk assessment instruments. Available at SSRN 3168500","author":"DeMichele Matthew","year":"2018","unstructured":"Matthew DeMichele, Megan Comfort, Shilpi Misra, Kelle Barrick, and Peter Baumgartner. 2018b. The intuitive-override model: Nudging judges toward pretrial risk assessment instruments. Available at SSRN 3168500 (2018)."},{"key":"e_1_2_2_24_1","volume-title":"Sarah E Duhart Clarke, and Evan M Lowder","author":"Desmarais Sarah L","year":"2020","unstructured":"Sarah L Desmarais, Samantha A Zottola, Sarah E Duhart Clarke, and Evan M Lowder. 2020. Predictive validity of pretrial risk assessments: A systematic review of the literature. Criminal Justice and Behavior (2020), 0093854820932959."},{"key":"e_1_2_2_25_1","unstructured":"William Dieterich Christina Mendoza and Tim Brennan. 2016. COMPAS Risk Scales: Demonstrating Accuracy Equity and Predictive Parity. (2016)."},{"key":"e_1_2_2_26_1","volume-title":"The accuracy, fairness, and limits of predicting recidivism. Science advances","author":"Dressel Julia","year":"2018","unstructured":"Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science advances, Vol. 4, 1 (2018), eaao5580."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/58.3.403"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10979-005-8380-7"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167205282152"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1559-1816.2001.tb02687.x"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.495"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780198525134.001.0001"},{"key":"e_1_2_2_33_1","volume-title":"Fairness Evaluation in Presence of Biased Noisy Labels. In International Conference on Artificial Intelligence and Statistics. PMLR, 2325--2336","author":"Fogliato Riccardo","year":"2020","unstructured":"Riccardo Fogliato, Alexandra Chouldechova, and Max G'Sell. 2020. Fairness Evaluation in Presence of Biased Noisy Labels. In International Conference on Artificial Intelligence and Statistics. PMLR, 2325--2336."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00057"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.539"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1037\/0096-3445.117.3.227"},{"key":"e_1_2_2_37_1","unstructured":"Glenn Grant. 2019. Report to the Governor and the Legislature. New Jersey Courts. (2019)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287563"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359152"},{"key":"e_1_2_2_40_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"3","author":"Nina","year":"2019","unstructured":"Nina Grgi\u0107-Hlavc a, Christoph Engel, and Krishna P Gummadi. 2019. Human decision making with machine assistance: An experiment on bailing and jailing. Proceedings of the ACM on Human-Computer Interaction, Vol. 3, CSCW (2019), 1--25."},{"key":"e_1_2_2_41_1","first-page":"777","article-title":"Inside the judicial mind","volume":"86","author":"Guthrie Chris","year":"2000","unstructured":"Chris Guthrie, Jeffrey J Rachlinski, and Andrew J Wistrich. 2000. Inside the judicial mind. Cornell L. Rev., Vol. 86 (2000), 777.","journal-title":"Cornell L. Rev."},{"key":"e_1_2_2_42_1","volume-title":"The use of risk assessment at sentencing: Implications for research and policy. Villanova Law\/Public Policy Research Paper 2017--1040","author":"Hyatt Jordan","year":"2016","unstructured":"Jordan Hyatt and Steven L Chanenson. 2016. The use of risk assessment at sentencing: Implications for research and policy. Villanova Law\/Public Policy Research Paper 2017--1040 (2016)."},{"key":"e_1_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Jongbin Jung Sharad Goel Jennifer Skeem et al. 2020. The limits of human predictions of recidivism. Science advances Vol. 6 7 (2020) eaaz0652.","DOI":"10.1126\/sciadv.aaz0652"},{"key":"e_1_2_2_44_1","doi-asserted-by":"crossref","unstructured":"Daniel Kahneman and Amos Tversky. 2013. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I. World Scientific 99--127.","DOI":"10.1142\/9789814417358_0006"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2020.3030335"},{"key":"e_1_2_2_46_1","unstructured":"Nathan Kallus and Angela Zhou. 2019. Assessing Disparate Impact of Personalized Interventions: Identifiability and Bounds. In Advances in Neural Information Processing Systems. 3426--3437."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.3386\/w23180"},{"key":"e_1_2_2_48_1","volume-title":"The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behavioral and brain sciences","author":"Koehler Jonathan J","year":"1996","unstructured":"Jonathan J Koehler. 1996. The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges. Behavioral and brain sciences, Vol. 19, 1 (1996), 1--17."},{"key":"e_1_2_2_49_1","unstructured":"Max Kuhn and Hadley Wickham. 2020. tidymodels: Easily Install and Load the 'Tidymodels' Packages. https:\/\/CRAN.R-project.org\/package=tidymodels R package version 0.1.0."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287590"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098066"},{"key":"e_1_2_2_52_1","volume-title":"General performance on a numeracy scale among highly educated samples. Medical decision making","author":"Lipkus Isaac M","year":"2001","unstructured":"Isaac M Lipkus, Greg Samsa, and Barbara K Rimer. 2001. General performance on a numeracy scale among highly educated samples. Medical decision making, Vol. 21, 1 (2001), 37--44."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376257"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-0262.2004.00537.x"},{"key":"e_1_2_2_55_1","volume-title":"Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods","author":"Mason Winter","year":"2012","unstructured":"Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods, Vol. 44, 1 (2012), 1--23."},{"key":"e_1_2_2_56_1","unstructured":"Cate Metz and Adam Satariano. 2020. An Algorithm that Grants Freedom or Takes it Away. https:\/\/www.nytimes.com\/2020\/02\/06\/technology\/predictive- algorithms-crime.html"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1006\/jesp.1999.1414"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-economics-072412-131310"},{"key":"e_1_2_2_59_1","volume-title":"Telling more than we can know: verbal reports on mental processes. Psychological review","author":"Nisbett Richard E","year":"1977","unstructured":"Richard E Nisbett and Timothy D Wilson. 1977. Telling more than we can know: verbal reports on mental processes. Psychological review, Vol. 84, 3 (1977), 231."},{"key":"e_1_2_2_60_1","volume-title":"Uncertain judgements: eliciting experts' probabilities","author":"O'Hagan Anthony","unstructured":"Anthony O'Hagan, Caitlin E Buck, Alireza Daneshkhah, J Richard Eiser, Paul H Garthwaite, David J Jenkinson, Jeremy E Oakley, and Tim Rakow. 2006. Uncertain judgements: eliciting experts' probabilities .John Wiley & Sons."},{"key":"e_1_2_2_61_1","unstructured":"Pennsylvania Commission on Sentencing. 2020 a. An Overview of the Sentence Risk Assessment Instrument. https:\/\/sentencing.umn.edu\/sites\/sentencing.umn.edu\/files\/ pennsylvania_overview_of_the_sentencing_risk_assessment_instrument_2017.pdf"},{"key":"e_1_2_2_62_1","volume-title":"2020 b. Judicial System General Provisions. Part VIII Criminal Sentencing","author":"The Pennsylvania Commission on Sentencing.","unstructured":"The Pennsylvania Commission on Sentencing. 2020 b. Judicial System General Provisions. Part VIII Criminal Sentencing. Chapter 305. Sentence Risk Assessment Instrument. http:\/\/pcs.la.psu.edu\/guidelines\/sentence-risk-assessment- instrument\/sentence-risk-assessment-instrument-guideline- text"},{"key":"e_1_2_2_63_1","doi-asserted-by":"crossref","unstructured":"Daniel Paravisini and Antoinette Schoar. 2013. The incentive effect of scores: Randomized evidence from credit committees. Technical Report. National Bureau of Economic Research.","DOI":"10.3386\/w19303"},{"key":"e_1_2_2_64_1","unstructured":"Pennsylvania Commission on Sentencing. 2020. Sentencing Guidelines and Implementation Manuals. http:\/\/pcs.la.psu.edu\/guidelines\/sentencing\/sentencing- guidelines-and-implementation-manuals"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445315"},{"key":"e_1_2_2_66_1","volume-title":"The probability weighting function. Econometrica","author":"Prelec Drazen","year":"1998","unstructured":"Drazen Prelec. 1998. The probability weighting function. Econometrica (1998), 497--527."},{"key":"e_1_2_2_67_1","volume-title":"R: A Language and Environment for Statistical Computing","author":"Team R Core","year":"2019","unstructured":"R Core Team. 2019. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https:\/\/www.R-project.org\/"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2019.00014"},{"key":"e_1_2_2_70_1","first-page":"D1","article-title":"Exploring slider vs. categorical response formats in web-based surveys","volume":"11","author":"Roster Catherine A","year":"2015","unstructured":"Catherine A Roster, Lorenzo Lucianetti, and Gerald Albaum. 2015. Exploring slider vs. categorical response formats in web-based surveys. Journal of Research Practice, Vol. 11, 1 (2015), D1--D1.","journal-title":"Journal of Research Practice"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376229"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-127-11-199712010-00003"},{"key":"e_1_2_2_73_1","volume-title":"The effect of framing actuarial risk probabilities on involuntary civil commitment decisions. Law and human behavior","author":"Scurich Nicholas","year":"2011","unstructured":"Nicholas Scurich and Richard S John. 2011. The effect of framing actuarial risk probabilities on involuntary civil commitment decisions. Law and human behavior, Vol. 35, 2 (2011), 83--91."},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372827"},{"key":"e_1_2_2_75_1","volume-title":"Using algorithms to address trade-offs inherent in predicting recidivism. Behavioral Sciences & the Law","author":"Skeem Jennifer","year":"2020","unstructured":"Jennifer Skeem and Christopher Lowenkamp. 2020. Using algorithms to address trade-offs inherent in predicting recidivism. Behavioral Sciences & the Law (2020)."},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1111\/1745-9125.12123"},{"key":"e_1_2_2_77_1","first-page":"303","article-title":"Assessing risk assessment in action","volume":"103","author":"Stevenson Megan","year":"2018","unstructured":"Megan Stevenson. 2018. Assessing risk assessment in action. Minn. L. Rev., Vol. 103 (2018), 303.","journal-title":"Minn. L. Rev."},{"key":"e_1_2_2_78_1","volume-title":"arXiv preprint arXiv:1808.09123","author":"Tan Sarah","year":"2018","unstructured":"Sarah Tan, Julius Adebayo, Kori Inkpen, and Ece Kamar. 2018. Investigating Human+Machine Complementarity for Recidivism Predictions. arXiv preprint arXiv:1808.09123 (2018)."},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"e_1_2_2_80_1","volume-title":"Judgment under uncertainty: Heuristics and biases. science","author":"Tversky Amos","year":"1974","unstructured":"Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. science, Vol. 185, 4157 (1974), 1124--1131."},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3358955.3363293"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1539-6924.2008.01137.x"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01686"},{"key":"e_1_2_2_84_1","volume-title":"Strangers to ourselves","author":"Wilson Timothy D","unstructured":"Timothy D Wilson. 2004. Strangers to ourselves. Harvard University Press."},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300509"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479572","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3479572","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T05:02:31Z","timestamp":1752469351000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3479572"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":86,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3479572"],"URL":"https:\/\/doi.org\/10.1145\/3479572","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}