{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T15:54:26Z","timestamp":1778082866459,"version":"3.51.4"},"reference-count":117,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,4,9]],"date-time":"2024-04-09T00:00:00Z","timestamp":1712620800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"University of Colorado Research and Engineering Center for Unmanned Vehicles (RECUV) and the Defense Advanced Research Projects Agency","award":["HR001120C0032"],"award-info":[{"award-number":["HR001120C0032"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>\n            Humans working with autonomous artificially intelligent systems may not be experts in the inner workings of their machine teammates, but need to understand when to employ, trust, and rely on the system. A critical challenge is to develop machine agents with the capacity to understand their own capabilities and limitations, and the ability to communicate this information to human partners. Self-assessment is an emerging field that tackles this challenge through the development of algorithms that enable autonomous agents to understand and communicate their competency. These methods can engender appropriate trust and align human expectations with autonomous assistant abilities. However, current research in self-assessment is dispersed across many fields, including artificial intelligence, robotics, and human factors. This survey connects work from these disparate areas and reviews state-of-the-art methods for algorithmic self-assessments that enable autonomous agents to estimate, understand, and communicate valuable information pertaining to their competency, with focus on methods that can improve interactions within human-machine teams. To better understand the landscape of self-assessment approaches, we present a framework for categorizing work in self-assessment based on underlying algorithm type:\n            <jats:italic>test-based<\/jats:italic>\n            ,\n            <jats:italic>learning-based<\/jats:italic>\n            , or\n            <jats:italic>knowledge-based<\/jats:italic>\n            . We synthesize common features across these approaches and discuss relevant future directions for research in this emerging space.\n          <\/jats:p>","DOI":"10.1145\/3616010","type":"journal-article","created":{"date-parts":[[2023,8,14]],"date-time":"2023-08-14T11:20:57Z","timestamp":1692012057000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["A Survey of Algorithmic Methods for Competency Self-Assessments in Human-Autonomy Teaming"],"prefix":"10.1145","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8262-2169","authenticated-orcid":false,"given":"Nicholas","family":"Conlon","sequence":"first","affiliation":[{"name":"University of Colorado at Boulder, Boulder, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7555-5671","authenticated-orcid":false,"given":"Nisar R.","family":"Ahmed","sequence":"additional","affiliation":[{"name":"University of Colorado at Boulder, Boulder, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1848-7884","authenticated-orcid":false,"given":"Daniel","family":"Szafir","sequence":"additional","affiliation":[{"name":"University of North Carolina at Chapel Hill, Chapel Hill, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,4,9]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2011.09004"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS47612.2022.9981991"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2206.10553"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSPEC.2022.9771355"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2870052"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.3390\/s21041210"},{"key":"e_1_3_2_8_2","volume-title":"Assured Human-Autonomy Interaction through Machine Self-Confidence","author":"Aitken Matthew","year":"2016","unstructured":"Matthew Aitken. 2016. Assured Human-Autonomy Interaction through Machine Self-Confidence. Master\u2019s Thesis. University of Colorado Boulder, Boulder, CO."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2019.8917111"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2016.7745145"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS45743.2020.9340841"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.5555\/3398761.3398781"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2007.11740"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/72.279181"},{"key":"e_1_3_2_15_2","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop Christopher M.","year":"2006","unstructured":"Christopher M. Bishop and Nasser M. Nasrabadi. 2006. Pattern Recognition and Machine Learning. Springer, New York, NY."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/8344.001.0001"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.06508"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2012.2186810"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.01546"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48891.2023.10160916"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2023.3262187"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2015.7333656"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.bica.2018.07.016"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICHMS49158.2020.9209477"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1080\/1463922X.2017.1315750"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.5555\/3463952.3463996"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/0022-247X(89)90335-1"},{"key":"e_1_3_2_28_2","volume-title":"Proceedings of the IEEE International Conference on Robotics and Automation Workshop on Human Multi-Robot System Interaction (ICRA\u201917)","author":"Chung Jen Jen","year":"2017","unstructured":"Jen Jen Chung, Lauren Milliken, Geoffrey A. Hollinger, and Kagan Tumer. 2017. When to ask for help: Introspection in multi-robot teams. In Proceedings of the IEEE International Conference on Robotics and Automation Workshop on Human Multi-Robot System Interaction (ICRA\u201917). IEEE, New York, NY."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.2514\/6.2022-2496"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.01646"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS47612.2022.9981653"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2005.10.009"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143874"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2018.2877482"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1710.00794"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.23919\/MIPRO.2018.8400040"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00038-7"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-017-2376-3_21"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2005.05.007"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.01544"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3219024"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/CTS.2007.4621745"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-25891-6_38"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967924"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-51758-8_15"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA46639.2022.9812030"},{"key":"e_1_3_2_47_2","first-page":"429","volume-title":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS\u201920)","author":"Gervits Felix","year":"2020","unstructured":"Felix Gervits, Dean Thurston, Ravenna Thielstrom, Terry Fong, Quinn Pham, and Matthias Scheutz. 2020. Toward genuine robot teammates: Improving human-robot team performance using robot shared mental models. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS\u201920). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 429\u2013437."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.5555\/3327345.3327436"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-50115-4_40"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.16462"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/FormaliSE.2017.9"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1057\/9780230280830_23"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1177\/1541931215591057"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.5772\/55919"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2014.11.003"},{"key":"e_1_3_2_56_2","volume-title":"Algorithmic Assurances and Self-Assessment of Competency Boundaries in Autonomous Systems","author":"Israelsen Brett","year":"2019","unstructured":"Brett Israelsen. 2019. Algorithmic Assurances and Self-Assessment of Competency Boundaries in Autonomous Systems. Ph. D. Dissertation. University of Colorado Boulder, Boudler, CO."},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-20454-9_21"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.11981"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3267338"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2013.00016"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2013.6706783"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.3.1.Johnson"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2013.6483548"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","unstructured":"Christos Kaplanis Claudia Clopath and Murray Shanahan. 2020. Continual reinforcement learning with multi-timescale replay. (2020). DOI:10.48550\/arXiv.2004.07530","DOI":"10.48550\/arXiv.2004.07530"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1611835114"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/10187.001.0001"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/IV47402.2020.9304844"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2020.3044813"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","unstructured":"Sreejan Kumar Ishita Dasgupta Jonathan D. Cohen Nathaniel Daw and Thomas Griffiths. 2021. Meta-learning of structured task distributions in humans and machines. International Conference on Learning Representations (2021). DOI:10.48550\/arXiv.2010.02317","DOI":"10.48550\/arXiv.2010.02317"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2018.1547842"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3028751"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","unstructured":"Sergey Levine Aviral Kumar G. Tucker and Justin Fu. 2020. Offline reinforcement learning: Tutorial review and perspectives on open problems. (2020). DOI:10.48550\/arXiv.2005.01643","DOI":"10.48550\/arXiv.2005.01643"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1093\/sysbio\/syw077"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2022.883321"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3342355"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2019.02.023"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.5555\/647636.733058"},{"key":"e_1_3_2_78_2","volume-title":"Approaches for the Computation of Generalized Self-Confidence Statements for Autonomous Vehicles","author":"McGinley Jamison","year":"2022","unstructured":"Jamison McGinley. 2022. Approaches for the Computation of Generalized Self-Confidence Statements for Autonomous Vehicles. Master\u2019s Thesis. University of Colorado Boulder, Boulder, CO."},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.5555\/2999792.2999959"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07458-0_19"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-67361-5_42"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3387166"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2021.3116319"},{"key":"e_1_3_2_84_2","volume-title":"Robotic Introspection for Exploration and Mapping of Subterranean Environments","author":"Morris Aaron Christopher","year":"2007","unstructured":"Aaron Christopher Morris. 2007. Robotic Introspection for Exploration and Mapping of Subterranean Environments. Ph. D. Dissertation. Carnegie Mellon University, Pittsburgh, PA."},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","unstructured":"Anusha Nagabandi Ignasi Clavera Simin Liu Ronald S. Fearing P. Abbeel Sergey Levine and Chelsea Finn. 2018. Learning to adapt in dynamic real-world environments through meta-reinforcement learning. DOI:10.48550\/arXiv.1803.11347","DOI":"10.48550\/arXiv.1803.11347"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3522579"},{"key":"e_1_3_2_87_2","first-page":"98","article-title":"Words or numbers? The evaluation of probability expressions in general practice.","volume":"39","author":"O\u2019Brien B. J.","year":"1989","unstructured":"B. J. O\u2019Brien. 1989. Words or numbers? The evaluation of probability expressions in general practice. The Journal of the Royal College of General Practitioners 39, 320 (1989), 98\u2013100.","journal-title":"The Journal of the Royal College of General Practitioners"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1177\/1541931213601780"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543886"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2011.5980404"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.matpr.2017.11.304"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISIT.2008.4595271"},{"key":"e_1_3_2_93_2","volume-title":"Critical Thinking for Strategic Intelligence","author":"Pherson R. H.","year":"2020","unstructured":"R. H. Pherson. 2020. Critical Thinking for Strategic Intelligence. CQ Press, Washington, DC."},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDL49984.2021.9515672"},{"key":"e_1_3_2_95_2","volume-title":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) Workshop on Open Source Robotics","author":"Quigley Morgan","year":"2009","unstructured":"Morgan Quigley, Brian Gerkey, Ken Conley, Josh Faust, Tully Foote, Jeremy Leibs, Eric Berger, Rob Wheeler, and Andrew Ng. 2009. ROS: An open-source robot operating system. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) Workshop on Open Source Robotics. IEEE, New York, NY."},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.18626"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.06776"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2022.3192612"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2017.8206438"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1117\/12.2519302"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.52842\/conf.ecaade.2017.2.269"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0893-6080(02)00228-9"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2019.2908665"},{"key":"e_1_3_2_104_2","volume-title":"Introduction to the Theory of Computation","author":"Sipser Michael","year":"2013","unstructured":"Michael Sipser. 2013. Introduction to the Theory of Computation (3rd. ed.). Course Technology, Boston, MA."},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561941"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967676"},{"key":"e_1_3_2_107_2","volume-title":"Proceedings of the IJCAI-ECAI Workshop on Architectures and Evaluation for Generality, Autonomy, and Progress in AI (AEGAP\u201918)","author":"Svegliato Justin","year":"2018","unstructured":"Justin Svegliato and Shlomo Zilberstein. 2018. Adaptive metareasoning for bounded rational agents. In Proceedings of the IJCAI-ECAI Workshop on Architectures and Evaluation for Generality, Autonomy, and Progress in AI (AEGAP\u201918). IJCAI, Stockholm, Sweden."},{"key":"e_1_3_2_108_2","doi-asserted-by":"publisher","DOI":"10.1007\/s43154-020-00019-0"},{"key":"e_1_3_2_109_2","doi-asserted-by":"publisher","unstructured":"Joaquin Vanschoren. 2018. Meta-learning: A survey. (2018). DOI:10.48550\/arXiv.1810.03548","DOI":"10.48550\/arXiv.1810.03548"},{"key":"e_1_3_2_110_2","doi-asserted-by":"publisher","DOI":"10.1109\/ETFA45728.2021.9613467"},{"key":"e_1_3_2_111_2","doi-asserted-by":"publisher","unstructured":"Jane X. Wang Zeb Kurth-Nelson Dhruva Tirumala Hubert Soyer Joel Z. Leibo Remi Munos Charles Blundell Dharshan Kumaran and Matt Botvinick. 2016. Learning to reinforcement learn. (2016). DOI:10.48550\/arXiv.1611.05763","DOI":"10.48550\/arXiv.1611.05763"},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919883338"},{"key":"e_1_3_2_113_2","volume-title":"Portfolio Selection and Lower Partial Moments","author":"Wojt Alexander","year":"2009","unstructured":"Alexander Wojt. 2009. Portfolio Selection and Lower Partial Moments. Master\u2019s Thesis. Royal Institute of Technology, Stockholm, Sweden."},{"key":"e_1_3_2_114_2","doi-asserted-by":"publisher","DOI":"10.1177\/1555343416653562"},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1109\/HUMANOIDS.2017.8246976"},{"key":"e_1_3_2_116_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2110.11334"},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2013.2259892"},{"key":"e_1_3_2_118_2","doi-asserted-by":"publisher","DOI":"10.1109\/SSCI47803.2020.9308437"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3616010","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3616010","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:29Z","timestamp":1750178189000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3616010"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,9]]},"references-count":117,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3616010"],"URL":"https:\/\/doi.org\/10.1145\/3616010","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,9]]},"assertion":[{"value":"2022-09-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}