{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T01:18:36Z","timestamp":1776129516740,"version":"3.50.1"},"reference-count":92,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["72271227"],"award-info":[{"award-number":["72271227"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Fundamental Research Funds"},{"name":"Central Universities","award":["E1E40810X2"],"award-info":[{"award-number":["E1E40810X2"]}]},{"DOI":"10.13039\/501100004739","name":"Youth Innovation Promotion Association CAS","doi-asserted-by":"crossref","award":["110800EAG2"],"award-info":[{"award-number":["110800EAG2"]}],"id":[{"id":"10.13039\/501100004739","id-type":"DOI","asserted-by":"crossref"}]},{"name":"MOE Social Science Laboratory of Digital Economic Forecasts and Policy Simulation at UCAS"},{"name":"China Inclusive Green Development Policy Evaluation Laboratory","award":["2024SYFH004"],"award-info":[{"award-number":["2024SYFH004"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Comput. Simul."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>Agent-Based Modeling (ABM) has gained significant traction due to its ability to represent diverse agents\u2019 behaviors and interactions accurately. However, ensuring the reliability and widespread acceptance of ABM necessitates rigorous validation. Unfortunately, existing literature often lacks integration of representative validation methods into a cohesive framework, hindering standardized validation practices. This study aims at proposing a comprehensive and pragmatic validation framework. Initially, we clarify three fundamental concepts: calibration, verification, and validation. Subsequently, we review 17 distinct validation approaches and categorize them based on their data requirements and suitability for various simulation methodologies. Aligned with the ABM procedures, we introduce a Hierarchical ABM Validation (HAV) framework structured across three tiers: agent level, model level, and output level. Each tier recommends appropriate validation methods contingent upon data availability, enhancing the HAV\u2019s applicability across diverse modeling scenarios. Finally, we develop an accessible Python code package (hav) and provide two examples of validating a traffic model and a wealth model. These cases exemplify the HAV\u2019s implementation and underscore its efficacy in promoting standardized validation practices within ABM research.<\/jats:p>","DOI":"10.1145\/3769857","type":"journal-article","created":{"date-parts":[[2025,9,27]],"date-time":"2025-09-27T11:14:41Z","timestamp":1758971681000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Towards Standardizing Validation Practices in Agent-Based Modeling: A Hierarchical ABM Validation Framework"],"prefix":"10.1145","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6288-7215","authenticated-orcid":false,"given":"Zhou","family":"He","sequence":"first","affiliation":[{"name":"University of the Chinese Academy of Sciences","place":["Beijing, China"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0020-8960","authenticated-orcid":false,"given":"Qi","family":"Song","sequence":"additional","affiliation":[{"name":"University of the Chinese Academy of Sciences","place":["Beijing, China"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-4925-0865","authenticated-orcid":false,"given":"Junchao","family":"Lian","sequence":"additional","affiliation":[{"name":"University of the Chinese Academy of Sciences","place":["Beijing, China"]}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2461-4165","authenticated-orcid":false,"given":"Yiming","family":"Liu","sequence":"additional","affiliation":[{"name":"Academy of Mathematics and Systems Science, Chinese Academy of Sciences","place":["Beijing, China"]}]}],"member":"320","published-online":{"date-parts":[[2026,1,18]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2016.06.001"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jebo.2016.06.002"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2020.105022"},{"issue":"4","key":"e_1_3_4_5_2","first-page":"28","article-title":"Agent-based modeling: Understanding our creations","volume":"9","author":"Axtell Robert L.","year":"1994","unstructured":"Robert L. Axtell and Joshua M. Epstein. 1994. Agent-based modeling: Understanding our creations. The Bulletin of the Santa Fe Institute 9, 4 (1994), 28\u201332.","journal-title":"The Bulletin of the Santa Fe Institute"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2013.6721512"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF02136828"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2021.1900914"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.2992473"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1040.0264"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70766-2"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijpe.2022.108500"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctt9qh59s.14"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.mbs.2005.07.006"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.5105"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.5095"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10614-021-10188-5"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.5555\/1161734.1161774"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1287\/isre.5.2.170"},{"key":"e_1_3_4_20_2","volume-title":"Who goes first? An Examination of the Impact of Activation on Outcome Behavior in Agent-based Models","author":"Comer Kenneth W.","year":"2014","unstructured":"Kenneth W. Comer. 2014. Who goes first? An Examination of the Impact of Activation on Outcome Behavior in Agent-based Models. Ph.D. Dissertation. George Mason University, Fairfax, VA, USA."},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.2828"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.42.6.891"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1093\/icc\/dty010"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11403-017-0193-4"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1063\/1.1571309"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/wsc.2005.1574355"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70766-2_31"},{"issue":"1","key":"e_1_3_4_28_2","first-page":"3","article-title":"Agent-based urban land markets: Agent\u015b pricing behavior, land prices and urban land use change","volume":"12","author":"Filatova Tatiana","year":"2009","unstructured":"Tatiana Filatova, Dawn Parker, and Anne van der Veen. 2009. Agent-based urban land markets: Agent\u015b pricing behavior, land prices and urban land use change. Journal of Artificial Societies and Social Simulation 12, 1 (2009), 3.","journal-title":"Journal of Artificial Societies and Social Simulation"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.13.7.525"},{"key":"e_1_3_4_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2018.03.011"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1287\/opre.31.4.603"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aap.2015.09.015"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2567897"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2009.26"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.35.2.127"},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1287\/deca.2017.0361"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2017.06.001"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1504\/IJSPM.2012.049817"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tra.2022.03.028"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10464-009-9268-2"},{"issue":"2","key":"e_1_3_4_41_2","first-page":"7","article-title":"Social simulation of stock markets: Taking it to the next level","volume":"10","author":"Hoffmann A. O. I.","year":"2007","unstructured":"A. O. I. Hoffmann, W. Jager, and J. H. Von Eije. 2007. Social simulation of stock markets: Taking it to the next level. Journal of Artificial Societies and Social Simulation 10, 2 (2007), 7. Retrieved from https:\/\/www.jasss.org\/10\/2\/7.html","journal-title":"Journal of Artificial Societies and Social Simulation"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2010.10.003"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.17.5.247"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2014.10.006"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2017.01.014"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2014.26"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1961.10489938"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.2897"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1051\/jp1:1992277"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/1363686.1363696"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11403-017-0206-3"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1080\/17477778.2020.1800422"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1093\/icc\/11.4.667"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00181-007-0159-9"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","DOI":"10.1080\/00207543.2019.1636321"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.14.2.B92"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2013.6721538"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11403-020-00289-z"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2012.26"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.nucengdes.2007.02.032"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-813543-3.00008-1"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.13140\/2.1.4420.3527"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC52266.2021.9715352"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2016.03.050"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/259207.259375"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2014.42"},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jedc.2020.103859"},{"key":"e_1_3_4_68_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.2616"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2011.6148057"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-15353-2"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10614-013-9406-7"},{"key":"e_1_3_4_72_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2012.20"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cie.2018.07.046"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2010.14"},{"key":"e_1_3_4_75_2","doi-asserted-by":"publisher","DOI":"10.2166\/wcc.2019.108"},{"key":"e_1_3_4_76_2","volume-title":"Business Dynamics: System Thinking and Modeling for a Complex World","author":"Sterman John","year":"2000","unstructured":"John Sterman. 2000. Business Dynamics: System Thinking and Modeling for a Complex World. McGraw-Hill Education, San Diego, CA. 1\u201329 pages."},{"issue":"3","key":"e_1_3_4_77_2","first-page":"1","article-title":"Agent-based approach to investors\u2019 behavior and asset price fluctuation in financial markets","volume":"6","author":"Takahashi Hiroshi","year":"2003","unstructured":"Hiroshi Takahashi and Takao Terano. 2003. Agent-based approach to investors\u2019 behavior and asset price fluctuation in financial markets. Journal of Artificial Societies and Social Simulation 6, 3 (2003), 1. Retrieved from https:\/\/www.jasss.org\/6\/3\/3.html","journal-title":"Journal of Artificial Societies and Social Simulation"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.2857"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.2503"},{"key":"e_1_3_4_80_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2021.10.034"},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10614-021-10191-w"},{"key":"e_1_3_4_82_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-publhealth-040617-014317"},{"key":"e_1_3_4_83_2","doi-asserted-by":"publisher","DOI":"10.1080\/17477778.2020.1753251"},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1287\/trsc.2019.0971"},{"key":"e_1_3_4_85_2","first-page":"1","article-title":"Empirical calibration of simulation models","volume":"200413","author":"Werker Claudia","year":"2004","unstructured":"Claudia Werker and Thomas Brenner. 2004. Empirical calibration of simulation models. Eindhoven Center for Innovation Studies (ECIS), Eindhoven Center for Innovation Studies (ECIS) Working Paper Series 200413(2004), 1\u201319.","journal-title":"Eindhoven Center for Innovation Studies (ECIS), Eindhoven Center for Innovation Studies (ECIS) Working Paper Series"},{"key":"e_1_3_4_86_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijresmar.2011.04.002"},{"key":"e_1_3_4_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564928"},{"issue":"2","key":"e_1_3_4_88_2","first-page":"8","article-title":"Empirical validation of agent-based models: Alternatives and prospects","volume":"10","author":"Windrum Paul","year":"2007","unstructured":"Paul Windrum, Giorgio Fagiolo, and Alessio Moneta. 2007. Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation 10, 2 (2007), 8. Retrieved from https:\/\/www.jasss.org\/10\/2\/8.html","journal-title":"Journal of Artificial Societies and Social Simulation"},{"key":"e_1_3_4_89_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11269-008-9328-7"},{"key":"e_1_3_4_90_2","doi-asserted-by":"publisher","DOI":"10.1080\/17477778.2019.1604465"},{"key":"e_1_3_4_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/2699713"},{"key":"e_1_3_4_92_2","doi-asserted-by":"publisher","DOI":"10.1155\/2020\/6946370"},{"key":"e_1_3_4_93_2","doi-asserted-by":"publisher","DOI":"10.1287\/ijoc.2016.0701"}],"container-title":["ACM Transactions on Modeling and Computer Simulation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3769857","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T07:22:26Z","timestamp":1768807346000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3769857"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,18]]},"references-count":92,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3769857"],"URL":"https:\/\/doi.org\/10.1145\/3769857","relation":{},"ISSN":["1049-3301","1558-1195"],"issn-type":[{"value":"1049-3301","type":"print"},{"value":"1558-1195","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,1,18]]},"assertion":[{"value":"2024-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-11","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-01-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}