{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T14:26:56Z","timestamp":1753885616338,"version":"3.41.2"},"reference-count":78,"publisher":"World Scientific Pub Co Pte Ltd","issue":"06","funder":[{"name":"City of Virginia Beach","award":["200152-010"],"award-info":[{"award-number":["200152-010"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Model. Simul. Sci. Comput."],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:p> Advances in computing allow for the construction of increasingly large and complex models and simulations. Exhaustive error checking of these intricate, large computational simulation models is daunting and potentially impractical. This paper explores an approach to error-checking simulation model components using an Acceptance Sampling methodology from the field of industrial manufacturing. We propose a systematic process in which a simulation inspector examines only a fraction of the computational model elements to measure the errors present. Our proposed process could support established verification processes by sampling the simulation components to identify whether the model is acceptably error free and which components require correcting. The proposed methodology relies on several statistical constraints but serves the interests of simulation professionals as part of the overall verification process. We illustrate the application and usefulness of our methodology through a real-world case study of a citywide microscopic transportation model. <\/jats:p>","DOI":"10.1142\/s1793962322500441","type":"journal-article","created":{"date-parts":[[2022,3,29]],"date-time":"2022-03-29T12:45:50Z","timestamp":1648557950000},"source":"Crossref","is-referenced-by-count":0,"title":["Acceptance sampling to aid in the verification of computational simulations"],"prefix":"10.1142","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8012-2272","authenticated-orcid":false,"given":"Andrew J.","family":"Collins","sequence":"first","affiliation":[{"name":"Engineering Management & Systems Engineering, Old Dominion University, Norfolk, Virginia, USA"}]},{"given":"Erika","family":"Frydenlund","sequence":"additional","affiliation":[{"name":"Virginia Modeling, Analysis and Simulation Center, Old Dominion University, Suffolk, Virginia, USA"}]},{"given":"Christopher J.","family":"Lynch","sequence":"additional","affiliation":[{"name":"Virginia Modeling, Analysis and Simulation Center, Old Dominion University, Suffolk, Virginia, USA"}]},{"given":"R. Michael","family":"Robinson","sequence":"additional","affiliation":[{"name":"Virginia Modeling, Analysis and Simulation Center, Old Dominion University, Suffolk, Virginia, USA"}]}],"member":"219","published-online":{"date-parts":[[2022,3,28]]},"reference":[{"key":"S1793962322500441BIB001","doi-asserted-by":"crossref","first-page":"1223","DOI":"10.1016\/j.ijdrr.2017.12.010","volume":"31","author":"Collins A. J.","year":"2018","journal-title":"Int. J. Disaster Risk Reduct."},{"volume-title":"The Complete Guide to Software Testing","year":"1990","author":"Hetzel W. C.","key":"S1793962322500441BIB002"},{"issue":"5","key":"S1793962322500441BIB003","first-page":"20","volume":"18","author":"Cook D. A.","year":"2005","journal-title":"J. Defense Softw. Eng."},{"key":"S1793962322500441BIB004","first-page":"39","volume-title":"Proc. 1999 Winter Simulation Conf.","author":"Sargent R. G.","year":"1999"},{"volume-title":"Passages from the Life of a Philosopher","year":"1864","author":"Babbage C.","key":"S1793962322500441BIB005"},{"volume-title":"Chaos: Making a New Science","year":"2011","author":"Gleick J.","key":"S1793962322500441BIB006"},{"volume-title":"Simulation Modeling & Analysis","year":"2007","author":"Law A. M.","key":"S1793962322500441BIB007"},{"key":"S1793962322500441BIB008","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpc.2009.09.018"},{"issue":"459","key":"S1793962322500441BIB009","doi-asserted-by":"crossref","first-page":"702","DOI":"10.1198\/016214502388618447","volume":"97","author":"Saltelli A.","year":"2002","journal-title":"J. Am. Stat. Assoc."},{"key":"S1793962322500441BIB010","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.14.2.B92"},{"key":"S1793962322500441BIB011","first-page":"53","volume-title":"Proc. 2004 Winter Simulation Conf.","author":"Balci O.","year":"2004"},{"key":"S1793962322500441BIB012","first-page":"209","volume":"14","author":"Senge P. M.","year":"1980","journal-title":"TIMS Studies Manag. Sci."},{"volume-title":"Proc. 29th Conf. Winter Simulation","year":"1997","author":"Robinson S.","key":"S1793962322500441BIB013"},{"issue":"4","key":"S1793962322500441BIB014","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1145\/358598.358609","volume":"24","author":"Balci O.","year":"1981","journal-title":"Commun. ACM"},{"key":"S1793962322500441BIB015","doi-asserted-by":"publisher","DOI":"10.3141\/2586-01"},{"key":"S1793962322500441BIB016","doi-asserted-by":"publisher","DOI":"10.1002\/9781118241042.ch2"},{"issue":"2","key":"S1793962322500441BIB017","first-page":"9","volume":"11","author":"Takadama K.","year":"2008","journal-title":"J. Artif. Soc. Soc. Simulat."},{"key":"S1793962322500441BIB018","first-page":"1","author":"Z\u00fcfle A.","year":"2021","journal-title":"Comput. Math. Organ. Theory"},{"issue":"4","key":"S1793962322500441BIB019","first-page":"425","volume":"2","author":"Collins A. J.","year":"2021","journal-title":"MDPI Model."},{"first-page":"1","volume-title":"2020 Winter Simulation Conf.","author":"Collins A. J.","key":"S1793962322500441BIB020"},{"issue":"3","key":"S1793962322500441BIB021","first-page":"1","volume":"5","author":"Courdier R.","year":"2002","journal-title":"J. Artif. Soc. Soc. Simulat."},{"issue":"4","key":"S1793962322500441BIB022","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1177\/0037549716659707","volume":"93","author":"Gore R. J.","year":"2017","journal-title":"Simulation"},{"key":"S1793962322500441BIB023","first-page":"47","volume-title":"Agent-Directed Simulation Conf.","author":"Xiang X.","year":"2005"},{"key":"S1793962322500441BIB024","first-page":"204","volume-title":"Complex Systems in the Social and Behavioral Sciences: Theory, Method and Application","author":"Diallo S. Y.","year":"2021"},{"issue":"9","key":"S1793962322500441BIB025","doi-asserted-by":"crossref","first-page":"e0133505","DOI":"10.1371\/journal.pone.0133505","volume":"10","author":"Gore R. J.","year":"2015","journal-title":"PLoS One"},{"key":"S1793962322500441BIB026","first-page":"1","volume-title":"Int. Conf. Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation","author":"Kavak H.","year":"2018"},{"issue":"6","key":"S1793962322500441BIB027","doi-asserted-by":"crossref","first-page":"e0198857","DOI":"10.1371\/journal.pone.0198857","volume":"13","author":"Padilla J. J.","year":"2018","journal-title":"PLoS One"},{"key":"S1793962322500441BIB029","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1109\/WSC.1991.185589","volume-title":"Proceedings of the 1991 Winter Simulation Conference","author":"Sargent R. G.","year":"1991"},{"key":"S1793962322500441BIB030","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2012.20"},{"key":"S1793962322500441BIB031","first-page":"559","volume-title":"Proc. 21st Conf. Winter Simulation","author":"Whitner R. B.","year":"1989"},{"volume-title":"An Assessment Procedure and a Set of Criteria for Use in the Evaluation of Computerized Models and Computer-Based Modeling Tools","year":"1981","author":"Sargent R. G.","key":"S1793962322500441BIB032"},{"key":"S1793962322500441BIB033","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2013.3"},{"key":"S1793962322500441BIB034","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1109\/WSC.2002.1172944","volume-title":"Proc. Winter Simulation Conf.","author":"Balci O.","year":"2002"},{"volume-title":"Verification and Validation in Computational Science and Engineering","year":"1998","author":"Roache P. J.","key":"S1793962322500441BIB036"},{"key":"S1793962322500441BIB037","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1145\/256562.256612","volume-title":"Proc. Winter Simulation Conf.","author":"Arthur J. D.","year":"1996"},{"issue":"3","key":"S1793962322500441BIB038","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1145\/259207.259375","volume":"7","author":"Page E. H.","year":"1997","journal-title":"ACM Trans. Model. Comput. Simulat."},{"issue":"3","key":"S1793962322500441BIB040","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1177\/003754978003400310","volume":"34","author":"Schruben L. W.","year":"1980","journal-title":"Simulation"},{"issue":"2","key":"S1793962322500441BIB041","first-page":"1","volume":"6","author":"Eek M.","year":"2015","journal-title":"Int. J. Model. Simul. Sci. Comput."},{"issue":"5","key":"S1793962322500441BIB042","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1097\/SIH.0000000000000298","volume":"13","author":"Barnes J. J.","year":"2018","journal-title":"Simulat. Healthcare"},{"key":"S1793962322500441BIB043","first-page":"1621","volume-title":"2015 Winter Simulation Conf. (WSC)","author":"Yilmaz L.","year":"2015"},{"key":"S1793962322500441BIB044","doi-asserted-by":"publisher","DOI":"10.1002\/9780470172445.ch10"},{"key":"S1793962322500441BIB045","doi-asserted-by":"publisher","DOI":"10.1177\/003754977903200304"},{"issue":"5","key":"S1793962322500441BIB046","doi-asserted-by":"crossref","first-page":"e0232929","DOI":"10.1371\/journal.pone.0232929","volume":"15","author":"Lynch C. J.","year":"2020","journal-title":"PLoS One"},{"issue":"2","key":"S1793962322500441BIB047","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1145\/384169.384173","volume":"11","author":"Uhrmacher A. M.","year":"2001","journal-title":"ACM Trans. Model. Comput. Simulat."},{"volume-title":"Model Checking","year":"2018","author":"Clarke E. M.","key":"S1793962322500441BIB048"},{"issue":"2","key":"S1793962322500441BIB049","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1145\/505145.505149","volume":"11","author":"Jackson D.","year":"2002","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"volume-title":"PVS: A Prototype Verification System","year":"1992","author":"Owre S.","key":"S1793962322500441BIB050"},{"key":"S1793962322500441BIB051","doi-asserted-by":"publisher","DOI":"10.1049\/sej.1993.0025"},{"volume-title":"Theory of Modeling and Simulation: Integrating Discrete Event and Continuous Complex Dynamic Systems","year":"2010","author":"Zeigler B. P.","key":"S1793962322500441BIB052"},{"issue":"7","key":"S1793962322500441BIB053","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1109\/TCAD.2008.923410","volume":"27","author":"D\u2019Silva V.","year":"2008","journal-title":"IEEE Trans. Comput.-Aided Design Integr. Circuits Syst."},{"key":"S1793962322500441BIB054","first-page":"685","volume-title":"Summer Computer Simulation Conf.","author":"Stevenson D. C.","year":"2003"},{"issue":"6","key":"S1793962322500441BIB055","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1177\/0037549717737159","volume":"94","author":"Padilla J. J.","year":"2018","journal-title":"Simulation"},{"first-page":"37","volume-title":"Proc. 2002 Int. Symp. Empirical Software Engineering","author":"Andersson C.","key":"S1793962322500441BIB056"},{"key":"S1793962322500441BIB057","first-page":"230","volume-title":"Proc. 1996 Winter Simulation Conf.","author":"Arthur J. D.","year":"1996"},{"issue":"3","key":"S1793962322500441BIB058","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2699722","volume":"25","author":"Gore R.","year":"2015","journal-title":"ACM Trans. Model. Comput. Simulat."},{"volume-title":"2012 34th Int. Conf. Software Engineering (ICSE)","year":"2012","author":"Gore R.","key":"S1793962322500441BIB060"},{"key":"S1793962322500441BIB061","first-page":"492","volume-title":"2011 26th IEEE\/ACM Int. Conf. Automated Software Engineering (ASE 2011)","author":"Gore R.","year":"2011"},{"issue":"1","key":"S1793962322500441BIB062","doi-asserted-by":"crossref","first-page":"1641001","DOI":"10.1142\/S1793962316410014","volume":"7","author":"Diallo S. Y.","year":"2016","journal-title":"Int. J. Model. Simul. Sci. Comput."},{"key":"S1793962322500441BIB063","volume-title":"Introduction to Statistical Quality Control","author":"Montgomery D. C.","year":"1991","edition":"2"},{"volume-title":"Economic Control of Quality of Manufactured Product","year":"1931","author":"Shewhart W. A.","key":"S1793962322500441BIB064"},{"issue":"2","key":"S1793962322500441BIB065","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1016\/S0016-0032(38)90436-3","volume":"226","author":"Shewhart W. A.","year":"1938","journal-title":"J. Franklin Inst."},{"issue":"1","key":"S1793962322500441BIB066","doi-asserted-by":"crossref","first-page":"80","DOI":"10.2307\/2983626","volume":"6","author":"Jennett W.","year":"1939","journal-title":"Suppl. J. R. Stat. Soc."},{"first-page":"849","volume-title":"2000 Winter Simulation Conf. Proc.","author":"Balci O.","key":"S1793962322500441BIB067"},{"volume-title":"e-Handbook of Statistical Methods","year":"2012","author":"NIST\/SEMATECH","key":"S1793962322500441BIB068"},{"key":"S1793962322500441BIB073","doi-asserted-by":"crossref","first-page":"11","DOI":"10.3141\/2459-02","volume":"2549","author":"Collins A. J.","year":"2014","journal-title":"Transport. Res. Rec."},{"key":"S1793962322500441BIB075","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1109\/WSC.2014.7020019","volume-title":"Proc. Winter Simulation Conf. 2014","author":"Lynch C.","year":"2014"},{"issue":"4","key":"S1793962322500441BIB076","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/(SICI)1099-1727(199924)15:4<379::AID-SDR179>3.0.CO;2-E","volume":"15","author":"Vennix J. A.","year":"1999","journal-title":"Syst. Dyn. Rev."},{"key":"S1793962322500441BIB077","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1007\/978-1-4684-2001-2_9","volume-title":"Complexity of Computer Computations: Proc. Symp. Complexity of Computer Computations","author":"Karp R. M.","year":"1972"},{"issue":"3","key":"S1793962322500441BIB078","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1080\/02626667.2014.903331","volume":"60","author":"Crochemore L.","year":"2015","journal-title":"Hydrol. Sci. J."},{"issue":"1","key":"S1793962322500441BIB079","doi-asserted-by":"crossref","DOI":"10.1109\/TEM.2003.822468","volume":"15","author":"Davis F. D.","year":"2004","journal-title":"IEEE Trans. Eng. Manag."},{"volume-title":"Lean-Agile Acceptance Test-Driven Development: Better Software Through Collaboration","year":"2010","author":"Pugh K.","key":"S1793962322500441BIB080"},{"key":"S1793962322500441BIB081","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780195172119.001.0001"},{"issue":"4","key":"S1793962322500441BIB082","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1108\/IJHMA-Jul-2012-0027","volume":"6","author":"Collins A. J.","year":"2013","journal-title":"Int. J. Housing Markets Anal."},{"issue":"2","key":"S1793962322500441BIB083","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1080\/15427560.2013.791296","volume":"14","author":"Gangel M.","year":"2013","journal-title":"J. Behav. Finance"},{"key":"S1793962322500441BIB084","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtbi.2008.04.011"},{"issue":"4","key":"S1793962322500441BIB085","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1057\/jos.2010.24","volume":"5","author":"Banks J.","year":"2011","journal-title":"J. Simulat."},{"volume-title":"Empirical Model Building and Response Surfaces","year":"1987","author":"Box G. E. P.","key":"S1793962322500441BIB087"},{"volume-title":"Proc. 18th Conf. Winter Simulation","year":"1986","author":"Balci O.","key":"S1793962322500441BIB088"}],"container-title":["International Journal of Modeling, Simulation, and Scientific Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S1793962322500441","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T08:21:49Z","timestamp":1674807709000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S1793962322500441"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,28]]},"references-count":78,"journal-issue":{"issue":"06","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["10.1142\/S1793962322500441"],"URL":"https:\/\/doi.org\/10.1142\/s1793962322500441","relation":{},"ISSN":["1793-9623","1793-9615"],"issn-type":[{"type":"print","value":"1793-9623"},{"type":"electronic","value":"1793-9615"}],"subject":[],"published":{"date-parts":[[2022,3,28]]},"article-number":"2250044"}}