{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,3]],"date-time":"2025-03-03T06:00:10Z","timestamp":1740981610983,"version":"3.38.0"},"reference-count":70,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T00:00:00Z","timestamp":1708560000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["SIMULATION"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p> A stochastic network simulation is verified when its distribution of outputs is aligned with the ground truth, while tolerating deviations due to variability in real-world measurements and the randomness of a stochastic simulation. However, comparing distributions may yield false positives, as erroneous simulations may have the expected distribution yet present aberrations in low-level patterns. For instance, the number of sick individuals may present the right trend over time, but the wrong individuals were infected. We previously proposed an approach that transforms simulation traces into images verified by machine learning algorithms that account for low-level patterns. We demonstrated the viability of this approach when many simulation traces are compared with a large ground truth data set. However, ground truth data are often limited. For example, a publication may include few images of their simulation as illustrations; hence, teams that independently re-implement the model can only compare low-level patterns with few cases. In this paper, we examine whether our approach can be utilized with very small data sets (e.g., 5\u201310 images), as provided in publications. Depending on the network simulation model (e.g., rumor spread, cascading failure, and disease spread), we show that results obtained with little data can even surpass results obtained with moderate amounts of data at the cost of variability. Although a good accuracy is obtained in detecting several forms of errors, this paper is only a first step in the use of this technique for verification; hence, future works should assess the applicability of our approach to other types of network simulations. <\/jats:p>","DOI":"10.1177\/00375497241229753","type":"journal-article","created":{"date-parts":[[2024,2,22]],"date-time":"2024-02-22T09:56:01Z","timestamp":1708595761000},"page":"545-561","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":1,"title":["Experimental evaluation of a machine learning approach to improve the reproducibility of network simulations"],"prefix":"10.1177","volume":"100","author":[{"given":"Luke","family":"Liang","sequence":"first","affiliation":[{"name":"Department of Computer Science and Software Engineering, Miami University, OH, USA"}]},{"given":"Hieu","family":"Phan","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Software Engineering, Miami University, OH, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6816-355X","authenticated-orcid":false,"given":"Philippe J","family":"Giabbanelli","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Software Engineering, Miami University, OH, USA"}]}],"member":"179","published-online":{"date-parts":[[2024,2,22]]},"reference":[{"key":"bibr1-00375497241229753","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1038\/nature17990","volume":"533","author":"Baker M","year":"2016","journal-title":"Nature"},{"key":"bibr2-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1177\/00375497221132566"},{"first-page":"749","volume-title":"2018 Winter Simulation Conference (WSC)","author":"Taylor SJ","key":"bibr3-00375497241229753"},{"key":"bibr4-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274786"},{"first-page":"535","volume-title":"2017 winter simulation conference (WSC)","author":"Taylor SJ","key":"bibr5-00375497241229753"},{"key":"bibr6-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1177\/00375497231178303"},{"key":"bibr7-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.envsoft.2018.08.028"},{"key":"bibr8-00375497241229753","unstructured":"Treasury H. The aqua book: guidance on producing quality analysis for government, 2015, https:\/\/www.gov.uk\/government\/publications\/the-aqua-book-guidance-on-producing-quality-analysis-for-government"},{"key":"bibr9-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-77980-1_31"},{"key":"bibr10-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-77967-2_54"},{"key":"bibr11-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/s11538-021-00958-5"},{"key":"bibr12-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1177\/0272989X211003081"},{"key":"bibr13-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.epidem.2023.100677"},{"key":"bibr14-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-88768-6"},{"key":"bibr15-00375497241229753","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.2897"},{"key":"bibr16-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1145\/3437959.3459256"},{"first-page":"653","volume-title":"2022 winter simulation conference (WSC)","author":"Wozniak MK","key":"bibr17-00375497241229753"},{"first-page":"1301","volume-title":"2016 winter simulation conference (WSC)","author":"Uhrmacher AM","key":"bibr18-00375497241229753"},{"key":"bibr19-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1002\/adts.202100343"},{"first-page":"1","volume-title":"2021 annual modeling and simulation conference (ANNSIM)","author":"Lutz CB","key":"bibr20-00375497241229753"},{"key":"bibr21-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78911-6_2"},{"key":"bibr22-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1137\/140996148"},{"key":"bibr23-00375497241229753","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2020.00230"},{"key":"bibr24-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104421"},{"key":"bibr25-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-69355-0_16"},{"key":"bibr26-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2011.12.008"},{"key":"bibr27-00375497241229753","volume":"13","author":"Angali A","year":"2021","journal-title":"Inte J Intell Syst Appl"},{"key":"bibr28-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.69.045104"},{"key":"bibr29-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1140\/epjb\/e2009-00347-4"},{"first-page":"128","volume-title":"MASCOTS\u201999. Proceedings of the seventh international symposium on modeling, analysis and simulation of computer and telecommunication systems","author":"Riley GF","key":"bibr30-00375497241229753"},{"key":"bibr31-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12935"},{"first-page":"636","volume-title":"2009 11th IEEE international conference on computer-aided design and computer graphics","author":"Fekete JD","key":"bibr32-00375497241229753"},{"key":"bibr33-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2022.3153838"},{"first-page":"345","volume-title":"2023 annual modeling and simulation conference (ANNSIM)","author":"Nguyen M","key":"bibr34-00375497241229753"},{"key":"bibr35-00375497241229753","first-page":"56","volume":"12","author":"Ta V","year":"2021","journal-title":"J Methods Meas Soc Sci"},{"key":"bibr36-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1561\/0600000027"},{"key":"bibr37-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-022-10275-5"},{"key":"bibr38-00375497241229753","volume":"81","author":"Maimon OZ","year":"2014","journal-title":"Data mining with decision trees: theory and applications"},{"key":"bibr39-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.isprsjprs.2011.11.002"},{"key":"bibr40-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2016.04.013"},{"journal-title":"Comput Econ","author":"Chu B","key":"bibr41-00375497241229753"},{"volume":"134","volume-title":"Machine Learning in Clinical Neuroscience","author":"Kernbach JM","key":"bibr42-00375497241229753"},{"first-page":"187","volume-title":"Workshop on computational organization design (AAAI spring symposium series)","author":"Talukdar S","key":"bibr43-00375497241229753"},{"volume-title":"Complex Social and Behavioral Systems. Encyclopedia of Complexity and Systems Science Series","year":"2009","author":"Parry HR","key":"bibr44-00375497241229753"},{"key":"bibr45-00375497241229753","first-page":"271","volume-title":"Agent-based models of geographical systems","author":"Parry HR","year":"2011"},{"key":"bibr46-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-29174-3"},{"first-page":"139","volume-title":"Advances in practical applications of survivable agents and multi-agent systems: the PAAMS collection: 17th international conference, PAAMS 2019","author":"Kosiachenko L","key":"bibr47-00375497241229753"},{"first-page":"280","volume-title":"2020 zooming innovation in consumer technologies conference (ZINC)","author":"Petrovi\u0107 N","key":"bibr48-00375497241229753"},{"first-page":"6863","volume-title":"Proceedings of the 56th Hawaii international conference on system sciences","author":"Ghumrawi K","key":"bibr49-00375497241229753"},{"key":"bibr50-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1038\/srep01905"},{"key":"bibr51-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1038\/nature02541"},{"key":"bibr52-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1093\/comnet\/cny004"},{"key":"bibr53-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2017.09.007"},{"key":"bibr54-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1109\/TPWRS.2015.2439237"},{"key":"bibr55-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-77980-1_50"},{"key":"bibr56-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-01604-7_42"},{"first-page":"382","volume-title":"2019 IEEE intelligent transportation systems conference (ITSC)","author":"Zhu L","key":"bibr57-00375497241229753"},{"key":"bibr58-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2019.123536"},{"key":"bibr59-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1098\/rsif.2021.0445"},{"journal-title":"arXiv","year":"2016","author":"Kipf TN","key":"bibr60-00375497241229753"},{"key":"bibr61-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.01.001"},{"journal-title":"arXiv","year":"2017","author":"Garcia V","key":"bibr62-00375497241229753"},{"journal-title":"arXiv","year":"2017","author":"Yu B","key":"bibr63-00375497241229753"},{"first-page":"1684","volume-title":"Proceedings of the 38th International Conference on Machine Learning","author":"Chen Y","key":"bibr64-00375497241229753"},{"key":"bibr65-00375497241229753","doi-asserted-by":"publisher","DOI":"10.3390\/math11010224"},{"key":"bibr66-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2765202"},{"key":"bibr67-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1109\/JAS.2017.7510583"},{"key":"bibr68-00375497241229753","first-page":"24","volume":"9","author":"Ismail N","year":"2022","journal-title":"Inf Process Agric"},{"key":"bibr69-00375497241229753","doi-asserted-by":"publisher","DOI":"10.3390\/foods9020113"},{"key":"bibr70-00375497241229753","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2019.2906821"}],"container-title":["SIMULATION"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00375497241229753","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/00375497241229753","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/00375497241229753","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,3]],"date-time":"2025-03-03T01:43:09Z","timestamp":1740966189000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/00375497241229753"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,22]]},"references-count":70,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1177\/00375497241229753"],"URL":"https:\/\/doi.org\/10.1177\/00375497241229753","relation":{},"ISSN":["0037-5497","1741-3133"],"issn-type":[{"type":"print","value":"0037-5497"},{"type":"electronic","value":"1741-3133"}],"subject":[],"published":{"date-parts":[[2024,2,22]]}}}