{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,17]],"date-time":"2025-04-17T11:30:27Z","timestamp":1744889427791,"version":"3.37.3"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T00:00:00Z","timestamp":1650240000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T00:00:00Z","timestamp":1650240000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["HR0011937661"],"award-info":[{"award-number":["HR0011937661"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Math Organ Theory"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world\u2019s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.<\/jats:p>","DOI":"10.1007\/s10588-021-09346-9","type":"journal-article","created":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T08:02:40Z","timestamp":1650268960000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["The Ground Truth program: simulations as test beds for social science research methods"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9203-9049","authenticated-orcid":false,"given":"Asmeret","family":"Naugle","sequence":"first","affiliation":[]},{"given":"Adam","family":"Russell","sequence":"additional","affiliation":[]},{"given":"Kiran","family":"Lakkaraju","sequence":"additional","affiliation":[]},{"given":"Laura","family":"Swiler","sequence":"additional","affiliation":[]},{"given":"Stephen","family":"Verzi","sequence":"additional","affiliation":[]},{"given":"Vicente","family":"Romero","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,4,18]]},"reference":[{"key":"9346_CR1","volume-title":"General features of complex systems. Encyclopedia of life support systems","author":"Y Bar-Yam","year":"2002","unstructured":"Bar-Yam Y (2002) General features of complex systems. Encyclopedia of life support systems. UNESCO, Oxford"},{"key":"9346_CR2","first-page":"1","volume":"20","author":"R Botvinik-Nezer","year":"2020","unstructured":"Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, Kirchler M, Iwanir R, Mumford JA, Adcock RA, Avesani P (2020) Variability in the analysis of a single neuroimaging dataset by many teams. Nature 20:1\u20137","journal-title":"Nature"},{"key":"9346_CR3","unstructured":"Branch A, Cardon E, Ellis D, Russell A (2021) We ignore the human domain at our own peril. Modern war institute. https:\/\/mwi.usma.edu\/we-ignore-the-human-domain-at-our-own-peril\/. Accessed 20 July 2021"},{"key":"9346_CR4","volume-title":"Military strategy and the 21st century","author":"S Bryant","year":"2018","unstructured":"Bryant S, Cleveland C, Jensen B, Arnel D (2018) Military strategy and the 21st century. Cambria Press, Amherst"},{"key":"9346_CR5","unstructured":"Castellani B (2014) FOCUS: complexity and the failure of quantitative social science. Discover society. https:\/\/discoversociety.org\/2014\/11\/04\/focus-complexity-and-the-failure-of-quantitative-social-science\/. Accessed 20 July 2021"},{"key":"9346_CR6","doi-asserted-by":"publisher","DOI":"10.7249\/CT468","volume-title":"Understanding Russian \u201chybrid warfare\u201d and what can be done about it","author":"CS Chivvis","year":"2017","unstructured":"Chivvis CS (2017) Understanding Russian \u201chybrid warfare\u201d and what can be done about it. RAND Corporation, California"},{"key":"9346_CR7","doi-asserted-by":"publisher","first-page":"1523","DOI":"10.1109\/TIT.2005.844059","volume":"51","author":"R Cilibrasi","year":"2005","unstructured":"Cilibrasi R, Vitanyi PMB (2005) Clustering by compression. IEEE Trans Inf Theory 51:1523\u20131545","journal-title":"IEEE Trans Inf Theory"},{"key":"9346_CR8","unstructured":"Gerber AS, Green DP (2011) Field experiments and natural experiments. The Oxford handbook of political science. https:\/\/www.oxfordhandbooks.com\/view\/10.1093\/oxfordhb\/9780199604456.001.0001\/oxfordhb-9780199604456-e-050. Accessed 20 July 2021"},{"key":"9346_CR9","unstructured":"Gray Zone Project (2020) Center for Strategic and International Studies. https:\/\/www.csis.org\/grayzone. Accessed 20 July 2021"},{"key":"9346_CR10","doi-asserted-by":"crossref","unstructured":"Graziul C, Belikov A, Chattopadyay I, Chen A, Fang H, Girdhar A, Jia X, Krafft P, Kleiman-Weiner M, Lewis C, Liang C, Muchovej J, Vient\u00f3s A, Evans J (2022) Does big data serve policy? Not without context. An experiment with in silico social science. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-022-09362-3"},{"issue":"2","key":"9346_CR11","first-page":"92","volume":"2","author":"H Gregg","year":"2016","unstructured":"Gregg H (2016) Human domain and influence operations in the 21st century. Spec Oper J 2(2):92\u2013105","journal-title":"Spec Oper J"},{"key":"9346_CR12","unstructured":"Jones RC (2019) Deterring \u201cCompetition short of war\u201d: are gray zones the ardennes of our modern maginot line of traditional deterrence? Small wars journal. https:\/\/smallwarsjournal.com\/jrnl\/art\/deterring-competition-short-war-are-gray-zones-ardennes-our-modern-maginot-line. Accessed 20 July 2021"},{"key":"9346_CR13","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s13194-012-0056-8","volume":"3","author":"J Ladyman","year":"2013","unstructured":"Ladyman J, Lambert J, Wiesner K (2013) What is a complex system? Eur J Philos Sci 3:33\u201367","journal-title":"Eur J Philos Sci"},{"key":"9346_CR14","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.1126\/science.aaz8170","volume":"369","author":"DMJ Lazer","year":"2020","unstructured":"Lazer DMJ, Pentland A, Watts DJ, Aral S, Athey S, Contractor N, Freelon N, Gonzalez-Bailon S, King G, Margetts H, Nelson A, Salganik M, Strohmaier M, Vespignani A, Wagner C (2020) Computational social science: obstacles and opportunities. Science 369:1060\u20131062","journal-title":"Science"},{"issue":"2","key":"9346_CR15","first-page":"13","volume":"8","author":"A Lo","year":"2010","unstructured":"Lo A, Mueller M (2010) Warning: physics envy may be hazardous to your wealth! J Invest Manage 8(2):13\u201363","journal-title":"J Invest Manage"},{"key":"9346_CR16","volume-title":"Complex systems theory and evolution. Encyclopedia of evolution","author":"M Mitchell","year":"2002","unstructured":"Mitchell M, Newman M (2002) Complex systems theory and evolution. Encyclopedia of evolution. Oxford University Press, Oxford"},{"key":"9346_CR17","doi-asserted-by":"publisher","first-page":"e33799","DOI":"10.1371\/journal.pone.0033799","volume":"7","author":"E Mones","year":"2012","unstructured":"Mones E, Vicsek L, Vicsek T (2012) Hierarchy measure for complex networks. PLoS ONE 7:e33799","journal-title":"PLoS ONE"},{"key":"9346_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41562-016-0021","volume":"1","author":"MR Munaf\u00f2","year":"2017","unstructured":"Munaf\u00f2 MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, Simonsohn U, Wagenmakers EJ, Ware JJ, Ioannidis JPA (2017) A manifesto for reproducible science. Nat Hum Behav 1:1\u20139","journal-title":"Nat Hum Behav"},{"key":"9346_CR19","volume-title":"Graph-based similarity metrics for comparing simulations and causal loop diagrams","author":"AB Naugle","year":"2019","unstructured":"Naugle AB, Swiler LP, Lakkaraju K, Verzi S, Warrender C, Romero V (2019) Graph-based similarity metrics for comparing simulations and causal loop diagrams. Sandia National Laboratories, Albuquerque"},{"key":"9346_CR20","doi-asserted-by":"crossref","unstructured":"Naugle AB, Krofcheck D, Warrender C, Lakkaraju K, Swiler L, Verzi S, Emery B, Murdock J, Bernard M, Romero V (2022) What can simulation test beds teach us about social science? Results of the ground truth program. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-021-09349-6"},{"key":"9346_CR21","doi-asserted-by":"publisher","first-page":"13","DOI":"10.3389\/fdata.2019.00013","volume":"2","author":"A Olteanu","year":"2019","unstructured":"Olteanu A, Castillo C, Diaz F, K\u0131c\u0131man E (2019) Social data: biases, methodological pitfalls, and ethical boundaries. Front Big Data 2:13","journal-title":"Front Big Data"},{"key":"9346_CR22","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1126\/science.263.5147.641","volume":"263","author":"N Oreskes","year":"1994","unstructured":"Oreskes N, Shrader-Frechette K, Belitz K (1994) Verification, validation, and confirmation of numerical models in the earth sciences. Science 263:641\u2013646","journal-title":"Science"},{"key":"9346_CR23","doi-asserted-by":"crossref","unstructured":"Parunak V (2022) SCAMP\u2019s Stigmergic model of social conflict. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-021-09347-8"},{"key":"9346_CR24","unstructured":"Prabhakar A (2020) In the realm of the barely feasible. Issues in science and technology 37(1): 34\u201340"},{"key":"9346_CR25","doi-asserted-by":"crossref","unstructured":"Pynadath D, Dilkina B, Jeong D, John R, Marsella S, Merchant C, Miller L, Read S (2022) Disaster world: decision-theoretic agents for simulating population responses to hurricanes. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-022-09359-y"},{"key":"9346_CR26","doi-asserted-by":"crossref","unstructured":"Rager S, Leung A, Pinegar S, Mangels J, Poole M, Contractor N (2022) Groups, governance, and greed: the ACCESS world model. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-021-09352-x"},{"key":"9346_CR27","volume-title":"Information retrieval","author":"CJV Rijsbergen","year":"1979","unstructured":"Rijsbergen CJV (1979) Information retrieval, 2nd edn. Butterworth-Heinemann, USA","edition":"2"},{"key":"9346_CR28","unstructured":"Russell A (2019) Fomenting (reproducible) revolutions: DARPA, replication, and high-risk, high-payoff research (Video). In: Metascience 2019 Symposium. https:\/\/www.metascience2019.org\/presentations\/adam-russell\/. Accessed 20 July 2021"},{"key":"9346_CR29","doi-asserted-by":"crossref","unstructured":"Salganik MJ, Lundberg I, Kindel AT, Ahearn CE, Al-Ghoneim K, Almaatouq A, Altschul DM, Brand JE, Carnegie NB, Compton RJ, Datta D (2020) Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences. 117(15):8398\u20138403. https:\/\/www.pnas.org\/content\/117\/15\/8398. Accessed 20 July 2021","DOI":"10.1073\/pnas.1915006117"},{"issue":"2","key":"9346_CR30","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1080\/0022250X.1971.9989794","volume":"1","author":"TC Schelling","year":"1971","unstructured":"Schelling TC (1971) Dynamic models of segregation. J Math Sociol 1(2):143\u2013186","journal-title":"J Math Sociol"},{"key":"9346_CR31","doi-asserted-by":"crossref","unstructured":"Schmidt A, Cameron C, Lowman C, Brul\u00e9 J, Deshpande A, Faemi S, Barash V, Greenberg A, Costello C, Sherman E, Bhattacharya R, McQuillan L, Perrone A, Kouskoulas Y, Fink C, Zhang J, Shpitser I, Macy M (2022) Searching for explanations: Testing social scientific methods in synthetic ground-truthed worlds. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-021-09353-w"},{"key":"9346_CR32","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/978-0-387-33532-2_2","volume-title":"Complex systems science in biomedicine","author":"CR Shalizi","year":"2006","unstructured":"Shalizi CR (2006) Methods and techniques of complex systems science: an overview. In: Deisboeck TS, Kresh JY (eds) Complex systems science in biomedicine. Springer, US, Boston, pp 33\u2013114"},{"key":"9346_CR33","unstructured":"Underwood K (2020) The army shapes joint all-domain operations. SIGNAL Magazine. https:\/\/www.afcea.org\/content\/army-shapes-joint-all-domain-operations. Accessed 20 July 2021"},{"key":"9346_CR34","unstructured":"United States Special Operations Command (2015) Operating in the human domain. Version 1.0. August 2015. https:\/\/nsiteam.com\/operating-in-the-human-domain-1-0"},{"key":"9346_CR35","volume-title":"Regression, ANOVA, and the general linear model: a statistics primer","author":"P Vik","year":"2013","unstructured":"Vik P (2013) Regression, ANOVA, and the general linear model: a statistics primer. SAGE Publications, California"},{"key":"9346_CR36","unstructured":"Volkova S, Arendt D, Saldanha E, Glenski M, Ayton E, Cottam J, Aksoy S, Jeffereson B, Shrivaram K (2022) Causal discovery and prediction of human behavior and social dynamics from observational data: Generalizability, reproducibility and robustness. Comput Math Organ Theory, this issue"},{"key":"9346_CR37","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41562-016-0015","volume":"1","author":"DJ Watts","year":"2017","unstructured":"Watts DJ (2017) Should social science be more solution-oriented? Nat Hum Behav 1:1\u20135","journal-title":"Nat Hum Behav"},{"key":"9346_CR38","unstructured":"White MD (2012) Which is \u201charder\u201d: social science or physical science? In: Economics and ethics. https:\/\/www.economicsandethics.org\/2012\/07\/which-is-harder-social-science-or-physical-science.html. Accessed 20 July 2021"},{"key":"9346_CR39","unstructured":"Wilensky U (1997) NetLogo Segregation model. http:\/\/ccl.northwestern.edu\/netlogo\/models\/Segregation. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL."},{"key":"9346_CR40","unstructured":"Wilensky U (1999) NetLogo. http:\/\/ccl.northwestern.edu\/netlogo\/. Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL."},{"key":"9346_CR41","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1023\/A:1009982220290","volume":"1","author":"Y Yang","year":"1999","unstructured":"Yang Y (1999) An evaluation of statistical approaches to text categorization. Inf Retrieval 1:69\u201390","journal-title":"Inf Retrieval"},{"key":"9346_CR42","doi-asserted-by":"publisher","first-page":"100145","DOI":"10.1016\/j.bdr.2020.100145","volume":"21","author":"J Zhang","year":"2020","unstructured":"Zhang J, Wang W, Xia F et al (2020) Data-driven computational social science: a survey. Big Data Res 21:100145","journal-title":"Big Data Res"},{"key":"9346_CR43","doi-asserted-by":"crossref","unstructured":"Z\u00fcfle A, Wenk C, Pfoser D, Crooks A, Kavak H, Kim J, Jin H (2022) Urban life: a model of people and places. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-021-09348-7"}],"container-title":["Computational and Mathematical Organization Theory"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10588-021-09346-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10588-021-09346-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10588-021-09346-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T09:27:49Z","timestamp":1679650069000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10588-021-09346-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,18]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["9346"],"URL":"https:\/\/doi.org\/10.1007\/s10588-021-09346-9","relation":{},"ISSN":["1381-298X","1572-9346"],"issn-type":[{"type":"print","value":"1381-298X"},{"type":"electronic","value":"1572-9346"}],"subject":[],"published":{"date-parts":[[2022,4,18]]},"assertion":[{"value":"13 October 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 April 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}