{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T12:35:24Z","timestamp":1740141324952,"version":"3.37.3"},"reference-count":18,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,30]],"date-time":"2022-04-30T00:00:00Z","timestamp":1651276800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,30]],"date-time":"2022-04-30T00:00:00Z","timestamp":1651276800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"crossref","award":["HR0011937661"],"award-info":[{"award-number":["HR0011937661"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Math Organ Theory"],"published-print":{"date-parts":[[2023,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.<\/jats:p>","DOI":"10.1007\/s10588-021-09349-6","type":"journal-article","created":{"date-parts":[[2022,5,2]],"date-time":"2022-05-02T13:11:50Z","timestamp":1651497110000},"page":"242-263","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["What can simulation test beds teach us about social science? Results of the ground truth program"],"prefix":"10.1007","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9203-9049","authenticated-orcid":false,"given":"Asmeret","family":"Naugle","sequence":"first","affiliation":[]},{"given":"Daniel","family":"Krofcheck","sequence":"additional","affiliation":[]},{"given":"Christina","family":"Warrender","sequence":"additional","affiliation":[]},{"given":"Kiran","family":"Lakkaraju","sequence":"additional","affiliation":[]},{"given":"Laura","family":"Swiler","sequence":"additional","affiliation":[]},{"given":"Stephen","family":"Verzi","sequence":"additional","affiliation":[]},{"given":"Ben","family":"Emery","sequence":"additional","affiliation":[]},{"given":"Jaimie","family":"Murdock","sequence":"additional","affiliation":[]},{"given":"Michael","family":"Bernard","sequence":"additional","affiliation":[]},{"given":"Vicente","family":"Romero","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,4,30]]},"reference":[{"key":"9349_CR1","doi-asserted-by":"publisher","first-page":"226","DOI":"10.2307\/2533213","volume":"50","author":"EJ Bedrick","year":"1994","unstructured":"Bedrick EJ, Tsai CL (1994) Model selection for multivariate regression in small samples. Biometrics 50:226\u2013331","journal-title":"Biometrics"},{"key":"9349_CR3","volume-title":"Model selection and multimodel inference: a practical information-theoretic approach","author":"KP Burnham","year":"2002","unstructured":"Burnham KP, Anderson DR (2002) Model selection and multimodel inference: a practical information-theoretic approach, 2nd edn. Springer, Heidelberg","edition":"2"},{"key":"9349_CR5","doi-asserted-by":"crossref","unstructured":"Deng J, Dong W, Socher R, et al. (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp 248\u2013255","DOI":"10.1109\/CVPR.2009.5206848"},{"issue":"2","key":"9349_CR6","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.strusafe.2008.06.020","volume":"31","author":"A Der Kiureghian","year":"2009","unstructured":"Der Kiureghian A, Ditlevsen O (2009) Aleatory or epistemic? Does it matter? Struct Saf 31(2):105\u2013112","journal-title":"Struct Saf"},{"key":"9349_CR7","doi-asserted-by":"crossref","unstructured":"Graziul C, Belikov A, Chattopadyay I, Chen Z, Fang H, Girdhar A, Jia X, Krafft PM, Kleiman-Weiner M, Lewis C, Liang C, Muchovej J, Vientos A, Young M, Evans J (2022) Does big data serve policy? Not without context. An experiment with in silico social science. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-022-09362-3"},{"issue":"8","key":"9349_CR11","doi-asserted-by":"publisher","first-page":"e124","DOI":"10.1371\/journal.pmed.0020124","volume":"2","author":"JP Ioannidis","year":"2005","unstructured":"Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2(8):e124. https:\/\/doi.org\/10.1371\/journal.pmed.0020124","journal-title":"PLoS Med"},{"issue":"10","key":"9349_CR12","doi-asserted-by":"publisher","first-page":"1002264","DOI":"10.1371\/journal.pbio.1002264","volume":"13","author":"JPA Ioannidis","year":"2015","unstructured":"Ioannidis JPA, Fanelli D, Dunne DD, Goodman SN (2015) Meta-research: evaluation and improvement of research methods and practices. PLOS Biol 13(10):1002264. https:\/\/doi.org\/10.1371\/journal.pbio.1002264","journal-title":"PLOS Biol"},{"key":"9349_CR13","volume-title":"Laboratory life: the construction of scientific facts","author":"B Latour","year":"1979","unstructured":"Latour B, Woolgar S (1979) Laboratory life: the construction of scientific facts. Princeton University Press, Princeton"},{"key":"9349_CR14","doi-asserted-by":"publisher","unstructured":"Naugle A, Russell A, Lakkaraju K, Swiler L, Verzi S, Romero V (2022) The ground truth program: simulations as test beds for social science research methods. Comput Math Organ Theory. https:\/\/doi.org\/10.1007\/s10588-021-09346-9","DOI":"10.1007\/s10588-021-09346-9"},{"key":"9349_CR15","unstructured":"Ossowska M and Ossowski S (1935) \"The Science of Science\", originally published in Polish as \"Nauka o nauce\" (\"The Science of Science\") in Nauka Polska (Polish Science). vol. XX, no. 3."},{"key":"9349_CR16","doi-asserted-by":"publisher","unstructured":"Parunak V (2022) SCAMP\u2019s stigmergic model of social conflict. Comput Math Organ Theory. https:\/\/doi.org\/10.1007\/s10588-021-09347-8","DOI":"10.1007\/s10588-021-09347-8"},{"key":"9349_CR17","doi-asserted-by":"crossref","unstructured":"Pynadath D, Bistra D, Jeong DC, John RS, Marsella SC, Merchant C, Miller LC, Read SJ (2022) Disaster world: decision-theoretic agents for simulating population responses to hurricanes. Comput Math Organ Theory, this issue","DOI":"10.1007\/s10588-022-09359-y"},{"key":"9349_CR18","doi-asserted-by":"publisher","unstructured":"Rager S, Alice L, Shannon P, Jennifer M, Marshall SP, Noshir C (2022) Groups, governance, and greed: the ACCESS world model. Comput Math Organ Theory. https:\/\/doi.org\/10.1007\/s10588-021-09352-x","DOI":"10.1007\/s10588-021-09352-x"},{"key":"9349_CR20","doi-asserted-by":"publisher","unstructured":"Schmidt A, Cameron C, Lowman C, Brule J, Deshpande AJ, Seyyed AF, Barash V, Greenberg AM, Costello CJ, Sherman ES, Bhattacharya R, McQuillan L, Perrone A, Kouskoulas YA, Find C, Zhang J, Shpitser I, Macy MW (2022) Searching for explanations: testing social scientific methods in synthetic ground-truthed worlds. Comput Math Organ Theory. https:\/\/doi.org\/10.1007\/s10588-021-09353-w","DOI":"10.1007\/s10588-021-09353-w"},{"key":"9349_CR21","doi-asserted-by":"crossref","unstructured":"Seabold S and Josef P (2010) Statsmodels: econometric and statistical modeling with python. In: Proceedings of the 9th Python in Science Conference. 2010.","DOI":"10.25080\/Majora-92bf1922-011"},{"key":"9349_CR22","series-title":"Wiley Series on Probability and Statistics","doi-asserted-by":"publisher","DOI":"10.1002\/9780471722199","volume-title":"Linear Regression Analysis","author":"GAF Seber","year":"2003","unstructured":"Seber GAF, Lee AJ (2003) Linear Regression Analysis, 2nd edn. Wiley Series on Probability and Statistics. Wiley, New York","edition":"2"},{"key":"9349_CR23","unstructured":"Volkova S, Arendt D, Saldanha E, Glenski M, Ayton E, Cottam J, Aksoy S, Jefferson B, Shrivaram K (2022) Causal discovery and prediction of human behavior and social dynamics from observational data: generalizability, reproducibility and robustness. Comput Math Organ Theory, this issue"},{"key":"9349_CR25","doi-asserted-by":"publisher","unstructured":"Z\u00fcfle A, Carola W, Dieter P, Andrew C, Hamdi K, Joon-Seok K, Hyunjee J (2022) Urban life: a model of people and places. Comput Math Organ Theory. https:\/\/doi.org\/10.1007\/s10588-021-09348-7","DOI":"10.1007\/s10588-021-09348-7"}],"container-title":["Computational and Mathematical Organization Theory"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10588-021-09349-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10588-021-09349-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10588-021-09349-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T09:27:16Z","timestamp":1679650036000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10588-021-09349-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,30]]},"references-count":18,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3]]}},"alternative-id":["9349"],"URL":"https:\/\/doi.org\/10.1007\/s10588-021-09349-6","relation":{},"ISSN":["1381-298X","1572-9346"],"issn-type":[{"type":"print","value":"1381-298X"},{"type":"electronic","value":"1572-9346"}],"subject":[],"published":{"date-parts":[[2022,4,30]]},"assertion":[{"value":"19 October 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 April 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}