{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,10]],"date-time":"2026-11-10T17:39:04Z","timestamp":1794332344041,"version":"4.9.0"},"reference-count":27,"publisher":"Ovid Technologies (Wolters Kluwer Health)","issue":"6","content-domain":{"domain":["lww.com","ovid.com"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11]]},"abstract":"<jats:sec>\n            <jats:title>Background:<\/jats:title>\n            <jats:p>The use of machine learning to estimate exposure effects introduces a dependence between the results of an empirical study and the value of the seed used to fix the pseudo-random number generator.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods:<\/jats:title>\n            <jats:p>We used data from 10,038 pregnant women and a 10% subsample (N = 1004) to examine the extent to which the risk difference for the relation between fruit and vegetable consumption and preeclampsia risk changes under different seed values. We fit an augmented inverse probability weighted estimator with two Super Learner algorithms: a simple algorithm including random forests and single-layer neural networks and a more complex algorithm with a mix of tree-based, regression-based, penalized, and simple algorithms. We evaluated the distributions of risk differences, standard errors, and <jats:italic toggle=\"yes\">P<\/jats:italic> values that result from 5000 different seed value selections.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results:<\/jats:title>\n            <jats:p>Our findings suggest important variability in the risk difference estimates, as well as an important effect of the stacking algorithm used. The interquartile range width of the risk differences in the full sample with the simple algorithm was 13 per 1000. However, all other interquartile ranges were roughly an order of magnitude lower. The medians of the distributions of risk differences differed according to the sample size and the algorithm used.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusions:<\/jats:title>\n            <jats:p>Our findings add another dimension of concern regarding the potential for \u201cp-hacking,\u201d and further warrant the need to move away from simplistic evidentiary thresholds in empirical research. When empirical results depend on pseudo-random number generator seed values, caution is warranted in interpreting these results.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1097\/ede.0000000000001785","type":"journal-article","created":{"date-parts":[[2024,8,16]],"date-time":"2024-08-16T13:00:28Z","timestamp":1723813228000},"page":"779-786","update-policy":"https:\/\/doi.org\/10.1097\/lww.0000000000001000","source":"Crossref","is-referenced-by-count":10,"title":["Pseudo-random Number Generator Influences on Average Treatment Effect Estimates Obtained with Machine Learning"],"prefix":"10.1097","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1510-8175","authenticated-orcid":false,"given":"Ashley I.","family":"Naimi","sequence":"first","affiliation":[{"name":"Department of Epidemiology, Emory University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ya-Hui","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, Emory University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lisa M.","family":"Bodnar","sequence":"additional","affiliation":[{"name":"Department of Epidemiology, University of Pittsburgh School of Public Health, Atlanta, GA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"276","published-online":{"date-parts":[[2024,8,16]]},"reference":[{"key":"R1-20250801","doi-asserted-by":"crossref","first-page":"4156","DOI":"10.1073\/pnas.1804597116","article-title":"Metalearners for estimating heterogeneous treatment effects using machine learning.","volume":"116","author":"K\u00fcnzel","year":"2019","journal-title":"Proc Natl Acad Sci USA"},{"key":"R2-20250801","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1093\/aje\/kwq439","article-title":"Doubly robust estimation of causal effects.","volume":"173","author":"Jonsson-Funk","year":"2011","journal-title":"Am J Epidemiol"},{"key":"R4-20250801","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","article-title":"Stacked generalization.","volume":"5","author":"Wolpert","year":"1992","journal-title":"Neural Netw"},{"key":"R5-20250801","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1007\/s10654-018-0390-z","article-title":"Stacked generalization: an introduction to super learning.","volume":"33","author":"Naimi","year":"2018","journal-title":"Eur J Epidemiol"},{"key":"R6-20250801","doi-asserted-by":"crossref","first-page":"1536","DOI":"10.1093\/aje\/kwab201","article-title":"Challenges in obtaining valid causal effect estimates with machine learning algorithms.","volume":"192","author":"Naimi","year":"2023","journal-title":"Am J Epidemiol"},{"key":"R7-20250801","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1097\/EDE.0000000000001332","article-title":"Machine learning for causal inference: on the use of cross-fit estimators.","volume":"32","author":"Zivich","year":"2021","journal-title":"Epidemiology"},{"key":"R9-20250801","doi-asserted-by":"crossref","first-page":"539.e1","DOI":"10.1016\/j.ajog.2015.01.019","article-title":"A description of the methods of the Nulliparous Pregnancy Outcomes Study: monitoring mothers-to-be (nuMoM2b).","volume":"212","author":"Haas","year":"2015","journal-title":"Am J Obstet Gynecol"},{"key":"R10-20250801","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1016\/j.ajcnut.2023.06.007","article-title":"Is the association between fruits and vegetables and preeclampsia due to higher dietary vitamin C and carotenoid intakes?","volume":"118","author":"Bodnar","year":"2023","journal-title":"Am J Clin Nutr"},{"key":"R14-20250801","doi-asserted-by":"crossref","first-page":"1591","DOI":"10.1016\/j.jand.2018.05.021","article-title":"Update of the Healthy Eating Index: HEI-2015.","volume":"118","author":"Krebs-Smith","year":"2018","journal-title":"J Acad Nutr Diet"},{"key":"R15-20250801","first-page":"1122","article-title":"Report of the American College of Obstetricians and Gynecologists\u2019 task force on hypertension in pregnancy.","volume":"122","year":"2013","journal-title":"Obstet Gynecol"},{"key":"R16-20250801","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1097\/AOG.0000000000001805","article-title":"Association between sleep-disordered breathing and hypertensive disorders of pregnancy and gestational diabetes mellitus.","volume":"129","author":"Facco","year":"2017","journal-title":"Obstet Gynecol"},{"key":"R17-20250801","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1093\/aje\/kwad012","article-title":"Defining and identifying average treatment effects.","volume":"192","author":"Naimi","year":"2023","journal-title":"Am J Epidemiol"},{"key":"R18-20250801","author":"Polley","year":"2024"},{"key":"R19-20250801","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v033.i01","article-title":"Regularization paths for generalized linear models via coordinate descent.","volume":"33","author":"Friedman","year":"2010","journal-title":"J Stat Softw"},{"key":"R20-20250801","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v077.i01","article-title":"Ranger: a fast implementation of random forests for high dimensional data in C++ and R.","volume":"77","author":"Wright","year":"2017","journal-title":"J Stat Softw"},{"key":"R21-20250801","author":"Chen"},{"key":"R22-20250801","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1038\/s41567-020-0921-x","article-title":"Tail risk of contagious diseases.","volume":"16","author":"Cirillo","year":"2020","journal-title":"Nat Phys"},{"key":"R24-20250801","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1093\/pan\/mpp036","article-title":"An introduction to the augmented inverse propensity weighted estimator.","volume":"18","author":"Glynn","year":"2010","journal-title":"Political Analysis"},{"key":"R25-20250801","doi-asserted-by":"crossref","first-page":"C1","DOI":"10.1111\/ectj.12097","article-title":"Double\/debiased machine learning for treatment and structural parameters.","volume":"21","author":"Chernozhukov","year":"2018","journal-title":"Econom J"},{"key":"R26-20250801","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v108.i03","article-title":"DoubleML - an object-oriented implementation of double machine learning in R.","volume":"108","author":"Bach","year":"2024","journal-title":"J Stat Softw"},{"key":"R27-20250801","first-page":"2002.06305","article-title":"Fine-tuning pretrained language models: weight initializations, data orders, and early stopping.","author":"Dodge","year":"2020","journal-title":"arXiv"},{"key":"R28-20250801","first-page":"2210.13393","article-title":"We need to talk about random seeds.","author":"Bethard","year":"2022","journal-title":"arXiv"},{"key":"R29-20250801","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1111\/ppe.12711","article-title":"Analysis goals, error-cost sensitivity, and analysis hacking: essential considerations in hypothesis testing and multiple comparisons.","volume":"35","author":"Greenland","year":"2021","journal-title":"Paediatr Perinat Epidemiol"},{"key":"R30-20250801","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/272991.272995","article-title":"Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator.","volume":"8","author":"Matsumoto","year":"1998","journal-title":"ACM Trans Model Comput Simul"},{"key":"R32-20250801","doi-asserted-by":"crossref","first-page":"Article 11","DOI":"10.2202\/1557-4679.1043","article-title":"Targeted maximum likelihood learning.","volume":"2","author":"van der Laan","year":"2006","journal-title":"Int J Biostat"},{"key":"R33-20250801","doi-asserted-by":"crossref","first-page":"1396","DOI":"10.1093\/aje\/kwac062","article-title":"Use of a doubly robust machine-learning-based approach to evaluate body mass index as a modifier of the association between fruit and vegetable intake and preeclampsia.","volume":"191","author":"Bodnar","year":"2022","journal-title":"Am J Epidemiol"},{"key":"R34-20250801","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1093\/ajcn\/nqaa027","article-title":"Machine learning as a strategy to account for dietary synergy: an illustration based on dietary intake and adverse pregnancy outcomes.","volume":"111","author":"Bodnar","year":"2020","journal-title":"Am J Clin Nutr"}],"container-title":["Epidemiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.lww.com\/10.1097\/EDE.0000000000001785","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T15:04:55Z","timestamp":1754060695000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.lww.com\/10.1097\/EDE.0000000000001785"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,16]]},"references-count":27,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024]]}},"URL":"https:\/\/doi.org\/10.1097\/ede.0000000000001785","relation":{},"ISSN":["1044-3983","1531-5487"],"issn-type":[{"value":"1044-3983","type":"print"},{"value":"1531-5487","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,8,16]]}}}