{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T02:42:38Z","timestamp":1768012958851,"version":"3.49.0"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA","license":[{"start":{"date-parts":[[2019,10,10]],"date-time":"2019-10-10T00:00:00Z","timestamp":1570665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2019,10,10]]},"abstract":"<jats:p>Online experiments have become a ubiquitous aspect of design and engineering processes within Internet firms. As the scale of experiments has grown, so has the complexity of their design and implementation. In response, firms have developed software frameworks for designing and deploying online experiments. Ensuring that experiments in these frameworks are correctly designed and that their results are trustworthy---referred to as internal validity---can be difficult. Currently, verifying internal validity requires manual inspection by someone with substantial expertise in experimental design.<\/jats:p>\n          <jats:p>We present the first approach for statically checking the internal validity of online experiments. Our checks are based on well-known problems that arise in experimental design and causal inference. Our analyses target PlanOut, a widely deployed, open-source experimentation framework that uses a domain-specific language to specify and run complex experiments. We have built a tool called PlanAlyzer that checks PlanOut programs for a variety of threats to internal validity, including failures of randomization, treatment assignment, and causal sufficiency. PlanAlyzer uses its analyses to automatically generate contrasts, a key type of information required to perform valid statistical analyses over the results of these experiments. We demonstrate PlanAlyzer's utility on a corpus of PlanOut scripts deployed in production at Facebook, and we evaluate its ability to identify threats to validity on a mutated subset of this corpus. PlanAlyzer has both precision and recall of 92% on the mutated corpus, and 82% of the contrasts it generates match hand-specified data.<\/jats:p>","DOI":"10.1145\/3360608","type":"journal-article","created":{"date-parts":[[2019,10,11]],"date-time":"2019-10-11T14:53:33Z","timestamp":1570805613000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["PlanAlyzer: assessing threats to the validity of online experiments"],"prefix":"10.1145","volume":"3","author":[{"given":"Emma","family":"Tosch","sequence":"first","affiliation":[{"name":"University of Massachusetts Amherst, USA"}]},{"given":"Eytan","family":"Bakshy","sequence":"additional","affiliation":[{"name":"Facebook, USA"}]},{"given":"Emery D.","family":"Berger","sequence":"additional","affiliation":[{"name":"University of Massachusetts Amherst, USA"}]},{"given":"David D.","family":"Jensen","sequence":"additional","affiliation":[{"name":"University of Massachusetts Amherst, USA"}]},{"given":"J. Eliot B.","family":"Moss","sequence":"additional","affiliation":[{"name":"University of Massachusetts Amherst, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,10,10]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Meeting of the Midwest Political Science Association.","author":"Ahler Douglas J","year":"2019","unstructured":"Douglas J Ahler , Carolyn E Roush , and Gaurav Sood . 2019 . The Micro-Task Market for Lemons: Data Quality on Amazon\u2019s Mechanical Turk . Presented at the Meeting of the Midwest Political Science Association. Douglas J Ahler, Carolyn E Roush, and Gaurav Sood. 2019. The Micro-Task Market for Lemons: Data Quality on Amazon\u2019s Mechanical Turk. Presented at the Meeting of the Midwest Political Science Association."},{"key":"e_1_2_2_2_1","volume-title":"Techniques","author":"Aho Alfred V","unstructured":"Alfred V Aho , Ravi Sethi , and Jeffrey D Ullman . 1986. Compilers , Principles , Techniques . Addison Wesley , 75 Arlington Street, Suite 300 in Boston, Mass. Alfred V Aho, Ravi Sethi, and Jeffrey D Ullman. 1986. Compilers, Principles, Techniques. Addison Wesley, 75 Arlington Street, Suite 300 in Boston, Mass."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1996.10476902"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567967"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026025"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2384616.2384663"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378704.1378723"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557139"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2012.6344535"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31128-4_15"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/115372.115320"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78800-3_24"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/360051.360056"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1200303"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/24039.24041"},{"key":"e_1_2_2_17_1","volume-title":"Design of experiments. Br Med J 1, 3923","author":"Fisher Ronald Aylmer","year":"1936","unstructured":"Ronald Aylmer Fisher . 1936. Design of experiments. Br Med J 1, 3923 ( 1936 ), 554\u2013554. Ronald Aylmer Fisher. 1936. Design of experiments. Br Med J 1, 3923 (1936), 554\u2013554."},{"key":"e_1_2_2_18_1","volume-title":"Refactoring: improving the design of existing code","author":"Fowler Martin","unstructured":"Martin Fowler . 2018. Refactoring: improving the design of existing code . Addison-Wesley Professional , Boston, MA, USA . Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley Professional, Boston, MA, USA."},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2603088.2603097"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010095604496"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.2307\/2111475"},{"key":"e_1_2_2_22_1","volume-title":"Field experiments: Design, analysis, and interpretation","author":"Gerber Alan S","unstructured":"Alan S Gerber and Donald P Green . 2012. Field experiments: Design, analysis, and interpretation . WW Norton , New York, NY, USA . Alan S Gerber and Donald P Green. 2012. Field experiments: Design, analysis, and interpretation. WW Norton, New York, NY, USA."},{"key":"e_1_2_2_23_1","volume-title":"Proc. 24th Conf. Uncertainty in Artificial Intelligence (UAI). JMLR: W&amp;CP, Online, 220\u2013229","author":"Goodman Noah D","year":"2008","unstructured":"Noah D Goodman , Vikash K Mansinghka , Daniel M Roy , Keith Bonawitz , and Joshua B Tenenbaum . 2008 . Church: A language for generative models . In Proc. 24th Conf. Uncertainty in Artificial Intelligence (UAI). JMLR: W&amp;CP, Online, 220\u2013229 . Noah D Goodman, Vikash K Mansinghka, Daniel M Roy, Keith Bonawitz, and Joshua B Tenenbaum. 2008. Church: A language for generative models. In Proc. 24th Conf. Uncertainty in Artificial Intelligence (UAI). JMLR: W&amp;CP, Online, 220\u2013229."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2535838.2535850"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593882.2593900"},{"key":"e_1_2_2_26_1","first-page":"1","article-title":"R package FrF2 for creating and analyzing fractional factorial 2-level designs","volume":"56","author":"Gr\u00f6nmping Ulrike","year":"2014","unstructured":"Ulrike Gr\u00f6nmping . 2014 . R package FrF2 for creating and analyzing fractional factorial 2-level designs . Journal of Statistical Software 56 , 1 (2014), 1 \u2013 56 . Ulrike Gr\u00f6nmping. 2014. R package FrF2 for creating and analyzing fractional factorial 2-level designs. Journal of Statistical Software 56, 1 (2014), 1\u201356.","journal-title":"Journal of Statistical Software"},{"key":"e_1_2_2_27_1","unstructured":"Ulrike Gr\u00f6nmping. 2016. FrF2: Fractional Factorial Designs with 2-Level Factors. http:\/\/CRAN.R-project.org\/package=FrF2  Ulrike Gr\u00f6nmping. 2016. FrF2: Fractional Factorial Designs with 2-Level Factors. http:\/\/CRAN.R-project.org\/package=FrF2"},{"key":"e_1_2_2_28_1","volume-title":"CRAN Task View: Design of Experiments (DoE) &amp","author":"Gr\u00f6nmping Ulrike","unstructured":"Ulrike Gr\u00f6nmping . 2017. CRAN Task View: Design of Experiments (DoE) &amp ; Analysis of Experimental Data . http:\/\/CRAN.R-project.org\/view=ExperimentalDesign Ulrike Gr\u00f6nmping. 2017. CRAN Task View: Design of Experiments (DoE) &amp; Analysis of Experimental Data. http:\/\/CRAN.R-project.org\/view=ExperimentalDesign"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.129"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2012.6405300"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9296-2"},{"key":"e_1_2_2_32_1","unstructured":"Miguel A Hern\u00e1n and Jamie M Robins. 2016. Causal Inference. Forthcoming.  Miguel A Hern\u00e1n and Jamie M Robins. 2016. Causal Inference. Forthcoming."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401938"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_2_2_35_1","volume-title":"Online controlled experiments and A\/B tests. Encyclopedia of Machine Learning and Data Mining","author":"Kohavi Ron","unstructured":"Ron Kohavi and Roger Longbotham . 2015. Online controlled experiments and A\/B tests. Encyclopedia of Machine Learning and Data Mining , C. Sammut and G. Webb, Eds. Ron Kohavi and Roger Longbotham. 2015. Online controlled experiments and A\/B tests. Encyclopedia of Machine Learning and Data Mining, C. Sammut and G. Webb, Eds."},{"key":"e_1_2_2_36_1","volume-title":"Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18, 1","author":"Kohavi Ron","year":"2009","unstructured":"Ron Kohavi , Roger Longbotham , Dan Sommerfield , and Randal M Henne . 2009. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18, 1 ( 2009 ), 140\u2013181. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. 2009. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery 18, 1 (2009), 140\u2013181."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNET.2013.2281542"},{"key":"e_1_2_2_38_1","volume-title":"Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Annual review of public health 21, 1","author":"Little Roderick J","year":"2000","unstructured":"Roderick J Little and Donald B Rubin . 2000. Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Annual review of public health 21, 1 ( 2000 ), 121\u2013145. Roderick J Little and Donald B Rubin. 2000. Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Annual review of public health 21, 1 (2000), 121\u2013145."},{"key":"e_1_2_2_40_1","volume-title":"Mediterranean diet, and cardiovascular disease. Current opinion in lipidology 25, 1","author":"Martinez-Gonzalez Miguel Angel","year":"2014","unstructured":"Miguel Angel Martinez-Gonzalez and Maira Bes-Rastrollo . 2014. Dietary patterns , Mediterranean diet, and cardiovascular disease. Current opinion in lipidology 25, 1 ( 2014 ), 20\u201326. Miguel Angel Martinez-Gonzalez and Maira Bes-Rastrollo. 2014. Dietary patterns, Mediterranean diet, and cardiovascular disease. Current opinion in lipidology 25, 1 (2014), 20\u201326."},{"key":"e_1_2_2_41_1","unstructured":"Alison McCook. 2018. Errors Trigger Retraction Of Study On Mediterranean Diet\u2019s Heart Benefits. https: \/\/www.npr.org\/sections\/health-shots\/2018\/06\/13\/619619302\/errors-trigger-retraction-of-study-on-mediterraneandiets-heart-benefits [Online; Last accessed 5 August 2019.].  Alison McCook. 2018. Errors Trigger Retraction Of Study On Mediterranean Diet\u2019s Heart Benefits. https: \/\/www.npr.org\/sections\/health-shots\/2018\/06\/13\/619619302\/errors-trigger-retraction-of-study-on-mediterraneandiets-heart-benefits [Online; Last accessed 5 August 2019.]."},{"key":"e_1_2_2_42_1","unstructured":"Tom Minka and John Winn. 2009. Gates. In Advances in Neural Information Processing Systems. JMLR: W&amp;CP Online 1073\u20131080.  Tom Minka and John Winn. 2009. Gates. In Advances in Neural Information Processing Systems. JMLR: W&amp;CP Online 1073\u20131080."},{"key":"e_1_2_2_43_1","unstructured":"T. Minka J.M. Winn J.P. Guiver S. Webster Y. Zaykov B. Yangel A. Spengler and J. Bronskill. 2014. Infer.NET 2.6. Microsoft Research Cambridge. http:\/\/research.microsoft.com\/infernet.  T. Minka J.M. Winn J.P. Guiver S. Webster Y. Zaykov B. Yangel A. Spengler and J. Bronskill. 2014. Infer.NET 2.6. Microsoft Research Cambridge. http:\/\/research.microsoft.com\/infernet."},{"key":"e_1_2_2_44_1","volume-title":"Counterfactuals and causal inference","author":"Morgan Stephen L","unstructured":"Stephen L Morgan and Christopher Winship . 2014. Counterfactuals and causal inference . Cambridge University Press , Cambridge, UK . Stephen L Morgan and Christopher Winship. 2014. Counterfactuals and causal inference. Cambridge University Press, Cambridge, UK."},{"key":"e_1_2_2_45_1","volume-title":"Advanced compiler design implementation. Morgan Kaufmann","author":"Muchnick Steven S","unstructured":"Steven S Muchnick . 1997. Advanced compiler design implementation. Morgan Kaufmann , Burlington, MA, USA . Steven S Muchnick. 1997. Advanced compiler design implementation. Morgan Kaufmann, Burlington, MA, USA."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9868.00389"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/287893.287899"},{"key":"e_1_2_2_48_1","volume-title":"Fourth Workshop on Human Computation (HCOMP\u201912)","author":"Parkes David C","year":"2012","unstructured":"David C Parkes , Andrew Mao , Yiling Chen , Krzysztof Z Gajos , Ariel Procaccia , and Haoqi Zhang . 2012 . Turkserver: Enabling synchronous and longitudinal online experiments . Fourth Workshop on Human Computation (HCOMP\u201912) . David C Parkes, Andrew Mao, Yiling Chen, Krzysztof Z Gajos, Ariel Procaccia, and Haoqi Zhang. 2012. Turkserver: Enabling synchronous and longitudinal online experiments. Fourth Workshop on Human Computation (HCOMP\u201912)."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161"},{"key":"e_1_2_2_50_1","volume-title":"Practical probabilistic programming","author":"Pfeffer Avi","unstructured":"Avi Pfeffer . 2016. Practical probabilistic programming . Manning Publications Co., Shelter Island, NY, USA . Avi Pfeffer. 2016. Practical probabilistic programming. Manning Publications Co., Shelter Island, NY, USA."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1198\/016214504000001880"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSAC.2002.806121"},{"key":"e_1_2_2_53_1","volume-title":"Reasoning about programs in continuation-passing style. Lisp and symbolic computation 6, 3-4","author":"Sabry Amr","year":"1993","unstructured":"Amr Sabry and Matthias Felleisen . 1993. Reasoning about programs in continuation-passing style. Lisp and symbolic computation 6, 3-4 ( 1993 ), 289\u2013360. Amr Sabry and Matthias Felleisen. 1993. Reasoning about programs in continuation-passing style. Lisp and symbolic computation 6, 3-4 (1993), 289\u2013360."},{"key":"e_1_2_2_54_1","volume-title":"JMP: Design of Experiments. https:\/\/www.jmp.com\/en_us\/about.html","author":"Sall John","year":"1989","unstructured":"John Sall . 1989 . JMP: Design of Experiments. https:\/\/www.jmp.com\/en_us\/about.html John Sall. 1989. JMP: Design of Experiments. https:\/\/www.jmp.com\/en_us\/about.html"},{"key":"e_1_2_2_55_1","volume-title":"Causal Reasoning: Disseminating New Curricula with Online Courseware. Presented at the American Education Research Association.","author":"Scheines Richard","year":"2003","unstructured":"Richard Scheines . 2003 . Causal Reasoning: Disseminating New Curricula with Online Courseware. Presented at the American Education Research Association. Richard Scheines. 2003. Causal Reasoning: Disseminating New Curricula with Online Courseware. Presented at the American Education Research Association."},{"key":"e_1_2_2_56_1","volume-title":"The Oxford Handbook of Political Methodology","author":"Sekhon Jasjeet S","unstructured":"Jasjeet S Sekhon . 2008. The Neyman-Rubin model of causal inference and estimation via matching methods . In The Oxford Handbook of Political Methodology . Oxford University Press , Oxford, UK , 271\u2013299. Jasjeet S Sekhon. 2008. The Neyman-Rubin model of causal inference and estimation via matching methods. In The Oxford Handbook of Political Methodology . Oxford University Press, Oxford, UK, 271\u2013299."},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081706.1081750"},{"key":"e_1_2_2_58_1","volume-title":"Campbell","author":"Shadish William R.","year":"2002","unstructured":"William R. Shadish , Thomas D. Cook , and Donald T . Campbell . 2002 . Experimental and quasi-experimental designs for generalized causal inference . Houghton Mifflin Company , Boston, MA, USA. William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference . Houghton Mifflin Company, Boston, MA, USA."},{"key":"e_1_2_2_59_1","volume-title":"Proceedings of the 2nd conference on Symposium on Networked Systems Design &amp; Implementation-Volume 2. USENIX Association","author":"Sherman Alex","year":"2005","unstructured":"Alex Sherman , Philip A Lisiecki , Andy Berkheimer , and Joel Wein . 2005 . ACMS: The Akamai configuration management system . In Proceedings of the 2nd conference on Symposium on Networked Systems Design &amp; Implementation-Volume 2. USENIX Association , Berkley, CA, USA, 245\u2013258. Alex Sherman, Philip A Lisiecki, Andy Berkheimer, and Joel Wein. 2005. ACMS: The Akamai configuration management system. In Proceedings of the 2nd conference on Symposium on Networked Systems Design &amp; Implementation-Volume 2. USENIX Association, Berkley, CA, USA, 245\u2013258."},{"key":"e_1_2_2_61_1","volume-title":"Henrik Toft S\u00f8rensen, and William J Blot","author":"Signorello Lisa B","year":"2002","unstructured":"Lisa B Signorello , Joseph K McLaughlin , Loren Lipworth , S\u00f8ren Friis , Henrik Toft S\u00f8rensen, and William J Blot . 2002 . Confounding by indication in epidemiologic studies of commonly used analgesics. American journal of therapeutics 9, 3 (2002), 199\u2013205. Lisa B Signorello, Joseph K McLaughlin, Loren Lipworth, S\u00f8ren Friis, Henrik Toft S\u00f8rensen, and William J Blot. 2002. Confounding by indication in epidemiologic studies of commonly used analgesics. American journal of therapeutics 9, 3 (2002), 199\u2013205."},{"key":"e_1_2_2_62_1","volume-title":"How Common is Breast Cancer? https:\/\/www.cancer.org\/cancer\/breast-cancer\/about\/howcommon-is-breast-cancer.html [Online","author":"American Cancer Society","year":"2019","unstructured":"American Cancer Society . 2019a. How Common is Breast Cancer? https:\/\/www.cancer.org\/cancer\/breast-cancer\/about\/howcommon-is-breast-cancer.html [Online ; Last accessed 12 August 2019 .]. American Cancer Society. 2019a. How Common is Breast Cancer? https:\/\/www.cancer.org\/cancer\/breast-cancer\/about\/howcommon-is-breast-cancer.html [Online; Last accessed 12 August 2019.]."},{"key":"e_1_2_2_63_1","volume-title":"Key Statistics for Breast Cancer in Men. https:\/\/www.cancer.org\/cancer\/breast-cancer-inmen\/about\/key-statistics.html [Online","author":"American Cancer Society","year":"2019","unstructured":"American Cancer Society . 2019b. Key Statistics for Breast Cancer in Men. https:\/\/www.cancer.org\/cancer\/breast-cancer-inmen\/about\/key-statistics.html [Online ; Last accessed 12 August 2019 .]. American Cancer Society. 2019b. Key Statistics for Breast Cancer in Men. https:\/\/www.cancer.org\/cancer\/breast-cancer-inmen\/about\/key-statistics.html [Online; Last accessed 12 August 2019.]."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623667"},{"key":"e_1_2_2_65_1","volume-title":"Category theory for the sciences","author":"Spivak David I","unstructured":"David I Spivak . 2014. Category theory for the sciences . MIT Press , Cambridge, MA, USA . David I Spivak. 2014. Category theory for the sciences. MIT Press, Cambridge, MA, USA."},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0024274"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1177012031"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098057"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815401"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v076.i12"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660193.2660206"},{"key":"e_1_2_2_73_1","volume-title":"The complexity of computing the permanent. Theoretical computer science 8, 2","author":"Valiant Leslie G","year":"1979","unstructured":"Leslie G Valiant . 1979. The complexity of computing the permanent. Theoretical computer science 8, 2 ( 1979 ), 189\u2013201. Leslie G Valiant. 1979. The complexity of computing the permanent. Theoretical computer science 8, 2 (1979), 189\u2013201."},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijforecast.2014.06.001"},{"key":"e_1_2_2_75_1","volume-title":"Proceedings of the 17th International Conference on Artificial Intelligence and Statistics","volume":"33","author":"Wood Frank","unstructured":"Frank Wood , Jan Willem van de Meent, and Vikash Mansinghka. 2014. A New Approach to Probabilistic Programming Inference . In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics , Vol. 33 . JMLR: W&amp;CP, Online, 1024\u20131032. Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. 2014. A New Approach to Probabilistic Programming Inference. In Proceedings of the 17th International Conference on Artificial Intelligence and Statistics, Vol. 33. JMLR: W&amp;CP, Online, 1024\u20131032."}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360608","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3360608","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:22:59Z","timestamp":1750202579000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360608"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,10]]},"references-count":72,"journal-issue":{"issue":"OOPSLA","published-print":{"date-parts":[[2019,10,10]]}},"alternative-id":["10.1145\/3360608"],"URL":"https:\/\/doi.org\/10.1145\/3360608","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,10]]},"assertion":[{"value":"2019-10-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}