{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:19:59Z","timestamp":1750220399970,"version":"3.41.0"},"reference-count":23,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2021,8,24]],"date-time":"2021-08-24T00:00:00Z","timestamp":1629763200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2021,9]]},"abstract":"<jats:p>\n            Online experiments are an integral part of the design and evaluation of software infrastructure at Internet firms. To handle the growing scale and complexity of these experiments, firms have developed software frameworks for their design and deployment. Ensuring that the results of experiments in these frameworks are trustworthy---referred to as\n            <jats:italic>internal validity<\/jats:italic>\n            ---can be difficult. Currently, verifying internal validity requires manual inspection by someone with substantial expertise in experimental design.\n          <\/jats:p>\n          <jats:p>We present the first approach for checking the internal validity of online experiments statically, that is, from code alone. We identify well-known problems that arise in experimental design and causal inference, which can take on unusual forms when expressed as computer programs: failures of randomization and treatment assignment, and causal sufficiency errors. Our analyses target PLANOUT, a popular framework that features a domain-specific language (DSL) to specify and run complex experiments. We have built PLANALYZER, a tool that checks PLANOUT programs for threats to internal validity, before automatically generating important data for the statistical analyses of a large class of experimental designs. We demonstrate PLANALYZER'S utility on a corpus of PLANOUT scripts deployed in production at Facebook, and we evaluate its ability to identify threats on a mutated subset of this corpus. PLANALYZER has both precision and recall of 92% on the mutated corpus, and 82% of the contrasts it generates match hand-specified data.<\/jats:p>","DOI":"10.1145\/3474385","type":"journal-article","created":{"date-parts":[[2021,8,24]],"date-time":"2021-08-24T14:28:46Z","timestamp":1629815326000},"page":"108-116","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["PlanAlyzer"],"prefix":"10.1145","volume":"64","author":[{"given":"Emma","family":"Tosch","sequence":"first","affiliation":[{"name":"University of Vermont, Burlington, VT"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eytan","family":"Bakshy","sequence":"additional","affiliation":[{"name":"Facebook, Inc., Menlo Park, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emery D.","family":"Berger","sequence":"additional","affiliation":[{"name":"University of Massachusetts, Amherst, MA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David D.","family":"Jensen","sequence":"additional","affiliation":[{"name":"University of Massachusetts, Amherst, MA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J. Eliot B.","family":"Moss","sequence":"additional","affiliation":[{"name":"University of Massachusetts, Amherst, MA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,8,24]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Techniques","author":"Aho A.V.","year":"1986","unstructured":"Aho , A.V. , Sethi , R. , Ullman , J.D. Compilers , Principles , Techniques . Addison Wesley , Boston, Massachusetts , 1986 . Aho, A.V., Sethi, R., Ullman, J.D. Compilers, Principles, Techniques. Addison Wesley, Boston, Massachusetts, 1986."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567967"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557139"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/864891"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78800-3_24"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/360051.360056"},{"key":"e_1_2_1_7_1","volume-title":"The program dependence graph and its use in optimization. ACM Trans. Programming Languages and Systems (TOPLAS) 9, 3","author":"Ferrante J.","year":"1987","unstructured":"Ferrante , J. , Ottenstein , K.J. , Warren , J.D. The program dependence graph and its use in optimization. ACM Trans. Programming Languages and Systems (TOPLAS) 9, 3 ( 1987 ), 319--349. Ferrante, J., Ottenstein, K.J., Warren, J.D. The program dependence graph and its use in optimization. ACM Trans. Programming Languages and Systems (TOPLAS) 9, 3 (1987), 319--349."},{"key":"e_1_2_1_8_1","volume-title":"Refactoring: Improving the Design of Existing Code","author":"Fowler M.","year":"2018","unstructured":"Fowler , M. Refactoring: Improving the Design of Existing Code . Addison-Wesley Professional , Boston, MA, USA , 2018 . Fowler, M. Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional, Boston, MA, USA, 2018."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2603088.2603097"},{"key":"e_1_2_1_10_1","first-page":"4","article-title":"Partial evaluation of computation process--an approach to a compiler-compiler","volume":"12","author":"Futamura Y","year":"1999","unstructured":"Futamura , Y . Partial evaluation of computation process--an approach to a compiler-compiler . Higher-Order Symbolic Comput. 12 , 4 ( 1999 ), 381--391. Futamura, Y. Partial evaluation of computation process--an approach to a compiler-compiler. Higher-Order Symbolic Comput. 12, 4 (1999), 381--391.","journal-title":"Higher-Order Symbolic Comput."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593882.2593900"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2488217"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-008-0114-1"},{"key":"e_1_2_1_14_1","volume-title":"Advances in Neural Information Processing Systems (Online","author":"Minka T.","year":"2009","unstructured":"Minka , T. , Winn , J. Gates . In Advances in Neural Information Processing Systems (Online , 2009 ), JMLR : W&CP, 1073--1080. Minka, T., Winn, J. Gates. In Advances in Neural Information Processing Systems (Online, 2009), JMLR: W&CP, 1073--1080."},{"key":"e_1_2_1_15_1","volume-title":"Morgan Kaufmann","author":"Muchnick S.S.","year":"1997","unstructured":"Muchnick , S.S. Advanced Compiler Design Implementation . Morgan Kaufmann , Burlington, MA, USA , 1997 . Muchnick, S.S. Advanced Compiler Design Implementation. Morgan Kaufmann, Burlington, MA, USA, 1997."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/3033232"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSAC.2002.806121"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01019462"},{"key":"e_1_2_1_19_1","volume-title":"Experimental and Quasi-Experimental Designs for Generalized Causal Inference","author":"Shadish W.R.","year":"2002","unstructured":"Shadish , W.R. , Cook , T.D. , Campbell , D.T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference . Houghton Mifflin Company , Boston, MA, USA , 2002 . Shadish, W.R., Cook, T.D., Campbell, D.T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, Boston, MA, USA, 2002."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation","volume":"2","author":"Sherman A.","year":"2005","unstructured":"Sherman , A. , Lisiecki , P.A. , Berkheimer , A. , Wein , J. Acms : The akamai configuration management system . In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation , Volume 2 (Berkley, CA, USA, 2005 ), USENIX Association, USA, 245--258. Sherman, A., Lisiecki, P.A., Berkheimer, A., Wein, J. Acms: The akamai configuration management system. In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, Volume 2 (Berkley, CA, USA, 2005), USENIX Association, USA, 245--258."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815401"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835810"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/0304-3975(79)90044-6"}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474385","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3474385","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:49Z","timestamp":1750191529000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474385"}},"subtitle":["assessing threats to the validity of online experiments"],"short-title":[],"issued":{"date-parts":[[2021,8,24]]},"references-count":23,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2021,9]]}},"alternative-id":["10.1145\/3474385"],"URL":"https:\/\/doi.org\/10.1145\/3474385","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"type":"print","value":"0001-0782"},{"type":"electronic","value":"1557-7317"}],"subject":[],"published":{"date-parts":[[2021,8,24]]},"assertion":[{"value":"2021-08-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}