{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T16:12:59Z","timestamp":1772554379120,"version":"3.50.1"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,5,23]],"date-time":"2020-05-23T00:00:00Z","timestamp":1590192000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,5,23]],"date-time":"2020-05-23T00:00:00Z","timestamp":1590192000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["Centre of Doctoral Training (CDT) in Modern Statistics and Machine Learning at Imperial College London and University of Oxford"],"award-info":[{"award-number":["Centre of Doctoral Training (CDT) in Modern Statistics and Machine Learning at Imperial College London and University of Oxford"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Sci. Eng."],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Experimentation and Measurement (E&amp;M) capabilities allow organizations to accurately assess the impact of new propositions and to experiment with many variants of existing products. However, until now, the question of measuring the measurer, or valuing the contribution of an E&amp;M capability to organizational success has not been addressed. We tackle this problem by analyzing how, by decreasing estimation uncertainty, E&amp;M platforms allow for better prioritization. We quantify this benefit in terms of expected relative improvement in the performance of all new propositions and provide guidance for how much an E&amp;M capability is worth and when organizations should invest in one.<\/jats:p>","DOI":"10.1007\/s41019-020-00121-5","type":"journal-article","created":{"date-parts":[[2020,5,23]],"date-time":"2020-05-23T10:02:52Z","timestamp":1590228172000},"page":"152-167","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["What is the Value of Experimentation and Measurement?"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6516-2364","authenticated-orcid":false,"given":"C. H. Bryan","family":"Liu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin Paul","family":"Chamberlain","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emma J.","family":"McCoy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,5,23]]},"reference":[{"key":"121_CR1","doi-asserted-by":"crossref","unstructured":"Tang D, Agarwal A, O\u2019Brien D, Meyer M (2010) Overlapping experiment infrastructure: more, better, faster experimentation. In: Proceedings of the 16th ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201910, ACM, New York, NY, USA, pp 17\u201326","DOI":"10.1145\/1835804.1835810"},{"key":"121_CR2","doi-asserted-by":"crossref","unstructured":"Xu Y, Chen N, Fernandez A, Sinno O, Bhasin A (2015) From infrastructure to culture: A\/b testing challenges in large scale social networks. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201915, ACM, New York, NY, USA, pp 2227\u20132236","DOI":"10.1145\/2783258.2788602"},{"key":"121_CR3","doi-asserted-by":"crossref","unstructured":"Kohavi R, Deng A, Frasca B, Walker T, Xu Y, Pohlmann N (2013) Online controlled experiments at large scale. In: Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201913, ACM, New York, NY, USA, pp 1168\u20131176","DOI":"10.1145\/2487575.2488217"},{"key":"121_CR4","doi-asserted-by":"crossref","unstructured":"Lee MR, Shen M (2018) Winner\u2019s curse: bias estimation for total effects of features in online controlled experiments. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201918, ACM, New York, NY, USA, pp 491\u2013499","DOI":"10.1145\/3219819.3219905"},{"key":"121_CR5","doi-asserted-by":"crossref","unstructured":"Xie H, Aurisset J (2016) Improving the sensitivity of online controlled experiments: Case studies at netflix. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201916, ACM, New York, NY, USA, pp 645\u2013654","DOI":"10.1145\/2939672.2939733"},{"key":"121_CR6","doi-asserted-by":"crossref","unstructured":"Poyarkov A, Drutsa A, Khalyavin A, Gusev G, Serdyukov P (2016) Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201916, ACM, New York, NY, USA, pp 235\u2013244","DOI":"10.1145\/2939672.2939688"},{"key":"121_CR7","doi-asserted-by":"crossref","unstructured":"Hill DN, Nassif H, Liu Y, Iyer A, Vishwanathan S (2017) An efficient bandit algorithm for realtime multivariate optimization. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201917, ACM, New York, NY, USA, pp 1813\u20131821","DOI":"10.1145\/3097983.3098184"},{"issue":"2","key":"121_CR8","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1287\/mksc.2018.1135","volume":"38","author":"BR Gordon","year":"2019","unstructured":"Gordon BR, Zettelmeyer F, Bhargava N, Chapsky D (2019) A comparison of approaches to advertising measurement: evidence from big field experiments at facebook. Market Sci 38(2):193\u2013225","journal-title":"Market Sci"},{"key":"121_CR9","doi-asserted-by":"crossref","unstructured":"Johari R, Koomen P, Pekelis L, Walsh D (2017) Peeking at a\/b tests: why it matters, and what to do about it. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201917, ACM, New York, NY, USA, pp 1517\u20131525","DOI":"10.1145\/3097983.3097992"},{"key":"121_CR10","unstructured":"Browne W, Jones MS (2017) What works in e-commerce\u2014a meta-analysis of 6700 online experiments. https:\/\/www.qubit.com\/wp-content\/uploads\/2017\/12\/qubit-research-meta-analysis.pdf"},{"issue":"1","key":"121_CR11","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1086\/294846","volume":"39","author":"WF Sharpe","year":"1966","unstructured":"Sharpe WF (1966) Mutual fund performance. J Bus 39(1):119\u2013138","journal-title":"J Bus"},{"key":"121_CR12","unstructured":"Johnson G, Lewis RA, Nubbemeyer E (2017) The online display ad effectiveness funnel and carryover: lessons from 432 field experiments. Working paper. https:\/\/marketing.wharton.upenn.edu\/wp-content\/uploads\/2017\/08\/Johnson-Garrett-PAPER-VERSION-2.pdf"},{"key":"121_CR13","doi-asserted-by":"crossref","unstructured":"Dmitriev P, Gupta S, Kim DW, Vaz G (2017) A dirty dozen: twelve common metric interpretation pitfalls in online controlled experiments. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201917, ACM, New York, NY, USA, pp 1427\u20131436","DOI":"10.1145\/3097983.3098024"},{"key":"121_CR14","doi-asserted-by":"crossref","unstructured":"Hohnhold H, O\u2019Brien D, Tang D (2015) Focusing on the long-term: it\u2019s good for users and business. In: Proceedings of the 21st ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201915, ACM, New York, NY, USA, pp 1849\u20131858","DOI":"10.1145\/2783258.2788583"},{"key":"121_CR15","doi-asserted-by":"crossref","unstructured":"Backstrom L, Kleinberg J (2011) Network bucket testing. In: Proceedings of the 20th international conference on world wide web, WWW \u201911, ACM, New York, NY, USA, pp 615\u2013624","DOI":"10.1145\/1963405.1963492"},{"key":"121_CR16","doi-asserted-by":"crossref","unstructured":"Bakshy E, Eckles D (2013) Uncertainty in online experiments with dependent data: an evaluation of bootstrap methods. In: Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, KDD \u201913, ACM, New York, NY, USA, pp 1303\u20131311","DOI":"10.1145\/2487575.2488218"},{"key":"121_CR17","doi-asserted-by":"publisher","first-page":"961","DOI":"10.1287\/opre.30.5.961","volume":"30","author":"D Bell","year":"1982","unstructured":"Bell D (1982) Regret in decision making under uncertainty. Oper Res 30:961\u201381","journal-title":"Oper Res"},{"issue":"1","key":"121_CR18","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1016\/j.pnsc.2008.07.005","volume":"19","author":"Y Xu","year":"2009","unstructured":"Xu Y, Tung Y-K, Li J, Niu S (2009) Alternative risk measure for decision-making under uncertainty in water management. Prog Nat Sci 19(1):115\u2013119","journal-title":"Prog Nat Sci"},{"issue":"1","key":"121_CR19","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1016\/0377-2217(87)90168-8","volume":"28","author":"M Weber","year":"1987","unstructured":"Weber M (1987) Decision making with incomplete information. Eur J Oper Res 28(1):44\u201357","journal-title":"Eur J Oper Res"},{"key":"121_CR20","doi-asserted-by":"crossref","unstructured":"Soliman MA, Ilyas IF (2009) Ranking with uncertain scores. In: 2009 IEEE 25th international conference on data engineering, pp 317\u2013328","DOI":"10.1109\/ICDE.2009.102"},{"key":"121_CR21","unstructured":"Zuk O, Ein-Dor L, Domany (2007) Ranking under uncertainty. In: Proceedings of the twenty-third conference on uncertainty in artificial intelligence, UAI\u201907, AUAI Press, Arlington, Virginia, United States, pp 466\u2013474"},{"issue":"1","key":"121_CR22","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1504\/IJMCDM.2013.052474","volume":"3","author":"G Mavrotas","year":"2013","unstructured":"Mavrotas G, Pechak O (2013) The trichotomic approach for dealing with uncertainty in project portfolio selection: combining mcda, mathematical programming and monte carlo simulation. Int J Multicrit Decis Mak 3(1):79\u201396","journal-title":"Int J Multicrit Decis Mak"},{"key":"121_CR23","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1016\/j.cie.2011.03.015","volume":"61","author":"M Shakhsi-Niaei","year":"2011","unstructured":"Shakhsi-Niaei M, Torabi SA, Iranmanesh SH (2011) A comprehensive framework for project selection problem under uncertainty and real-world constraints. Comput Ind Eng 61:226\u2013237","journal-title":"Comput Ind Eng"},{"key":"121_CR24","unstructured":"Blom G (1958) Statistical estimates and transformed beta-variables. PhD thesis, Stockholm College"},{"key":"121_CR25","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1093\/biomet\/48.1-2.151","volume":"48","author":"HL Harter","year":"1961","unstructured":"Harter HL (1961) Expected values of normal order statistics. Biometrika 48:151\u2013165","journal-title":"Biometrika"},{"key":"121_CR26","doi-asserted-by":"crossref","unstructured":"David HA, Nagaraja HN (2004) Order statistics. In: Encyclopedia of statistical sciences","DOI":"10.1002\/0471667196.ess6023"},{"key":"121_CR27","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1093\/biomet\/41.1-2.284-a","volume":"41","author":"FN David","year":"1954","unstructured":"David FN, Johnson NL (1954) Statistical treatment of censored data: Part i. Fundamental formulae. Biometrika 41:228\u2013240","journal-title":"Biometrika"},{"key":"121_CR28","doi-asserted-by":"crossref","unstructured":"David H, Nagaraja H (1998) 18 concomitants of order statistics. In: Order statistics: theory and methods, vol 16 of handbook of statistics, Elsevier, pp 487\u2013513","DOI":"10.1016\/S0169-7161(98)16020-0"},{"key":"121_CR29","unstructured":"Talts S, Betancourt M, Simpson D, Vehtari A, Gelman A (2018) Validating Bayesian inference algorithms with simulation-based calibration"},{"key":"121_CR30","unstructured":"Liu CHB, Bettaney EM, Chamberlain BP (2018) Designing experiments to measure incrementality on facebook. In: AdKDD and TargetAd workshop"},{"key":"121_CR31","volume-title":"The Bayesian choice: from decision-theoretic foundations to computational implementation","author":"C Robert","year":"2007","unstructured":"Robert C (2007) The Bayesian choice: from decision-theoretic foundations to computational implementation. Springer, Berlin"},{"issue":"4","key":"121_CR32","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1080\/03610918008812164","volume":"9","author":"DB Owen","year":"1980","unstructured":"Owen DB (1980) A table of normal integrals. Commun Stat Simul Comput 9(4):389\u2013419","journal-title":"Commun Stat Simul Comput"},{"key":"121_CR33","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.1214\/aoms\/1177728074","volume":"27","author":"DB Owen","year":"1956","unstructured":"Owen DB (1956) Tables for computing bivariate normal probabilities. Ann Math Stat 27:1075\u20131090","journal-title":"Ann Math Stat"},{"issue":"5","key":"121_CR34","first-page":"1","volume":"5","author":"M Patefield","year":"2000","unstructured":"Patefield M, Tandy D (2000) Fast and accurate calculation of owen\u2019s t function. J Stat Softw Artic 5(5):1\u201325","journal-title":"J Stat Softw Artic"},{"key":"121_CR35","doi-asserted-by":"crossref","unstructured":"Liu CHB, Chamberlain BP (2019) What is the value of experimentation & measurement? In: 2019 IEEE international conference on data mining (ICDM), pp 1222\u20131227","DOI":"10.1109\/ICDM.2019.00151"}],"container-title":["Data Science and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-020-00121-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41019-020-00121-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-020-00121-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,22]],"date-time":"2021-05-22T23:46:24Z","timestamp":1621727184000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41019-020-00121-5"}},"subtitle":["Quantifying the Value and Risk of Reducing Uncertainty to Make Better Decisions"],"short-title":[],"issued":{"date-parts":[[2020,5,23]]},"references-count":35,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["121"],"URL":"https:\/\/doi.org\/10.1007\/s41019-020-00121-5","relation":{},"ISSN":["2364-1185","2364-1541"],"issn-type":[{"value":"2364-1185","type":"print"},{"value":"2364-1541","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,23]]},"assertion":[{"value":"15 March 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 April 2020","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 April 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 May 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with Ethical Standards"}},{"value":"The authors declare they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}