{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T02:32:59Z","timestamp":1776825179290,"version":"3.51.2"},"reference-count":113,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,11,24]],"date-time":"2023-11-24T00:00:00Z","timestamp":1700784000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"EPSRC CITCoM","award":["EP\/T030526\/1"],"award-info":[{"award-number":["EP\/T030526\/1"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,1,31]]},"abstract":"<jats:p>\n            From simulating galaxy formation to viral transmission in a pandemic, scientific models play a pivotal role in developing scientific theories and supporting government policy decisions that affect us all. Given these critical applications, a poor modelling assumption or bug could have far-reaching consequences. However, scientific models possess several properties that make them notoriously difficult to test, including a complex input space, long execution times, and non-determinism, rendering existing testing techniques impractical. In fields such as epidemiology, where researchers seek answers to challenging causal questions, a statistical methodology known as Causal inference has addressed similar problems, enabling the inference of causal conclusions from noisy, biased, and sparse data instead of costly experiments. This article introduces the causal testing framework: a framework that uses causal inference techniques to establish causal effects from existing data, enabling users to conduct software testing activities concerning the effect of a change, such as metamorphic testing,\n            <jats:italic>a posteriori<\/jats:italic>\n            . We present three case studies covering real-world scientific models, demonstrating how the causal testing framework can infer metamorphic test outcomes from reused, confounded test data to provide an efficient solution for testing scientific modelling software.\n          <\/jats:p>","DOI":"10.1145\/3607184","type":"journal-article","created":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T11:44:34Z","timestamp":1689162274000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Testing Causality in Scientific Modelling Software"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6830-0566","authenticated-orcid":false,"given":"Andrew G.","family":"Clark","sequence":"first","affiliation":[{"name":"The University of Sheffield, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8233-9873","authenticated-orcid":false,"given":"Michael","family":"Foster","sequence":"additional","affiliation":[{"name":"The University of Sheffield, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2952-0208","authenticated-orcid":false,"given":"Benedikt","family":"Prifling","sequence":"additional","affiliation":[{"name":"Ulm University, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2134-6548","authenticated-orcid":false,"given":"Neil","family":"Walkinshaw","sequence":"additional","affiliation":[{"name":"The University of Sheffield, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4771-1446","authenticated-orcid":false,"given":"Robert M.","family":"Hierons","sequence":"additional","affiliation":[{"name":"The University of Sheffield, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7451-8185","authenticated-orcid":false,"given":"Volker","family":"Schmidt","sequence":"additional","affiliation":[{"name":"Ulm University, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1353-1404","authenticated-orcid":false,"given":"Robert D.","family":"Turner","sequence":"additional","affiliation":[{"name":"The University of Sheffield, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2023,11,24]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1080\/07350015.2014.975555"},{"issue":"9952","key":"e_1_3_3_3_2","doi-asserted-by":"crossref","first-page":"1423","DOI":"10.1016\/S0140-6736(14)61734-7","article-title":"Randomised controlled trials for Ebola: Practical and ethical issues","volume":"384","author":"Adebamowo Clement","year":"2014","unstructured":"Clement Adebamowo, Oumou Bah-Sow, Fred Binka, Roberto Bruzzone, Arthur Caplan, Jean-Fran\u00e7ois Delfraissy, David Heymann, et\u00a0al. 2014. Randomised controlled trials for Ebola: Practical and ethical issues. Lancet 384, 9952 (2014), 1423\u20131424.","journal-title":"Lancet"},{"key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/AST52587.2021.00012","volume-title":"Proceedings of the 2021 IEEE\/ACM International Conference on Automation of Software Test (AST\u201921)","author":"Arrieta Aitor","year":"2021","unstructured":"Aitor Arrieta, Jon Ayerdi, Miren Illarramendi, Aitor Agirre, Goiuria Sagardui, and Maite Arratibel. 2021. Using machine learning to build test oracles: An industrial case study on elevators dispatching algorithms. In Proceedings of the 2021 IEEE\/ACM International Conference on Automation of Software Test (AST\u201921). IEEE, Los Alamitos, CA, 30\u201339."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1353\/obs.2019.0001"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473920"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3520304.3534077"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1831708.1831717"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025136"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102597"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1510507113"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"issue":"1","key":"e_1_3_3_13_2","first-page":"23","article-title":"Linear regression models with logarithmic transformations","volume":"22","author":"Benoit Kenneth","year":"2011","unstructured":"Kenneth Benoit. 2011. Linear regression models with logarithmic transformations. London School of Economics, London 22, 1 (2011), 23\u201336.","journal-title":"London School of Economics, London"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2009.06.006"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/b137241"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12228"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","unstructured":"Michael J. Butler Philipp K\u00f6rner Sebastian Krings Thierry Lecomte Michael Leuschel Luis-Fernando Mejia and Laurent Voisin. 2020. The first twenty-five years of industrial use of the B-Method. In Formal Methods for Industrial Critical Systems . Lecture Notes in Computer Science Vol. 12327. Springer 189\u2013209.","DOI":"10.1007\/978-3-030-58298-2_8"},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1111\/j.1365-2753.2010.01382.x","article-title":"The limitations of randomized controlled trials in predicting effectiveness.","volume":"16","author":"Cartwright Nancy","year":"2010","unstructured":"Nancy Cartwright and Eileen Munro. 2010. The limitations of randomized controlled trials in predicting effectiveness. Journal of Evaluation in Clinical Practice 16 2 (2010), 260\u2013266.","journal-title":"Journal of Evaluation in Clinical Practice"},{"key":"e_1_3_3_19_2","unstructured":"cellML. 2022. cellML: Luo-Rudy 1991. Retrieved July 14 2023 from https:\/\/models.cellml.org\/exposure\/456b07d6a7a5b45ed71caad0ea2c0b9d"},{"issue":"6","key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"e0130252","DOI":"10.1371\/journal.pone.0130252","article-title":"Bayesian sensitivity analysis of a cardiac cell model using a Gaussian process emulator","volume":"10","author":"Chang Eugene T. Y.","year":"2015","unstructured":"Eugene T. Y. Chang, Mark Strong, and Richard H. Clayton. 2015. Bayesian sensitivity analysis of a cardiac cell model using a Gaussian process emulator. PLoS One 10, 6 (2015), e0130252.","journal-title":"PLoS One"},{"key":"e_1_3_3_21_2","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases","author":"Chen Tsong Y.","year":"1998","unstructured":"Tsong Y. Chen, Shing C. Cheung, and Shiu Ming Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01. The Hong Kong University of Science and Technology."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/TWC.2018.2824832"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","DOI":"10.1002\/9781118658222","volume-title":"Stochastic Geometry and Its Applications (3rd ed.)","author":"Chiu Sung Nok","year":"2013","unstructured":"Sung Nok Chiu, Dietrich Stoyan, W. S. Kendall, and Joseph Mecke. 2013. Stochastic Geometry and Its Applications (3rd ed.). John Wiley & Sons, Chichester, West Sussex, United Kingdom."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2103.03622"},{"issue":"3","key":"e_1_3_3_25_2","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1109\/TSE.1978.231496","article-title":"Testing software design modeled by finite-state machines","author":"Chow Tsun S.","year":"1978","unstructured":"Tsun S. Chow. 1978. Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering 3 (1978), 178\u2013187.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1111\/rssb.12348"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1101\/2020.09.08.20190942"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jspi.2009.08.006"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1093\/jnci\/22.1.173"},{"key":"e_1_3_3_30_2","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1007\/978-3-540-78800-3_24","volume-title":"Tools and Algorithms for the Construction and Analysis of Systems","author":"Moura Leonardo de","year":"2008","unstructured":"Leonardo de Moura and Nikolaj Bj\u00f8rner. 2008. Z3: An efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems, C. R. Ramakrishnan and Jakob Rehof (Eds.). Springer, Berlin, Germany, 337\u2013340."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jspi.2011.09.016"},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1007\/BFb0024651","volume-title":"FME\u201993: Industrial-Strength Formal Methods","author":"Dick J.","year":"1993","unstructured":"J. Dick and A. Faivre. 1993. Automating the generation and sequencing of test cases from model-based specifications. In FME\u201993: Industrial-Strength Formal Methods. Lecture Notes in Computer Science, Vol. 670, 268\u2013284."},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1007\/3-540-45848-4_57","volume-title":"Graph Drawing","author":"Ellson John","year":"2002","unstructured":"John Ellson, Emden Gansner, Lefteris Koutsofios, Stephen C. North, and Gordon Woodhull. 2002. Graphviz\u2014Open source graph drawing tools. In Graph Drawing, Petra Mutzel, Michael J\u00fcnger, and Sebastian Leipert (Eds.). Springer, Berlin, Germany, 483\u2013484."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1061\/(ASCE)CP.1943-5487.0000233"},{"key":"e_1_3_3_35_2","unstructured":"Institute for Disease Modeling. 2022. Covasim: Vaccine Tests. Retrieved July 14 2023 from https:\/\/github.com\/InstituteforDiseaseModeling\/covasim\/blob\/master\/tests\/test_interventions.py"},{"key":"e_1_3_3_36_2","unstructured":"Institute for Disease Modelling. 2022. Covasim. Retrieved July 14 2023 from https:\/\/github.com\/InstituteforDiseaseModeling\/covasim"},{"key":"e_1_3_3_37_2","doi-asserted-by":"crossref","unstructured":"Marie-Claude Gaudel. 1995. Testing can be formal too. In TAPSOFT\u201995: Theory and Practice of Software Development . Lecture Notes in Computer Science Vol. 915. Springer 82\u201396.","DOI":"10.1007\/3-540-59293-8_188"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2303.01302"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.3389\/fgene.2019.00524"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2012.6227169"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1097\/00001648-199901000-00008"},{"key":"e_1_3_3_42_2","unstructured":"Michael H. Grider Rishita Jessu and Rian Kabir. 2019. Physiology Action Potential . StatPearls [Internet]. Treasure Island FL."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2007.4385527"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1093\/bjps\/axi147"},{"key":"e_1_3_3_45_2","volume-title":"Causal Inference: What if","author":"Hern\u00e1n Miguel A.","year":"2020","unstructured":"Miguel A. Hern\u00e1n and James M. Robins. 2020. Causal Inference: What if. Chapman & Hall\/CRC, Boca Raton, FL."},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1099-1689(199703)7:1<19::AID-STVR124>3.0.CO;2-N"},{"issue":"2","key":"e_1_3_3_47_2","first-page":"Article 9, 76 p","article-title":"Using formal specifications to support testing","volume":"41","author":"Hierons Robert M.","year":"2009","unstructured":"Robert M. Hierons, Kirill Bogdanov, Jonathan P. Bowen, Rance Cleaveland, John Derrick, Jeremy Dick, Marian Gheorghe, et\u00a0al. 2009. Using formal specifications to support testing. ACM Computing Surveys 41, 2 (2009), Article 9, 76 pages.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1986.10478354"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1312"},{"key":"e_1_3_3_50_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1145\/3377811.3380377","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering","author":"Johnson Brittany","year":"2020","unstructured":"Brittany Johnson, Yuriy Brun, and Alexandra Meliou. 2020. Causal testing: Understanding defects\u2019 root causes. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. IEEE, Los Alamitos, CA, 87\u201399."},{"key":"e_1_3_3_51_2","first-page":"1","volume-title":"Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE\u201913)","author":"Kanewala Upulee","year":"2013","unstructured":"Upulee Kanewala and James M. Bieman. 2013. Using machine learning techniques to detect metamorphic relations for programs without test oracles. In Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE\u201913). IEEE, Los Alamitos, CA, 1\u201310."},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.05.006"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1093\/pan\/mpv007"},{"key":"e_1_3_3_54_2","unstructured":"Diane Kelly and Rebecca Sanders. 2008. The challenge of testing scientific software. In Proceedings of the 3rd Annual Conference of the Association for Software Testing (CAST\u201908) . 30\u201336."},{"issue":"2","key":"e_1_3_3_55_2","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1136\/emj.20.2.164","article-title":"Designing a research project: Randomised controlled trials and their principles","volume":"20","author":"Kendall John","year":"2003","unstructured":"John Kendall. 2003. Designing a research project: Randomised controlled trials and their principles. Emergency Medicine Journal: EMJ 20, 2 (2003), 164.","journal-title":"Emergency Medicine Journal: EMJ"},{"issue":"1","key":"e_1_3_3_56_2","first-page":"1","article-title":"Controlling COVID-19 via test-trace-quarantine","volume":"12","author":"Kerr Cliff C.","year":"2021","unstructured":"Cliff C. Kerr, Dina Mistry, Robyn M. Stuart, Katherine Rosenfeld, Gregory R. Hart, Rafael C. N\u00fa\u00f1ez, Jamie A. Cohen, et\u00a0al. 2021. Controlling COVID-19 via test-trace-quarantine. Nature Communications 12, 1 (2021), 1\u201312.","journal-title":"Nature Communications"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1009149"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/0377-2217(94)00016-6"},{"key":"e_1_3_3_59_2","volume-title":"Principles and Practice of Structural Equation Modeling","author":"Kline Rex B.","year":"2015","unstructured":"Rex B. Kline. 2015. Principles and Practice of Structural Equation Modeling. Guilford Publications."},{"key":"e_1_3_3_60_2","unstructured":"Konstantin Kreyman David Lorge Parnas and Sanzheng Qiao. 1999. Inspection Procedures for Critical Programs That Model Physical Phenomena . Technical Report. McMaster University Hamilton Canada."},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.533956"},{"key":"e_1_3_3_62_2","unstructured":"Seongmin Lee Dave Binkley Robert Feldt Nicolas Gold and Shin Yoo. 2021. Causal program dependence analysis. arXiv:2104.09107 (2021)."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1161\/01.RES.68.6.1501"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1111\/phc3.12470"},{"key":"e_1_3_3_65_2","volume-title":"Spline Regression Models","author":"Marsh Lawrence C.","year":"2001","unstructured":"Lawrence C. Marsh and David R. Cormier. 2001. Spline Regression Models. No. 137. Sage."},{"key":"e_1_3_3_66_2","volume-title":"Automata Studies","author":"Moore E. F.","year":"1956","unstructured":"E. F. Moore. 1956. Gedanken-experiments. In Automata Studies, C. Shannon and J. McCarthy (Eds.). Princeton University Press, Princeton, NJ."},{"key":"e_1_3_3_67_2","first-page":"337","volume-title":"Proceedings of the 2012 10th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt\u201912)","author":"Morlot Fr\u00e9d\u00e9ric","year":"2012","unstructured":"Fr\u00e9d\u00e9ric Morlot. 2012. A population model based on a Poisson line tessellation. In Proceedings of the 2012 10th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt\u201912). IEEE, Los Alamitos, CA, 337\u2013342."},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.3929531"},{"key":"e_1_3_3_69_2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1145\/222124.222136","volume-title":"Proceedings of the 3rd ACM SIGSOFT Symposium on Foundations of Software Engineering","author":"Murphy Gail C.","year":"1995","unstructured":"Gail C. Murphy, David Notkin, and Kevin Sullivan. 1995. Software reflexion models: Bridging the gap between source and high-level models. In Proceedings of the 3rd ACM SIGSOFT Symposium on Foundations of Software Engineering. IEEE, Los Alamitos, CA, 18\u201328."},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.5555\/983238"},{"issue":"21","key":"e_1_3_3_71_2","doi-asserted-by":"crossref","first-page":"6833","DOI":"10.1109\/JLT.2021.3106714","article-title":"Physics-informed Gaussian process regression for optical fiber communication systems","volume":"39","author":"Nevin Josh W.","year":"2021","unstructured":"Josh W. Nevin, F. J. Vaquero-Caballero, David J. Ives, and Seb J. Savory. 2021. Physics-informed Gaussian process regression for optical fiber communication systems. Journal of Lightwave Technology 39, 21 (2021), 6833\u20136844.","journal-title":"Journal of Lightwave Technology"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.5121\/ijesa.2012.2204"},{"key":"e_1_3_3_73_2","first-page":"46","volume-title":"Proceedings of the 1995 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON\u201995)","author":"Nursimulu Khenaidoo","year":"1995","unstructured":"Khenaidoo Nursimulu and Robert L. Probert. 1995. Cause-effect graphing analysis and validation of requirements. In Proceedings of the 1995 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON\u201995). 46."},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2004.05304.x"},{"issue":"7","key":"e_1_3_3_75_2","doi-asserted-by":"crossref","first-page":"1680","DOI":"10.1111\/trf.13635","article-title":"How do I interpret a confidence interval?","volume":"56","author":"O\u2019Brien Sheila F.","year":"2016","unstructured":"Sheila F. O\u2019Brien and Qi Long Yi. 2016. How do I interpret a confidence interval? Transfusion 56, 7 (2016), 1680\u20131683.","journal-title":"Transfusion"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-021-00078-9"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1016\/S2352-4642(20)30250-9"},{"key":"e_1_3_3_78_2","doi-asserted-by":"crossref","unstructured":"Jasmina Panovska-Griffiths Cliff C. Kerr William Waites Robyn Margaret Stuart Dina Mistry Derek Foster Daniel J. Klein Russell M. Viner and Chris Bonell. 2021. The potential contribution of face coverings to the control of SARS-CoV-2 transmission in schools and broader society in the UK: A modelling study. Scientific Reports 11 (2021) 8747.","DOI":"10.1101\/2020.09.28.20202937"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/82.4.669"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1214\/09-SS057"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.5555\/1642718"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1515\/jci-2018-2001"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.5555\/3238230"},{"key":"e_1_3_3_84_2","first-page":"789","volume-title":"Studies in Logic and the Foundations of Mathematics","author":"Pearl Judea","year":"1995","unstructured":"Judea Pearl and Thomas S. Verma. 1995. A theory of inferred causation. In Studies in Logic and the Foundations of Mathematics. Vol. 134. Elsevier, 789\u2013811."},{"key":"e_1_3_3_85_2","doi-asserted-by":"crossref","first-page":"382","DOI":"10.1109\/ICSME46990.2020.00044","volume-title":"Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201920)","author":"Podgurski Andy","year":"2020","unstructured":"Andy Podgurski and Yi\u011fit K\u00fc\u00e7\u00fck. 2020. CounterFault: Value-based fault localization by modeling and predicting counterfactual outcomes. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201920). IEEE, Los Alamitos, CA, 382\u2013393."},{"key":"e_1_3_3_86_2","first-page":"10","volume-title":"Proceedings of the 2018 IEEE\/ACM 3rd International Workshop on Metamorphic Testing (MET\u201918)","author":"Rahman Karishma","year":"2018","unstructured":"Karishma Rahman and Upulee Kanewala. 2018. Predicting metamorphic relations for matrix calculation programs. In Proceedings of the 2018 IEEE\/ACM 3rd International Workshop on Metamorphic Testing (MET\u201918). IEEE, Los Alamitos, CA, 10\u201313."},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437479.3437483"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-28650-9_4"},{"key":"e_1_3_3_89_2","volume-title":"Gaussian Processes for Machine Learning","year":"2006","unstructured":"Carl Edward Rasmussen and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. Vol. 1. Springer."},{"issue":"1","key":"e_1_3_3_90_2","doi-asserted-by":"crossref","first-page":"144","DOI":"10.2105\/AJPH.2004.059204","article-title":"Causation and causal inference in epidemiology","volume":"95","author":"Rothman Kenneth J.","year":"2005","unstructured":"Kenneth J. Rothman and Sander Greenland. 2005. Causation and causal inference in epidemiology. American Journal of Public Health 95, S1 (2005), 144\u2013150.","journal-title":"American Journal of Public Health"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1198\/016214504000001880"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.envsoft.2016.02.005"},{"issue":"2","key":"e_1_3_3_93_2","doi-asserted-by":"crossref","first-page":"79","DOI":"10.5694\/mja2.50845","article-title":"Modelling the impact of reducing control measures on the COVID-19 pandemic in a low transmission setting","volume":"214","author":"Scott Nick","year":"2020","unstructured":"Nick Scott, Anna Palmer, Dominic Delport, Romesh Abeysuriya, Robyn Stuart, Cliff C. Kerr, Dina Mistry, et\u00a0al. 2020. Modelling the impact of reducing control measures on the COVID-19 pandemic in a low transmission setting. Medical Journal of Australia 214, 2 (2020), 79\u201383.","journal-title":"Medical Journal of Australia"},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_3_95_2","first-page":"1053","volume-title":"Proceedings of the 2013 IEEE Global Conference on Signal and Information Processing","author":"Shin Dongeek","year":"2013","unstructured":"Dongeek Shin, Ahmed Kirmani, Andrea Cola\u00e7o, and Vivek K. Goyal. 2013. Parametric Poisson process imaging. In Proceedings of the 2013 IEEE Global Conference on Signal and Information Processing. IEEE, Los Alamitos, CA, 1053\u20131056."},{"key":"e_1_3_3_96_2","first-page":"124","volume-title":"Proceedings of the 2013 IEEE 6th International Conference on Software Testing, Verification, and Validation","author":"Shu Gang","year":"2013","unstructured":"Gang Shu, Boya Sun, Andy Podgurski, and Feng Cao. 2013. MFL: Method-level fault localization with causal inference. In Proceedings of the 2013 IEEE 6th International Conference on Software Testing, Verification, and Validation. IEEE, Los Alamitos, CA, 124\u2013133."},{"key":"e_1_3_3_97_2","volume-title":"The Z Notation: A Reference Manual (2nd ed.)","author":"Spivey J. M.","year":"1992","unstructured":"J. M. Spivey. 1992. The Z Notation: A Reference Manual (2nd ed.). Prentice Hall."},{"key":"e_1_3_3_98_2","first-page":"391","volume-title":"Proceedings of the 2011 33rd International Conference on Software Engineering (ICSE\u201911)","author":"Staats Matt","year":"2011","unstructured":"Matt Staats, Michael W. Whalen, and Mats P. E. Heimdahl. 2011. Programs, tests, and oracles: The foundations of testing revisited. In Proceedings of the 2011 33rd International Conference on Software Engineering (ICSE\u201911). IEEE, Los Alamitos, CA, 391\u2013400."},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.2307\/1269769"},{"key":"e_1_3_3_100_2","volume-title":"Introduction to Econometrics","year":"2003","unstructured":"James H. Stock and Mark W. Watson. 2003. Introduction to Econometrics. Vol. 104. Addison Wesley, Boston, MA."},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1101\/2020.09.02.20186742v1"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1093\/ije\/dyaa213"},{"issue":"1","key":"e_1_3_3_103_2","first-page":"1","article-title":"Epidemiological models are important tools for guiding COVID-19 interventions","volume":"18","author":"Thompson Robin N.","year":"2020","unstructured":"Robin N. Thompson. 2020. Epidemiological models are important tools for guiding COVID-19 interventions. BMC Medicine 18, 1 (2020), 1\u20134.","journal-title":"BMC Medicine"},{"key":"e_1_3_3_104_2","doi-asserted-by":"crossref","unstructured":"Jan Tretmans. 2008. Model based testing with labelled transition systems. In Formal Methods and Testing . Lecture Notes in Computer Science Vol. 4949. Springer 1\u201338.","DOI":"10.1007\/978-3-540-78917-8_1"},{"key":"e_1_3_3_105_2","volume-title":"Practical Model-Based Testing: A Tools Approach","author":"Utting Mark","year":"2010","unstructured":"Mark Utting and Bruno Legeard. 2010. Practical Model-Based Testing: A Tools Approach. Elsevier."},{"key":"e_1_3_3_106_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.456"},{"issue":"4","key":"e_1_3_3_107_2","doi-asserted-by":"crossref","first-page":"268","DOI":"10.7326\/M16-2607","article-title":"Sensitivity analysis in observational research: Introducing the E-value","volume":"167","author":"VanderWeele Tyler J.","year":"2017","unstructured":"Tyler J. VanderWeele and Peng Ding. 2017. Sensitivity analysis in observational research: Introducing the E-value. Annals of Internal Medicine 167, 4 (2017), 268\u2013274.","journal-title":"Annals of Internal Medicine"},{"key":"e_1_3_3_108_2","doi-asserted-by":"publisher","DOI":"10.1214\/12-STS412"},{"key":"e_1_3_3_109_2","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1319839"},{"key":"e_1_3_3_110_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9367-7"},{"key":"e_1_3_3_111_2","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/25.4.465"},{"key":"e_1_3_3_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/69575.357231"},{"key":"e_1_3_3_113_2","author":"Wolf Christof","year":"2013","unstructured":"Christof Wolf and Henning Best. 2013. The SAGE Handbook of Regression Analysis and Causal Inference. SAGE.","journal-title":"The SAGE Handbook of Regression Analysis and Causal Inference."},{"key":"e_1_3_3_114_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2521368"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3607184","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3607184","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:34Z","timestamp":1750178254000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3607184"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,24]]},"references-count":113,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1,31]]}},"alternative-id":["10.1145\/3607184"],"URL":"https:\/\/doi.org\/10.1145\/3607184","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,24]]},"assertion":[{"value":"2022-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-20","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}