{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,5]],"date-time":"2026-11-05T08:20:40Z","timestamp":1793866840778,"version":"4.6.1"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,2,10]],"date-time":"2020-02-10T00:00:00Z","timestamp":1581292800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,2,10]],"date-time":"2020-02-10T00:00:00Z","timestamp":1581292800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Trials"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Retrospective exploratory analyses of randomised controlled trials (RCTs) seeking to identify treatment effect heterogeneity (TEH) are prone to bias and false positives. Yet the desire to learn all we can from exhaustive data measurements on trial participants motivates the inclusion of such analyses within RCTs. Moreover, widespread advances in machine learning (ML) methods hold potential to utilise such data to identify subjects exhibiting heterogeneous treatment response.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      We present a novel analysis strategy for detecting TEH in randomised data using ML methods, whilst ensuring proper control of the false positive discovery rate. Our approach uses random data partitioning with statistical or ML-based prediction on held-out data. This method can test for both crossover TEH (switch in optimal treatment) and non-crossover TEH (systematic variation in benefit across patients). The former is done via a two-sample hypothesis test measuring overall predictive performance. The latter is done via \u2018stacking\u2019 the ML predictors alongside a classical statistical model to formally test the added benefit of the ML algorithm. An adaptation of recent statistical theory allows for the construction of a valid aggregate\n                      <jats:italic>p<\/jats:italic>\n                      value. This testing strategy is independent of the choice of ML method.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We demonstrate our approach with a re-analysis of the SEAQUAMAT trial, which compared quinine to artesunate for the treatment of severe malaria in Asian adults. We find no evidence for any subgroup who would benefit from a change in treatment from the current standard of care, artesunate, but strong evidence for significant TEH within the artesunate treatment group. In particular, we find that artesunate provides a differential benefit to patients with high numbers of circulating ring stage parasites.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>ML analysis plans using computational notebooks (documents linked to a programming language that capture the model parameter settings, data processing choices, and evaluation criteria) along with version control can improve the robustness and transparency of RCT exploratory analyses. A data-partitioning algorithm allows researchers to apply the latest ML techniques safe in the knowledge that any declared associations are statistically significant at a user-defined level.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s13063-020-4076-y","type":"journal-article","created":{"date-parts":[[2020,2,10]],"date-time":"2020-02-10T11:02:47Z","timestamp":1581332567000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Machine learning analysis plans for randomised controlled trials: detecting treatment effect heterogeneity with strict control of type I error"],"prefix":"10.1186","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5524-0325","authenticated-orcid":false,"given":"James A.","family":"Watson","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6667-4943","authenticated-orcid":false,"given":"Chris C.","family":"Holmes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,2,10]]},"reference":[{"issue":"9454","key":"4076_CR1","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1016\/S0140-6736(05)17709-5","volume":"365","author":"P Rothwell","year":"2005","unstructured":"Rothwell P. Subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet. 2005; 365(9454):176\u201386.","journal-title":"Lancet"},{"issue":"9","key":"4076_CR2","doi-asserted-by":"publisher","first-page":"506","DOI":"10.1038\/nrclinonc.2015.133","volume":"12","author":"D Altman","year":"2015","unstructured":"Altman D. Clinical trials: subgroup analyses in randomized trials \u2013 more rigour needed. Nat Rev Clin Oncol. 2015; 12(9):506\u20137.","journal-title":"Nat Rev Clin Oncol"},{"key":"4076_CR3","unstructured":"Brown D. The press-release conviction of a biotech CEO and its impact on scientific research. Wash Post. 2013. https:\/\/www.washingtonpost.com\/national\/health-science\/the-press-release-crime-of-a-biotech-ceo-and-its-impact-on-scientific-research\/2013\/09\/23\/9b4a1a32-007a-11e3-9a3e-916de805f65d_story.html."},{"issue":"3","key":"4076_CR4","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1214\/ss\/1009213726","volume":"16","author":"L Breiman","year":"2001","unstructured":"Breiman L. Statistical modeling: the two cultures (with comments and a rejoinder by the author). Stat Sci. 2001; 16(3):199\u2013231.","journal-title":"Stat Sci"},{"key":"4076_CR5","unstructured":"Murphy S. J R Stat Soc Ser B (Stat Methodol). 2003; 65(2):331\u201355."},{"issue":"3","key":"4076_CR6","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1162\/rest.90.3.389","volume":"90","author":"RK Crump","year":"2008","unstructured":"Crump RK, Hotz VJ, Imbens GW, Mitnik OA. Nonparametric tests for treatment effect heterogeneity. Rev Econ Stat. 2008; 90(3):389\u2013405.","journal-title":"Rev Econ Stat"},{"issue":"Feb","key":"4076_CR7","first-page":"141","volume":"10","author":"X Su","year":"2009","unstructured":"Su X, Tsai C-L, Wang H, Nickerson D, Li B. Subgroup analysis via recursive partitioning. J Mach Learn Res. 2009; 10(Feb):141\u201358.","journal-title":"J Mach Learn Res"},{"issue":"2","key":"4076_CR8","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1093\/biomet\/asq012","volume":"97","author":"T Cai","year":"2010","unstructured":"Cai T, Tian L, Uno H, Solomon S, Wei L. Calibrating parametric subject-specific risk estimation. Biometrika. 2010; 97(2):389\u2013404.","journal-title":"Biometrika"},{"issue":"24","key":"4076_CR9","doi-asserted-by":"publisher","first-page":"2867","DOI":"10.1002\/sim.4322","volume":"30","author":"J Foster","year":"2011","unstructured":"Foster J, Taylor J, Ruberg S. Subgroup identification from randomized clinical trial data. Stat Med. 2011; 30(24):2867\u201380.","journal-title":"Stat Med"},{"issue":"499","key":"4076_CR10","doi-asserted-by":"publisher","first-page":"1106","DOI":"10.1080\/01621459.2012.695674","volume":"107","author":"Y Zhao","year":"2012","unstructured":"Zhao Y, Zeng D, Rush A, Kosorok M. Estimating individualized treatment rules using outcome weighted learning. J Am Stat Assoc. 2012; 107(499):1106\u201318.","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"4076_CR11","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1214\/12-AOAS593","volume":"7","author":"K Imai","year":"2013","unstructured":"Imai K, Ratkovic M. Estimating treatment effect heterogeneity in randomized program evaluation. Ann Appl Stat. 2013; 7(1):443\u201370.","journal-title":"Ann Appl Stat"},{"issue":"27","key":"4076_CR12","doi-asserted-by":"publisher","first-page":"7353","DOI":"10.1073\/pnas.1510489113","volume":"113","author":"S Athey","year":"2016","unstructured":"Athey S, Imbens G. Recursive partitioning for heterogeneous causal effects. Proc Natl Acad Sci. 2016; 113(27):7353\u201360.","journal-title":"Proc Natl Acad Sci"},{"issue":"1","key":"4076_CR13","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1002\/sim.7064","volume":"36","author":"I Lipkovich","year":"2017","unstructured":"Lipkovich I, Dmitrienko A, D\u2019Agostino B. Tutorial in biostatistics: data-driven subgroup identification and analysis in clinical trials. Stat Med. 2017; 36(1):136\u201396.","journal-title":"Stat Med"},{"issue":"2","key":"4076_CR14","doi-asserted-by":"publisher","first-page":"1148","DOI":"10.1214\/18-AOS1709","volume":"47","author":"S Athey","year":"2019","unstructured":"Athey S, Tibshirani J, Wager S. Generalized random forests. Ann Stat. 2019; 47(2):1148\u201378.","journal-title":"Ann Stat"},{"key":"4076_CR15","doi-asserted-by":"crossref","unstructured":"Chernozhukov V, Demirer M, Duflo E, Fernadezval I. Generic machine learning inference on heterogeneous treatment effects in randomized experiments. 2019. arXiv:1712.04802v4.","DOI":"10.3386\/w24678"},{"issue":"33","key":"4076_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3310\/hta5330","volume":"5","author":"ST Brookes","year":"2001","unstructured":"Brookes ST, Whitley E, Peters TJ, Mulheran PA, Egger M, Davey Smith G. Subgroup analyses in randomised controlled trials: quantifying the risks of false-positives and false-negatives. Health Technol Assess. 2001; 5(33):1\u201356.","journal-title":"Health Technol Assess"},{"issue":"3","key":"4076_CR17","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1016\/j.jclinepi.2003.08.009","volume":"57","author":"ST Brookes","year":"2004","unstructured":"Brookes ST, Whitely E, Egger M, Smith GD, Mulheran PA, Peters TJ. Subgroup analyses in randomized trials: risks of subgroup-specific analyses;: power and sample size for the interaction test. J Clin Epidemiol. 2004; 57(3):229\u201336.","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"4076_CR18","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1186\/1745-6215-11-85","volume":"11","author":"DM Kent","year":"2010","unstructured":"Kent DM, Rothwell PM, Ioannidis JP, Altman DG, Hayward RA. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010; 11(1):85.","journal-title":"Trials"},{"key":"4076_CR19","doi-asserted-by":"publisher","unstructured":"Watson JA, Holmes C. Exploratory subgroup analysis of the SEAQUAMAT trial using Random Forests: a generic template for the ML analysis of RCT data with binary outcomes. 2018. https:\/\/doi.org\/10.24433\/CO.271758d1-893d-4d24-9cd0-89d162b722b9.","DOI":"10.24433\/CO.271758d1-893d-4d24-9cd0-89d162b722b9"},{"issue":"1","key":"4076_CR20","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random forests. Mach Learn. 2001; 45(1):5\u201332.","journal-title":"Mach Learn"},{"issue":"9487","key":"4076_CR21","first-page":"717","volume":"366","author":"A Dondorp","year":"2004","unstructured":"Dondorp A, Nosten F, Stepniewska K, Day N, White N. Artesunate versus quinine for treatment of severe falciparum malaria: a randomised trial. The Lancet (London, England). 2004; 366(9487):717\u201325.","journal-title":"The Lancet (London, England)"},{"issue":"2","key":"4076_CR22","doi-asserted-by":"publisher","first-page":"361","DOI":"10.2307\/2530862","volume":"41","author":"M Gail","year":"1985","unstructured":"Gail M, Simon R. Testing for qualitative interactions between treatment effects and patient subsets. Biometrics. 1985; 41(2):361\u201372.","journal-title":"Biometrics"},{"issue":"6","key":"4076_CR23","doi-asserted-by":"publisher","first-page":"460","DOI":"10.1511\/2014.111.460","volume":"102","author":"A Gelman","year":"2014","unstructured":"Gelman A, Loken E. The statistical crisis in science. Data-dependent analysis\u2014a garden of forking paths\u2014explains why many statistically significant comparisons don\u2019t hold up. Am Sci. 2014; 102(6):460.","journal-title":"Am Sci"},{"key":"4076_CR24","doi-asserted-by":"publisher","unstructured":"Burke JF, Sussman JB, Kent DM, Hayward RA. Three simple rules to ensure reasonably credible subgroup analyses. Br Med J. 2015;351. https:\/\/doi.org\/10.1136\/bmj.h5651.","DOI":"10.1136\/bmj.h5651"},{"issue":"488","key":"4076_CR25","doi-asserted-by":"publisher","first-page":"1671","DOI":"10.1198\/jasa.2009.tm08647","volume":"104","author":"N Meinshausen","year":"2009","unstructured":"Meinshausen N, Meier L, B\u00fchlmann P. P-values for high-dimensional regression. J Am Stat Assoc. 2009; 104(488):1671\u201381.","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"4076_CR26","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1186\/1471-2288-6-18","volume":"6","author":"RA Hayward","year":"2006","unstructured":"Hayward RA, Kent DM, Vijan S, Hofer TP. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006; 6(1):18.","journal-title":"BMC Med Res Methodol"},{"key":"4076_CR27","doi-asserted-by":"publisher","first-page":"4245","DOI":"10.1136\/bmj.k4245","volume":"363","author":"DM Kent","year":"2018","unstructured":"Kent DM, Steyerberg E, van Klaveren D. Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. Br Med J. 2018; 363:4245.","journal-title":"Br Med J"},{"key":"4076_CR28","volume-title":"Data mining: practical machine learning tools and techniques, 2nd ed","author":"IH Witten","year":"2016","unstructured":"Witten IH, Frank E, Hall MA, Pal CJ. Data mining: practical machine learning tools and techniques, 2nd ed. Burlington: Morgan Kaufmann; 2016."},{"issue":"523","key":"4076_CR29","doi-asserted-by":"publisher","first-page":"1228","DOI":"10.1080\/01621459.2017.1319839","volume":"113","author":"S Wager","year":"2018","unstructured":"Wager S, Athey S. Estimation and inference of heterogeneous treatment effects using random forests. J Am Stat Assoc. 2018; 113(523):1228\u201342. https:\/\/doi.org\/10.1080\/01621459.2017.1319839.","journal-title":"J Am Stat Assoc"},{"key":"4076_CR30","doi-asserted-by":"crossref","unstructured":"Spiegelhalter D. J R Stat Soc Ser A (Stat Soc). 2017; 180(4):1\u201316.","DOI":"10.1111\/rssa.12302"},{"key":"4076_CR31","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7","volume-title":"The elements of statistical learning, 2nd ed","author":"T Hastie","year":"2009","unstructured":"Hastie T, Tibshirani R, Friedman J. The elements of statistical learning, 2nd ed. New York: Springer; 2009."},{"issue":"3","key":"4076_CR32","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1214\/08-AOAS169","volume":"2","author":"H Ishwaran","year":"2008","unstructured":"Ishwaran H, Kogalur UB, Blackstone EH, Lauer MS. Random survival forests. Ann Appl Stat. 2008; 2(3):841\u201360.","journal-title":"Ann Appl Stat"},{"issue":"2","key":"4076_CR33","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1086\/589287","volume":"47","author":"AM Dondorp","year":"2008","unstructured":"Dondorp AM, Lee SJ, Faiz M, Mishra S, Price R, Tjitra E, Than M, Htut Y, Mohanty S, Yunus EB. The relationship between age and the manifestations of and mortality associated with severe malaria. Clin Infect Dis. 2008; 47(2):151\u20137.","journal-title":"Clin Infect Dis"},{"issue":"9753","key":"4076_CR34","doi-asserted-by":"publisher","first-page":"1647","DOI":"10.1016\/S0140-6736(10)61924-1","volume":"376","author":"AM Dondorp","year":"2010","unstructured":"Dondorp AM, Fanello CI, Hendriksen IC, Gomes E, Seni A, Chhaganlal KD, Bojang K, Olaosebikan R, Anunobi N, Maitland K, et al.Artesunate versus quinine in the treatment of severe falciparum malaria in African children (AQUAMAT): an open-label, randomised trial. Lancet. 2010; 376(9753):1647\u201357.","journal-title":"Lancet"},{"issue":"9918","key":"4076_CR35","doi-asserted-by":"publisher","first-page":"723","DOI":"10.1016\/S0140-6736(13)60024-0","volume":"383","author":"NJ White","year":"2014","unstructured":"White NJ, Pukrittayakamee S, Hien TT, Faiz MA, Mokuolu OA, Dondorp AM. Malaria. The Lancet. 2014; 383(9918):723\u201335. https:\/\/doi.org\/10.1016\/S0140-6736(13)60024-0.","journal-title":"The Lancet"},{"issue":"5","key":"4076_CR36","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1086\/649928","volume":"50","author":"J Hanson","year":"2010","unstructured":"Hanson J, Lee SJ, Mohanty S, Faiz M, Anstey NM, Charunwatthana Pk, Yunus EB, Mishra SK, Tjitra E, Price RN, et al. A simple score to predict the outcome of severe malaria in adults. Clin Infect Dis. 2010; 50(5):679\u201385.","journal-title":"Clin Infect Dis"},{"issue":"5","key":"4076_CR37","doi-asserted-by":"publisher","first-page":"411","DOI":"10.1056\/NEJMoa1314981","volume":"371","author":"EA Ashley","year":"2014","unstructured":"Ashley EA, Dhorda M, Fairhurst RM, Amaratunga C, Lim P, Suon S, Sreng S, Anderson JM, Mao S, Sam B, et al. Spread of artemisinin resistance in Plasmodium falciparum malaria. N Engl J Med. 2014; 371(5):411\u201323.","journal-title":"N Engl J Med"},{"issue":"1","key":"4076_CR38","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1186\/1475-2875-10-278","volume":"10","author":"N White","year":"2011","unstructured":"White N. The parasite clearance curve. Malar J. 2011; 10(1):278.","journal-title":"Malar J"}],"container-title":["Trials"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13063-020-4076-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s13063-020-4076-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13063-020-4076-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,2,8]],"date-time":"2021-02-08T20:54:18Z","timestamp":1612817658000},"score":1,"resource":{"primary":{"URL":"https:\/\/trialsjournal.biomedcentral.com\/articles\/10.1186\/s13063-020-4076-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,10]]},"references-count":38,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["4076"],"URL":"https:\/\/doi.org\/10.1186\/s13063-020-4076-y","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/330795","asserted-by":"object"}]},"ISSN":["1745-6215"],"issn-type":[{"value":"1745-6215","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,10]]},"assertion":[{"value":"22 March 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 January 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 February 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"All patients in the SEAQUAMAT study gave full informed consent for study participation. The study was approved by the Oxford University ethics review board and local ethics committees.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"156"}}