{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T11:13:34Z","timestamp":1792494814660,"version":"4.5.0"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2017,8,7]],"date-time":"2017-08-07T00:00:00Z","timestamp":1502064000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2017,8,7]],"date-time":"2017-08-07T00:00:00Z","timestamp":1502064000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The prediction of relapse in childhood acute lymphoblastic leukemia (ALL) is a critical factor for successful treatment and follow-up planning. Our goal was to construct an ALL relapse prediction model based on machine learning algorithms. Monte Carlo cross-validation nested by 10-fold cross-validation was used to rank clinical variables on the randomly split training sets of 336 newly diagnosed ALL children, and a forward feature selection algorithm was employed to find the shortest list of most discriminatory variables. To enable an unbiased estimation of the prediction model to new patients, besides the split test sets of 150 patients, we introduced another independent data set of 84 patients to evaluate the model. The Random Forest model with 14 features achieved a cross-validation accuracy of 0.827\u2009\u00b1\u20090.031 on one set and an accuracy of 0.798 on the other, with the area under the curve of 0.902\u2009\u00b1\u20090.027 and 0.904, respectively. The model performed well across different risk-level groups, with the best accuracy of 0.829 in the standard-risk group. To our knowledge, this is the first study to use machine learning models to predict childhood ALL relapse based on medical data from Electronic Medical Record, which will further facilitate stratification treatments.<\/jats:p>","DOI":"10.1038\/s41598-017-07408-0","type":"journal-article","created":{"date-parts":[[2017,8,1]],"date-time":"2017-08-01T10:58:10Z","timestamp":1501585090000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":102,"title":["Machine learning applications for prediction of relapse in childhood acute lymphoblastic leukemia"],"prefix":"10.1038","volume":"7","author":[{"given":"Liyan","family":"Pan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guangjian","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fangqin","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuling","family":"Zhong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huimin","family":"Xia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xin","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Huiying","family":"Liang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2017,8,7]]},"reference":[{"key":"7408_CR1","doi-asserted-by":"publisher","first-page":"2625","DOI":"10.1200\/JCO.2009.27.0421","volume":"28","author":"MA Smith","year":"2010","unstructured":"Smith, M. A. et al. Outcomes for children and adolescents with cancer: challenges for the twenty-first century. Journal of clinical oncology: official journal of the American Society of Clinical Oncology \n                           28, 2625\u20132634, doi:10.1200\/JCO.2009.27.0421 (2010).","journal-title":"Journal of clinical oncology: official journal of the American Society of Clinical Oncology"},{"key":"7408_CR2","doi-asserted-by":"publisher","first-page":"2730","DOI":"10.1056\/NEJMoa0900386","volume":"360","author":"CH Pui","year":"2009","unstructured":"Pui, C. H. et al. Treating childhood acute lymphoblastic leukemia without cranial irradiation. The New England journal of medicine \n                           360, 2730\u20132741, doi:10.1056\/NEJMoa0900386 (2009).","journal-title":"The New England journal of medicine"},{"key":"7408_CR3","doi-asserted-by":"publisher","first-page":"1541","DOI":"10.1056\/NEJMra1400972","volume":"373","author":"SP Hunger","year":"2015","unstructured":"Hunger, S. P. & Mullighan, C. G. Acute Lymphoblastic Leukemia in Children. The New England journal of medicine \n                           373, 1541\u20131552, doi:10.1056\/NEJMra1400972 (2015).","journal-title":"The New England journal of medicine"},{"key":"7408_CR4","doi-asserted-by":"publisher","first-page":"e205","DOI":"10.1016\/S1470-2045(12)70580-6","volume":"14","author":"D Bhojwani","year":"2013","unstructured":"Bhojwani, D. & Pui, C. H. Relapsed childhood acute lymphoblastic leukaemia. The Lancet. Oncology \n                           14, e205\u2013217, doi:10.1016\/S1470-2045(12)70580-6 (2013).","journal-title":"The Lancet. Oncology"},{"key":"7408_CR5","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1016\/S0933-3657(02)00086-6","volume":"27","author":"JM Jerez-Aragones","year":"2003","unstructured":"Jerez-Aragones, J. M., Gomez-Ruiz, J. A., Ramos-Jimenez, G., Munoz-Perez, J. & Alba-Conejo, E. A combined neural network and decision trees model for prognosis of breast cancer relapse. Artificial intelligence in medicine \n                           27, 45\u201363 (2003).","journal-title":"Artificial intelligence in medicine"},{"key":"7408_CR6","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1007\/s11899-016-0337-y","volume":"11","author":"LM Vrooman","year":"2016","unstructured":"Vrooman, L. M. & Silverman, L. B. Treatment of Childhood Acute Lymphoblastic Leukemia: Prognostic Factors and Clinical Advances. Current hematologic malignancy reports \n                           11, 385\u2013394, doi:10.1007\/s11899-016-0337-y (2016).","journal-title":"Current hematologic malignancy reports"},{"key":"7408_CR7","doi-asserted-by":"publisher","first-page":"606","DOI":"10.1111\/bjh.12442","volume":"162","author":"DT Teachey","year":"2013","unstructured":"Teachey, D. T. & Hunger, S. P. Predicting relapse risk in childhood acute lymphoblastic leukaemia. British journal of haematology \n                           162, 606\u2013620, doi:10.1111\/bjh.12442 (2013).","journal-title":"British journal of haematology"},{"key":"7408_CR8","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/S2215-0366(15)00549-0","volume":"3","author":"IC Passos","year":"2016","unstructured":"Passos, I. C., Mwangi, B. & Kapczinski, F. Big data analytics and machine learning: 2015 and beyond. The lancet. Psychiatry \n                           3, 13\u201315, doi:10.1016\/S2215-0366(15)00549-0 (2016).","journal-title":"The lancet. Psychiatry"},{"key":"7408_CR9","first-page":"59","volume":"2","author":"JA Cruz","year":"2007","unstructured":"Cruz, J. A. & Wishart, D. S. Applications of machine learning in cancer prediction and prognosis. Cancer informatics \n                           2, 59\u201377 (2007).","journal-title":"Cancer informatics"},{"key":"7408_CR10","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.csbj.2014.11.005","volume":"13","author":"K Kourou","year":"2015","unstructured":"Kourou, K., Exarchos, T. P., Exarchos, K. P., Karamouzis, M. V. & Fotiadis, D. I. Machine learning applications in cancer prognosis and prediction. Computational and structural biotechnology journal \n                           13, 8\u201317, doi:10.1016\/j.csbj.2014.11.005 (2015).","journal-title":"Computational and structural biotechnology journal"},{"key":"7408_CR11","unstructured":"Q., W. Application of Machine Learning Techniques to Acute Myeloid Leukemia Master thesis, University of Washington (2016)."},{"key":"7408_CR12","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1016\/j.jbi.2016.02.017","volume":"60","author":"EJ deAndres-Galiana","year":"2016","unstructured":"deAndres-Galiana, E. J. et al. Analysis of clinical prognostic variables for Chronic Lymphocytic Leukemia decision-making problems. Journal of biomedical informatics \n                           60, 342\u2013351, doi:10.1016\/j.jbi.2016.02.017 (2016).","journal-title":"Journal of biomedical informatics"},{"key":"7408_CR13","doi-asserted-by":"publisher","first-page":"3144","DOI":"10.1200\/JCO.2014.59.1339","volume":"33","author":"R Shouval","year":"2015","unstructured":"Shouval, R. et al. Prediction of Allogeneic Hematopoietic Stem-Cell Transplantation Mortality 100 Days After Transplantation Using a Machine Learning Algorithm: A European Group for Blood and Marrow Transplantation Acute Leukemia Working Party Retrospective Data Mining Study. Journal of clinical oncology: official journal of the American Society of Clinical Oncology \n                           33, 3144\u20133151, doi:10.1200\/JCO.2014.59.1339 (2015).","journal-title":"Journal of clinical oncology: official journal of the American Society of Clinical Oncology"},{"key":"7408_CR14","doi-asserted-by":"publisher","first-page":"7219","DOI":"10.2147\/OTT.S115257","volume":"9","author":"Y Hu","year":"2016","unstructured":"Hu, Y. et al. E2A-PBX1 exhibited a promising prognosis in pediatric acute lymphoblastic leukemia treated with the CCLG-ALL2008 protocol. OncoTargets and therapy \n                           9, 7219\u20137225, doi:10.2147\/OTT.S115257 (2016).","journal-title":"OncoTargets and therapy"},{"key":"7408_CR15","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1016\/S1535-6108(02)00032-6","volume":"1","author":"EJ Yeoh","year":"2002","unstructured":"Yeoh, E. J. et al. Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling. Cancer cell \n                           1, 133\u2013143 (2002).","journal-title":"Cancer cell"},{"key":"7408_CR16","doi-asserted-by":"publisher","first-page":"1270","DOI":"10.1038\/sj.leu.2403392","volume":"18","author":"H Willenbrock","year":"2004","unstructured":"Willenbrock, H., Juncker, A., Schmiegelow, K., Knudsen, S. & Ryder, L. Prediction of immunophenotype, treatment response, and relapse in childhood acute lymphoblastic leukemia using DNA microarrays. Leukemia \n                           18, 1270\u20131277 (2004).","journal-title":"Leukemia"},{"key":"7408_CR17","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1038\/leu.2010.275","volume":"25","author":"E Waanders","year":"2011","unstructured":"Waanders, E. et al. Integrated use of minimal residual disease classification and IKZF1 alteration status accurately predicts 79% of relapses in pediatric acute lymphoblastic leukemia. Leukemia \n                           25, 254\u2013258 (2011).","journal-title":"Leukemia"},{"key":"7408_CR18","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.jclinepi.2015.10.002","volume":"71","author":"F Bagherzadeh-Khiabani","year":"2016","unstructured":"Bagherzadeh-Khiabani, F. et al. A tutorial on variable selection for clinical prediction models: feature selection methods in data mining could improve the results. Journal of clinical epidemiology \n                           71, 76\u201385, doi:10.1016\/j.jclinepi.2015.10.002 (2016).","journal-title":"Journal of clinical epidemiology"},{"key":"7408_CR19","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1007\/s10654-009-9411-2","volume":"24","author":"S Walter","year":"2009","unstructured":"Walter, S. & Tiemeier, H. Variable selection: current practice in epidemiological studies. European journal of epidemiology \n                           24, 733\u2013736, doi:10.1007\/s10654-009-9411-2 (2009).","journal-title":"European journal of epidemiology"},{"key":"7408_CR20","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1586\/17474086.2015.978281","volume":"8","author":"F Ceppi","year":"2015","unstructured":"Ceppi, F., Cazzaniga, G., Colombini, A., Biondi, A. & Conter, V. Risk factors for relapse in childhood acute lymphoblastic leukemia: prediction and prevention. Expert review of hematology \n                           8, 57\u201370, doi:10.1586\/17474086.2015.978281 (2015).","journal-title":"Expert review of hematology"},{"key":"7408_CR21","doi-asserted-by":"publisher","first-page":"e205","DOI":"10.1016\/S1470-2045(12)70580-6","volume":"14","author":"D Bhojwani","year":"2013","unstructured":"Bhojwani, D. & Pui, C.-H. Relapsed childhood acute lymphoblastic leukaemia. The Lancet Oncology \n                           14, e205\u2013e217, doi:10.1016\/s1470-2045(12)70580-6 (2013).","journal-title":"The Lancet Oncology"},{"key":"7408_CR22","doi-asserted-by":"publisher","first-page":"3","DOI":"10.3329\/cbmj.v2i1.14174","volume":"2","author":"M Tawfique","year":"2013","unstructured":"Tawfique, M. et al. Relationship of Initial Lactate Dehydrogenase (LDH) with initial tumor load and bone marrow remission in childhood Acute Lymphoblastic Leukemia (ALL). Community Based Medical Journal \n                           2, 3\u20138 (2013).","journal-title":"Community Based Medical Journal"},{"key":"7408_CR23","doi-asserted-by":"publisher","first-page":"441","DOI":"10.1182\/blood-2005-07-3011","volume":"108","author":"JM Hilden","year":"2006","unstructured":"Hilden, J. M. et al. Analysis of prognostic factors of acute lymphoblastic leukemia in infants: report on CCG 1953 from the Children\u2019s Oncology Group. Blood \n                           108, 441\u2013451 (2006).","journal-title":"Blood"},{"key":"7408_CR24","doi-asserted-by":"publisher","first-page":"1692","DOI":"10.1038\/leu.2008.152","volume":"22","author":"B Gruhn","year":"2008","unstructured":"Gruhn, B. et al. Prenatal origin of childhood acute lymphoblastic leukemia, association with birth weight and hyperdiploidy. Leukemia \n                           22, 1692\u20131697, doi:10.1038\/leu.2008.152 (2008).","journal-title":"Leukemia"},{"key":"7408_CR25","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1111\/j.1365-2141.2009.07769.x","volume":"146","author":"C Mitchell","year":"2009","unstructured":"Mitchell, C. et al. The impact of risk stratification by early bone-marrow response in childhood lymphoblastic leukaemia: results from the United Kingdom Medical Research Council trial ALL97 and ALL97\/99. British journal of haematology \n                           146, 424\u2013436, doi:10.1111\/j.1365-2141.2009.07769.x (2009).","journal-title":"British journal of haematology"},{"key":"7408_CR26","doi-asserted-by":"publisher","first-page":"4477","DOI":"10.1182\/blood-2007-09-112920","volume":"111","author":"A M\u00f6ricke","year":"2008","unstructured":"M\u00f6ricke, A. et al. Risk-adjusted therapy of acute lymphoblastic leukemia can decrease treatment burden and improve survival: treatment results of 2169 unselected pediatric and adolescent patients enrolled in the trial ALL-BFM 95. Blood \n                           111, 4477\u20134489 (2008).","journal-title":"Blood"},{"key":"7408_CR27","doi-asserted-by":"publisher","first-page":"2690","DOI":"10.1182\/blood-2004-04-1616","volume":"104","author":"C-H Pui","year":"2004","unstructured":"Pui, C.-H. et al. Improved outcome for children with acute lymphoblastic leukemia: results of Total Therapy Study XIIIB at St Jude Children\u2019s Research Hospital. Blood \n                           104, 2690\u20132696 (2004).","journal-title":"Blood"},{"key":"7408_CR28","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1016\/j.jmoldx.2012.04.006","volume":"14","author":"I Schrijver","year":"2012","unstructured":"Schrijver, I. et al. Opportunities and challenges associated with clinical diagnostic genome sequencing: a report of the Association for Molecular Pathology. The Journal of molecular diagnostics: JMD \n                           14, 525\u2013540, doi:10.1016\/j.jmoldx.2012.04.006 (2012).","journal-title":"The Journal of molecular diagnostics: JMD"},{"key":"7408_CR29","doi-asserted-by":"publisher","first-page":"1463","DOI":"10.1001\/jama.2013.1465","volume":"309","author":"GS Ginsburg","year":"2013","unstructured":"Ginsburg, G. S. Realizing the opportunities of genomics in health care. Jama \n                           309, 1463\u20131464, doi:10.1001\/jama.2013.1465 (2013).","journal-title":"Jama"},{"key":"7408_CR30","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1016\/S0140-6736(05)17866-0","volume":"365","author":"S Michiels","year":"2005","unstructured":"Michiels, S., Koscielny, S. & Hill, C. Prediction of cancer outcome with microarrays: a multiple random validation strategy. Lancet \n                           365, 488\u2013492, doi:10.1016\/S0140-6736(05)17866-0 (2005).","journal-title":"Lancet"},{"key":"7408_CR31","doi-asserted-by":"crossref","first-page":"77","DOI":"10.4137\/CIN.S408","volume":"6","author":"AL Boulesteix","year":"2008","unstructured":"Boulesteix, A. L., Strobl, C., Augustin, T. & Daumer, M. Evaluating microarray-based classifiers: an overview. Cancer informatics \n                           6, 77\u201397 (2008).","journal-title":"Cancer informatics"},{"key":"7408_CR32","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1016\/j.compbiomed.2010.03.005","volume":"40","author":"DC Li","year":"2010","unstructured":"Li, D. C., Liu, C. W. & Hu, S. C. A learning method for the class imbalance problem with medical data sets. Computers in biology and medicine \n                           40, 509\u2013518, doi:10.1016\/j.compbiomed.2010.03.005 (2010).","journal-title":"Computers in biology and medicine"},{"key":"7408_CR33","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","volume":"16","author":"NV Chawla","year":"2002","unstructured":"Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. SMOTE: Synthetic minority over-sampling technique. J Artif Intell Res \n                           16, 321\u2013357 (2002).","journal-title":"J Artif Intell Res"},{"key":"7408_CR34","first-page":"1157","volume":"3","author":"I Guyon","year":"2003","unstructured":"Guyon, I. & Elisseeff, A. An introduction to variable and feature selection. Journal of machine learning research \n                           3, 1157\u20131182 (2003).","journal-title":"Journal of machine learning research"},{"key":"7408_CR35","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1023\/A:1012487302797","volume":"46","author":"I Guyon","year":"2002","unstructured":"Guyon, I., Weston, J., Barnhill, S. & Vapnik, V. Gene selection for cancer classification using support vector machines. Machine learning \n                           46, 389\u2013422 (2002).","journal-title":"Machine learning"},{"key":"7408_CR36","first-page":"163","volume":"1","author":"S Vijayarani","year":"2016","unstructured":"Vijayarani, S. & Sylviaa, M. S. M. Dimensionality Reduction-A Study. International Journal of Engineering Applied Sciences and Technology \n                           1, 163\u2013170 (2016).","journal-title":"International Journal of Engineering Applied Sciences and Technology"},{"key":"7408_CR37","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research \n                           12, 2825\u20132830 (2011).","journal-title":"Journal of Machine Learning Research"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-07408-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-07408-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-07408-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T07:59:13Z","timestamp":1671782353000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-017-07408-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,7]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,12]]}},"alternative-id":["7408"],"URL":"https:\/\/doi.org\/10.1038\/s41598-017-07408-0","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,8,7]]},"assertion":[{"value":"28 February 2017","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 June 2017","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 August 2017","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"7402"}}