{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T16:34:20Z","timestamp":1790699660687,"version":"4.1.0"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,7,1]],"date-time":"2021-07-01T00:00:00Z","timestamp":1625097600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,1]],"date-time":"2021-07-01T00:00:00Z","timestamp":1625097600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100009395","name":"CancerCare Manitoba Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009395","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Cancer"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Algorithms that use administrative health and electronic medical record (EMR) data to determine cancer recurrence have the potential to replace chart reviews. This study evaluated algorithms to determine breast and colorectal cancer recurrence in a Canadian province with a universal health care system.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>Individuals diagnosed with stage I-III breast or colorectal cancer diagnosed from 2004 to 2012 in Manitoba, Canada were included. Pre-specified and conditional inference tree algorithms using administrative health and structured EMR data were developed. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) correct classification, and scaled Brier scores were measured.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>The weighted pre-specified variable algorithm for the breast cancer validation cohort (<jats:italic>N<\/jats:italic>\u00a0=\u20091181, 167 recurrences) demonstrated 81.1% sensitivity, 93.2% specificity, 61.4% PPV, 97.4% NPV, 91.8% correct classification, and scaled Brier score of 0.21. The weighted conditional inference tree algorithm demonstrated 68.5% sensitivity, 97.0% specificity, 75.4% PPV, 95.8% NPV, 93.6% correct classification, and scaled Brier score of 0.39. The weighted pre-specified variable algorithm for the colorectal validation cohort (<jats:italic>N<\/jats:italic>\u00a0=\u2009693, 136 recurrences) demonstrated 77.7% sensitivity, 92.8% specificity, 70.7% PPV, 94.9% NPV, 90.1% correct classification, and scaled Brier score of 0.33. The conditional inference tree algorithm demonstrated 62.6% sensitivity, 97.8% specificity, 86.4% PPV, 92.2% NPV, 91.4% correct classification, and scaled Brier score of 0.42.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Algorithms developed in this study using administrative health and structured EMR data to determine breast and colorectal cancer recurrence had moderate sensitivity and PPV, high specificity, NPV, and correct classification, but low accuracy. The accuracy is similar to other algorithms developed to classify recurrence only (i.e., distinguished from second primary) and inferior to algorithms that do not make this distinction. The accuracy of algorithms for determining cancer recurrence only must improve before replacing chart reviews.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12885-021-08526-9","type":"journal-article","created":{"date-parts":[[2021,7,1]],"date-time":"2021-07-01T17:04:02Z","timestamp":1625159042000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Evaluation of algorithms using administrative health and structured electronic medical record data to determine breast and colorectal cancer recurrence in a Canadian province"],"prefix":"10.1186","volume":"21","author":[{"given":"Pascal","family":"Lambert","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marshall","family":"Pitz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harminder","family":"Singh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0044-7545","authenticated-orcid":false,"given":"Kathleen","family":"Decker","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,7,1]]},"reference":[{"key":"8526_CR1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-1348-7_14","volume-title":"Epidemiology of Cancer recurrence, second primary Cancer, and comorbidity among Cancer survivors","author":"X Yu","year":"2011","unstructured":"Yu X. In: Feuerstein M, Ganz P, editors. Epidemiology of Cancer recurrence, second primary Cancer, and comorbidity among Cancer survivors. New York: Springer; 2011."},{"key":"8526_CR2","unstructured":"North American Association of Central Cancer Registries. APPENDIX C - Data Quality Indicators by Year and Registry. In: Hotes Ellison J, Wu XC, McLaughlin C, Lake A, Firth R, et al., editors. Cancer In North America: 1999\u20132003 Volume One: Incidence. Springfield: North American Association of Cancer Registries Inc.; 2006. p. II-325."},{"key":"8526_CR3","doi-asserted-by":"crossref","unstructured":"Lamont EB, Hernon JE, Weeks JC, Henderson C, Earle CR, Schilsky RL, et al. Measuring disease-free survival and cancer relapse using medicare claims from CALGB breast cancer trial participants (Companion to 9344). J Natl Cancer Inst. 2006;98(18)1335-8.","DOI":"10.1093\/jnci\/djj363"},{"key":"8526_CR4","doi-asserted-by":"crossref","unstructured":"Livaudais-Toman J, Franco R, Prasad-Hayes M, Howell EA, Wisnivesky J, Bickell NA. A validation of administrative claims data to measure ovarian cancer recurrence and secondary debluking surgery. EGEMS. 2016;4(1):1208.","DOI":"10.13063\/2327-9214.1208"},{"issue":"12","key":"8526_CR5","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1093\/jnci\/djs233","volume":"104","author":"J Chubak","year":"2012","unstructured":"Chubak J, Yu O, Pocobelli G, Lamerato L, Webster J, Prout MN, et al. Administrative data algorithms to identify second breast cancer events following early-stage invasive cancer. J Natl Cancer Inst. 2012;104(12):931\u201340. https:\/\/doi.org\/10.1093\/jnci\/djs233.","journal-title":"J Natl Cancer Inst"},{"key":"8526_CR6","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/j.canep.2019.01.016","volume":"59","author":"LA Rasmussen","year":"2019","unstructured":"Rasmussen LA, Jensen H, Flytkjaer Virgilsen L, Beck Jellesmark Thorsen L, Vrou Offersen B, Vedsted P. A validated algorithm for register-based identification of patients with recurrence of breast cancer - based on Danish Breast Cancer Group (DBCG) data. Cancer Epidemiol. 2019;59:129\u201334.","journal-title":"Cancer Epidemiol"},{"issue":"10","key":"8526_CR7","doi-asserted-by":"publisher","first-page":"e65","DOI":"10.1097\/MLR.0b013e318277eb6f","volume":"52","author":"MJ Hassett","year":"2014","unstructured":"Hassett MJ, Ritzwoller DP, Taback N, Carroll N, Cronin AM, Ting GV, et al. Validating billing\/encounter codes as indicators of lung, colorectal, breast, and prostate cancer recurrence using 2 large contemporary cohorts. Med Care. 2014;52(10):e65\u201373. https:\/\/doi.org\/10.1097\/MLR.0b013e318277eb6f.","journal-title":"Med Care"},{"issue":"12","key":"8526_CR8","doi-asserted-by":"publisher","first-page":"e88","DOI":"10.1097\/MLR.0000000000000404","volume":"55","author":"MJ Hassett","year":"2017","unstructured":"Hassett MJ, Uno H, Cronin AM, Hornbrook MC, Ritzwoller DP. Detecting lung and colorectal cancer recurrence using structured clinical\/administrative data to enable outcomes research and population health management. Med Care. 2017;55(12):e88\u201398. https:\/\/doi.org\/10.1097\/MLR.0000000000000404.","journal-title":"Med Care"},{"key":"8526_CR9","doi-asserted-by":"crossref","unstructured":"Xu Y, Kong S, Cheung WY, Bouchard-Fortier A, Dort JC, Quan H, et al. Development and validation of case-finding algorithms for recurrence of breast cancer using routinely collected administrative data. BMC Cancer. 2019;19(210):210.","DOI":"10.1186\/s12885-019-5432-8"},{"issue":"3","key":"8526_CR10","doi-asserted-by":"publisher","first-page":"e343","DOI":"10.3747\/co.27.5861","volume":"27","author":"ZF Cairncross","year":"2020","unstructured":"Cairncross ZF, Nelson G, Shack L, Metcalfe A. Validation in Alberta of an administrative data algorithm to identify cancer recurrence. Curr Oncol. 2020;27(3):e343\u2013e6. https:\/\/doi.org\/10.3747\/co.27.5861.","journal-title":"Curr Oncol"},{"key":"8526_CR11","unstructured":"Manitoba Health, Seniors and Active Living.\u00a0Population Report, June 1, 2019.\u00a0Winnipeg:\u00a0Manitoba Health, Seniors and Active Living;\u00a02019."},{"issue":"9","key":"8526_CR12","doi-asserted-by":"publisher","first-page":"932","DOI":"10.1097\/00005650-199709000-00006","volume":"35","author":"JR Robinson","year":"1997","unstructured":"Robinson JR, Young TK, Roos LL, Gelskey DE. Estimating the burden of disease. Comparing administrative data and self-reports. Med Care. 1997;35(9):932\u201347. https:\/\/doi.org\/10.1097\/00005650-199709000-00006.","journal-title":"Med Care"},{"issue":"3","key":"8526_CR13","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1097\/00005650-199303000-00002","volume":"31","author":"LL Roos","year":"1993","unstructured":"Roos LL, Mustard CA, Nicol JP, McLarran DF, Malenka DJ, et al. Registries and administrative data: organization and accuracy. Med Care. 1993;31(3):201\u201312. https:\/\/doi.org\/10.1097\/00005650-199303000-00002.","journal-title":"Med Care"},{"issue":"4","key":"8526_CR14","doi-asserted-by":"publisher","first-page":"1167","DOI":"10.1111\/j.1475-6773.2005.00407.x","volume":"40","author":"LL Roos","year":"2005","unstructured":"Roos LL, Walld R, Uhanova J, Bond R. Physician visits, hospitalizations, and socioeconomic status: ambulatory case sensitive condintions in a Canadian setting. Health Serv Res. 2005;40(4):1167\u201385. https:\/\/doi.org\/10.1111\/j.1475-6773.2005.00407.x.","journal-title":"Health Serv Res"},{"issue":"29","key":"8526_CR15","doi-asserted-by":"publisher","first-page":"3885","DOI":"10.1200\/JCO.2011.36.1105","volume":"29","author":"ND Arvold","year":"2011","unstructured":"Arvold ND, Taghaian AG, Niemierko A, Abi Raad RF, Srreedhara M, Nguyen PL, et al. Age, breast cancer subtype approximation, and local recurrence after breast-conserving therapy. J Clin Oncol. 2011;29(29):3885\u201391. https:\/\/doi.org\/10.1200\/JCO.2011.36.1105.","journal-title":"J Clin Oncol"},{"issue":"3","key":"8526_CR16","doi-asserted-by":"publisher","first-page":"177","DOI":"10.3322\/caac.21395","volume":"67","author":"R Siegel","year":"2017","unstructured":"Siegel R, Miller K, Fedewa S, Ahnen D, Meester R, Barzi A, et al. Colorectal cancer statistics, 2017. CA Cancer J Clin. 2017;67(3):177\u201393. https:\/\/doi.org\/10.3322\/caac.21395.","journal-title":"CA Cancer J Clin"},{"issue":"3","key":"8526_CR17","doi-asserted-by":"publisher","first-page":"419","DOI":"10.2307\/2983440","volume":"158","author":"C Chatfield","year":"1995","unstructured":"Chatfield C. Model uncertainty, data mining and statistical inference. J R Stat Soc A. 1995;158(3):419\u201366. https:\/\/doi.org\/10.2307\/2983440.","journal-title":"J R Stat Soc A"},{"issue":"382","key":"8526_CR18","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1080\/01621459.1983.10477973","volume":"78","author":"B Efron","year":"1983","unstructured":"Efron B. Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc. 1983;78(382):316\u201331. https:\/\/doi.org\/10.1080\/01621459.1983.10477973.","journal-title":"J Am Stat Assoc"},{"key":"8526_CR19","doi-asserted-by":"publisher","unstructured":"Harrell FE, Lee K, Mark D. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing erros. Stat Med. 1996;15(4):361\u201387. https:\/\/doi.org\/10.1002\/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4.","DOI":"10.1002\/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4"},{"issue":"3","key":"8526_CR20","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1198\/106186006X133933","volume":"15","author":"T Hothorn","year":"2006","unstructured":"Hothorn T, Hornik K, Zeileis A. Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat. 2006;15(3):651\u201374. https:\/\/doi.org\/10.1198\/106186006X133933.","journal-title":"J Comput Graph Stat"},{"key":"8526_CR21","doi-asserted-by":"crossref","unstructured":"Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-38.","DOI":"10.1097\/EDE.0b013e3181c30fb2"},{"key":"8526_CR22","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-77244-8","volume-title":"Clinical prediction models: a practical approach to development, validation, and udating","author":"EW Steyerberg","year":"2009","unstructured":"Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and udating. New York: Springer-Verlag; 2009. https:\/\/doi.org\/10.1007\/978-0-387-77244-8."},{"issue":"sep18 1","key":"8526_CR23","doi-asserted-by":"publisher","first-page":"a1343","DOI":"10.1136\/bmj.a1343","volume":"337","author":"S Albrechtsen","year":"2008","unstructured":"Albrechtsen S, Rasmussen M, Thoresen S, Irgens LM, Iversen OE. Pregnancy outcomes in women before and after cervical conisation: population based cohort study. BMJ. 2008;337(sep18 1):a1343. https:\/\/doi.org\/10.1136\/bmj.a1343.","journal-title":"BMJ."},{"key":"8526_CR24","doi-asserted-by":"crossref","unstructured":"Banerjee I, Bozkurt S, Caswell-Jin JL, Kurian AW, Rubin DL. Natural language processing approaches to detect the timeline of metastatic recurrence of breast cancer. JCO Cllin Cancer Inform. 2019:1\u201312.","DOI":"10.1200\/CCI.19.00034"},{"issue":"6","key":"8526_CR25","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1093\/aje\/kwt441","volume":"179","author":"DS Carrell","year":"2013","unstructured":"Carrell DS, Halgrim S, Tran D-T, Buist DS, Chubak J, Chapman WW, et al. Using natural language processing to improve efficiency of manual chart abstraction in research: the case of breast cancer recurrence. Am J Epidemiol. 2013;179(6):749\u201358.","journal-title":"Am J Epidemiol"},{"issue":"4","key":"8526_CR26","doi-asserted-by":"publisher","first-page":"528","DOI":"10.1093\/jamiaopen\/ooz040","volume":"2","author":"AY Ling","year":"2019","unstructured":"Ling AY, Kurian AW, Caswell-Jin JL, Sledge GW Jr, Shah NH, Tamang SR. Using natural language processing to construct a metastatic breast cancer cohort from linked cancer registry and electronic medical records data. JAMIA Open. 2019;2(4):528\u201337. https:\/\/doi.org\/10.1093\/jamiaopen\/ooz040.","journal-title":"JAMIA Open"},{"issue":"S17","key":"8526_CR27","doi-asserted-by":"publisher","first-page":"498","DOI":"10.1186\/s12859-018-2466-x","volume":"19","author":"Z Zeng","year":"2018","unstructured":"Zeng Z, Espino S, Roy A, Li X, Khan SA, Clare SE, et al. Using natural language processing and machine learning to identify breast cancer local recurrence. BMC Bioinformatics. 2018;19(S17):498\u2013508. https:\/\/doi.org\/10.1186\/s12859-018-2466-x.","journal-title":"BMC Bioinformatics"},{"issue":"6","key":"8526_CR28","first-page":"JS264","volume":"37","author":"LL Roos","year":"1999","unstructured":"Roos LL, Traverse D, Turner D. Delivering prevention: the role of public programs in delivering care to high-risk populations. Med Care. 1999;37(6):JS264\u2013JS78.","journal-title":"Med Care"},{"issue":"5","key":"8526_CR29","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1006\/pmed.2002.1096","volume":"35","author":"SJ McPhee","year":"2002","unstructured":"McPhee SJ, Nguyen TT, Shema SJ, Nguyen B, Somkin C, Vo P, et al. Validation of recall of breast and cervical cancer screening by women in an ethcinally diverse population. Prev Med. 2002;35(5):463\u201373. https:\/\/doi.org\/10.1006\/pmed.2002.1096.","journal-title":"Prev Med"},{"issue":"3","key":"8526_CR30","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh ML. Interrater reliability: the kappa statistic. Biochem Med. 2012;22(3):276\u201382.","journal-title":"Biochem Med"}],"container-title":["BMC Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12885-021-08526-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12885-021-08526-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12885-021-08526-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,1]],"date-time":"2021-07-01T17:05:24Z","timestamp":1625159124000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmccancer.biomedcentral.com\/articles\/10.1186\/s12885-021-08526-9"}},"subtitle":["Using algorithms to determine breast and colorectal cancer recurrence"],"short-title":[],"issued":{"date-parts":[[2021,7,1]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["8526"],"URL":"https:\/\/doi.org\/10.1186\/s12885-021-08526-9","relation":{},"ISSN":["1471-2407"],"issn-type":[{"value":"1471-2407","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,1]]},"assertion":[{"value":"22 September 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 June 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study was approved by the University of Manitoba\u2019s Health Research Ethics Board, Manitoba Health\u2019s Health Information and Privacy Committee, and CancerCare Manitoba\u2019s Research and Resource Impact Committee. Because data were de-identified, informed consent was not required.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"763"}}