{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:01:46Z","timestamp":1782842506394,"version":"3.54.5"},"reference-count":37,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>In 2021, the European Union reported &gt;270,000 excess deaths, including &gt;16,000 in Portugal. The Portuguese Directorate-General of Health developed a deep neural network, AUTOCOD, which determines the primary causes of death by analyzing the free text of physicians\u2019 death certificates (DCs). Although AUTOCOD\u2019s performance has been established, it remains unclear whether its performance remains consistent over time, particularly during periods of excess mortality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to assess the sensitivity and other performance metrics of AUTOCOD in classifying underlying causes of death compared with manual coding to identify specific causes of death during periods of excess mortality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We included all DCs between 2016 and 2019. AUTOCOD\u2019s performance was evaluated by calculating various performance metrics, such as sensitivity, specificity, positive predictive value (PPV), and F1-score, using a confusion matrix. This compared International Statistical Classification of Diseases and Health-Related Problems, 10th Revision (ICD-10), classifications of DCs by AUTOCOD with those by human coders at the Directorate-General of Health (gold standard). Subsequently, we compared periods without excess mortality with periods of excess, severe, and extreme excess mortality. We defined excess mortality as 2 consecutive days with a Z score above the 95% baseline limit, severe excess mortality as 2 consecutive days with a Z score &gt;4 SDs, and extreme excess mortality as 2 consecutive days with a Z score &gt;6 SDs. Finally, we repeated the analyses for the 3 most common ICD-10 chapters focusing on block-level classification.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We analyzed a large data set comprising 330,098 DCs classified by both human coders and AUTOCOD. AUTOCOD demonstrated high sensitivity (\u22650.75) for 10 ICD-10 chapters examined, with values surpassing 0.90 for the more prevalent chapters (chapter II\u2014\u201cNeoplasms,\u201d chapter IX\u2014\u201cDiseases of the circulatory system,\u201d and chapter X\u2014\u201cDiseases of the respiratory system\u201d), accounting for 67.69% (223,459\/330,098) of all human-coded causes of death. No substantial differences were observed in these high-sensitivity values when comparing periods without excess mortality with periods of excess, severe, and extreme excess mortality. The same holds for specificity, which exceeded 0.96 for all chapters examined, and for PPV, which surpassed 0.75 in 9 chapters, including the more prevalent ones. When considering block classification within the 3 most common ICD-10 chapters, AUTOCOD maintained a high performance, demonstrating high sensitivity (\u22650.75) for 13 ICD-10 blocks, high PPV for 9 blocks, and specificity of &gt;0.98 in all blocks, with no significant differences between periods without excess mortality and those with excess mortality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Our findings indicate that, during periods of excess and extreme excess mortality, AUTOCOD\u2019s performance remains unaffected by potential text quality degradation because of pressure on health services. Consequently, AUTOCOD can be dependably used for real-time cause-specific mortality surveillance even in extreme excess mortality situations.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/40965","type":"journal-article","created":{"date-parts":[[2023,6,2]],"date-time":"2023-06-02T07:37:50Z","timestamp":1685691470000},"page":"e40965","source":"Crossref","is-referenced-by-count":3,"title":["Real-Time Classification of Causes of Death Using AI: Sensitivity Analysis"],"prefix":"10.2196","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4460-4549","authenticated-orcid":false,"given":"Patr\u00edcia","family":"Pita Ferreira","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0818-3644","authenticated-orcid":false,"given":"Diogo","family":"Godinho Sim\u00f5es","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7653-9887","authenticated-orcid":false,"given":"Constan\u00e7a","family":"Pinto de Carvalho","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1800-9942","authenticated-orcid":false,"given":"Francisco","family":"Duarte","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4547-6358","authenticated-orcid":false,"given":"Eug\u00e9nia","family":"Fernandes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9772-806X","authenticated-orcid":false,"given":"Pedro","family":"Casaca Carvalho","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1168-8827","authenticated-orcid":false,"given":"Jos\u00e9 Francisco","family":"Loff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9890-0924","authenticated-orcid":false,"given":"Ana Paula","family":"Soares","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9646-4988","authenticated-orcid":false,"given":"Maria Jo\u00e3o","family":"Albuquerque","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5939-2577","authenticated-orcid":false,"given":"Pedro","family":"Pinto-Leite","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1390-2076","authenticated-orcid":false,"given":"Andr\u00e9","family":"Peralta-Santos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2023,11,22]]},"reference":[{"key":"ref1","unstructured":"Graphs and mapsEuroMOMO2023-04-13https:\/\/www.euromomo.eu\/graphs-and-maps\/"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.2807\/1560-7917.ES.2020.25.26.2001214"},{"issue":"2","key":"ref3","first-page":"1","volume":"22","author":"Pinto, CS","year":"2016","journal-title":"Eurohealth (Lond)"},{"key":"ref4","unstructured":"SICO - eVMVigil\u00e2ncia de Mortalidade2021-11-08https:\/\/evm.min-saude.pt\/#shiny-tab-info_eVM"},{"key":"ref5","unstructured":"International Statistical Classification of Diseases and Health-Related Problems, 10th Revision, 5th Edition, 2016World Health Organization20152021-11-09https:\/\/apps.who.int\/iris\/handle\/10665\/246208"},{"issue":"1","key":"ref6","first-page":"E070104.1","volume":"12","author":"Mazick, A","year":"2007","journal-title":"Euro Surveill"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.20344\/amp.13928"},{"issue":"13","key":"ref8","doi-asserted-by":"crossref","first-page":"19529","DOI":"10.2807\/ese.15.13.19529-en","volume":"15","author":"Nogueira, PJ","year":"2010","journal-title":"Euro Surveill"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.2807\/ese.15.43.19697-en"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1017\/S0950268811000410"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1097\/00001648-199707000-00007"},{"key":"ref12","unstructured":"DuarteFRAutomated classification of causes of mortality [Thesis]Instituto Superior T\u00e9cnico201792021-10-26http:\/\/\/C:\/Users\/HP\/Downloads\/_Thesis___Automated_Classification_of_Causes_of_Mortality%20(1).pdf"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2018.02.011"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.2196\/17125"},{"key":"ref15","unstructured":"Impact of the Implementation of IRIS software for ICD-10 cause of death coding on mortality statistics, England and WalesOffice of National Statistics2014882021-11-16https:\/\/tinyurl.com\/42d6hs9y"},{"key":"ref16","unstructured":"GergonneBMazickAO\u2019DonnellJOzaACoxBWuillaumeFKaufmanZVirtanenMGreenHHardelidPAndrewsNPebodyRHolmbergMDetsisMDanisCUphoffHJosseranLFouilletANunesBNogueiraPJunkerCvan AstenLvan KloosterTSimonFFloresVMTomsicSSpiteriGNielsenJM\u00f8lbakKA European algorithm for a common monitoring of mortality across EuropeEuroMOMO2021-10-11https:\/\/www.euromomo.eu\/uploads\/pdf\/wp7_report.pdf"},{"key":"ref17","unstructured":"\"Westgard rules\" and multirulesWestgard QC2022-03-21https:\/\/www.westgard.com\/mltirule.htm"},{"key":"ref18","unstructured":"KuhnMWingJWestonSWilliamsAKeeferCEngelhardtACooperTMayerZKenkelBBenestyMLescarbeauRZiemAScruccaLTangYCandanCHuntTcaret: classification and regression trainingThe Comprehensive R Archive Network2022-03-20https:\/\/CRAN.R-project.org\/package=caret"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v028.i05"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1016\/j.advengsoft.2011.12.014"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.21105\/joss.00765"},{"key":"ref22","unstructured":"R Core TeamR: a language and environment for statistical computingR Foundation for Statistical Computing20172022-04-05https:\/\/www.R-project.org\/"},{"key":"ref23","unstructured":"WickhamHChangWHenryLPedersenTLTakahashiKWilkeCWooKYutaniHDunningtonDggplot2: create elegant data visualisations using the grammar of graphicsThe Comprehensive R Archive Network2022-04-05https:\/\/CRAN.R-project.org\/package=ggplot2"},{"key":"ref24","unstructured":"WickhamHVaughanDGirlichMUsheyKtidyr: tidy messy dataThe Comprehensive R Archive Network2022-04-05https:\/\/CRAN.R-project.org\/package=tidyr"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.21105\/joss.01686"},{"key":"ref26","unstructured":"Passo 1: check-list de auto-avalia\u00e7\u00e3o \u00e9ticaEscola Nacional de Sa\u00fade P\u00fablica20212023-04-21https:\/\/www.ensp.unl.pt\/wp-content\/uploads\/2021\/08\/passo1-doc1-check-list-de-auto-avaliacao-etica-final-2.pdf"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.2196\/26353"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1016\/j.imu.2020.100456"},{"key":"ref29","unstructured":"SonntagDAssessing the quality of natural language text dataDaimlerChrysler Research and Technology2021-11-16https:\/\/www.dfki.de\/~sonntag\/text_quality_short.pdf"},{"key":"ref30","unstructured":"LouisAPredicting text quality: metrics for content, organization and reader interestUniversity of Pennsylvania20132022-04-18https:\/\/repository.upenn.edu\/edissertations\/665"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2016.06.006"},{"key":"ref32"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1109\/jbhi.2022.3163013"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1007\/s10654-010-9445-5"},{"key":"ref35","author":"Schwaighofer, A","year":"2008","journal-title":"Dataset Shift in Machine Learning"},{"key":"ref36","unstructured":"Lei n.\u00ba 15\/2012Di\u00e1rio da Rep\u00fablica2021-10-26https:\/\/dre.pt\/web\/guest\/pesquisa\/-\/search\/554389\/details\/maximized"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1016\/s2589-7500(22)00004-8"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2023\/1\/e40965","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:40:08Z","timestamp":1782841208000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2023\/1\/e40965"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,22]]},"references-count":37,"URL":"https:\/\/doi.org\/10.2196\/40965","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.40965","asserted-by":"object"}]},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,22]]}}}