{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T10:48:18Z","timestamp":1777027698700,"version":"3.51.4"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,6,9]],"date-time":"2020-06-09T00:00:00Z","timestamp":1591660800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,6,9]],"date-time":"2020-06-09T00:00:00Z","timestamp":1591660800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Patients increasingly turn to search engines and online content before, or in place of, talking with a health professional. Low quality health information, which is common on the internet, presents risks to the patient in the form of misinformation and a possibly poorer relationship with their physician. To address this, the DISCERN criteria (developed at University of Oxford) are used to evaluate the quality of online health information. However, patients are unlikely to take the time to apply these criteria to the health websites they visit.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>We built an automated implementation of the DISCERN instrument (Brief version) using machine learning models. We compared the performance of a traditional model (Random Forest) with that of a hierarchical encoder attention-based neural network (HEA) model using two language embeddings, BERT and BioBERT.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>The HEA BERT and BioBERT models achieved average F1-macro scores across all criteria of 0.75 and 0.74, respectively, outperforming the Random Forest model (average F1-macro = 0.69). Overall, the neural network based models achieved 81% and 86% average accuracy at 100% and 80% coverage, respectively, compared to 94% manual rating accuracy. The attention mechanism implemented in the HEA architectures not only provided \u2019model explainability\u2019 by identifying reasonable supporting sentences for the documents fulfilling the Brief DISCERN criteria, but also boosted F1 performance by 0.05 compared to the same architecture without an attention mechanism.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>Our research suggests that it is feasible to automate online health information quality assessment, which is an important step towards empowering patients to become informed partners in the healthcare process.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s12911-020-01131-z","type":"journal-article","created":{"date-parts":[[2020,6,9]],"date-time":"2020-06-09T12:03:02Z","timestamp":1591704182000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks"],"prefix":"10.1186","volume":"20","author":[{"given":"Laura","family":"Kinkead","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0871-1977","authenticated-orcid":false,"given":"Ahmed","family":"Allam","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Krauthammer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,6,9]]},"reference":[{"issue":"22","key":"1131_CR1","doi-asserted-by":"publisher","first-page":"2618","DOI":"10.1001\/archinte.165.22.2618","volume":"165","author":"BW Hesse","year":"2005","unstructured":"Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, Viswanath K. Trust and Sources of Health Information. Arch Intern Med. 2005; 165(22):2618. https:\/\/doi.org\/10.1001\/archinte.165.22.2618.","journal-title":"Arch Intern Med"},{"issue":"1","key":"1131_CR2","doi-asserted-by":"publisher","first-page":"24","DOI":"10.4066\/AMJ.2014.1900","volume":"7","author":"E Fahy","year":"2014","unstructured":"Fahy E. Quality of patient health information on the internet: reviewing a complex and. Australas Med J. 2014; 7(1):24\u20138. https:\/\/doi.org\/10.4066\/AMJ.2014.1900.","journal-title":"Australas Med J"},{"key":"1131_CR3","doi-asserted-by":"publisher","unstructured":"Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results. J Assoc Inf Sci Technol. 2015; 66(10). https:\/\/doi.org\/10.1002\/asi.23311.","DOI":"10.1002\/asi.23311"},{"key":"1131_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1200\/CCI.18.00010","volume":"2","author":"CH Saunders","year":"2018","unstructured":"Saunders CH, Petersen CL, Durang M-A, Bagley PJ, Elywn G. Bring on the Machines: Could Machine Learning Improve the Quality of Patient Education Materials? A Systematic Search and Rapid Review. JCO Clin Cancer Inform. 2018; 2:1\u201316. https:\/\/doi.org\/10.1200\/cci.18.00010.","journal-title":"JCO Clin Cancer Inform"},{"issue":"3","key":"1131_CR5","doi-asserted-by":"publisher","first-page":"17","DOI":"10.2196\/jmir.5.3.e17","volume":"5","author":"E Murray","year":"2003","unstructured":"Murray E, Lo B, Pollack L, Donelan K, Catania J, Lee K, Zapert K, Turner R. The Impact of Health Information on the Internet on Health Care and the Physician-Patient Relationship: National U.S. Survey among 1.050 U.S. Physicians. J Med Internet Res. 2003; 5(3):17. https:\/\/doi.org\/10.2196\/jmir.5.3.e17.","journal-title":"J Med Internet Res"},{"issue":"4","key":"1131_CR6","doi-asserted-by":"publisher","first-page":"100","DOI":"10.2196\/jmir.2642","volume":"16","author":"A Allam","year":"2014","unstructured":"Allam A, Schulz PJ, Nakamoto K. The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: two experiments manipulating Google output,. J Med Internet Res. 2014; 16(4):100. https:\/\/doi.org\/10.2196\/jmir.2642.","journal-title":"J Med Internet Res"},{"issue":"6","key":"1131_CR7","doi-asserted-by":"publisher","first-page":"137","DOI":"10.2196\/jmir.5430","volume":"18","author":"R Ludolph","year":"2016","unstructured":"Ludolph R, Allam A, Schulz PJ. Manipulating Google\u2019s Knowledge Graph Box to Counter Biased Information Processing During an Online Search on Vaccination: Application of a Technological Debiasing Strategy. J Med Internet Res. 2016; 18(6):137. https:\/\/doi.org\/10.2196\/jmir.5430.","journal-title":"J Med Internet Res"},{"issue":"12","key":"1131_CR8","first-page":"699","volume":"108","author":"SA Iverson","year":"2008","unstructured":"Iverson SA, Howard KB, Penney BK. Impact of internet use on health-related behaviors and the patient-physician relationship: a survey-based study and review. J Am Osteopath Assoc. 2008; 108(12):699\u2013711.","journal-title":"J Am Osteopath Assoc"},{"issue":"3","key":"1131_CR9","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1016\/j.pec.2007.05.016","volume":"68","author":"HS Wald","year":"2007","unstructured":"Wald HS, Dube CE, Anthony DC. Untangling the Web-The impact of Internet use on health care and the physician-patient relationship. Patient Educ Couns. 2007; 68(3):218\u201324. https:\/\/doi.org\/10.1016\/j.pec.2007.05.016.","journal-title":"Patient Educ Couns"},{"key":"1131_CR10","doi-asserted-by":"publisher","unstructured":"Risk A, Dzenowagis J. Review of Internet health information quality initiatives. J Med Internet Res. 2001. https:\/\/doi.org\/10.2196\/jmir.3.4.e28.","DOI":"10.2196\/jmir.3.4.e28"},{"issue":"5","key":"1131_CR11","doi-asserted-by":"publisher","first-page":"1209","DOI":"10.1002\/widm.1209","volume":"7","author":"M Viviani","year":"2017","unstructured":"Viviani M, Pasi G. Credibility in social media: opinions, news, and health information-a survey. Wiley Interdiscip Rev Data Min Knowl Disc. 2017; 7(5):1209. https:\/\/doi.org\/10.1002\/widm.1209.","journal-title":"Wiley Interdiscip Rev Data Min Knowl Disc"},{"issue":"2","key":"1131_CR12","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1136\/jech.53.2.105","volume":"53","author":"D Charnock","year":"1999","unstructured":"Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices,. J Epidemiol Community Health. 1999; 53(2):105\u201311. https:\/\/doi.org\/10.1136\/jech.53.2.105.","journal-title":"J Epidemiol Community Health"},{"key":"1131_CR13","doi-asserted-by":"publisher","unstructured":"Boyer C, Dolamic L. Feasibility of automated detection of HONcode conformity for health-related websites. Int J Adv Comput Sci Appl. 2014; 5(3). https:\/\/doi.org\/10.14569\/IJACSA.2014.050309.","DOI":"10.14569\/IJACSA.2014.050309"},{"issue":"6","key":"1131_CR14","doi-asserted-by":"publisher","first-page":"135","DOI":"10.2196\/jmir.3831","volume":"17","author":"C Boyer","year":"2015","unstructured":"Boyer C, Dolamic L. Automated Detection of HONcode Website Conformity Compared to Manual Detection: An Evaluation,. J Med Internet Res. 2015; 17(6):135. https:\/\/doi.org\/10.2196\/jmir.3831.","journal-title":"J Med Internet Res"},{"key":"1131_CR15","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I. Attention Is All You Need. 2017. http:\/\/arxiv.org\/abs\/1706.03762. Accessed 14 Oct 2019."},{"key":"1131_CR16","doi-asserted-by":"crossref","unstructured":"Luong M-T, Pham H, Manning CD. Effective Approaches to Attention-based Neural Machine Translation. 2015. http:\/\/arxiv.org\/abs\/1508.04025. Accessed 15 Oct 2019.","DOI":"10.18653\/v1\/D15-1166"},{"key":"1131_CR17","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Brew J. Transformers: State-of-the-art Natural Language Processing. 2019. http:\/\/arxiv.org\/abs\/1910.03771.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"1131_CR18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-5004","volume-title":"Proceedings of the 2019 Conference of the North","author":"S Ruder","year":"2019","unstructured":"Ruder S, Peters ME, Swayamdipta S, Wolf T. Transfer Learning in Natural Language Processing. In: Proceedings of the 2019 Conference of the North. Stroudsburg: Association for Computational Linguistics: 2019. p. 15\u201318. https:\/\/doi.org\/10.18653\/v1\/N19-5004."},{"key":"1131_CR19","doi-asserted-by":"publisher","unstructured":"Rees CE, Ford JE, Sheard CE. Evaluating the reliability of DISCERN: A tool for assessing the quality of written patient information on treatment choices. Patient Educ Couns. 2002. https:\/\/doi.org\/10.1016\/S0738-3991(01)00225-7.","DOI":"10.1016\/S0738-3991(01)00225-7"},{"key":"1131_CR20","doi-asserted-by":"publisher","unstructured":"Khazaal Y, Chatton A, Cochand S, Coquard O, Fernandez S, Khan R, Billieux J, Zullino D. Brief DISCERN, six questions for the evaluation of evidence-based content of health-related websites. Patient Educ Couns. 2009. https:\/\/doi.org\/10.1016\/j.pec.2009.02.016.","DOI":"10.1016\/j.pec.2009.02.016"},{"key":"1131_CR21","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018. http:\/\/arxiv.org\/abs\/1810.04805."},{"key":"1131_CR22","doi-asserted-by":"publisher","unstructured":"Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, Kang J. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2019. https:\/\/doi.org\/10.1093\/bioinformatics\/btz682.","DOI":"10.1093\/bioinformatics\/btz682"},{"issue":"3","key":"1131_CR23","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1093\/jamia\/ocw140","volume":"24","author":"A Allam","year":"2017","unstructured":"Allam A, Schulz PJ, Krauthammer M. Toward automated assessment of health Web page quality using the DISCERN instrument. J Am Med Informa Assoc. 2017; 24(3):481\u201387. https:\/\/doi.org\/10.1093\/jamia\/ocw140.","journal-title":"J Am Med Informa Assoc"},{"key":"1131_CR24","unstructured":"Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S. PyTorch: An Imperative Style, High-Performance Deep Learning Library In: Wallach H, Larochelle H, Beygelzimer A, Alch\u00e9-Buc F, Fox E, Garnett R, editors. Advances in Neural Information Processing Systems 32. Curran Associates, Inc.: 2019. p. 8024\u20138035. http:\/\/papers.neurips.cc\/paper\/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf."},{"key":"1131_CR25","doi-asserted-by":"publisher","unstructured":"Hochreiter S, Schmidhuber J. Long Short-Term Memory. Neural Comput. 1997; 9(8):1735\u201380. https:\/\/doi.org\/10.1162\/neco.1997.9.8.1735.","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"1131_CR26","doi-asserted-by":"publisher","unstructured":"Bengio Y, Simard P, Frasconi P. Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw. 1994; 5(2):157\u201366. https:\/\/doi.org\/10.1109\/72.279181.","DOI":"10.1109\/72.279181"},{"key":"1131_CR27","unstructured":"Graves A. Supervised Sequence Labelling with Recurrent Neural Networks. Berlin, Heidelberg: Springer; 2012. https:\/\/doi.org\/10.1007\/978-3-642-24797-2. http:\/\/link.springer.com\/10.1007\/978-3-642-24797-2."},{"key":"1131_CR28","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"K Cho","year":"2014","unstructured":"Cho K, Van Merri\u00ebnboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha: Association for Computational Linguistics: 2014. p. 1724\u20131734. http:\/\/aclweb.org\/anthology\/D14-1179. Accessed 01 Nov 2019."},{"key":"1131_CR29","unstructured":"Chung J, Gulcehre C, Cho K, Bengio Y. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. 2014. http:\/\/arxiv.org\/abs\/1412.3555. Accessed 01 Nov 2018."},{"key":"1131_CR30","unstructured":"Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate. 2014. http:\/\/arxiv.org\/abs\/1409.0473. Accessed 18 Dec 2019."},{"key":"1131_CR31","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J Mach Learn Res. 2014; 15:1929\u201358.","journal-title":"J Mach Learn Res"},{"key":"1131_CR32","doi-asserted-by":"publisher","unstructured":"Bergstra JAMESBERGSTRA J, Yoshua Bengio YOSHUABENGIO U. Random Search for HyperParameter Optimization. J Mach Learn Res. 2012. https:\/\/doi.org\/10.1162\/153244303322533223.","DOI":"10.1162\/153244303322533223"},{"key":"1131_CR33","doi-asserted-by":"publisher","unstructured":"Demner-Fushman D, Rogers WJ, Aronson AR. MetaMap Lite: an evaluation of a new Java implementation of MetaMap. J Am Med Inform Assoc. 2017; 177. https:\/\/doi.org\/10.1093\/jamia\/ocw177.","DOI":"10.1093\/jamia\/ocw177"},{"key":"1131_CR34","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E. Scikit-learn: Machine learning in Python. J Mach Learn Res. 2011; 12:2825\u201330.","journal-title":"J Mach Learn Res"},{"key":"1131_CR35","unstructured":"Berthelot D, Carlini N, Goodfellow I, Papernot N, Oliver A, Raffel C. MixMatch: A Holistic Approach to Semi-Supervised Learning. 2019. http:\/\/arxiv.org\/abs\/1905.02249. Accessed 18 Dec 2019."},{"key":"1131_CR36","unstructured":"Xie Q, Dai Z, Hovy E, Luong M-T, Le QV. Unsupervised Data Augmentation for Consistency Training. 2019. http:\/\/arxiv.org\/abs\/1904.12848."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01131-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-020-01131-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-01131-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,8]],"date-time":"2021-06-08T23:22:19Z","timestamp":1623194539000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-01131-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,9]]},"references-count":36,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1131"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-01131-z","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,9]]},"assertion":[{"value":"30 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 May 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 June 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not Applicable","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not Applicable \u2026","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"104"}}