{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T10:48:19Z","timestamp":1777027699747,"version":"3.51.4"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"3","funder":[{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation","doi-asserted-by":"publisher","award":["P2TIP1-161635"],"award-info":[{"award-number":["P2TIP1-161635"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,5,1]]},"abstract":"<jats:p>Background: As the Internet becomes the number one destination for obtaining health-related information, there is an increasing need to identify health Web pages that convey an accurate and current view of medical knowledge. In response, the research community has created multicriteria instruments for reliably assessing online medical information quality. One such instrument is DISCERN, which measures health Web page quality by assessing an array of features. In order to scale up use of the instrument, there is interest in automating the quality evaluation process by building machine learning (ML)-based DISCERN Web page classifiers.<\/jats:p>\n               <jats:p>Objective: The paper addresses 2 key issues that are essential before constructing automated DISCERN classifiers: (1) generation of a robust DISCERN training corpus useful for training classification algorithms, and (2) assessment of the usefulness of the current DISCERN scoring schema as a metric for evaluating the performance of these algorithms.<\/jats:p>\n               <jats:p>Methods: Using DISCERN, 272 Web pages discussing treatment options in breast cancer, arthritis, and depression were evaluated and rated by trained coders. First, different consensus models were compared to obtain a robust aggregated rating among the coders, suitable for a DISCERN ML training corpus. Second, a new DISCERN scoring criterion was proposed (features-based score) as an ML performance metric that is more reflective of the score distribution across different DISCERN quality criteria.<\/jats:p>\n               <jats:p>Results: First, we found that a probabilistic consensus model applied to the DISCERN instrument was robust against noise (random ratings) and superior to other approaches for building a training corpus. Second, we found that the established DISCERN scoring schema (overall score) is ill-suited to measure ML performance for automated classifiers.<\/jats:p>\n               <jats:p>Conclusion: Use of a probabilistic consensus model is advantageous for building a training corpus for the DISCERN instrument, and use of a features-based score is an appropriate ML metric for automated DISCERN classifiers.<\/jats:p>\n               <jats:p>Availability: The code for the probabilistic consensus model is available at https:\/\/bitbucket.org\/A_2\/em_dawid\/.<\/jats:p>","DOI":"10.1093\/jamia\/ocw140","type":"journal-article","created":{"date-parts":[[2016,10,6]],"date-time":"2016-10-06T01:49:54Z","timestamp":1475718594000},"page":"481-487","source":"Crossref","is-referenced-by-count":37,"title":["Toward automated assessment of health Web page quality using the DISCERN instrument"],"prefix":"10.1093","volume":"24","author":[{"given":"Ahmed","family":"Allam","sequence":"first","affiliation":[{"name":"Department of Pathology, Yale University School of Medicine, New Haven, CT, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peter J","family":"Schulz","sequence":"additional","affiliation":[{"name":"Institute of Communication and Health, Faculty of Communication Sciences, University of Lugano (Universit\u00e0 della Svizzera Italiana), Lugano, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Krauthammer","sequence":"additional","affiliation":[{"name":"Program for Computational Biology and Bioinformatics, Department of Pathology, Yale University School of Medicine, New Haven, CT, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2016,10,5]]},"reference":[{"key":"2020110612441018200_ocw140-B1","volume-title":"Health Online","author":"Fox","year":"2013"},{"key":"2020110612441018200_ocw140-B2","author":"Dutton"},{"key":"2020110612441018200_ocw140-B3","doi-asserted-by":"crossref","first-page":"2618","DOI":"10.1001\/archinte.165.22.2618","article-title":"Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey","volume":"165","author":"Hesse","journal-title":"Arch Intern Med"},{"key":"2020110612441018200_ocw140-B4","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1056\/NEJMc0909595","article-title":"Surveys of physicians and electronic health information","volume":"362","author":"Hesse","year":"2010","journal-title":"N Engl J Med"},{"key":"2020110612441018200_ocw140-B5","article-title":"Literature review on health information\u2013seeking behaviour on the web: a health consumer and health professional perspective","author":"Higgins","year":"2011"},{"key":"2020110612441018200_ocw140-B6","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1186\/1471-2458-7-53","article-title":"European citizens\u2019 use of E-health services: a study of seven countries","volume":"7","author":"Andreassen","year":"2007","journal-title":"BMC Public Health"},{"key":"2020110612441018200_ocw140-B7","doi-asserted-by":"crossref","first-page":"e17","DOI":"10.2196\/jmir.5.3.e17","article-title":"The impact of health information on the Internet on health care and the physician-patient relationship: national U.S","volume":"5","author":"Murray","year":"2003","journal-title":"survey among 1.050 U.S. physicians. J Med Internet Res"},{"key":"2020110612441018200_ocw140-B8","doi-asserted-by":"crossref","first-page":"1781","DOI":"10.1002\/asi.21115","article-title":"A model for online consumer health information quality","volume":"60","author":"Stvilia","year":"2009","journal-title":"J Am Soc Inf Sci Technol"},{"key":"2020110612441018200_ocw140-B9","doi-asserted-by":"crossref","first-page":"1150","DOI":"10.1136\/bmj.328.7449.1150","article-title":"Evaluation of ehealth systems and services","volume":"328","author":"Gustafson","year":"2004","journal-title":"BMJ"},{"issue":"20","key":"2020110612441018200_ocw140-B10","doi-asserted-by":"crossref","first-page":"2612","DOI":"10.1001\/jama.285.20.2612","article-title":"Health Information on the Internet: Accessibility, Quality, and Readability in English and Spanish","volume":"285","author":"Berland","year":"2001","journal-title":"J JAMA"},{"key":"2020110612441018200_ocw140-B11","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1136\/bmj.324.7337.581","article-title":"Accuracy of information on apparently credible websites: survey of five common health topics","volume":"324","author":"Kunst","year":"2002","journal-title":"BMJ"},{"key":"2020110612441018200_ocw140-B12","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/j.pec.2007.05.016","article-title":"Untangling the Web: the impact of Internet use on health care and the physician-patient relationship","volume":"68","author":"Wald","year":"2007","journal-title":"Patient Educ Couns"},{"key":"2020110612441018200_ocw140-B13","first-page":"699","article-title":"Impact of internet use on health-related behaviors and the patient-physician relationship: a survey-based study and review","volume":"108","author":"Iverson","year":"2008","journal-title":"J Am Osteopath Assoc"},{"issue":"1","key":"2020110612441018200_ocw140-B14","first-page":"e5","article-title":"Survey of Doctors\u2019 Experience of Patients Using the Internet","volume":"4","author":"Potts","year":"2002","journal-title":"J Med Int Res"},{"key":"2020110612441018200_ocw140-B15","first-page":"13","article-title":"Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform","volume":"74","author":"Bernstam","journal-title":"2005"},{"key":"2020110612441018200_ocw140-B16","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1136\/bmj.324.7337.569","article-title":"Examination of instruments used to rate quality of health information on the internet: chronicle of a voyage with an unclear destination","volume":"324","author":"Gagliardi","year":"2002","journal-title":"BMJ"},{"key":"2020110612441018200_ocw140-B17","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1145\/505248.506007","article-title":"Information quality benchmarks","volume":"45","author":"Kahn","year":"2002","journal-title":"Commun ACM"},{"key":"2020110612441018200_ocw140-B18","first-page":"76","article-title":"A review of information quality research\u2014develop a research agenda","author":"Ge","year":"2007","journal-title":"Int Conf Inf Qual"},{"key":"2020110612441018200_ocw140-B19","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1136\/jech.53.2.105","article-title":"DISCERN: an instrument for judging the quality of written consumer health information on treatment choices","volume":"53","author":"Charnock","year":"1999","journal-title":"J Epidemiol Community Health"},{"key":"2020110612441018200_ocw140-B20","doi-asserted-by":"crossref","first-page":"e59","DOI":"10.2196\/jmir.7.5.e59","article-title":"Automated assessment of the quality of depression websites","volume":"7","author":"Griffiths","year":"2005","journal-title":"J Med Internet Res"},{"key":"2020110612441018200_ocw140-B21","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/S0738-3991(01)00225-7","article-title":"Evaluating the reliability of DISCERN: a tool for assessing the quality of written patient information on treatment choices","volume":"47","author":"Rees","year":"2002","journal-title":"Patient Educ Couns"},{"key":"2020110612441018200_ocw140-B22","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/j.pec.2009.02.016","article-title":"Brief DISCERN, six questions for the evaluation of evidence-based content of health-related websites","volume":"77","author":"Khazaal","year":"2009","journal-title":"Patient Educ Couns"},{"key":"2020110612441018200_ocw140-B23","doi-asserted-by":"crossref","first-page":"2691","DOI":"10.1001\/jama.287.20.2691","article-title":"Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review","volume":"287","author":"Eysenbach","year":"2002","journal-title":"JAMA"},{"key":"2020110612441018200_ocw140-B24","doi-asserted-by":"crossref","first-page":"159","DOI":"10.28945\/493","article-title":"Developing a framework for assessing information quality on the World Wide Web","volume":"8","author":"Knight","year":"2005","journal-title":"Informing Sci"},{"key":"2020110612441018200_ocw140-B25","doi-asserted-by":"crossref","first-page":"2071","DOI":"10.1002\/asi.23311","article-title":"Quality of health information for consumers on the web: a systematic review of indicators, criteria, tools, and evaluation results","volume":"66","author":"Zhang","year":"2015","journal-title":"J Assoc Inf Sci Technol"},{"key":"2020110612441018200_ocw140-B26"},{"key":"2020110612441018200_ocw140-B27"},{"key":"2020110612441018200_ocw140-B28","first-page":"20","article-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm","volume":"28","author":"Dawid","year":"1979","journal-title":"J R Stat Soc Ser C Appl Stat"},{"key":"2020110612441018200_ocw140-B29","volume-title":"Inferring Ground Truth from Subjective Labelling of Venus Images","author":"Smyth","year":"1995"},{"key":"2020110612441018200_ocw140-B30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1553374.1553488","article-title":"Supervised learning from multiple experts","volume-title":"Proc 26th Annu Int Conf Mach Learn - ICML \u201909","author":"Raykar","year":"2009"},{"key":"2020110612441018200_ocw140-B31","first-page":"1297","article-title":"Learning from crowds","volume":"11","author":"Raykar","year":"2010","journal-title":"J Mach Learn Res"},{"key":"2020110612441018200_ocw140-B32","first-page":"2013","article-title":"Learning whom to trust with MACE","volume-title":"NAACL-HLT \u201913","author":"Hovy"},{"key":"2020110612441018200_ocw140-B33"},{"key":"2020110612441018200_ocw140-B34","volume-title":"R: a Language and Environment for Statistical Computing, 3.2.1","author":"R Core Development Team."},{"key":"2020110612441018200_ocw140-B35","first-page":"25","article-title":"Online crowdsourcing: rating annotators and obtaining cost-effective labels","volume-title":"2010 IEEE Comput Soc Conf Comput Vis Pattern Recognit \u2013 Work","author":"Welinder","year":"2010"},{"key":"2020110612441018200_ocw140-B36","first-page":"1","article-title":"Whose vote should count more: optimal integration of labels from labelers of unknown expertise","volume":"22","author":"Whitehill","year":"2009","journal-title":"Adv Neural Inf Process Syst"},{"key":"2020110612441018200_ocw140-B37","article-title":"Inferring truth from multiple annotators for social interaction analysis","volume":"4","author":"Chittaranjan","year":"2011","journal-title":"Neural Information Processing Systems (NIPS) Workshop on Modeling Human Communication Dynamics (HCD)"},{"key":"2020110612441018200_ocw140-B38","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1016\/j.jbi.2013.08.007","article-title":"Learning classification models from multiple experts","volume":"46","author":"Valizadegan","year":"2013","journal-title":"J Biomed Inform"},{"key":"2020110612441018200_ocw140-B39","first-page":"254","article-title":"Cheap and fast\u2014but is it good?: evaluating non-expert annotations for natural language tasks","author":"Snow","year":"2008","journal-title":"Proc Conf Empir Methods Nat Lang Process."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/3\/481\/34148839\/ocw140.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/3\/481\/34148839\/ocw140.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T18:17:46Z","timestamp":1604686666000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/24\/3\/481\/2907912"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,10,5]]},"references-count":39,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2016,10,5]]},"published-print":{"date-parts":[[2017,5,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocw140","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,5]]},"published":{"date-parts":[[2016,10,5]]}}}