{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T11:11:48Z","timestamp":1769598708445,"version":"3.49.0"},"reference-count":44,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T00:00:00Z","timestamp":1762905600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004826","name":"Natural Science Foundation of Beijing","doi-asserted-by":"publisher","award":["7222306"],"award-info":[{"award-number":["7222306"]}],"id":[{"id":"10.13039\/501100004826","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,2,1]]},"abstract":"Abstract<\/jats:title>\n \n Objective<\/jats:title>\n To develop the first comprehensive, standardized annotated corpus of Chinese online health information (OHI) using the full 16-item DISCERN instrument and to establish a reliable annotation process that supports automated quality assessment.<\/jats:p>\n <\/jats:sec>\n \n Materials and Methods<\/jats:title>\n We assembled 510 web-sourced articles on breast cancer, arthritis, and depression. All the articles were independently annotated by three trained raters using the DISCERN scale. Annotation followed a four-step workflow: data collection and preprocessing, rater training, iterative annotation, and quality control. Raters calibrated through consensus sessions and calibration articles. The Dawid\u2013Skene model aggregated individual annotations into final consensus scores. Original five-point ratings were retained and binarized (scores 1-3 as low quality, 4-5 as high quality) to enable both fine-grained and coarse evaluation for machine learning.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Initial annotation of a 60-article pilot produced low agreement (mean Krippendorff\u2019s \u03b1\u2009\u2248\u20090.022) due to subjective variability. Successive calibration exercises improved agreement markedly, culminating in a corpus-wide Krippendorff\u2019s \u03b1 of 0.834. Consensus scores correlated strongly with individual rater scores, confirming annotation robustness. The dual-scale design yielded a relatively balanced distribution of labels across topics, with roughly equal representation of low- and high-quality articles, and preserved granularity for detailed DISCERN analysis.<\/jats:p>\n <\/jats:sec>\n \n Discussion<\/jats:title>\n Our iterative calibration approach and consensus modeling effectively addressed the subjective ambiguity inherent in quality assessment. The binary and five-class labeling strategies facilitate flexible downstream applications, allowing automated systems to perform both broad filtering and nuanced quality differentiation. The high inter-rater reliability demonstrates that rigorous training and consensus methods can overcome domain-specific annotation challenges.<\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n The resulting Chinese OHI corpus, annotated via a standardized DISCERN framework and refined through iterative calibration, provides a robust benchmark for training and evaluating machine learning models. This resource lays the foundation for scalable, reliable automated quality assessment of OHI in Chinese public health settings.<\/jats:p>\n <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf175","type":"journal-article","created":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T17:18:22Z","timestamp":1762967902000},"page":"316-325","source":"Crossref","is-referenced-by-count":0,"title":["Development of a robust corpus for automated evaluation of online health information in Chinese using the DISCERN scale"],"prefix":"10.1093","volume":"33","author":[{"given":"Ting","family":"E","sequence":"first","affiliation":[{"name":"Bloomberg School of Public Health,Johns Hopkins University, MD, 21205,","place":["United States"]}]},{"given":"Xingxi","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering, Tsinghua University , Beijing, 100084,","place":["China"]}]},{"given":"Jun","family":"Liang","sequence":"additional","affiliation":[{"name":"Department of AI and IT, Second Affiliated Hospital, School of Medicine, Zhejiang University , Hangzhou, Zhejiang Province, 310000,","place":["China"]}]},{"given":"Junhao","family":"Ma","sequence":"additional","affiliation":[{"name":"School of Public Health, Hangzhou Medical College , Hangzhou, Zhejiang Province, 310053,","place":["China"]}]},{"given":"Qichuan","family":"Fang","sequence":"additional","affiliation":[{"name":"School of Medical Technology and Information Engineering, Zhejiang Chinese Medical University , Hangzhou, Zhejiang Province, 310053,","place":["China"]}]},{"given":"Shanli","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Public Health, Southwest Medical University , Luzhou, Sichuan Province, 646000,","place":["China"]}]},{"given":"Jianbo","family":"Lei","sequence":"additional","affiliation":[{"name":"Clinical Research Center, Affiliated Hospital of Southwest Medical University , Liuzhou, 646000,","place":["China"]},{"name":"School of Medical Information and Engineering, Southwest Medical University , Luzhou, 646000,","place":["China"]},{"name":"Center for Medical Informatics, Advanced Institute of Clinical Medicine, Peking University , Beijing, 100191,","place":["China"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5437-2545","authenticated-orcid":false,"given":"Christopher G","family":"Chute","sequence":"additional","affiliation":[{"name":"Bloomberg School of Public Health,Johns Hopkins University, MD, 21205,","place":["United States"]},{"name":"School of Medicine, Johns Hopkins University , Baltimore, MD, 21206,","place":["United States"]},{"name":"School of Nursing, Johns Hopkins University , Baltimore, MD, 21206,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2025,11,12]]},"reference":[{"key":"2026012716173669300_ocaf175-B1","author":"CNNIC"},{"key":"2026012716173669300_ocaf175-B2","doi-asserted-by":"crossref","first-page":"1740","DOI":"10.3390\/healthcare9121740","article-title":"Online health information seeking behavior: a systematic review","volume":"9","author":"Jia","year":"2021","journal-title":"Healthcare"},{"key":"2026012716173669300_ocaf175-B3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12905-024-03509-x","article-title":"Online health information seeking behavior among breast cancer patients and survivors: a scoping review","volume":"25","author":"Chen","year":"2025","journal-title":"BMC Womens Health"},{"key":"2026012716173669300_ocaf175-B4","doi-asserted-by":"publisher","first-page":"e40778","DOI":"10.2196\/40778","article-title":"The relation between eHealth literacy and health-related behaviors: systematic review and meta-analysis","volume":"25","author":"Kim","year":"2023","journal-title":"J Med Internet Res"},{"key":"2026012716173669300_ocaf175-B5","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1016\/j.ejogrb.2022.02.010","article-title":"Online health information on induction of labour: a systematic review and quality assessment study","volume":"271","author":"Ewington","year":"2022","journal-title":"Eur J Obstet Gynecol Reprod Biol"},{"key":"2026012716173669300_ocaf175-B6","doi-asserted-by":"publisher","first-page":"e19985","DOI":"10.2196\/19985","article-title":"Online health information seeking by parents for their children: systematic review and agenda for further research","volume":"22","author":"Kubb","year":"2020","journal-title":"J Med Internet Res"},{"key":"2026012716173669300_ocaf175-B7","doi-asserted-by":"publisher","first-page":"e22","DOI":"10.2196\/jmir.1030","article-title":"Medicine 2.0: social networking, collaboration, participation, apomediation, and openness","volume":"10","author":"Eysenbach","year":"2008","journal-title":"J Med Internet Res"},{"key":"2026012716173669300_ocaf175-B8","doi-asserted-by":"crossref","first-page":"893","DOI":"10.1177\/1461444810386774","article-title":"Internet skills and the digital divide","volume":"13","author":"Dijk","year":"2011","journal-title":"New Media Soc"},{"key":"2026012716173669300_ocaf175-B9","doi-asserted-by":"publisher","first-page":"770","DOI":"10.1016\/j.pec.2020.11.016","article-title":"The influence of online health information on health decisions: a systematic review","volume":"104","author":"Thapa","year":"2021","journal-title":"Patient Educ Couns"},{"key":"2026012716173669300_ocaf175-B10","doi-asserted-by":"publisher","first-page":"2524","DOI":"10.1080\/10410236.2023.2275921","article-title":"Is there a relationship between online health informationseeking and health anxiety? A systematic review and meta-analysis","volume":"39","author":"Wang","year":"2024","journal-title":"Health Commun"},{"key":"2026012716173669300_ocaf175-B11","doi-asserted-by":"crossref","first-page":"1163","DOI":"10.1080\/10410236.2020.1748829","article-title":"Online health information seeking: a review and meta-analysis","volume":"36","author":"Wang","year":"2020","journal-title":"Health Commun"},{"key":"2026012716173669300_ocaf175-B12","doi-asserted-by":"publisher","first-page":"2055207620948996","DOI":"10.1177\/2055207620948996","article-title":"Factors affecting the quality and reliability of online health information","volume":"6","author":"Battineni","year":"2020","journal-title":"Digit Health"},{"key":"2026012716173669300_ocaf175-B13","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.ijmedinf.2005.07.032","article-title":"Influences, usage, and outcomes of Internet health information searching: Multivariate results from the Pew surveys","volume":"75","author":"Rice","year":"2006","journal-title":"Int J Med Inform"},{"key":"2026012716173669300_ocaf175-B14","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1007\/s10389-021-01604-z","article-title":"In the digital age: a systematic literature review of the e-health literacy and influencing factors among Chinese older adults","volume":"31","author":"Shi","year":"2023","journal-title":"Z Gesundh Wiss"},{"key":"2026012716173669300_ocaf175-B15","author":"WHO","year":"2025"},{"key":"2026012716173669300_ocaf175-B16","doi-asserted-by":"publisher","first-page":"20552076231212296","DOI":"10.1177\/20552076231212296","article-title":"Evaluating online health information quality using machine learning and deep learning: a systematic literature review","volume":"9","author":"Baqraf","year":"2023","journal-title":"Digit Health"},{"key":"2026012716173669300_ocaf175-B17","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1093\/jamia\/ocw140","article-title":"Toward automated assessment of health Web page quality using the DISCERN instrument","volume":"24","author":"Allam","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2026012716173669300_ocaf175-B18","doi-asserted-by":"publisher","first-page":"104321","DOI":"10.1016\/j.ijmedinf.2020.104321","article-title":"Interventions to support consumer evaluation of online health information credibility: a scoping review","volume":"145","author":"Song","year":"2021","journal-title":"Int J Med Inform"},{"key":"2026012716173669300_ocaf175-B19","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1016\/j.ijhcs.2006.02.007","article-title":"A framework for understanding trust factors in web-based health advice","volume":"64","author":"Sillence","year":"2006","journal-title":"Int J Hum-Comput Stud"},{"key":"2026012716173669300_ocaf175-B20","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1093\/rheumatology\/keh498","article-title":"Accessibility, nature and quality of health information on the internet: a survey on osteoarthritis","volume":"44","author":"Maloney","year":"2005","journal-title":"Rheumatology (Oxford)"},{"key":"2026012716173669300_ocaf175-B21","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1136\/jech.53.2.105","article-title":"DISCERN: an instrument for judging the quality of written consumer health information on treatment choices","volume":"53","author":"Charnock","year":"1999","journal-title":"J Epidemiol Community Health"},{"key":"2026012716173669300_ocaf175-B22","year":"2025"},{"key":"2026012716173669300_ocaf175-B23","first-page":"1004","article-title":"Accessing reliable health information on the web: a review of the HON approach","volume":"245","author":"Boyer","year":"2017","journal-title":"Stud Health Technol Inform"},{"key":"2026012716173669300_ocaf175-B24","doi-asserted-by":"publisher","first-page":"e3","DOI":"10.2196\/aging","article-title":"A tool that assesses the evidence, transparency, and usability of online health information: development and reliability assessment","volume":"1","author":"Dobbins","year":"2018","journal-title":"JMIR Aging"},{"key":"2026012716173669300_ocaf175-B25","doi-asserted-by":"publisher","first-page":"1244","DOI":"10.1001\/jama.1997.03540390074039","article-title":"Assessing, controlling, and assuring the quality of medical information on the internet: caveant lector et viewor\u2014let the reader and viewer beware","volume":"277","author":"Silberg","year":"1997","journal-title":"JAMA"},{"key":"2026012716173669300_ocaf175-B26","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1016\/j.pec.2016.08.028","article-title":"Understanding online health information: Evaluation, tools, and strategies","volume":"100","author":"Beaunoyer","year":"2017","journal-title":"Patient Educ Couns"},{"key":"2026012716173669300_ocaf175-B27","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-020-01131-z","article-title":"AutoDiscern: rating the quality of online health information with hierarchical encoder attention-based neural networks","volume":"20","author":"Kinkead","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026012716173669300_ocaf175-B28","doi-asserted-by":"crossref","first-page":"1105","DOI":"10.1080\/10410236.2015.1045236","article-title":"Modeling online health information-seeking behavior in china: the roles of source characteristics, reward assessment, and internet self-efficacy","volume":"31","author":"Cao","year":"2016","journal-title":"Health Commun"},{"key":"2026012716173669300_ocaf175-B29","doi-asserted-by":"publisher","first-page":"e9","DOI":"10.2196\/jmir.5729","article-title":"Internet health information seeking and the patient-physician relationship: a systematic review","volume":"19","author":"Tan","year":"2017","journal-title":"J Med Internet Res"},{"key":"2026012716173669300_ocaf175-B30","doi-asserted-by":"publisher","first-page":"1099","DOI":"10.1089\/tmj.2018.0217","article-title":"The impact of individuals\u2019 attitudes toward health websites on their perceived quality of health information: an empirical study","volume":"25","author":"Liu","year":"2019","journal-title":"Telemed J E Health"},{"key":"2026012716173669300_ocaf175-B31","doi-asserted-by":"publisher","first-page":"e36463","DOI":"10.2196\/36463","article-title":"Consumers\u2019 evaluation of web-based health information quality: meta-analysis","volume":"24","author":"Zhang","year":"2022","journal-title":"J Med Internet Res"},{"key":"2026012716173669300_ocaf175-B32","doi-asserted-by":"publisher","first-page":"e25783","DOI":"10.2196\/25783","article-title":"Assessing the quality of online health information about breast cancer from Chinese language websites: quality assessment survey","volume":"7","author":"Sun","year":"2021","journal-title":"JMIR Cancer"},{"key":"2026012716173669300_ocaf175-B33","doi-asserted-by":"publisher","first-page":"667","DOI":"10.3233\/JAD-231339","article-title":"Evaluation of the quality and readability of online information about Alzheimer\u2019s disease in China","volume":"99","author":"Chu","year":"2024","journal-title":"J Alzheimers Dis"},{"key":"2026012716173669300_ocaf175-B34","doi-asserted-by":"crossref","first-page":"e21820","DOI":"10.2196\/21820","article-title":"How to fight an infodemic: the four pillars of infodemic management","volume":"22","author":"Eysenbach","year":"2020","journal-title":"J Med Internet Res"},{"key":"2026012716173669300_ocaf175-B35","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1016\/S0140-6736(20)30461-X","article-title":"How to fight an infodemic","volume":"395","author":"Zarocostas","year":"2020","journal-title":"Lancet"},{"key":"2026012716173669300_ocaf175-B36","doi-asserted-by":"publisher","DOI":"10.2196\/52995","article-title":"Automated credibility assessment of web-based health information considering Health on the Net Foundation Code of Conduct (HONcode): model development and validation study","volume":"7","author":"Bayani","year":"2023","journal-title":"JMIR Form Res"},{"key":"2026012716173669300_ocaf175-B37","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1080\/19312450709336664","article-title":"Answering the call for a standard reliability measure for coding data","volume":"1","author":"Hayes","year":"2007","journal-title":"Commun Methods Meas"},{"key":"2026012716173669300_ocaf175-B38","doi-asserted-by":"publisher","first-page":"20","DOI":"10.2307\/2346806","article-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm","volume":"28","author":"Dawid","year":"1979","journal-title":"J R Stat Soc Ser C (Appl Stat)"},{"key":"2026012716173669300_ocaf175-B39","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1016\/j.eswa.2016.12.035","article-title":"Learning from class-imbalanced data: review of methods and applications","volume":"73","author":"Guo","year":"2017","journal-title":"Expert Syst Appl"},{"key":"2026012716173669300_ocaf175-B40","volume-title":"Learning from Imbalanced Data Sets","author":"Alberto","year":"2018"},{"key":"2026012716173669300_ocaf175-B41","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1109\/ICACCI.2017","article-title":"Handling class imbalance problem using oversampling techniques: a review","author":"Gosain","year":"2017","journal-title":"2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI)"},{"key":"2026012716173669300_ocaf175-B42","doi-asserted-by":"publisher","first-page":"4368","DOI":"10.1109\/IJCNN.2016.7727770","article-title":"Training deep neural networks on imbalanced data sets","author":"Wang","year":"2016","journal-title":"2016 International Joint Conference on Neural Networks (IJCNN)"},{"key":"2026012716173669300_ocaf175-B43","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1016\/j.ins.2019.11.004","article-title":"Data imbalance in classification: experimental evaluation","volume":"513","author":"Fadi","year":"2020","journal-title":"Inf Sci"},{"key":"2026012716173669300_ocaf175-B44","doi-asserted-by":"publisher","first-page":"105478","DOI":"10.1016\/j.ijmedinf.2024.105478","article-title":"Have we found a solution for health misinformation? A ten-year systematic review of health misinformation literature 2013\u20132022","volume":"188","author":"Zhang","year":"2024","journal-title":"Int J Med Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/2\/316\/65277256\/ocaf175.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/2\/316\/65277256\/ocaf175.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T21:17:44Z","timestamp":1769548664000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/2\/316\/8321896"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,12]]},"references-count":44,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,11,12]]},"published-print":{"date-parts":[[2026,2,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf175","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,2]]},"published":{"date-parts":[[2025,11,12]]}}}