{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T16:46:20Z","timestamp":1792082780854,"version":"4.4.0"},"reference-count":62,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Social media offers valuable information for public health surveillance, but detecting valid symptom expressions in noisy and informal text remains challenging. Fine-tuned language models can achieve strong test accuracy while relying on spurious patterns, limiting their reliability in real-world applications.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to develop a more robust symptom detection model by incorporating surveillance-relevant attributes into the training objective and introducing a counterfactual-based evaluation framework to assess attribute sensitivity.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We proposed a concept-aware learning framework (CALF), which combines multiattribute counterfactual data augmentation with multitask learning. We compared standard fine-tuning (baseline) with data-level interventions, a model-level intervention, hybrid strategies, and prompt-based methods across multiple encoders and LLMs for data augmentation. Performance was evaluated on the following 3 datasets: the MedWeb test using macro F1-score (m-F1), the MedWeb checklist using flip accuracy (FA), and a supplementary test set using F1-score for flu-or-fever detection (f-F1). FA measures sensitivity to surveillance-relevant attributes, and the supplementary test score serves as an indicator for out-of-distribution performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The baseline achieved an m-F1-score of 0.871 on the test set but a low overall FA on the checklist (FA=0.342). It also has limited sensitivity to recovery status, timing of occurrence, and uncertainty of expression. Compared with the baseline, CALF has a comparable MedWeb test score (m-F1=0.880 vs 0.871; P=.18), but a substantially higher FA (FA=0.830 vs 0.342; P=.001) and supplementary test score (f-F1=0.856 vs 0.797; P=.007). CALF also improved sensitivity to recovery, timing, and uncertainty across pretrained encoders. The gains were generally larger when GPT-4.1 (OpenAI) was used for data augmentation. Under GPT-4.1, CALF often achieved the highest overall FA among encoder-based methods. When Qwen2.5-7B-Instruct (Alibaba Cloud) was used, an alternative hybrid strategy may achieve comparable or slightly better results. Checklist performance was strongly associated with supplementary test performance (\u03c1=0.811), whereas MedWeb test performance showed weak association with supplementary test performance (\u03c1=\u20130.042). Prompting GPT-4.1 achieved the highest FA overall (FA=0.920) and the highest supplementary test score (f-F1=0.910), but a lower test set score than the baseline (m-F1=0.776 vs 0.871).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Standard held-out metrics alone do not adequately characterize robustness to dataset-specific or spuriously correlated features. Models may perform well on the test set while remaining insensitive to attributes that influence whether a symptom mention is relevant to surveillance. The MedWeb checklist complements conventional evaluation by measuring sensitivity to controlled attribute changes and may indicate out-of-distribution performance. CALF improved sensitivity to surveillance-relevant attributes while maintaining strong classification performance, supporting concept-aware learning as a practical strategy for developing more robust encoder-based models.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/94769","type":"journal-article","created":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T00:20:06Z","timestamp":1786580406000},"page":"e94769","source":"Crossref","is-referenced-by-count":0,"title":["Concept-Aware Learning Framework for Detecting Common Symptoms in Social Media Text: Model Development and Evaluation Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-5776-2308","authenticated-orcid":false,"given":"Michael Van","family":"Supranes","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4020-9100","authenticated-orcid":false,"given":"Shaowen","family":"Peng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1538-8266","authenticated-orcid":false,"given":"Tomohiro","family":"Nishiyama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9371-1340","authenticated-orcid":false,"given":"Shoko","family":"Wakamiya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0201-3609","authenticated-orcid":false,"given":"Eiji","family":"Aramaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,15]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.32604\/cmes.2023.043921"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1057\/s41599-024-03589-y"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.3390\/ijerph22071104"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.2196\/29413"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1109\/jbhi.2021.3123192"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1109\/iccworkshops50388.2021.9473830"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1016\/j.heliyon.2022.e09683"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1109\/aimhc59811.2024.00011"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1109\/iv64223.2024.00051"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1016\/j.pcad.2024.10.010"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2025.1623922"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.fever-1.4"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n18-2017"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.135"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.666"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.329"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.4"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.woah-1.25"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.gebnlp-1.12"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.wnut-1.29"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.semeval-1.15"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-98281-9"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2503.23673"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2024.3492143"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.474"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1109\/CI-NLPSoMeCompanion65206.2025.10977896"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1109\/access.2025.3589503"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.244"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2026.acl-long.1408"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.26"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.28"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.10"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1109\/satml54575.2023.00028"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.221"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.440"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.acl-long.260"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.28"},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.347"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.386"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-023-11368-0"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.54254\/2755-2721\/105\/2024TJ0057"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i1.19950"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.553"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.1109\/ispcem64498.2024.00020"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.20965\/jaciii.2024.p0029"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.2196\/12783"},{"key":"ref48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.720"},{"key":"ref49","doi-asserted-by":"publisher","DOI":"10.20736\/0002001327"},{"key":"ref50","doi-asserted-by":"publisher","DOI":"10.5260\/chara.21.2.8"},{"key":"ref51","unstructured":"AramakiEWakamiyaSMoritaMKanoYOhkumaTMedWeb: NTCIR-13 medWeb task overview (Version 2.0)20172026-03-06https:\/\/mednlp.jp\/medweb\/NTCIR-13\/doc\/en-ver2.0.pdf"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.5260\/chara.21.2.8"},{"key":"ref54","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.84"},{"key":"ref55","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1009"},{"key":"ref56","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.57"},{"key":"ref57","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.647"},{"key":"ref58","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.732"},{"key":"ref59","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2433"},{"key":"ref60","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.39"},{"key":"ref61","doi-asserted-by":"publisher","DOI":"10.2307\/2532163"},{"key":"ref62","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2015.1089789"},{"key":"ref63","unstructured":"MedWeb Checklist DatasetGitHub2026-09-15https:\/\/github.com\/mvsupranes\/MedWeb-Checklist"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e94769","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T15:50:08Z","timestamp":1792079408000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e94769"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,15]]},"references-count":62,"URL":"https:\/\/doi.org\/10.2196\/94769","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.94769","asserted-by":"object"}]},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,15]]}}}