{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,17]],"date-time":"2026-09-17T15:29:35Z","timestamp":1789658975358,"version":"4.0.1"},"reference-count":20,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T00:00:00Z","timestamp":1789430400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"State Administration for Market Regulation Science and Technology Plan Project","award":["2024MK033"],"award-info":[{"award-number":["2024MK033"]}],"id":[{"id":"https:\/\/ror.org\/00aaqtw69","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"\n Cross-jurisdictional pharmaceutical compliance requires comparison of regulatory requirements across administrative systems such as the US Food and Drug Administration and China's National Medical Products Administration. Although large language models (LLMs) are increasingly explored for healthcare and regulatory applications, their performance in cross-jurisdictional pharmaceutical regulation has not been systematically evaluated using a dedicated benchmark. We introduce Sino-US-DrugQA, a bilingual multiple-choice benchmark covering Monolingual, Comparative, and Parallel regulatory question\u2013answer tasks. The 11,871 candidate items underwent deterministic structural validation and full-dataset semantic quality screening, followed by risk-stratified independent review of 1,432 items by two regulatory experts. The final release comprised 11,444 items, including 10,122 classified as pass and 1,322 as borderline. Among 500 items sampled from the semantic screen-negative population, 18 were subsequently classified as material errors, corresponding to an observed residual material-error proportion of 3.60% (Wilson 95% confidence interval, 2.29%\u20135.62%). Four representative LLMs\u2014gpt-5.6-terra, gemini-3.6-flash, deepseek-v4-flash, and qwen-3.5-max\u2014were evaluated under a standardized zero-shot protocol. Overall accuracy ranged from 83.21% to 86.43%. Comparative accuracy was consistently lower than Monolingual accuracy, with absolute differences of 4.42\u20138.98 percentage points across models. The two highest-scoring models, GPT and Gemini, did not differ significantly after adjustment for multiple comparisons. These results indicate that explicit comparison across non-equivalent regulatory systems remains more challenging than single-jurisdiction question answering. Sino-US-DrugQA provides a validated and reproducible resource for evaluating bilingual regulatory reasoning. The findings support further investigation of expert-supervised decision-support workflows rather than autonomous regulatory interpretation. The stable dataset release and evaluation resources are available at\n https:\/\/github.com\/DodgeLU\/Sino-US-DrugQA<\/jats:ext-link>\n .\n <\/jats:p>","DOI":"10.1371\/journal.pone.0343858","type":"journal-article","created":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T17:55:37Z","timestamp":1789494937000},"page":"e0343858","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Sino-US-DrugQA: A benchmark for evaluating large language models in cross-jurisdictional pharmaceutical regulation"],"prefix":"10.1371","volume":"21","author":[{"given":"Xuejing","family":"Fu","sequence":"first","affiliation":[{"name":"Information Application Research Center of Shanghai Municipal Administration for Market Regulation","place":["Shanghai, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhen","family":"Chen","sequence":"additional","affiliation":[{"name":"Information Application Research Center of Shanghai Municipal Administration for Market Regulation","place":["Shanghai, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0763-213X","authenticated-orcid":true,"given":"Wentao","family":"Lu","sequence":"additional","affiliation":[{"name":"Shanghai Artificial Intelligence Research Institute Co., Ltd.","place":["Shanghai, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,9,15]]},"reference":[{"key":"pone.0343858.ref001","author":"G Sengar","year":"2012"},{"key":"pone.0343858.ref002","first-page":"8100","article-title":"Benchmarking drug regulatory systems for capacity building: an integrative review of tools, practice, and recommendations","volume":"12","author":"J Shi","year":"2023","journal-title":"Int J Health Policy Manag"},{"issue":"4","key":"pone.0343858.ref003","doi-asserted-by":"crossref","first-page":"1222","DOI":"10.1111\/cts.12987","article-title":"Evolving drug regulatory landscape in China: a clinical pharmacology perspective","volume":"14","author":"W Tang","year":"2021","journal-title":"Clin Transl Sci"},{"key":"pone.0343858.ref004","article-title":"Global analysis of regulatory frameworks and drug safety standards in the drug approval process","author":"VS Gomase","year":"2025","journal-title":"Curr Drug Saf"},{"issue":"9","key":"pone.0343858.ref005","doi-asserted-by":"crossref","first-page":"1212","DOI":"10.51594\/gjabr.v3i9.157","article-title":"Designing intelligent compliance systems for evolving global regulatory landscapes","volume":"3","author":"E Iboro Akpan","year":"2025","journal-title":"GJABR"},{"issue":"6","key":"pone.0343858.ref006","first-page":"4509","article-title":"Risk management and compliance in a globalized economy: navigating regulatory challenges and strategic adaptations","volume":"5","author":"A Apooyin","year":"2025","journal-title":"J Law Politic Humanit"},{"issue":"6","key":"pone.0343858.ref007","doi-asserted-by":"crossref","first-page":"588","DOI":"10.3390\/healthcare13060588","article-title":"From command-control to lifecycle regulation: balancing innovation and safety in China\u2019s pharmaceutical legislation","volume":"13","author":"J Zhang","year":"2025","journal-title":"Healthcare (Basel)"},{"issue":"1","key":"pone.0343858.ref008","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1038\/s41746-023-00873-0","article-title":"The imperative for regulatory oversight of large language models (or generative AI) in healthcare","volume":"6","author":"B Mesk\u00f3","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"11","key":"pone.0343858.ref009","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1007\/s10462-024-10921-0","article-title":"Large language models in medical and healthcare fields: applications, advances, and challenges","volume":"57","author":"D Wang","year":"2024","journal-title":"Artif Intell Rev"},{"issue":"1","key":"pone.0343858.ref010","doi-asserted-by":"crossref","first-page":"13755","DOI":"10.1038\/s41598-025-98483-1","article-title":"Industrial applications of large language models","volume":"15","author":"M Raza","year":"2025","journal-title":"Sci Rep"},{"key":"pone.0343858.ref011","doi-asserted-by":"crossref","unstructured":"Fei Z, Shen X, Zhu D, Zhou F, Han Z, Huang A, et al. LawBench: benchmarking legal knowledge of large language models. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. 7933\u201362. https:\/\/doi.org\/10.18653\/v1\/2024.emnlp-main.452","DOI":"10.18653\/v1\/2024.emnlp-main.452"},{"key":"pone.0343858.ref012","doi-asserted-by":"crossref","unstructured":"Chalkidis I, Jana A, Hartung D, Bommarito M, Androutsopoulos I, Katz D, et al. LexGLUE: a benchmark dataset for legal language understanding in English. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022. 4310\u201330. https:\/\/doi.org\/10.18653\/v1\/2022.acl-long.297","DOI":"10.18653\/v1\/2022.acl-long.297"},{"key":"pone.0343858.ref013","unstructured":"Pal A, Umapathi LK, Sankarasubbu M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In: Proceedings of the Conference on Health, Inference, and Learning. 2022. 248\u201360."},{"key":"pone.0343858.ref014","unstructured":"Rasiah V, Stern R, Matoshi V, St\u00fcrmer M, Chalkidis I, Ho DE, et al. One law, many languages: benchmarking multilingual legal reasoning for judicial support. In: ICLR 2024 Workshop on Data-centric Machine Learning Research (DMLR): Harnessing Momentum for Science. Vienna, Austria, 2024. https:\/\/openreview.net\/forum?id=99e01aa79a3c122f099ffd7d640f81570c0e112e"},{"issue":"5","key":"pone.0343858.ref015","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbae354","article-title":"Harnessing large language models\u2019 zero-shot and few-shot learning capabilities for regulatory research","volume":"25","author":"H Meshkin","year":"2024","journal-title":"Brief Bioinform"},{"key":"pone.0343858.ref016","author":"D Hendrycks","year":"2020"},{"issue":"3","key":"pone.0343858.ref017","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","author":"ML McHugh","year":"2012","journal-title":"Biochem Med (Zagreb)"},{"key":"pone.0343858.ref018","doi-asserted-by":"crossref","unstructured":"Zhu W, Huang S, Yuan F, She S, Chen J, Birch A. Question translation training for better multilingual reasoning. In: Findings of the Association for Computational Linguistics ACL 2024, 2024. 8411\u201323. https:\/\/doi.org\/10.18653\/v1\/2024.findings-acl.498","DOI":"10.18653\/v1\/2024.findings-acl.498"},{"key":"pone.0343858.ref019","unstructured":"International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use ICH. ICH Q7 Guideline: Good Manufacturing Practice for Active Pharmaceutical Ingredients. Geneva: ICH; 2000."},{"key":"pone.0343858.ref020","unstructured":"Organisation for Economic Co-operation and Development. OECD Principles of Good Laboratory Practice (GLP). Paris: OECD; 1998."}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0343858","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T17:55:43Z","timestamp":1789494943000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0343858"}},"subtitle":[],"editor":[{"given":"Issa","family":"Atoum","sequence":"first","affiliation":[{"name":"Philadelphia University","place":["JORDAN"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,9,15]]},"references-count":20,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2026,9,15]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0343858","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,15]]}}}