{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T19:57:26Z","timestamp":1782935846892,"version":"3.54.5"},"reference-count":36,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["EJIHPE"],"abstract":"<jats:p>Background: Large language models (LLMs) are increasingly integrated into real-world medical practice as chatbots for answering clinical queries. However, the perceptions of this technology among its end-users remain understudied. Existing research on physicians\u2019 attitudes toward LLMs relies on non-validated questionnaires, raising concerns about the accuracy and reliability of the findings. The aim of this study is to develop and validate a questionnaire to assess physicians\u2019 attitudes toward LLM-based chatbots used as a reference tool for answering queries. Methods: The instrument was based on the previously developed and validated ATRAI-14 questionnaire assessing radiologists\u2019 attitudes toward artificial intelligence. Items for the new questionnaire were formulated and refined through focus group testing. Validation involved 562 physicians of various specialties working in medical institutions within the Moscow healthcare system. Some respondents had prior experience working with medical LLMs. We assessed face, content, construct, and criterion validity. Criterion validity was evaluated through correlation between respondents\u2019 self-assessed attitudes toward LLMs measured by visual analogue scale (VAS), and construct validity through confirmatory factor analysis. Results: The resulting ATRAI-LLM questionnaire comprised 19 items (8 in the background part and 11 in the main part). The questionnaire demonstrated acceptable internal consistency (Cronbach\u2019s \u03b1 = 0.770, McDonald\u2019s \u03c9t = 0.830). It encompasses three domains: \u201cWillingness to Use\u201d, \u201cImplementation Perspective\u201d, and \u201cHopes and Fears.\u201d Confirmatory factor analysis supported the three-factor structure, with satisfactory fit indices achieved (RMSEA = 0.05, CFI = 0.97, TLI = 0.96, SRMR = 0.03). Criterion validity was confirmed as acceptable with moderate correlation between the final score and VAS scores (Spearman\u2019s rho 0.68, p &lt; 0.001). Conclusions: ATRAI-LLM is a validated instrument for assessing physicians\u2019 attitudes toward LLMs as a knowledge base.<\/jats:p>","DOI":"10.3390\/ejihpe16070094","type":"journal-article","created":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T03:06:34Z","timestamp":1782875194000},"page":"94","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Development and Validation of the ATRAI Questionnaire to Assess Attitudes Toward Large Language Models in Clinical Setting (ATRAI-LLM)"],"prefix":"10.3390","volume":"16","author":[{"given":"Roman V.","family":"Reshetnikov","sequence":"first","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuriy A.","family":"Vasilev","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8521-4045","authenticated-orcid":false,"given":"Yuliya F.","family":"Shumskaya","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7705-9754","authenticated-orcid":false,"given":"Dina A.","family":"Akhmedzyanova","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yulya A.","family":"Alymova","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2990-7736","authenticated-orcid":false,"given":"Anton V.","family":"Vladzymyrskyy","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ilya A.","family":"Tyrov","sequence":"additional","affiliation":[{"name":"Moscow Health Care Department, 127006 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0245-4431","authenticated-orcid":false,"given":"Olga V.","family":"Omelyanskaya","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2681-9378","authenticated-orcid":false,"given":"Ivan A.","family":"Blokhin","sequence":"additional","affiliation":[{"name":"Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department, 127051 Moscow, Russia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1016\/0749-5978(91)90020-T","article-title":"The theory of planned behavior","volume":"50","author":"Ajzen","year":"1991","journal-title":"Organizational Behavior and Human Decision Processes"},{"key":"ref_2","first-page":"e37432","article-title":"Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references","volume":"15","author":"Athaluri","year":"2023","journal-title":"Cureus"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Barbella, M., and Tortora, G. (2022). Rouge metric evaluation for text summarization techniques. SSRN Electronic Journal.","DOI":"10.2139\/ssrn.4120317"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e68998","DOI":"10.2196\/68998","article-title":"Scientific evidence for clinical text summarization using large language models: Scoping review","volume":"27","author":"Bednarczyk","year":"2025","journal-title":"Journal of Medical Internet Research"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"e74428","DOI":"10.2196\/74428","article-title":"Generative artificial intelligence in primary care: Qualitative study of UK general practitioners\u2019 views","volume":"27","author":"Blease","year":"2025","journal-title":"Journal of Medical Internet Research"},{"key":"ref_6","unstructured":"Brown, T. (2015). Confirmatory factor analysis for applied research, Guilford Publications. [2nd ed.]."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1186\/1748-5908-7-37","article-title":"Validation of the theoretical domains framework for use in behaviour change and implementation research","volume":"7","author":"Cane","year":"2012","journal-title":"Implementation Science"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1023\/A:1023254226592","article-title":"Pretesting survey instruments: An overview of cognitive methods","volume":"12","author":"Collins","year":"2003","journal-title":"Quality of Life Research"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Conger, N., Scollar, N., Davaslioglu, K., Sagduyu, Y., and Kompella, S. (2025). How to discover knowledge for FutureG: Contextual RAG and LLM prompting for O-RAN. arXiv.","DOI":"10.1109\/CCNC65079.2026.11366352"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Conner, M. (2001). Health behaviors. International encyclopedia of the social & behavioral sciences, Elsevier.","DOI":"10.1016\/B0-08-043076-7\/03871-7"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"127","DOI":"10.2466\/pr0.1991.69.1.127","article-title":"Single-item vs multiple-item measures of health-related quality of life","volume":"69","author":"Cunny","year":"1991","journal-title":"Psychological Reports"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1016\/j.jclinepi.2013.12.002","article-title":"Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies","volume":"67","author":"Diamond","year":"2014","journal-title":"Journal of Clinical Epidemiology"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1111\/bjop.12046","article-title":"From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation","volume":"105","author":"Dunn","year":"2014","journal-title":"British Journal of Psychology"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"98","DOI":"10.1177\/01466216251401213","article-title":"On the unreliability of test\u2013retest reliability","volume":"50","author":"Groh","year":"2026","journal-title":"Applied Psychological Measurement"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3148","DOI":"10.1109\/TAI.2025.3567369","article-title":"TrumorGPT: Graph-based retrieval-augmented large language model for fact-checking","volume":"6","author":"Hang","year":"2025","journal-title":"IEEE Transactions on Artificial Intelligence"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1186\/s12929-025-01131-z","article-title":"Impact of large language model (ChatGPT) in healthcare: An umbrella review and evidence synthesis","volume":"32","author":"Iqbal","year":"2025","journal-title":"Journal of Biomedical Science"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1017\/cts.2018.18","article-title":"Impact of survey length and compensation on validity, reliability, and sample characteristics for ultrashort-, short-, and long-research participant perception surveys","volume":"2","author":"Kost","year":"2018","journal-title":"Journal of Clinical and Translational Science"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lavie, A., and Agarwal, A. (2007). METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. Proceedings of the WMT@ACL, Association for Computational Linguistics.","DOI":"10.3115\/1626355.1626389"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"109713","DOI":"10.1016\/j.isci.2024.109713","article-title":"The application of large language models in medicine: A scoping review","volume":"27","author":"Meng","year":"2024","journal-title":"iScience"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1207\/s15327574ijt0502_4","article-title":"Minimum sample size recommendations for conducting factor analyses","volume":"5","author":"Mundfrom","year":"2005","journal-title":"International Journal of Testing"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1162\/coli_a_00322","article-title":"A structured review of the validity of BLEU","volume":"44","author":"Reiter","year":"2018","journal-title":"Computational Linguistics"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Reshetnikov, R. V., Tyrov, I., Vasilev, Y., Shumskaya, Y., Vladzymyrskyy, A. V., Akhmedzyanova, D., Bezhenova, K., Varyukhina, M., Sokolova, M. V., Blokhin, I., Voytenko, D. A., Mynko, O. I., Kodenko, M. R., and Omelyanskaya, O. V. (2025). Assessing the quality of large generative models for basic healthcare applications. Medical Doctor and Information Technology, 64\u201375.","DOI":"10.25881\/18110193_2025_3_64"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1633","DOI":"10.1213\/ANE.0000000000005773","article-title":"Statistics from A (agreement) to Z (z score): A guide to interpreting common measures of association, agreement, diagnostic accuracy, effect size, heterogeneity, and reliability in medical research","volume":"133","author":"Schober","year":"2021","journal-title":"Anesthesia & Analgesia"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Shankar, R., Bundele, A., Yap, A., and Mukhopadhyay, A. (2025). Development and feasibility testing of an AI-powered chatbot for early detection of caregiver burden: Protocol for a mixed methods feasibility study. Frontiers in Psychiatry, 16.","DOI":"10.3389\/fpsyt.2025.1553494"},{"key":"ref_25","unstructured":"Soper, D. (2026). A-priori sample size calculator for structural equation models [Computer software], Daniel Soper. Available online: https:\/\/www.danielsoper.com\/statcalc."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"306","DOI":"10.1055\/a-2281-7092","article-title":"A survey of clinicians\u2019 views of the utility of large language models","volume":"15","author":"Spotnitz","year":"2024","journal-title":"Applied Clinical Informatics"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2909","DOI":"10.1038\/s41598-024-53335-2","article-title":"Attitudes towards AI: Measurement and associations with personality","volume":"14","author":"Stein","year":"2024","journal-title":"Scientific Reports"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e67383","DOI":"10.2196\/67383","article-title":"Perspectives and experiences with large language models in health care: Survey study","volume":"27","author":"Sumner","year":"2025","journal-title":"Journal of Medical Internet Research"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","year":"2024","journal-title":"npj Digital Medicine"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"e50658","DOI":"10.2196\/50658","article-title":"Using ChatGPT for clinical practice and medical education: Cross-sectional survey of medical students\u2019 and physicians\u2019 perceptions","volume":"9","author":"Tangadulrat","year":"2023","journal-title":"JMIR Medical Education"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Unger, Z., Soffer, S., Nadkarni, G., Efros, O., Chan, L., and Klang, E. (2025). Clinical applications and limitations of large language models in nephrology: A systematic review. Clinical Kidney Journal, 18.","DOI":"10.1093\/ckj\/sfaf243"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"268","DOI":"10.17816\/DD678373","article-title":"Application of large language models in radiological diagnostics: A scoping review","volume":"6","author":"Vasilev","year":"2025a","journal-title":"Digital Diagnostics"},{"key":"ref_33","unstructured":"Vasilev, Y., and Vladzymyrskyy, A. (2025). Iskusstvennyy intellekt v luchevoy diagnostike: Per aspera ad astra [Artificial intelligence in radiology: Per aspera ad astra], Izdatelskie Resheniya."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Vasilev, Y., Vladzymyrskyy, A. V., Alymova, Y., Akhmedzyanova, D., Blokhin, I., Romanenko, M., Seradzhi, S., Suchilova, M., Shumskaya, Y., and Reshetnikov, R. V. (2024). Development and validation of a questionnaire to assess the radiologists\u2019 views on the implementation of artificial intelligence in radiology (ATRAI-14). Healthcare, 12.","DOI":"10.3390\/healthcare12192011"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Vasilev, Y., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Alymova, Y., Akhmedzyanova, D., Shumskaya, Y., Kodenko, M., Blokhin, I., and Reshetnikov, R. V. (2025b). Development and validation of a questionnaire to evaluate AI-generated summaries for radiologists: ELEGANCE (expert-led evaluation of generative AI competence and excellence). AI, 6.","DOI":"10.3390\/ai6110287"},{"key":"ref_36","first-page":"e71946","article-title":"A pilot study of medical student opinions on large language models","volume":"16","author":"Xu","year":"2024","journal-title":"Cureus"}],"container-title":["European Journal of Investigation in Health, Psychology and Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2254-9625\/16\/7\/94\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T18:14:26Z","timestamp":1782929666000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2254-9625\/16\/7\/94"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":36,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["ejihpe16070094"],"URL":"https:\/\/doi.org\/10.3390\/ejihpe16070094","relation":{},"ISSN":["2254-9625"],"issn-type":[{"value":"2254-9625","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}