{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T16:08:38Z","timestamp":1773331718756,"version":"3.50.1"},"reference-count":33,"publisher":"BMJ","issue":"2","license":[{"start":{"date-parts":[[2022,2,9]],"date-time":"2022-02-09T00:00:00Z","timestamp":1644364800000},"content-version":"unspecified","delay-in-days":8,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006041","name":"Innovate UK","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100006041","id-type":"DOI","asserted-by":"publisher"}]},{"name":"UK Engineering and Physical Sciences Research Council Impact Acceleration Fund"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2021,12,20]]},"published-print":{"date-parts":[[2022,2]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Global, COVID-driven restrictions around face-to-face interviews for healthcare student selection have forced admission staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first automated interview grounded in multiple mini-interview (MMI) methodology. This study aimed to explore test\u2013retest reliability, acceptability and usability of the system.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design, setting and participants<\/jats:title>\n                    <jats:p>Multimethod feasibility study in Physician Associate programmes from two UK and one US university during 2019\u20132020.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Primary, secondary outcomes<\/jats:title>\n                    <jats:p>Feasibility measures (test\u2013retest reliability, acceptability and usability) were assessed using intraclass correlation (ICC), descriptive statistics, thematic and content analysis.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Volunteers took (T1), then repeated (T2), the automated MMI, with a 7-day interval (\u00b12) then completed an evaluation questionnaire. Admission staff participated in focus group discussions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Sixty-two students and seven admission staff participated; 34 students and 4 staff from UK and 28 students and 3 staff from US universities. Good-excellent test\u2013retest reliability was observed at two sites (US and UK2) with T1 and T2 ICC between 0.65 and 0.81 (p&lt;0.001) when assessed by individual total scores (range 80.6\u2013119), station total scores 0.6\u20130.91, p&lt;0.005 and individual site (\u22650.79\u2009p&lt;0.001). Mean test re-test ICC across all three sites was 0.82\u2009p&lt;0.001 (95% CI 0.7 to 0.9). Admission staff reported potential to reduce resource costs and bias through a more objective screening tool for preselection or to replace some MMI stations in a \u2018hybrid model\u2019. Maintaining human interaction through \u2018touch points\u2019 was considered essential. Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>These preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjopen-2021-050394","type":"journal-article","created":{"date-parts":[[2022,2,9]],"date-time":"2022-02-09T11:36:37Z","timestamp":1644406597000},"page":"e050394","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":6,"title":["Feasibility of an automated interview grounded in multiple mini interview (MMI) methodology for selection into the health professions: an international multimethod evaluation"],"prefix":"10.1136","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9617-909X","authenticated-orcid":false,"given":"Alison","family":"Callwood","sequence":"first","affiliation":[{"name":"Faculty of Health and Medical Sciences, University of Surrey, Guildford, Surrey, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lee","family":"Gillam","sequence":"additional","affiliation":[{"name":"Faculty of Engineering and Physical Sciences, University of Surrey, Guildford, Surrey, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Angelos","family":"Christidis","sequence":"additional","affiliation":[{"name":"Faculty of Engineering and Physical Sciences, University of Surrey, Guildford, Surrey, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jia","family":"Doulton","sequence":"additional","affiliation":[{"name":"Faculty of Health and Medical Sciences, University of Surrey, Guildford, Surrey, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jenny","family":"Harris","sequence":"additional","affiliation":[{"name":"Faculty of Health and Medical Sciences, University of Surrey, Guildford, Surrey, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marianne","family":"Piano","sequence":"additional","affiliation":[{"name":"Melbourne School of Health Sciences, The University of Melbourne, Melbourne, Victoria, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Angela","family":"Kubacki","sequence":"additional","affiliation":[{"name":"St George's, University of London, London, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul A","family":"Tiffin","sequence":"additional","affiliation":[{"name":"Department of Health Sciences, University of York, York, North Yorkshire, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karen","family":"Roberts","sequence":"additional","affiliation":[{"name":"Brighton and Sussex Medical School, Brighton, Brighton and Hove, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Drew","family":"Tarmey","sequence":"additional","affiliation":[{"name":"The University of Manchester School of Medical Sciences, Manchester, Manchester, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Doris","family":"Dalton","sequence":"additional","affiliation":[{"name":"Department of Family and Preventive Medicine, The University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Virginia L","family":"Valentin","sequence":"additional","affiliation":[{"name":"Department of Family and Preventive Medicine, The University of Utah, Salt Lake City, Utah, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"239","published-online":{"date-parts":[[2022,2,9]]},"reference":[{"key":"2025121515161090000_12.2.e050394.1","unstructured":"Top Universities . University admissions- how will Covid 19 affect my application, 2020. Available: https:\/\/www.topuniversities.com\/student-info\/admissions-advice\/university-admissions-covid-19-coronavirus [Accessed 20 Oct 2020]."},{"key":"2025121515161090000_12.2.e050394.2","doi-asserted-by":"crossref","unstructured":"Turpin C , Steele K , Matuk-Villazon O . Rapid Transition to a Virtual Multiple Mini-Interview Admissions Process: A New Medical School\u2019s Experience During the COVID-19 Pandemic. Acad Med 2021. [Epub ahead of print: 25 May 2021].doi:10.1097\/ACM.0000000000004179","DOI":"10.1097\/ACM.0000000000004179"},{"key":"2025121515161090000_12.2.e050394.3","doi-asserted-by":"crossref","first-page":"764","DOI":"10.1111\/medu.14207","article-title":"Virtual multiple Mini-Interview during the COVID-19 pandemic","volume":"54","author":"Ungtrakul","year":"2020","journal-title":"Med Educ"},{"key":"2025121515161090000_12.2.e050394.4","first-page":"1","article-title":"Implementation and evaluation of virtual multiple mini interviews as a selection tool for entry into paediatric postgraduate training: a Queensland experience","author":"Sabesan","year":"2021","journal-title":"Med Teach"},{"key":"2025121515161090000_12.2.e050394.5","doi-asserted-by":"crossref","first-page":"776","DOI":"10.1080\/0142159X.2020.1762851","article-title":"COVID 19: designing and conducting an online mini-multiple interview (mmi) in a dynamic landscape","volume":"42","author":"Cleland","year":"2020","journal-title":"Med Teach"},{"key":"2025121515161090000_12.2.e050394.6","first-page":"281","article-title":"Conducting multiple mini-interviews in the midst of COVID-19 pandemic","volume":"32","author":"Kok","year":"2020","journal-title":"Korean J Med Educ"},{"key":"2025121515161090000_12.2.e050394.7","doi-asserted-by":"crossref","first-page":"1091","DOI":"10.1080\/0142159X.2018.1498589","article-title":"2018 Ottawa consensus statement: selection and recruitment to the healthcare professions","volume":"40","author":"Patterson","year":"2018","journal-title":"Med Teach"},{"key":"2025121515161090000_12.2.e050394.8","doi-asserted-by":"publisher","DOI":"10.1046\/j.1365-2923.2004.01776.x"},{"key":"2025121515161090000_12.2.e050394.9","first-page":"203","article-title":"Multiple mini interview as an admission tool in higher education: insights from a systematic review","volume":"14","author":"Yusoff","year":"2019","journal-title":"J Taibah Univ Med Sci"},{"key":"2025121515161090000_12.2.e050394.10","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1111\/medu.12224","article-title":"Internet-Based multiple mini-interviews for candidate selection for graduate entry programmes","volume":"47","author":"Tiller","year":"2013","journal-title":"Med Educ"},{"key":"2025121515161090000_12.2.e050394.11","doi-asserted-by":"crossref","first-page":"285","DOI":"10.15694\/mep.2018.0000285.1","article-title":"An exploration of applicant perceptions of asynchronous video MMIs in medical selection","volume":"7","author":"Zibarras","year":"2018","journal-title":"MedEdPublish"},{"key":"2025121515161090000_12.2.e050394.12","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1002\/lary.27878","article-title":"Adding long-term value to the residency selection and assessment process","volume":"130","author":"Villwock","year":"2020","journal-title":"Laryngoscope"},{"key":"2025121515161090000_12.2.e050394.13","unstructured":"An untidy room could cost you your DREAM job, new research finds. Available: https:\/\/bmmagazine.co.uk\/in-business\/an-untidy-room-could-cost-you-your-dream-job-new-research-finds\/"},{"key":"2025121515161090000_12.2.e050394.14","unstructured":"HireVue . Video interviewing software and platform that makes hiring simple, 2020. Available: https:\/\/www.hirevue.com\/ [Accessed 10 Oct 2020]."},{"key":"2025121515161090000_12.2.e050394.15","unstructured":"United Nations . Sustainable development goals, 2015. Available: https:\/\/www.undp.org\/content\/undp\/en\/home\/sustainable-development-goals.html [Accessed 26 Oct 2020]."},{"key":"2025121515161090000_12.2.e050394.16","doi-asserted-by":"publisher","DOI":"10.3102\/01623737011003255"},{"key":"2025121515161090000_12.2.e050394.17","unstructured":"Olsen E , Eoyang G . Facilitating organisational change: lessons from complexity science. San Francisco: Jossey-Bass, 2001."},{"key":"2025121515161090000_12.2.e050394.18","unstructured":"Salkind N . Test-retest reliability. In: Saljind N , ed. Encyclopaedia of research design. Volume 1. London: Sage publications, 2015."},{"key":"2025121515161090000_12.2.e050394.19","doi-asserted-by":"crossref","unstructured":"Nielsen J . Usability engineering. London: AP professional, 1993.","DOI":"10.1016\/B978-0-08-052029-2.50009-7"},{"key":"2025121515161090000_12.2.e050394.20","doi-asserted-by":"crossref","unstructured":"Streiner D , Norman G , Cairney J . Health measurement scales: a practical guide to their development and use. 5th ed. Oxford: Oxford University, 2014.","DOI":"10.1093\/med\/9780199685219.001.0001"},{"key":"2025121515161090000_12.2.e050394.21","doi-asserted-by":"crossref","unstructured":"Szabo F . The linear algebra survival guide. Massachusetts, US: Academic Press, 2015.","DOI":"10.1016\/B978-0-12-409520-5.50020-5"},{"key":"2025121515161090000_12.2.e050394.22","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"2025121515161090000_12.2.e050394.23","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"2025121515161090000_12.2.e050394.24","unstructured":"University clinical aptitude test (UCAT), 2020. Available: https:\/\/www.ucat.ac.uk\/ [Accessed 12 Jan 2021]."},{"key":"2025121515161090000_12.2.e050394.25","unstructured":"Casper, 2020. Available: https:\/\/altusassessments.com\/casper\/how-it-works\/ [Accessed 12 Jan 2021]."},{"key":"2025121515161090000_12.2.e050394.26","doi-asserted-by":"crossref","DOI":"10.1186\/s12909-018-1128-z","article-title":"Multiple mini interview (mmi) for general practice training selection in Australia: interviewers' motivation","volume":"18","author":"Burgess","year":"2018","journal-title":"BMC Med Educ"},{"key":"2025121515161090000_12.2.e050394.27","first-page":"179","article-title":"Interviewers\u2019 experiences with two multiple mini-interview scoring methods used for admission to a Master of Physical Therapy programme","volume":"68","author":"Spuy","year":"2016","journal-title":"Physiother"},{"key":"2025121515161090000_12.2.e050394.28","unstructured":"European Commission . Ethical guidelines for trustworthy AI, 2019. Available: https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/ethics-guidelines-trustworthy-ai [Accessed 26 Oct 2020]."},{"key":"2025121515161090000_12.2.e050394.29","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1111\/emip.12359","article-title":"Internet\u2010based proctored assessment: security and fairness issues","volume":"39","author":"Langenfeld","year":"2020","journal-title":"Educational Measurement: Issues and Practice"},{"key":"2025121515161090000_12.2.e050394.30","doi-asserted-by":"crossref","first-page":"2","DOI":"10.25035\/pad.2016.002","article-title":"Technology in the employment interview: a meta-analysis and future research agenda","volume":"2","author":"Blacksmith","year":"2016","journal-title":"Personnel Assessment and Decisions"},{"key":"2025121515161090000_12.2.e050394.31","unstructured":"Coronavirus hiring: how recruiters are selecting and interviewing job candidates during the pandemic. Available: https:\/\/www.cnbc.com\/2020\/05\/24\/how-recruiters-select-and-interview-job-candidates-amid-coronavirus.html"},{"key":"2025121515161090000_12.2.e050394.32","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1111\/j.1468-2389.2011.00553.x","article-title":"Exploring the construct of perceived job discrimination and a model of applicant propensity for case initiation in selection","volume":"19","author":"Patterson","year":"2011","journal-title":"International Journal of Selection & Assessment"},{"key":"2025121515161090000_12.2.e050394.33","unstructured":"Zaiontz C . 2020 real statistics. Available: https:\/\/www.real-statistics.com\/reliability\/interrater-reliability\/intraclass-correlation\/icc-for-test-retest-reliability\/ [Accessed 12 December 2020]."}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2021-050394","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T23:16:23Z","timestamp":1765840583000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2021-050394"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2]]},"references-count":33,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2,9]]},"published-print":{"date-parts":[[2022,2]]}},"alternative-id":["10.1136\/bmjopen-2021-050394"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2021-050394","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.02.28.21251817","asserted-by":"object"}]},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2]]}}}