{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T19:02:39Z","timestamp":1773687759314,"version":"3.50.1"},"reference-count":50,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2020,4,28]],"date-time":"2020-04-28T00:00:00Z","timestamp":1588032000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The projected introduction of conditional automated driving systems to the market has sparked multifaceted research on human\u2013machine interfaces (HMIs) for such systems. By moderating the roles of the human driver and the driving automation system, the HMI is indispensable in avoiding side effects of automation such as mode confusion, misuse, and disuse. In addition to safety aspects, the usability of HMIs plays a vital role in improving the trust and acceptance of the automated driving system. This paper aggregates common research methods and findings based on an extensive literature review. Empirical studies, frameworks, and review articles are included. Findings and conclusions are presented with a focus on study characteristics such as test cases, dependent variables, testing environments, or participant samples. These methods and findings are discussed critically, taking into consideration requirements for usability assessments of HMIs in the context of conditional automated driving. The paper concludes with a derivation of recommended study characteristics framing best practice advice for the design of experiments. The advised selection of scenarios and metrics will be applied in a future validation study series comprising a driving simulator experiment and three real driving experiments on test tracks in Germany, the USA, and Japan.<\/jats:p>","DOI":"10.3390\/info11050240","type":"journal-article","created":{"date-parts":[[2020,4,28]],"date-time":"2020-04-28T09:57:21Z","timestamp":1588067841000},"page":"240","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Usability Evaluation\u2014Advances in Experimental Design in the Context of Automated Driving Human\u2013Machine Interfaces"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9923-0724","authenticated-orcid":false,"given":"Deike","family":"Albers","sequence":"first","affiliation":[{"name":"Chair of Ergonomics, Technical University of Munich, Boltzmannstra\u00dfe 15, 85748 Garching, Germany"}]},{"given":"Jonas","family":"Radlmayr","sequence":"additional","affiliation":[{"name":"Chair of Ergonomics, Technical University of Munich, Boltzmannstra\u00dfe 15, 85748 Garching, Germany"}]},{"given":"Alexandra","family":"Loew","sequence":"additional","affiliation":[{"name":"Chair of Ergonomics, Technical University of Munich, Boltzmannstra\u00dfe 15, 85748 Garching, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4061-8273","authenticated-orcid":false,"given":"Sebastian","family":"Hergeth","sequence":"additional","affiliation":[{"name":"BMW Group, Knorrstra\u00dfe 147, 80937 Munich, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1712-6473","authenticated-orcid":false,"given":"Frederik","family":"Naujoks","sequence":"additional","affiliation":[{"name":"BMW Group, Knorrstra\u00dfe 147, 80937 Munich, Germany"}]},{"given":"Andreas","family":"Keinath","sequence":"additional","affiliation":[{"name":"BMW Group, Knorrstra\u00dfe 147, 80937 Munich, Germany"}]},{"given":"Klaus","family":"Bengler","sequence":"additional","affiliation":[{"name":"Chair of Ergonomics, Technical University of Munich, Boltzmannstra\u00dfe 15, 85748 Garching, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2020,4,28]]},"reference":[{"key":"ref_1","unstructured":"SAE (2018). Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, Society of Automotive Engineers."},{"key":"ref_2","unstructured":"Lorenz, L., Kerschbaum, P., Hergeth, S., Gold, C., and Radlmayr, J. (2015, January 25\u201326). Der Fahrer im Hochautomatisierten Fahrzeug. Vom Dual-Task zum Sequential-Task Paradigma. Proceedings of the 7. Tagung Fahrerassistenz, M\u00fcnchen, Germany."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., and The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med., 6.","DOI":"10.1371\/journal.pmed.1000097"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Forster, Y., Hergeth, S., Naujoks, F., Krems, J., and Keinath, A. (2019). User Education in Automated Driving: Owner\u2019s Manual and Interactive Tutorial Support Mental Model Formation and Human-Automation Interaction. Information, 10.","DOI":"10.3390\/info10040143"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Forster, Y., Hergeth, S., Naujoks, F., Beggiato, M., Krems, J.F., and Keinath, A. (2019, January 24\u201327). Learning and Development of mental models during interactions with driving automation: A simulator study. Proceedings of the Tenth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Santa Fe, NM, USA.","DOI":"10.17077\/drivingassessment.1724"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1016\/j.trf.2019.02.013","article-title":"Learning to use automation: Behavioral changes in interaction with automated driving systems","volume":"62","author":"Forster","year":"2019","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Forster, Y., Hergeth, S., Naujoks, F., Krems, J.F., and Keinath, A. (2019). Self-report measures for the assessment of human\u2013machine interfaces in automated driving. Cogn. Technol. Work.","DOI":"10.1007\/s10111-019-00599-8"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1016\/j.trf.2017.04.006","article-title":"Cooperation between driver and automated driving system: Implementation and evaluation","volume":"61","author":"Guo","year":"2019","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kettwich, C., Haus, R., Temme, G., and Schieben, A. (2016, January 19\u201322). Validation of a HMI Concept Indicating the Status of the Traffic Light Signal in the Context of Automated Driving in Urban Environment. Proceedings of the 2016 IEEE Intelligent Vehicles Symposium (IV), Gothenburg, Sweden.","DOI":"10.1109\/IVS.2016.7535569"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Morgan, P.L., Voinescu, A., Alford, C., and Caleb-Solly, P. (2018, January 21\u201325). Exploring the Usability of a Connected Autonomous Vehicle Human Machine Interface Designed for Older Adults. Proceedings of the International Conference on Applied Human Factors and Ergonomics, Orlando, FL, USA.","DOI":"10.1007\/978-3-319-93885-1_54"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Stanton, N.A., Landry, S., Di Bucchianico, G., and Vallicelli, A. (2017). A Human-Machine Interface for Cooperative Highly Automated Driving. Advances in Human Aspects of Transportation, Springer International Publishing.","DOI":"10.1201\/9781315370460"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Richardson, N.T., Lehmer, C., Lienkamp, M., and Michel, B. (2018, January 26\u201330). Conceptual Design and Evaluation of a Human Machine Interface for Highly Automated Truck Driving. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500520"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Forster, Y., Hergeth, S., Naujoks, F., and Krems, J.F. (2018, January 23\u201325). How Usability Can Save the Day\u2014Methodological Considerations for Making Automated Driving a Success Story. Proceedings of the 10th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI \u201918), Toronto, ON, Canada.","DOI":"10.1145\/3239060.3239076"},{"key":"ref_14","first-page":"541","article-title":"Automotive HMI design and participatory user involvement: Review and perspectives","volume":"60","author":"Osiurak","year":"2016","journal-title":"Ergonomics"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Gold, C., Naujoks, F., Radlmayr, J., Bellem, H., and Jarosch, O. (2017, January 17\u201321). Testing Scenarios for Human Factors Research in Level 3 Automated Vehicles. Proceedings of the International Conference on Applied Human Factors and Ergonomics, Los Angeles, CA, USA.","DOI":"10.1007\/978-3-319-60441-1_54"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Naujoks, F., Hergeth, S., Wiedemann, K., Sch\u00f6mig, N., and Keinath, A. (2018, January 1\u20135). Use Cases for Assessing, Testing, and Validating the Human Machine Interface of Automated Driving Systems. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Philadelphia, PA, USA.","DOI":"10.1177\/1541931218621426"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"S146","DOI":"10.1080\/15389588.2019.1603374","article-title":"Test procedure for evaluating the human-machine interface of vehicles with automated driving systems","volume":"20","author":"Naujoks","year":"2019","journal-title":"Traffic Inj. Prev."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.trf.2018.10.012","article-title":"Towards guidelines and verification methods for automated vehicle HMIs","volume":"60","author":"Naujoks","year":"2019","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Pauzie, A., and Orfila, O. (2016). Methodologies to assess usability and safety of ADAS and automated vehicle. IFAC PapersOnLine, 72\u201377.","DOI":"10.1016\/j.ifacol.2016.12.192"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jordan Patrick, W., Thomas, B., Weerdmeester, B.A., and McClelland, I.L. (1996). SUS: A \u2018Quick and Dirty\u2019 Usability Scale. Usability Evaluation in Industry, Taylor & Francis.","DOI":"10.1201\/9781498710411"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1207\/S15327590IJHC143&4_11","article-title":"Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies","volume":"14","author":"Lewis","year":"2002","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/S0968-090X(96)00025-3","article-title":"A Simple Procedure for the Assessment of Acceptance of Advanced Transport Telematics","volume":"5","author":"Heino","year":"1997","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_23","unstructured":"ISO 9241-11: 2018 (2018). Ergonomics of Human-System Interaction. Part. 11: Usability: Definitions and Concepts, International Organization of Standardization."},{"key":"ref_24","unstructured":"NHTSA (2017). Automated Driving Systems 2.0: A Vision for Safety."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1993). Usability Engineering, Elsevier.","DOI":"10.1016\/B978-0-08-052029-2.50007-3"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Fisher, D.L., Rizzo, M., Caird, J.K., and Lee, J.D. (2011). Twelve Practical and Twelve Practical and Useful Questions About Driving Simulation. Handbook of Driving Simulation for Engineering, Medicine, and Psychology, CRC Press.","DOI":"10.1201\/b10836-2"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fisher, D.L., Rizzo, M., Caird, J.K., and Lee, J.D. (2011). Simulator Validity: Behaviors Observed on the Simulator and on the Road. Handbook of Driving Simulation for Engineering, Medicine, and Psychology, CRC Press.","DOI":"10.1201\/b10836"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Nielsen, J. (1994). Usability Inspection Methods. Conference Companion on Human Factors in Computing Systems, Association for Computing Machinery.","DOI":"10.1145\/259963.260531"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"621","DOI":"10.1016\/j.ergon.2008.02.012","article-title":"Web Evaluation: Heuristic Evaluation vs. User Testing","volume":"39","author":"Tan","year":"2009","journal-title":"Int. J. Ind. Ergon."},{"key":"ref_30","unstructured":"NHTSA (2014). Visual\u2013Manual NHTSA Driver Distraction Guidelines for In-Vehicle Electronic Devices."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1017\/S0140525X0999152X","article-title":"The weirdest people in the world?","volume":"33","author":"Henrich","year":"2010","journal-title":"Behav. Brain Sci."},{"key":"ref_32","unstructured":"Pr\u00fcmper, J., and Anft, M. (2017, November 02). ISONORM 9241\/110 (Langfassung): Beurteilung von Software auf Grundlage der Internationalen Ergonomie-Norm DIN EN ISO 9241-110. Available online: http:\/\/people.f3.htw-berlin.de\/Professoren\/Pruemper\/instrumente\/ISONORM%209241-110-L.pdf."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"425","DOI":"10.2307\/30036540","article-title":"User Acceptance of Information Technology: Toward a Unified View","volume":"27","author":"Ventakesh","year":"2003","journal-title":"MIS Q."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1985347.1985353","article-title":"Trust in a specific Technology","volume":"2","author":"Mcknight","year":"2011","journal-title":"ACM Trans. Manag. Inf. Syst."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Rau, P.L.R. (2014). Towards the Development of An Inter-Cultural Scale to measure Trust in Automation. Cross-Cultural Design, Springer International Publishing.","DOI":"10.1007\/978-3-319-07308-8"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/S0166-4115(08)62386-9","article-title":"Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research","volume":"52","author":"Hart","year":"1988","journal-title":"Adv. Psychol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1518\/001872095779049499","article-title":"Measurement of Situation Awareness in Dynamic Systems","volume":"37","author":"Endsley","year":"1995","journal-title":"Hum. Factors"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.trf.2015.10.005","article-title":"Learning and Development of Trust, Acceptance and the Mental Model of ACC. A longitudinal on-road Study","volume":"35","author":"Beggiato","year":"2015","journal-title":"Transp. Res. Part F Traffic Psychol. Behav."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1109\/47.867942","article-title":"Thinking Aloud: Reconciling Theory and Practice","volume":"43","author":"Boren","year":"2000","journal-title":"IEEE Trans. Prof. Commun."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Nielsen, J., and Molich, R. (1990, January 1\u20135). Heuristic Evaluation of User Interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA, USA.","DOI":"10.1145\/97243.97281"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jian, J.-Y., Bisantz, A.M., Drury, C.G., and Llinas, J. (2000). Foundations for an Empirically Determined Scale of Trust in Automated Systems, Air Force Research laboratory.","DOI":"10.1207\/S15327566IJCE0401_04"},{"key":"ref_42","unstructured":"Szwillus, G., and Ziegler, J. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualit\u00e4t. Mensch & Computer 2003: Interaktion in Bewegung, B. G. Teubner."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and Evaluation of a User Experience Questionnaire. HCI Usability Educ. Work, 63\u201376.","DOI":"10.1007\/978-3-540-89350-9_6"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Soares, M., Falc\u00e3o, C., and Ahram, T.Z. (2016). The meCUE Questionnaire. A Modular Tool for Measuring User Experience. Advances in Ergonomics Modeling, Usability & Special Populations, Proceedings of the 7th Applied Human Factors and Ergonomics Society Conference, Orlando, FL, USA, 27\u201331 July 2016, Springer International Press.","DOI":"10.1007\/978-3-319-41685-4"},{"key":"ref_45","unstructured":"Salas, E., and Dietz, A.S. (2016). Situational Awareness Rating Technique (SART): The Development of a Tool for Aircrew Systems Design. In \u201cSituation Awareness in Aerospace Operations\u201d. Situational Awareness, Routledge."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1093\/geronj\/47.4.P250","article-title":"Influence of Direct Computer Experience on Older Adults\u2019 Attitudes toward Computers","volume":"47","author":"Jay","year":"1992","journal-title":"J. Gerontol. Psychol. Sci."},{"key":"ref_47","unstructured":"Van den Beukel, A.P., and Van der Voort, M.C. (2014, January 2\u20136). Design Considerations on User-Interaction for Semi-Automated Driving. Proceedings of the FISITA 2014 World Automotive Congress, Maastricht, The Netherlands."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1049\/iet-its:20080023","article-title":"A Method to assess the Driver Mental Workload: The Driving Activity Load Index (DALI)","volume":"2","year":"2008","journal-title":"IET Intell. Transp. Syst."},{"key":"ref_49","unstructured":"ISO 15007-2 (2013). Road Vehicles\u2014Measurement of Driver Visual Behaviour with Respect to Transport Information and Control Systems. Part. 2: Equipment and Procedures, International Organization for Standardization."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1518\/001872097778543886","article-title":"Humans and Automation: Use, Misuse, Disuse, Abuse","volume":"39","author":"Parasuraman","year":"1997","journal-title":"Hum. Factors"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/5\/240\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:51:29Z","timestamp":1760363489000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/5\/240"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,28]]},"references-count":50,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,5]]}},"alternative-id":["info11050240"],"URL":"https:\/\/doi.org\/10.3390\/info11050240","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,28]]}}}