{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T07:57:23Z","timestamp":1773820643608,"version":"3.50.1"},"reference-count":99,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2022,5,26]],"date-time":"2022-05-26T00:00:00Z","timestamp":1653523200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,5,26]],"date-time":"2022-05-26T00:00:00Z","timestamp":1653523200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005713","name":"Technische Universit\u00e4t M\u00fcnchen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005713","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Cogn Tech Work"],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Over the past two decades, systemic-based risk assessment methods have garnered more attention, and their use and popularity are growing. In particular, the functional resonance analysis method (FRAM) is one of the most widely used systemic methods for risk assessment and accident analysis. FRAM has been progressively evolved since its starting point and is considered to be the most recent and promising step in understanding socio-technical systems. However, there is currently a lack of any formal testing of the reliability and validity of FRAM, something which applies to Human Factors and Ergonomics research as a whole, where validation is both a particularly challenging issue and an ongoing concern. Therefore, this paper aims to define a more formal approach to achieving and demonstrating the reliability and validity of an FRAM model, as well as to apply this formal approach partly to an existing FRAM model so as to prove its validity. At the same time, it hopes to evaluate the general applicability of this approach to potentially improve the performance and value of the FRAM method. Thus, a formal approach was derived by transferring both the general understanding and definitions of reliability and validity as well as concrete methods and techniques to the concept of FRAM. Consequently, predictive validity, which is the highest maxim of validation, was assessed for a specific FRAM model in a driving simulator study using the signal detection theory. The results showed that the predictive validity of the FRAM model is limited and a generalisation with changing system conditions is impossible without some adaptations of the model. The applicability of the approach is diminished because of several methodological limitations. Therefore, the reliability and validity framework can be utilised to calibrate rather than validate an FRAM model.<\/jats:p>","DOI":"10.1007\/s10111-022-00701-7","type":"journal-article","created":{"date-parts":[[2022,5,26]],"date-time":"2022-05-26T04:02:48Z","timestamp":1653537768000},"page":"483-508","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Assessing the reliability and validity of an FRAM model: the case of driving in an overtaking scenario"],"prefix":"10.1007","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0405-4845","authenticated-orcid":false,"given":"Niklas","family":"Grabbe","sequence":"first","affiliation":[]},{"given":"Almin","family":"Arifagic","sequence":"additional","affiliation":[]},{"given":"Klaus","family":"Bengler","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,5,26]]},"reference":[{"key":"701_CR1","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1016\/j.ssci.2019.02.016","volume":"115","author":"B Accou","year":"2019","unstructured":"Accou B, Reniers G (2019) Developing a method to improve safety management systems based on accident investigations: the SAfety FRactal ANalysis. Saf Sci 115:285\u2013293","journal-title":"Saf Sci"},{"issue":"12","key":"701_CR2","doi-asserted-by":"publisher","first-page":"1598","DOI":"10.1080\/00140139.2019.1661527","volume":"62","author":"A Adriaensen","year":"2019","unstructured":"Adriaensen A, Patriarca R, Smoker A, Bergstr\u00f6m J (2019) A socio-technical analysis of functional properties in a joint cognitive system: a case study in an aircraft cockpit. Ergonomics 62(12):1598\u20131616","journal-title":"Ergonomics"},{"issue":"7","key":"701_CR3","doi-asserted-by":"publisher","first-page":"28","DOI":"10.3102\/0013189X031007028","volume":"31","author":"VA Anfara Jr","year":"2002","unstructured":"Anfara VA Jr, Brown KM, Mangione TL (2002) Qualitative analysis on stage: making the research process more public. Educ Res 31(7):28\u201338","journal-title":"Educ Res"},{"issue":"2","key":"701_CR01","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1080\/14639220210124067","volume":"3","author":"J Annett","year":"2002","unstructured":"Annett J (2002) A note on the validity and reliability of ergonomics methods. Theor Issues Ergon Sci 3(2):228\u2013232","journal-title":"Theor Issues Ergon Sci"},{"key":"701_CR4","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.ress.2016.10.004","volume":"158","author":"F Anvarifar","year":"2017","unstructured":"Anvarifar F, Voorendt MZ, Zevenbergen C, Thissen W (2017) An application of the functional resonance analysis method (FRAM) to risk analysis of multifunctional flood defences in the Netherlands. Reliab Eng Syst Saf 158:130\u2013141","journal-title":"Reliab Eng Syst Saf"},{"issue":"11","key":"701_CR5","doi-asserted-by":"publisher","first-page":"1923","DOI":"10.1080\/00140139408964958","volume":"37","author":"C Baber","year":"1994","unstructured":"Baber C, Stanton NA (1994) Task analysis for error identification: a methodology for designing error-tolerant consumer products. Ergonomics 37(11):1923\u20131941","journal-title":"Ergonomics"},{"issue":"2","key":"701_CR6","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/0003-6870(95)00067-4","volume":"27","author":"C Baber","year":"1996","unstructured":"Baber C, Stanton NA (1996) Human error identification techniques applied to public technology: predictions compared with observed use. Appl Ergon 27(2):119\u2013131","journal-title":"Appl Ergon"},{"key":"701_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.apergo.2021.103583","volume":"98","author":"C Baber","year":"2022","unstructured":"Baber C, Young MS (2022) Making ergonomics accountable: reliability, validity and utility in ergonomics methods. Appl Ergon 98:103583","journal-title":"Appl Ergon"},{"issue":"8","key":"701_CR8","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1002\/9780470172445.ch10","volume":"10","author":"O Balci","year":"1998","unstructured":"Balci O (1998) Verification, validation, and testing. Handb Simul 10(8):335\u2013393","journal-title":"Handb Simul"},{"key":"701_CR9","unstructured":"Banks J, Gerstein D, Searles SP (1987) Modeling processes, validation, and verification of complex simulations: a survey. In: 1987 SCS simulators conference, p 13\u201318"},{"issue":"6","key":"701_CR10","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1016\/j.apergo.2011.01.009","volume":"42","author":"MT Baysari","year":"2011","unstructured":"Baysari MT, Caponecchia C, McIntosh AS (2011) A reliability and usability study of TRACEr-RAV: the technique for the retrospective analysis of cognitive errors\u2013for rail, Australian version. Appl Ergon 42(6):852\u2013859","journal-title":"Appl Ergon"},{"key":"701_CR11","unstructured":"Blana E (1996) Driving simulator validation studies: a literature review. Working paper, Institute of Transport Studies, University of Leeds, Leeds, UK"},{"issue":"2","key":"701_CR12","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1177\/0018720817743851","volume":"60","author":"KE Bridges","year":"2018","unstructured":"Bridges KE, Corballis PM, Hollnagel E (2018) \u201cFailure-to-Identify\u201d hunting incidents: a resilience engineering approach. Hum Factors 60(2):141\u2013159","journal-title":"Hum Factors"},{"key":"701_CR13","volume-title":"Discrete system simulation","author":"WG Bulgren","year":"1982","unstructured":"Bulgren WG (1982) Discrete system simulation. Prentice Hall, Upper Saddle River"},{"issue":"1","key":"701_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12864-019-6413-7","volume":"21","author":"D Chicco","year":"2020","unstructured":"Chicco D, Jurman G (2020) The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom 21(1):1\u201313","journal-title":"BMC Genom"},{"issue":"6","key":"701_CR15","doi-asserted-by":"publisher","first-page":"1484","DOI":"10.1016\/j.apergo.2014.04.010","volume":"45","author":"M Cornelissen","year":"2014","unstructured":"Cornelissen M, McClure R, Salmon PM, Stanton NA (2014) Validating the strategies analysis diagram: assessing the reliability and validity of a formative method. Appl Ergon 45(6):1484\u20131494","journal-title":"Appl Ergon"},{"issue":"2","key":"701_CR16","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1080\/00140136308930686","volume":"6","author":"A Crawford","year":"1963","unstructured":"Crawford A (1963) The overtaking driver. Ergonomics 6(2):153\u2013170","journal-title":"Ergonomics"},{"issue":"3","key":"701_CR17","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1207\/s15430421tip3903_2","volume":"39","author":"JW Creswell","year":"2000","unstructured":"Creswell JW, Miller DL (2000) Determining validity in qualitative inquiry. Theory Pract 39(3):124\u2013130","journal-title":"Theory Pract"},{"key":"701_CR18","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1016\/j.ssci.2017.03.012","volume":"119","author":"C Dallat","year":"2017","unstructured":"Dallat C, Salmon PM, Goode N (2017) Risky systems versus risky people: to what extent do risk assessment methods consider the systems approach to accident causation? A review of the literature. Saf Sci 119:266\u2013279","journal-title":"Saf Sci"},{"key":"701_CR19","unstructured":"Espi\u00e9 S, Gauriat P, Duraz M (2005) Driving simulators validation: the issue of transferability of results acquired on simulator. In: Driving simulation conference North-America (DSC-NA 2005), Orlondo, FL"},{"key":"701_CR20","first-page":"280","volume-title":"Elemente der Psychophysik [elements of psychophysics]","author":"GT Fechner","year":"1860","unstructured":"Fechner GT (1860) Elemente der Psychophysik [elements of psychophysics]. Breitkopf und H\u00e4rtel, Leipzig, pp 280\u2013286"},{"key":"701_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10111-019-00540-z","volume":"21","author":"PN Ferreira","year":"2019","unstructured":"Ferreira PN, Ca\u00f1as JJ (2019) Assessing operational impacts of automation using functional resonance analysis method. Cognit Technol Work 21:1\u201318","journal-title":"Cognit Technol Work"},{"key":"701_CR22","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.apergo.2017.04.014","volume":"64","author":"N Goode","year":"2017","unstructured":"Goode N, Salmon PM, Taylor NZ, Lenn\u00e9 MG, Finch CF (2017) Developing a contributing factor classification scheme for Rasmussen\u2019s AcciMap: reliability and validity evaluation. Appl Ergon 64:14\u201326","journal-title":"Appl Ergon"},{"key":"701_CR23","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2020.104665","volume":"126","author":"N Grabbe","year":"2020","unstructured":"Grabbe N, Kellnberger A, Aydin B, Bengler K (2020) Safety of automated driving: the need for a systems approach and application of the functional resonance analysis method. Saf Sci 126:104665","journal-title":"Saf Sci"},{"issue":"1","key":"701_CR24","doi-asserted-by":"publisher","first-page":"3","DOI":"10.3390\/safety8010003","volume":"8","author":"N Grabbe","year":"2022","unstructured":"Grabbe N, Gales A, H\u00f6cher M, Bengler K (2022) Functional resonance analysis in an overtaking situation in road traffic: comparing the performance variability mechanisms between human and automation. Safety 8(1):3","journal-title":"Safety"},{"key":"701_CR25","volume-title":"Einf\u00fchrung in die Systemforschung","author":"F Hanssmann","year":"2018","unstructured":"Hanssmann F (2018) Einf\u00fchrung in die Systemforschung. Oldenbourg Wissenschaftsverlag, Munich"},{"issue":"6","key":"701_CR26","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1016\/j.ast.2005.04.002","volume":"9","author":"D Harris","year":"2005","unstructured":"Harris D, Stanton NA, Marshall A, Young MS, Demagalski J, Salmon P (2005) Using SHERPA to predict design-induced error on the flight deck. Aerosp Sci Technol 9(6):525\u2013532","journal-title":"Aerosp Sci Technol"},{"key":"701_CR27","unstructured":"Hill R, Boult M, Sujan M, Hollnagel E, Slater D (2020) Predictive analysis of complex systems\u2019 behaviour. https:\/\/www.researchgate.net\/profile\/David-Slater\/publication\/343944100_PREDICTIVE_ANALYSIS_OF_COMPLEX_SYSTEMS'_BEHAVIOUR_SWIFTFRAM\/links\/5f4907e0299bf13c5047f8d3\/PREDICTIVE-ANALYSIS-OF-COMPLEX-SYSTEMS-BEHAVIOUR-SWIFTFRAM.pdf. Accessed 18 Nov 2021"},{"key":"701_CR28","volume-title":"Barriers and accident prevention","author":"E Hollnagel","year":"2004","unstructured":"Hollnagel E (2004) Barriers and accident prevention. Ashgate, Hampshire"},{"key":"701_CR29","volume-title":"FRAM: the functional resonance analysis method: modelling complex socio-technical systems","author":"E Hollnagel","year":"2012","unstructured":"Hollnagel E (2012) FRAM: the functional resonance analysis method: modelling complex socio-technical systems. CRC Press, Boca Raton"},{"key":"701_CR30","volume-title":"Safety\u2013I and safety\u2013II: the past and future of safety management","author":"E Hollnagel","year":"2014","unstructured":"Hollnagel E (2014) Safety\u2013I and safety\u2013II: the past and future of safety management. CRC Press, Boca Raton"},{"key":"701_CR31","unstructured":"Hollnagel E (2020) FRAM model interpreter. https:\/\/functionalresonance.com\/onewebmedia\/FMI%20basicPlus%20V3.pdf. 09 Nov 2021"},{"key":"701_CR32","unstructured":"Hollnagel E, Hounsgaard J, Colligan L (2014) FRAM\u2014the functional resonance analysis method\u2014a handbook for the practical use of the method. https:\/\/functionalresonance.com\/onewebmedia\/FRAM_handbook_web-2.pdf. 17 Nov 2021"},{"issue":"3","key":"701_CR33","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1504\/IJHVS.2006.010020","volume":"13","author":"AH Hoskins","year":"2006","unstructured":"Hoskins AH, El-Gindy M (2006) Technical report: Literature survey on driving simulator validation studies. Int J Heavy Veh Syst 13(3):241\u2013252","journal-title":"Int J Heavy Veh Syst"},{"key":"701_CR34","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1016\/j.aap.2014.06.003","volume":"74","author":"BP Hughes","year":"2015","unstructured":"Hughes BP, Newstead S, Anund A, Shu CC, Falkmer T (2015) A review of models relevant to road safety. Accid Anal Prev 74:250\u2013270","journal-title":"Accid Anal Prev"},{"key":"701_CR35","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1016\/j.ssci.2019.04.016","volume":"117","author":"A Hulme","year":"2019","unstructured":"Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM (2019) What do applications of systems thinking accident analysis methods tell us about accident causation? A systematic review of applications between 1990 and 2018. Saf Sci 117:164\u2013183","journal-title":"Saf Sci"},{"key":"701_CR36","doi-asserted-by":"crossref","unstructured":"Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM (2021a) Testing the reliability and validity of Net-HARMS: a new systems-based risk assessment method in HFE. In: Congress of the International Ergonomics Association, Springer, Cham, p 354\u2013362","DOI":"10.1007\/978-3-030-74602-5_51"},{"key":"701_CR37","first-page":"1","volume":"65","author":"A Hulme","year":"2021","unstructured":"Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM (2021b) Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics. Ergonomics 65:1\u201322","journal-title":"Ergonomics"},{"key":"701_CR38","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2021.105454","volume":"144","author":"A Hulme","year":"2021","unstructured":"Hulme A, Stanton NA, Walker GH, Waterson P, Salmon PM (2021c) Are accident analysis methods fit for purpose? Testing the criterion-referenced concurrent validity of AcciMap, STAMP-CAST and AcciNet. Saf Sci 144:105454","journal-title":"Saf Sci"},{"key":"701_CR39","unstructured":"ISO Standard 26262 (2018) Road vehicles\u2014functional safety\u2014part 3: concept phase. https:\/\/www.iso.org\/standard\/68385.html. Accessed 16 Dec 2021"},{"key":"701_CR40","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1016\/j.ress.2017.11.018","volume":"171","author":"A Jensen","year":"2018","unstructured":"Jensen A, Aven T (2018) A new definition of complexity in a risk analysis setting. Reliab Eng Syst Saf 171:169\u2013173","journal-title":"Reliab Eng Syst Saf"},{"issue":"1","key":"701_CR41","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1177\/0361198196155000105","volume":"1550","author":"NA Kaptein","year":"1996","unstructured":"Kaptein NA, Theeuwes J, Van Der Horst R (1996) Driving simulator validity: some considerations. Transp Res Rec 1550(1):30\u201336","journal-title":"Transp Res Rec"},{"key":"701_CR42","doi-asserted-by":"publisher","first-page":"835","DOI":"10.1016\/j.ssci.2019.06.020","volume":"118","author":"GK Kaya","year":"2019","unstructured":"Kaya GK, Ovali HF, Ozturk F (2019) Using the functional resonance analysis method on the drug administration process to assess performance variability. Saf Sci 118:835\u2013840","journal-title":"Saf Sci"},{"issue":"1","key":"701_CR43","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/S0003-6870(96)00045-2","volume":"28","author":"B Kirwan","year":"1997","unstructured":"Kirwan B, Kennedy R, Taylor-Adams S, Lambert B (1997) The validation of three human reliability quantification techniques\u2014THERP, HEART and JHEDI: Part II\u2014results of validation exercise. Appl Ergon 28(1):17\u201325","journal-title":"Appl Ergon"},{"key":"701_CR44","doi-asserted-by":"crossref","unstructured":"Laaraj N, Jawab F (2018) Road accident modeling approaches: literature review. In: 2018 International colloquium on Lo-1769 gistics and supply chain management (LOGISTIQUA). IEEE, p 188\u2013193","DOI":"10.1109\/LOGISTIQUA.2018.8428276"},{"issue":"9","key":"701_CR45","doi-asserted-by":"publisher","first-page":"1167","DOI":"10.1016\/j.ssci.2009.10.006","volume":"48","author":"P Larsson","year":"2010","unstructured":"Larsson P, Dekker SW, Tingvall C (2010) The need for a systems theory approach to road safety. Saf Sci 48(9):1167\u20131174","journal-title":"Saf Sci"},{"issue":"6","key":"701_CR46","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1177\/0018720818783507","volume":"60","author":"GS Larue","year":"2018","unstructured":"Larue GS, Wullems C, Sheldrake M, Rakotonirainy A (2018) Validation of a driving simulator study on driver behavior at passive rail level crossings. Hum Factors 60(6):743\u2013754","journal-title":"Hum Factors"},{"issue":"4","key":"701_CR47","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/S0925-7535(03)00047-X","volume":"42","author":"N Leveson","year":"2004","unstructured":"Leveson N (2004) A new accident model for engineering safer systems. Saf Sci 42(4):237\u2013270","journal-title":"Saf Sci"},{"key":"701_CR48","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1016\/j.ress.2019.02.012","volume":"186","author":"W Li","year":"2019","unstructured":"Li W, He M, Sun Y, Cao Q (2019) A proactive operational risk identification and analysis framework based on the integration of ACAT and FRAM. Reliab Eng Syst Saf 186:101\u2013109","journal-title":"Reliab Eng Syst Saf"},{"key":"701_CR49","volume-title":"Simulation","author":"F Liebl","year":"2018","unstructured":"Liebl F (2018) Simulation. Oldenbourg Wissenschaftsverlag, Munich"},{"issue":"1","key":"701_CR50","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s41077-021-00166-0","volume":"6","author":"RJ MacKinnon","year":"2021","unstructured":"MacKinnon RJ, Pukk-H\u00e4renstam K, Kennedy C, Hollnagel E, Slater D (2021) A novel approach to explore safety-I and safety-II perspectives in in situ simulations\u2014the structured what if functional resonance analysis methodology. Adv Simul 6(1):1\u201313","journal-title":"Adv Simul"},{"issue":"1","key":"701_CR51","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1136\/qshc.2007.022491","volume":"17","author":"MA Makeham","year":"2008","unstructured":"Makeham MA, Stromer S, Bridges-Webb C, Mira M, Saltman DC, Cooper C, Kidd MR (2008) Patient safety events reported in general practice: a taxonomy. BMJ Qual Saf 17(1):53\u201357","journal-title":"BMJ Qual Saf"},{"issue":"2","key":"701_CR52","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","volume":"405","author":"BW Matthews","year":"1975","unstructured":"Matthews BW (1975) Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct 405(2):442\u2013451","journal-title":"Biochim Biophys Acta (BBA) Protein Struct"},{"key":"701_CR53","first-page":"1","volume-title":"Handbook of driving simulation for engineering, medicine and psychology","author":"N Mullen","year":"2011","unstructured":"Mullen N, Charlton J, Devlin A, Bedard M (2011) Simulator validity: behaviours observed on the simulator and on the road. In: Fisher DL, Rizzo M, Caird JK, Lee JD (eds) Handbook of driving simulation for engineering, medicine and psychology, 1st edn. CRC Press, Boca Raton, pp 1\u201318","edition":"1"},{"issue":"15","key":"701_CR54","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1007\/s10111-012-0246-3","volume":"1","author":"C Nemeth","year":"2013","unstructured":"Nemeth C (2013) Erik Hollnagel: FRAM: the functional resonance analysis method, modeling complex socio-technical systems. Cogn Technol Work 1(15):117\u2013118","journal-title":"Cogn Technol Work"},{"key":"701_CR55","doi-asserted-by":"crossref","unstructured":"Nilsson L (1993) Behavioural research in an advanced driving simulator-experiences of the VTI system. In: Proceedings of the human factors and ergonomics society annual meeting, vol 37, no 9. Sage: Los Angeles, p 612\u2013616","DOI":"10.1177\/154193129303700921"},{"issue":"6","key":"701_CR56","doi-asserted-by":"publisher","first-page":"599","DOI":"10.3357\/ASEM.2228.2008","volume":"79","author":"P O'Connor","year":"2008","unstructured":"O\u2019Connor P (2008) HFACS with an additional layer of granularity: validity and utility in accident analysis. Aviat Space Environ Med 79(6):599\u2013606","journal-title":"Aviat Space Environ Med"},{"issue":"2","key":"701_CR57","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1016\/j.apergo.2012.06.009","volume":"44","author":"NS Olsen","year":"2013","unstructured":"Olsen NS (2013) Reliability studies of incident coding systems in high hazard industries: a narrative review of study methodology. Appl Ergon 44(2):175\u2013184","journal-title":"Appl Ergon"},{"issue":"2","key":"701_CR58","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1016\/j.aap.2009.09.005","volume":"42","author":"NS Olsen","year":"2010","unstructured":"Olsen NS, Shorrock ST (2010) Evaluation of the HFACS-ADF safety classification system: inter-coder consensus and intra-coder consistency. Accid Anal Prev 42(2):437\u2013444","journal-title":"Accid Anal Prev"},{"issue":"4","key":"701_CR59","doi-asserted-by":"publisher","first-page":"711","DOI":"10.1007\/s10111-017-0426-2","volume":"19","author":"R Patriarca","year":"2017","unstructured":"Patriarca R, Bergstr\u00f6m J (2017) Modelling complexity in everyday operations: functional resonance in maritime mooring at quay. Cogn Technol Work 19(4):711\u2013729","journal-title":"Cogn Technol Work"},{"key":"701_CR60","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.ress.2017.03.032","volume":"165","author":"R Patriarca","year":"2017","unstructured":"Patriarca R, Bergstr\u00f6m J, Di Gravio G (2017) Defining the functional resonance analysis space: combining abstraction hierarchy and FRAM. Reliab Eng Syst Saf 165:34\u201346","journal-title":"Reliab Eng Syst Saf"},{"key":"701_CR61","doi-asserted-by":"publisher","DOI":"10.1016\/j.ssci.2020.104827","volume":"129","author":"R Patriarca","year":"2020","unstructured":"Patriarca R, Di Gravio G, Woltjer R, Costantino F, Praetorius G, Ferreira P, Hollnagel E (2020) Framing the FRAM: a literature review on the functional resonance analysis method. Saf Sci 129:104827","journal-title":"Saf Sci"},{"key":"701_CR62","unstructured":"Pereira AG (2013) Introduction to the Use of FRAM on the effectiveness assessment of a radiopharmaceutical dispatches process. In: International nuclear Atlantic conference"},{"key":"701_CR63","first-page":"267","volume":"16","author":"M Pollatschek","year":"2005","unstructured":"Pollatschek M, Polus A (2005) Modelling impatience of driver in passing manuevers. Transp Traffic Theory 16:267\u2013279","journal-title":"Transp Traffic Theory"},{"key":"701_CR64","unstructured":"Qureshi ZH (2007) A review of accident modelling approaches for complex socio-technical systems. In: Proceedings of the 1757 twelfth Australian workshop on Safety critical systems and software and safety-related programmable systems, vol 86. Australian Computer Society, Inc., p 1758 47\u201359"},{"issue":"2\u20133","key":"701_CR65","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1016\/S0925-7535(97)00052-0","volume":"27","author":"J Rasmussen","year":"1997","unstructured":"Rasmussen J (1997) Risk management in a dynamic society: a modelling problem. Saf Sci 27(2\u20133):183\u2013213","journal-title":"Saf Sci"},{"key":"701_CR66","doi-asserted-by":"crossref","unstructured":"Rastegary H, Landy FJ (1993) The interactions among time urgency, uncertainty, and time pressure. In: Time pressure and stress in human judgment and decision making. Springer, Boston, p 217\u2013239","DOI":"10.1007\/978-1-4757-6846-6_15"},{"key":"701_CR67","doi-asserted-by":"publisher","first-page":"294","DOI":"10.1016\/j.apergo.2017.12.005","volume":"68","author":"A Ross","year":"2018","unstructured":"Ross A, Sherriff A, Kidd J, Gnich W, Anderson J, Deas L, Macpherson L (2018) A systems approach using the functional resonance analysis method to support fluoride varnish application for children attending general dental practice. Appl Ergon 68:294\u2013303","journal-title":"Appl Ergon"},{"issue":"1","key":"701_CR68","doi-asserted-by":"publisher","first-page":"118","DOI":"10.1002\/hfm.20874","volume":"31","author":"V Salehi","year":"2021","unstructured":"Salehi V, Veitch B, Smith D (2021) Modeling complex socio-technical systems using the FRAM: a literature review. Hum Factors Ergon Manuf Serv Ind 31(1):118\u2013142","journal-title":"Hum Factors Ergon Manuf Serv Ind"},{"issue":"9","key":"701_CR69","doi-asserted-by":"publisher","first-page":"1829","DOI":"10.1016\/j.ssci.2012.04.011","volume":"50","author":"PM Salmon","year":"2012","unstructured":"Salmon PM, McClure R, Stanton NA (2012) Road transport in drift? Applying contemporary systems thinking to road safety. Saf Sci 50(9):1829\u20131838","journal-title":"Saf Sci"},{"key":"701_CR70","doi-asserted-by":"publisher","DOI":"10.1002\/hfm.20873","author":"PM Salmon","year":"2020","unstructured":"Salmon PM, Read GJ, Walker GH, Stevens NJ, Hulme A, McLean S, Stanton NA (2020) Methodological issues in systems Human Factors and Ergonomics: perspectives on the research\u2013practice gap, reliability and validity, and prediction. Hum Factors Ergon Manuf Serv Ind. https:\/\/doi.org\/10.1002\/hfm.20873","journal-title":"Hum Factors Ergon Manuf Serv Ind"},{"key":"701_CR71","unstructured":"Sargent RG (1984) A tutorial on verification and validation of simulation models. In: Proceedings of the 16th conference on Winter simulation, pp 115\u2013121. https:\/\/repository.lib.ncsu.edu\/bitstream\/handle\/1840.4\/4929\/1984_0017.pdf?sequence=1"},{"issue":"2","key":"701_CR72","doi-asserted-by":"publisher","first-page":"B-104","DOI":"10.1287\/mnsc.14.2.B104","volume":"14","author":"WE Schrank","year":"1967","unstructured":"Schrank WE, Holt CC (1967) Critique of: \u201cVerification of computer simulation models.\u201d Manag Sci 14(2):B-104","journal-title":"Manag Sci"},{"key":"701_CR73","volume-title":"Telerobotics, automation, and human supervisory control","author":"TB Sheridan","year":"1992","unstructured":"Sheridan TB (1992) Telerobotics, automation, and human supervisory control. MIT Press, Cambridge"},{"issue":"1","key":"701_CR74","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1080\/1463922X.2013.778355","volume":"15","author":"NA Stanton","year":"2014","unstructured":"Stanton NA (2014) Commentary on the paper by Heimrich Kanis entitled \u2018Reliability and validity of findings in ergonomics research\u2019: where is the methodology in ergonomics methods? Theor Issues Ergon Sci 15(1):55\u201361","journal-title":"Theor Issues Ergon Sci"},{"issue":"4","key":"701_CR75","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1080\/1463922X.2015.1117688","volume":"17","author":"NA Stanton","year":"2016","unstructured":"Stanton NA (2016) On the reliability and validity of, and training in, ergonomics methods: a challenge revisited. Theor Issues Ergon Sci 17(4):345\u2013353","journal-title":"Theor Issues Ergon Sci"},{"issue":"9","key":"701_CR76","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1080\/00140130500219726","volume":"48","author":"NA Stanton","year":"2005","unstructured":"Stanton NA, Baber C (2005) Validating task analysis for error identification: reliability and validity of a human error prediction technique. Ergonomics 48(9):1097\u20131113","journal-title":"Ergonomics"},{"issue":"11","key":"701_CR77","doi-asserted-by":"publisher","first-page":"1737","DOI":"10.1080\/001401398186162","volume":"41","author":"NA Stanton","year":"1998","unstructured":"Stanton NA, Stevenage SV (1998) Learning to predict human error: issues of acceptability, reliability and validity. Ergonomics 41(11):1737\u20131756","journal-title":"Ergonomics"},{"issue":"6733","key":"701_CR78","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1038\/20298","volume":"399","author":"NA Stanton","year":"1999","unstructured":"Stanton NA, Young MS (1999a) What price ergonomics? Nature 399(6733):197\u2013198","journal-title":"Nature"},{"key":"701_CR79","volume-title":"A guide to methodology in ergonomics: designing for human use","author":"NA Stanton","year":"1999","unstructured":"Stanton NA, Young MS (1999b) A guide to methodology in ergonomics: designing for human use. Taylor & Francis, London"},{"issue":"5","key":"701_CR80","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1016\/S0003-6870(03)00067-X","volume":"34","author":"NA Stanton","year":"2003","unstructured":"Stanton NA, Young MS (2003) Giving ergonomics away? The application of ergonomics methods by novices. Appl Ergon 34(5):479\u2013490","journal-title":"Appl Ergon"},{"issue":"3","key":"701_CR81","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1016\/j.apergo.2008.10.005","volume":"40","author":"NA Stanton","year":"2009","unstructured":"Stanton NA, Salmon P, Harris D, Marshall A, Demagalski J, Young MS, Dekker S et al (2009) Predicting pilot error: testing a new methodology and a multi-methods and analysts approach. Appl Ergon 40(3):464\u2013471","journal-title":"Appl Ergon"},{"key":"701_CR82","volume-title":"Human factors methods: a practical guide for engineering and design","author":"NA Stanton","year":"2013","unstructured":"Stanton NA, Salmon PM, Rafferty LA, Walker GH, Baber C, Jenkins DP (2013) Human factors methods: a practical guide for engineering and design. CRC Press, Boca Raton"},{"key":"701_CR83","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1201\/9781003050841-19","volume-title":"Designing interaction and interfaces for automated vehicles","author":"NA Stanton","year":"2021","unstructured":"Stanton NA, Brown JW, Revell KM, Clark JR, Richardson J, Langdon P et al (2021a) Modelling automation-human driver interactions in vehicle takeovers using OESDs. Designing interaction and interfaces for automated vehicles. CRS Press, Boca Raton, pp 299\u2013320"},{"key":"701_CR84","first-page":"1","volume":"24","author":"NA Stanton","year":"2021","unstructured":"Stanton NA, Brown JW, Revell KM, Kim J, Richardson J, Langdon P et al (2021b) OESDs in an on-road study of semi-automated vehicle to human driver handovers. Cognit Technol Work 24:1\u201316","journal-title":"Cognit Technol Work"},{"key":"701_CR85","doi-asserted-by":"crossref","unstructured":"Stanton NA, Brown JW, Revell KM, Kim J, Richardson J, Langdon P et al (2021c) Validating OESDs in an on-road study of semi-automated vehicle-to-human driver takeovers. In: Designing interaction and interfaces for automated vehicles. CRC Press, Boca Raton, p 443\u2013464b","DOI":"10.1201\/9781003050841-25"},{"key":"701_CR86","doi-asserted-by":"crossref","unstructured":"Stanton NA, Brown JW, Revell KM, Langdon P, Bradley M, Politis I et al (2021d) Validating operator event sequence diagrams: the case of automated vehicle-to-human driver takeovers. In: Designing interaction and interfaces for automated vehicles. CRC Press, Boca Raton, p 137\u2013157","DOI":"10.1201\/9781003050841-10"},{"key":"701_CR87","volume-title":"Ohitukset kaksikaistaisilla teilla (Summary in English)","author":"J Tapio","year":"2003","unstructured":"Tapio J (2003) Ohitukset kaksikaistaisilla teilla (Summary in English). Finnish Road Administration, Helsinki"},{"key":"701_CR88","unstructured":"Underwood P, Waterson P (2012) A critical review of the STAMP, FRAM and Accimap systemic accident analysis models. In: Advances in human aspects of road and rail transportation. CRC Press, Boca Raton, pp 385\u2013394"},{"issue":"5","key":"701_CR89","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1287\/mnsc.17.5.247","volume":"17","author":"RL Van Horn","year":"1971","unstructured":"Van Horn RL (1971) Validation of simulation results. Manag Sci 17(5):247\u2013258","journal-title":"Manag Sci"},{"issue":"6","key":"701_CR90","doi-asserted-by":"publisher","first-page":"776","DOI":"10.15837\/ijccc.2014.6.1484","volume":"9","author":"F Vanderhaegen","year":"2014","unstructured":"Vanderhaegen F (2014) Dissonance engineering: a new challenge to analyse risky knowledge when using a system. Int J Comput Commun Control 9(6):776\u2013785","journal-title":"Int J Comput Commun Control"},{"key":"701_CR91","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1016\/j.eswa.2016.08.071","volume":"65","author":"F Vanderhaegen","year":"2016","unstructured":"Vanderhaegen F (2016) A rule-based support system for dissonance discovery and control applied to car driving. Expert Syst Appl 65:361\u2013371","journal-title":"Expert Syst Appl"},{"key":"701_CR92","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2021.103867","volume":"146","author":"F Vanderhaegen","year":"2021","unstructured":"Vanderhaegen F (2021) Heuristic-based method for conflict discovery of shared control between humans and autonomous systems\u2014a driving automation case study. Robot Auton Syst 146:103867","journal-title":"Robot Auton Syst"},{"key":"701_CR93","first-page":"S.3","volume-title":"Behavioural adaptation in response to driving assistance technologies: a literature review. Driver adaptation to information and assistance systems","author":"CA Wege","year":"2014","unstructured":"Wege CA, Pereira M, Victor TW, Krems JF, Stevens A, Brusque C (2014) Behavioural adaptation in response to driving assistance technologies: a literature review. Driver adaptation to information and assistance systems. The Institution of Engineering and Technology, London, p S.3-34"},{"key":"701_CR94","unstructured":"Wienen HCA, Bukhsh FA, Vriezekolk E, Wieringa RJ (2017) Accident analysis methods and models\u2014a systematic literature 1767 review. In: Centre for Telematics and Information Technology (CTIT), p 1768"},{"key":"701_CR95","unstructured":"Woltjer R, Hollnagel E (2008) Functional modeling for risk assessment of automation in a changing air traffic management environment. In: Proceedings of the 4th international conference working on safety, vol 30"},{"key":"701_CR96","doi-asserted-by":"publisher","first-page":"162","DOI":"10.1016\/j.ssci.2016.12.006","volume":"93","author":"Q Yang","year":"2017","unstructured":"Yang Q, Tian J, Zhao T (2017) Safety is an emergent property: illustrating functional resonance in air traffic management with formal verification. Saf Sci 93:162\u2013177","journal-title":"Saf Sci"},{"key":"701_CR97","doi-asserted-by":"crossref","unstructured":"Yang Q, Tian J (2015) Model-based safety assessment using FRAM for complex systems. In: Proceedings of the 25th European safety and reliability conference","DOI":"10.1201\/b19094-520"},{"key":"701_CR98","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1016\/j.patrec.2020.03.030","volume":"136","author":"Q Zhu","year":"2020","unstructured":"Zhu Q (2020) On the performance of Matthews correlation coefficient (MCC) for imbalanced dataset. Pattern Recognit Lett 136:71\u201380","journal-title":"Pattern Recognit Lett"}],"container-title":["Cognition, Technology &amp; Work"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10111-022-00701-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10111-022-00701-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10111-022-00701-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,6]],"date-time":"2023-02-06T07:36:29Z","timestamp":1675668989000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10111-022-00701-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,26]]},"references-count":99,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["701"],"URL":"https:\/\/doi.org\/10.1007\/s10111-022-00701-7","relation":{},"ISSN":["1435-5558","1435-5566"],"issn-type":[{"value":"1435-5558","type":"print"},{"value":"1435-5566","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,26]]},"assertion":[{"value":"21 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 May 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 May 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to dis-close. In addition, no funds, grants or other support were received.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}},{"value":"The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of the Ethics Committee of the Technical University of Munich (protocol code 509\/21 S and date of approval 23 August 2021).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Institutional review board statement"}},{"value":"Informed consent was obtained from all subjects involved in the study.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}}]}}