{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T14:28:51Z","timestamp":1742912931554,"version":"3.40.3"},"publisher-location":"Cham","reference-count":22,"publisher":"Springer Nature Switzerland","isbn-type":[{"type":"print","value":"9783031830716"},{"type":"electronic","value":"9783031830723"}],"license":[{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":71,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>\n            <jats:bold>Background.<\/jats:bold> In recent years,\u00a0cyber security user studies have been scrutinized for their reporting completeness, statistical reporting fidelity, statistical reliability and biases. It remains an open question what strength\u00a0of evidence positive reports of such studies actually yield. We\u00a0focus on the extent to which positive reports indicate relations true\u00a0in reality, that is, a probabilistic assessment. <jats:bold>Aim.<\/jats:bold> This study aims at quantifying overall strength of evidence in cyber security user studies. <jats:bold>Method.<\/jats:bold> Based on 431 coded statistical inferences in 146 cyber security user studies from a published SLR covering the years 2006\u20132016, we first compute a simulation of the <jats:italic>a posteriori<\/jats:italic> false positive risk based on parametrized\u00a0prior probability, biases and effect size thresholds. Second, we establish the observed likelihood ratios for positive reports. Third,\u00a0we compute the reverse Bayesian argument on the observed positive reports by computing the prior required for a fixed <jats:italic>a posteriori<\/jats:italic> false positive rate. <jats:bold>Results.<\/jats:bold> We obtain a comprehensive analysis of the strength of evidence of the field. The simulations show that even in face of well-controlled conditions and high prior likelihoods, only few studies achieve good <jats:italic>a posteriori<\/jats:italic> probabilities. <jats:bold>Conclusions.<\/jats:bold> This\u00a0work constitutes a \u201cWhat if?\u201d analysis, which permits the reader\u00a0to evaluate the consequences of their assumptions on the state of\u00a0the field. One may stop short at the bleak conclusion that the strength of evidence of the field leaves something to be desired and\u00a0that most positive reports are likely false. At the same time, the \u201cWhat if?\u201d analysis offers a way forward to sensitize researchers to\u00a0the effects of investigating many relations and incurring biases. It, thereby, allows them to plan better ahead for future studies.<\/jats:p>","DOI":"10.1007\/978-3-031-83072-3_3","type":"book-chapter","created":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T19:00:03Z","timestamp":1741806003000},"page":"31-51","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Why Most Results of\u00a0Socio-Technical Security User Studies are False"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7766-2454","authenticated-orcid":false,"given":"Thomas","family":"Gro\u00df","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"issue":"3","key":"3_CR1","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.140216","volume":"1","author":"D Colquhoun","year":"2014","unstructured":"Colquhoun, D.: An investigation of the false discovery rate and the misinterpretation of p-values. R. Soc. Open Sci. 1(3), 140216 (2014)","journal-title":"R. Soc. Open Sci."},{"issue":"12","key":"3_CR2","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.171085","volume":"4","author":"D Colquhoun","year":"2017","unstructured":"Colquhoun, D.: The reproducibility of research and the misinterpretation of p-values. R. Soc. Open Sci. 4(12), 171085 (2017)","journal-title":"R. Soc. Open Sci."},{"key":"3_CR3","unstructured":"Coopamootoo, K., Gro\u00df, T.: Systematic evaluation for evidence-based methods in cyber security. Technical report TR-1528, Newcastle University (2017)"},{"key":"3_CR4","unstructured":"Coopamootoo, K.P., Gro\u00df, T.: A codebook for experimental research: the nifty nine indicators v1.0. Technical report. TR-1514, Newcastle University (2017)"},{"key":"3_CR5","doi-asserted-by":"crossref","unstructured":"Coopamootoo, K.P., Gro\u00df, T.: Cyber security and privacy experiments: a design and reporting toolkit. In: IFIP International Summer School on Privacy and Identity Management, pp. 243\u2013262. Springer, Cham (2017)","DOI":"10.1007\/978-3-319-92925-5_17"},{"key":"3_CR6","doi-asserted-by":"crossref","unstructured":"Coopamootoo, K.P., Gro\u00df, T.: Evidence-based methods for privacy and identity management. In: 11th International IFIP Summer School on Privacy and Identity Management. Springer, Cham (2017)","DOI":"10.1007\/978-3-319-55783-0_9"},{"key":"3_CR7","doi-asserted-by":"crossref","unstructured":"Cumming, G.: Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis. Routledge (2013)","DOI":"10.4324\/9780203807002"},{"key":"3_CR8","unstructured":"Fisher, R.A.: Statistical Methods for Research Workers. Genesis Publishing Pvt. Ltd. (1925)"},{"key":"3_CR9","unstructured":"Gro\u00df, T.: Fidelity of statistical reporting in 10 years of cyber security user studies. In: Proceedings of the 9th International Workshop on Socio-Technical Aspects in Security (STAST 2019). LNCS, vol. 11739, pp. 1\u201324. Springer, Cham (2019)"},{"key":"3_CR10","doi-asserted-by":"crossref","unstructured":"Gro\u00df, T.: Fidelity of statistical reporting in 10 years of cyber security user studies [extended version]. arXiv Report arXiv:2004.06672, Newcastle University (2020)","DOI":"10.1007\/978-3-030-55958-8_1"},{"key":"3_CR11","unstructured":"Gro\u00df, T.: Statistical reliability of 10 years of cybersecurity user studies. In: Proceedings of the 10th International Workshop on Socio-Technical Aspects in Security (STAST\u20192020). LNCS, vol. 12812, pp. 157\u2013176. Springer, Cham (2020)"},{"key":"3_CR12","unstructured":"Gro\u00df, T.: Statistical reliability of 10 years of cybersecurity user studies [extended version]. arXiv Report arXiv:2010.02117, Newcastle University (2020)"},{"key":"3_CR13","unstructured":"Howson, C., Urbach, P.: Scientific Reasoning: The Bayesian Approach. Open Court Publishing (2006)"},{"issue":"8","key":"3_CR14","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124","volume":"2","author":"JP Ioannidis","year":"2005","unstructured":"Ioannidis, J.P.: Why most published research findings are false. PLoS Med. 2(8), e124 (2005)","journal-title":"PLoS Med."},{"key":"3_CR15","unstructured":"Joyce, J.: Bayes\u2019 theorem. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2021 edn. (2021)"},{"key":"3_CR16","unstructured":"Kennedy, J.E., Watt, C.A.: How to plan falsifiable confirmatory research (2018). http:\/\/jeksite.org\/psi\/falsifiable_research.pdf"},{"key":"3_CR17","unstructured":"Lehmann, E.L., Romano, J.P.: Testing Statistical Hypotheses. Springer Texts in Statistics (2005)"},{"issue":"1","key":"3_CR18","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1016\/S0378-3758(00)00232-9","volume":"94","author":"RA Matthews","year":"2001","unstructured":"Matthews, R.A.: Why should clinicians care about Bayesian methods? J. Stat. Plan. Inference 94(1), 43\u201358 (2001)","journal-title":"J. Stat. Plan. Inference"},{"issue":"2","key":"3_CR19","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0040028","volume":"4","author":"R Moonesinghe","year":"2007","unstructured":"Moonesinghe, R., Khoury, M.J., Janssens, A.C.J.: Most published research findings are false\u2013but a little replication goes a long way. PLoS Med. 4(2), e28 (2007)","journal-title":"PLoS Med."},{"issue":"2","key":"3_CR20","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1037\/1082-989X.5.2.241","volume":"5","author":"RS Nickerson","year":"2000","unstructured":"Nickerson, R.S.: Null hypothesis significance testing: a review of an old and continuing controversy. Psychol. Methods 5(2), 241 (2000)","journal-title":"Psychol. Methods"},{"key":"3_CR21","doi-asserted-by":"crossref","unstructured":"Nuijten, M.B., van Assen, M.A., Hartgerink, C.H., Epskamp, S., Wicherts, J.: The validity of the tool \u201cstatcheck\u201d in discovering statistical reporting inconsistencies (2017). https:\/\/psyarxiv.com\/tcxaj\/","DOI":"10.31234\/osf.io\/tcxaj"},{"issue":"6","key":"3_CR22","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1093\/jnci\/djh075","volume":"96","author":"S Wacholder","year":"2004","unstructured":"Wacholder, S., Chanock, S., Garcia-Closas, M., Rothman, N., et al.: Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J. Natl Cancer Inst. 96(6), 434\u2013442 (2004)","journal-title":"J. Natl Cancer Inst."}],"container-title":["Lecture Notes in Computer Science","Socio-Technical Aspects in Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-83072-3_3","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T19:00:12Z","timestamp":1741806012000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-83072-3_3"}},"subtitle":["And What to Do About it"],"short-title":[],"issued":{"date-parts":[[2025]]},"ISBN":["9783031830716","9783031830723"],"references-count":22,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-83072-3_3","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2025]]},"assertion":[{"value":"13 March 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"STAST","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Workshop on Socio-Technical Aspects in Security","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Copenhagen","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Denmark","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2022","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 September 2022","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 September 2022","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"12","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"stast2022","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/stast.uni.lu\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Easychair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"20","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"6","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"30% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"1 external reviewers involved (outside the PC)","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}