{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T10:08:52Z","timestamp":1780654132674,"version":"3.54.1"},"update-to":[{"DOI":"10.12688\/wellcomeopenres.22288.1","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T00:00:00Z","timestamp":1778112000000}}],"reference-count":76,"publisher":"F1000 Research Ltd","license":[{"start":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T00:00:00Z","timestamp":1778112000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["214251"],"award-info":[{"award-number":["214251"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["203139"],"award-info":[{"award-number":["203139"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000319","name":"ARUK","doi-asserted-by":"crossref","award":["21537"],"award-info":[{"award-number":["21537"]}],"id":[{"id":"10.13039\/501100000319","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000266","name":"EPSRC","doi-asserted-by":"crossref","award":["EP\/W03509X\/1"],"award-info":[{"award-number":["EP\/W03509X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"crossref"}]},{"name":"IITP","award":["MSIT2019-0-01371"],"award-info":[{"award-number":["MSIT2019-0-01371"]}]},{"name":"MRC 713","award":["MR\/W027593\/1"],"award-info":[{"award-number":["MR\/W027593\/1"]}]},{"name":"JSPS KAKENHI","award":["22H04998"],"award-info":[{"award-number":["22H04998"]}]}],"content-domain":{"domain":["wellcomeopenresearch.org"],"crossmark-restriction":false},"short-container-title":["Wellcome Open Res"],"abstract":"<ns5:p>Bayesian statistics offers a flexible framework that supports iterative updating of hypotheses and the incorporation of prior information, amongst other advantages. Although well established for retrospective analysis, the application of Bayesian methods to prospective analysis is less well developed, especially when used in combination with computational model-based analysis of behavioural data in cognitive neuroscience. It is therefore important to establish effective methods for testing and optimising experimental designs for these purposes. One framework for a prospective approach is Bayes factor design analysis (BFDA), which can be used alongside latent variable modelling to evaluate and visualise the distribution of Bayes factors for a given experimental design. This paper provides a tutorial-style analysis combining BFDA with latent variable modelling to evaluate exploration-exploitation trade-offs in the binary multi-armed bandit task (MAB). This is a complex example of human decision-making with which to investigate the feasibility of differentiating latent variables between groups as a function of different design parameters. We examined how sample size, number of games per participant and effect size affect the strength of evidence supporting a difference in means between two groups. To further assess how these parameters affect experimental results, metrics of error were evaluated. Using simulations, we demonstrated how BFDA can be combined with latent variable modelling to evaluate and optimise parameter estimation of exploration in the MAB task, allowing inference of the mean degree of random exploration in a population, as well as between groups. However, BFDA indicated that, even with large samples and effect sizes, there may be some circumstances where there is a high likelihood of errors and a low probability of detecting evidence in favour of a difference when comparing random exploration between two groups performing the bandit task. In summary, we show how BFDA can prospectively inform design and power of human behavioural tasks.<\/ns5:p>","DOI":"10.12688\/wellcomeopenres.22288.2","type":"journal-article","created":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T10:14:07Z","timestamp":1778148847000},"page":"423","update-policy":"https:\/\/doi.org\/10.12688\/wellcomeopenres.crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Enhancing experimental design through Bayes factor design analysis: insights from multi-armed bandit tasks"],"prefix":"10.12688","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-7827-9682","authenticated-orcid":false,"given":"Sarah","family":"Schreiber","sequence":"first","affiliation":[{"name":"Institute of Biomedical Engineering Oxford England OX37DQ University of Oxford"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2245-4962","authenticated-orcid":false,"given":"Danielle","family":"Hewitt","sequence":"additional","affiliation":[{"name":"Wellcome Centre for Integrative Neuroimaging Oxford England OX39DU University of Oxford"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"Seymour","sequence":"additional","affiliation":[{"name":"Institute of Biomedical Engineering Oxford England OX37DQ University of Oxford"},{"name":"Wellcome Centre for Integrative Neuroimaging Oxford England OX39DU University of Oxford"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9273-1617","authenticated-orcid":false,"given":"Wako","family":"Yoshida","sequence":"additional","affiliation":[{"name":"Wellcome Centre for Integrative Neuroimaging Oxford England OX39DU University of Oxford"},{"name":"Department of Neural Computation for Decision-making Kyoto Advanced Telecommunications Research Institute International"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"2560","published-online":{"date-parts":[[2026,5,7]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","first-page":"561-563","DOI":"10.1038\/s41562-019-0807-z","article-title":"Discussion points for Bayesian inference.","volume":"4","author":"B Aczel","year":"2020","journal-title":"Nature Human Behaviour."},{"key":"ref2","doi-asserted-by":"publisher","first-page":"3589-3631","DOI":"10.1002\/sim.2672","article-title":"Bayesian statistics in medicine: a 25 year review.","volume":"25","author":"D Ashby","year":"2006","journal-title":"Stat Med."},{"key":"ref3","doi-asserted-by":"publisher","first-page":"103576","DOI":"10.1016\/j.brat.2020.103576","article-title":"Psychophysiological modelling and the measurement of fear conditioning.","volume":"127","author":"D Bach","year":"2020","journal-title":"Behaviour Research and Therapy."},{"key":"ref4","doi-asserted-by":"publisher","first-page":"1229-1235","DOI":"10.1038\/s41562-020-00976-8","article-title":"Calibrating the experimental measurement of psychological attributes.","volume":"4","author":"D Bach","year":"2020","journal-title":"Nature Human Behaviour."},{"key":"ref5","doi-asserted-by":"publisher","first-page":"105146","DOI":"10.1016\/j.neubiorev.2023.105146","article-title":"Consensus design of a calibration experiment for human fear conditioning.","volume":"148","author":"D Bach","year":"2023","journal-title":"Neurosci Biobehav Rev."},{"key":"ref6","doi-asserted-by":"publisher","first-page":"295-314","DOI":"10.1037\/met0000337","article-title":"Power contours: optimising sample size and precision in experimental psychology and human neuroscience.","volume":"26","author":"D Baker","year":"2021","journal-title":"Psychol Methods."},{"key":"ref7","first-page":"157-171","article-title":"An essay towards solving a problem in the doctrine of chances. 1763.","volume":"8","author":"T Bayes","year":"1991","journal-title":"MD Comput."},{"key":"ref8","doi-asserted-by":"publisher","first-page":"6-10","DOI":"10.1038\/s41562-017-0189-z","article-title":"Redefine statistical significance.","volume":"2","author":"D Benjamin","year":"2018","journal-title":"Nat Hum Behav."},{"key":"ref9","doi-asserted-by":"publisher","first-page":"1025-1026","DOI":"10.1214\/aos\/1176345895","article-title":"A Bayes but Not Classically Sufficient Statistic.","volume":"10","author":"D Blackwell","year":"1982","journal-title":"Ann Stat."},{"key":"ref10","doi-asserted-by":"publisher","first-page":"4111-4122","DOI":"10.1016\/j.jspi.2009.05.041","article-title":"Comparison of Bayesian sample size criteria: ACC, ALC, and WOC.","volume":"139","author":"J Cao","year":"2009","journal-title":"J Stat Plan Inference."},{"key":"ref11","doi-asserted-by":"publisher","first-page":"29-46","DOI":"10.1016\/j.neubiorev.2018.12.022","article-title":"Neuroimaging of individual differences: a latent variable modeling perspective.","volume":"98","author":"S Cooper","year":"2019","journal-title":"Neurosci Biobehav Rev."},{"key":"ref12","doi-asserted-by":"publisher","first-page":"ENEURO.0484-0423.2024","DOI":"10.1523\/ENEURO.0484-23.2024","article-title":"Practical Bayesian Inference in Neuroscience: Or How I Learned to Stop Worrying and Embrace the Distribution.","volume":"11","author":"B Coventry","year":"2024","journal-title":"eNeuro."},{"key":"ref13","doi-asserted-by":"publisher","first-page":"591-602","DOI":"10.1152\/jn.00379.2009","article-title":"Hierarchical Bayesian Modeling and Markov Chain Monte Carlo Sampling for Tuning-Curve Analysis.","volume":"103","author":"B Cronin","year":"2010","journal-title":"Journal of Neurophysiology."},{"key":"ref14","doi-asserted-by":"publisher","first-page":"171-198","DOI":"10.1137\/1.9780898719765.ch14","author":"V Czitrom","year":"2012","journal-title":"Statistical Case Studies for Industrial Process Improvement."},{"key":"ref15","doi-asserted-by":"publisher","first-page":"547-563","DOI":"10.1007\/s42113-022-00139-0","article-title":"Parameter and model recovery of reinforcement learning models for restless bandit problems.","volume":"5","author":"L Danwitz","year":"2022","journal-title":"Comput Brain Behav."},{"key":"ref16","doi-asserted-by":"publisher","first-page":"876-879","DOI":"10.1038\/nature04766","article-title":"Cortical substrates for exploratory decisions in humans.","volume":"441","author":"N Daw","year":"2006","journal-title":"Nature."},{"key":"ref17","doi-asserted-by":"publisher","first-page":"135-144","DOI":"10.1080\/21622965.2012.737760","article-title":"Higher-order factor structures for the WISC-IV: implications for neuropsychological test interpretation.","volume":"3","author":"S Decker","year":"2014","journal-title":"Applied Neuropsychology: Child."},{"key":"ref18","doi-asserted-by":"publisher","first-page":"71-95","DOI":"10.1016\/j.compchemeng.2017.05.010","article-title":"Design of computer experiments: A review.","volume":"106","author":"S Garud","year":"2017","journal-title":"Computers & Chemical Engineering."},{"key":"ref19","doi-asserted-by":"publisher","first-page":"515-534","DOI":"10.1214\/06-BA117A","article-title":"Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper).","volume":"1","author":"A Gelman","year":"2006","journal-title":"Bayes Anal."},{"key":"ref20","doi-asserted-by":"publisher","first-page":"641-651","DOI":"10.1177\/1745691614551642","article-title":"Beyond power calculations: assessing Type S (Sign) and Type M (Magnitude) errors.","volume":"9","author":"A Gelman","year":"2014","journal-title":"Perspect Psychol Sci."},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511790942","author":"A Gelman","year":"2006","journal-title":"Data Analysis Using Regression and Multilevel\/Hierarchical Models."},{"key":"ref22","doi-asserted-by":"publisher","first-page":"165-173","DOI":"10.2307\/271064","article-title":"Avoiding Model Selection in Bayesian Social Research.","volume":"25","author":"A Gelman","year":"1995","journal-title":"Sociol Methodol."},{"key":"ref23","doi-asserted-by":"publisher","first-page":"8-38","DOI":"10.1111\/j.2044-8317.2011.02037.x","article-title":"Philosophy and the practice of Bayesian statistics.","volume":"66","author":"A Gelman","year":"2013","journal-title":"British Journal of Mathematical and Statistical Psychology."},{"key":"ref24","doi-asserted-by":"publisher","first-page":"277-286","DOI":"10.1037\/dec0000101","article-title":"Uncertainty and exploration.","volume":"6","author":"S Gershman","year":"2019","journal-title":"Decision."},{"key":"ref25","doi-asserted-by":"publisher","first-page":"102725","DOI":"10.1016\/j.jmp.2022.102725","article-title":"Neuro-cognitive models of single-trial EEG measures describe latent effects of spatial attention during perceptual decision making.","volume":"111","author":"A Ghaderi-Kangavari","year":"2022","journal-title":"J Math Psychol."},{"key":"ref26","doi-asserted-by":"publisher","first-page":"336-338","DOI":"10.1017\/S0140525X04310083","article-title":"The irrationality paradox.","volume":"27","author":"G Gigerenzer","year":"2004","journal-title":"Behav Brain Sci."},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1017\/9781108890984","author":"T Hudson","year":"2021","journal-title":"Bayesian Data Analysis for the Behavioral and Neural Sciences: Non-Calculus Fundamentals."},{"key":"ref28","doi-asserted-by":"publisher","first-page":"166-175","DOI":"10.1016\/S0140-6736(13)62227-8","article-title":"Increasing value and reducing waste in research design, conduct, and analysis.","volume":"383","author":"J Ioannidis","year":"2014","journal-title":"Lancet."},{"key":"ref29","doi-asserted-by":"publisher","first-page":"986-1000","DOI":"10.1038\/s41562-022-01510-8","article-title":"Modelling human behaviour in cognitive tasks with latent dynamical systems.","volume":"7","author":"P Jaffe","year":"2023","journal-title":"Nat Hum Behav."},{"key":"ref30","doi-asserted-by":"publisher","first-page":"203-222","DOI":"10.1017\/S030500410001330X","article-title":"Some tests of significance, treated by the theory of probability.","volume":"31","author":"H Jeffreys","year":"1935","journal-title":"Math Proc Camb Philos Soc."},{"key":"ref32","doi-asserted-by":"publisher","first-page":"e2217331120","DOI":"10.1073\/pnas.2217331120","article-title":"Bayes factor functions for reporting outcomes of hypothesis tests.","volume":"120","author":"V Johnson","year":"2023","journal-title":"Proc Natl Acad Sci U S A."},{"key":"ref33","doi-asserted-by":"publisher","first-page":"209-226","DOI":"10.1111\/1467-9884.00077","article-title":"Bayesian sample size determination for normal means and differences between normal means.","volume":"46","author":"L Joseph","year":"1997","journal-title":"Journal of the Royal Statistical Society: Series D (The Statistician)."},{"key":"ref34","doi-asserted-by":"publisher","first-page":"773-795","DOI":"10.2307\/2291091","article-title":"Bayes factors.","volume":"90","author":"R Kass","year":"1995","journal-title":"J Am Stat Assoc."},{"key":"ref35","doi-asserted-by":"publisher","first-page":"293-300","DOI":"10.1016\/j.tics.2010.05.001","article-title":"What to believe: Bayesian methods for data analysis.","volume":"14","author":"J Kruschke","year":"2010","journal-title":"Trends in Cognitive Sciences."},{"key":"ref36","doi-asserted-by":"publisher","first-page":"573-603","DOI":"10.1037\/a0029146","article-title":"Bayesian estimation supersedes the t test.","volume":"142","author":"J Kruschke","year":"2013","journal-title":"J Exp Psychol Gen."},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-405888-0.09999-2","author":"J Kruschke","year":"2014","journal-title":"Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan, second edition."},{"key":"ref38","doi-asserted-by":"publisher","first-page":"99-120","DOI":"10.1016\/B978-0-12-405888-0.00005-2","author":"J Kruschke","year":"2015a","journal-title":"Doing Bayesian Data Analysis."},{"key":"ref39","doi-asserted-by":"publisher","first-page":"265-296","DOI":"10.1016\/B978-0-12-405888-0.00010-6","author":"J Kruschke","year":"2015b","journal-title":"Doing Bayesian Data Analysis."},{"key":"ref40","doi-asserted-by":"publisher","first-page":"722-752","DOI":"10.1177\/1094428112457829","article-title":"The time has come: Bayesian methods for data analysis in the organizational sciences.","volume":"15","author":"J Kruschke","year":"2012","journal-title":"Organ Res Methods."},{"key":"ref41","doi-asserted-by":"publisher","first-page":"33-39","DOI":"10.1016\/j.ijpsycho.2022.08.001","article-title":"The role of intolerance of uncertainty when solving the exploration-exploitation dilemma.","volume":"181","author":"A Krypotos","year":"2022a","journal-title":"Int J Psychophysiol."},{"key":"ref42","doi-asserted-by":"publisher","first-page":"e215-e233","DOI":"10.1097\/j.pain.0000000000002352","article-title":"The exploration-exploitation dilemma in pain: an experimental investigation.","volume":"163","author":"A Krypotos","year":"2022b","journal-title":"Pain."},{"key":"ref43","doi-asserted-by":"publisher","first-page":"2145-2149","DOI":"10.1097\/j.pain.0000000000003222","article-title":"The dynamics of pain avoidance: the exploration-exploitation dilemma.","volume":"165","author":"A Krypotos","year":"2024","journal-title":"Pain."},{"key":"ref44","doi-asserted-by":"publisher","first-page":"1982-1991","DOI":"10.1016\/j.neuroimage.2012.02.002","article-title":"Using confirmatory factor analysis to measure contemporaneous activation of defined neuronal networks in functional magnetic resonance imaging.","volume":"60","author":"B Lahey","year":"2012","journal-title":"Neuroimage."},{"key":"ref45","doi-asserted-by":"publisher","first-page":"87","DOI":"10.3389\/fnins.2020.00087","article-title":"Latent factor decoding of multi-channel EEG for emotion recognition through autoencoder-like neural networks.","volume":"14","author":"X Li","year":"2020","journal-title":"Front Neurosci."},{"key":"ref46","doi-asserted-by":"publisher","first-page":"155-167","DOI":"10.1016\/j.tics.2017.01.006","article-title":"Neuroadaptive Bayesian optimization and hypothesis testing.","volume":"21","author":"R Lorenz","year":"2017","journal-title":"Trends Cogn Sci."},{"key":"ref47","doi-asserted-by":"publisher","first-page":"101-104","DOI":"10.1016\/S0140-6736(13)62329-6","article-title":"Biomedical research: increasing value, reducing waste.","volume":"383","author":"M Macleod","year":"2014","journal-title":"Lancet."},{"key":"ref48","doi-asserted-by":"publisher","first-page":"e1007593","DOI":"10.1371\/journal.pcbi.1007593","article-title":"Computational optimization of associative learning experiments.","volume":"16","author":"F Melinscak","year":"2020","journal-title":"PLOS Computational Biology."},{"key":"ref49","doi-asserted-by":"publisher","first-page":"88-101","DOI":"10.1037\/pag0000791","article-title":"Differential impacts of healthy cognitive aging on directed and random exploration.","volume":"39","author":"J Mizell","year":"2024","journal-title":"Psychol Aging."},{"key":"ref50","doi-asserted-by":"publisher","first-page":"406-419","DOI":"10.1037\/a0024377","article-title":"Bayes factor approaches for testing interval null hypotheses.","volume":"16","author":"R Morey","year":"2011","journal-title":"Psychol Methods."},{"key":"ref51","doi-asserted-by":"publisher","DOI":"10.4324\/9781315134918","author":"D Morrison","year":"2017","journal-title":"The significance test controversy: a reader."},{"key":"ref52","doi-asserted-by":"publisher","first-page":"0021","DOI":"10.1038\/s41562-016-0021","article-title":"A manifesto for reproducible science.","volume":"1","author":"M Munaf\u00f2","year":"2017","journal-title":"Nature Human Behaviour."},{"key":"ref53","doi-asserted-by":"publisher","first-page":"40-45","DOI":"10.20982\/tqmp.17.1.p040","article-title":"Variability of Bayes factor estimates in bayesian Analysis of Variance.","volume":"17","author":"R Pfister","year":"2021","journal-title":"Quant Method Psychol."},{"key":"ref54","doi-asserted-by":"publisher","first-page":"19-26","DOI":"10.1177\/2515245917745058","article-title":"Power, dominance, and constraint: a note on the appeal of different design traditions.","volume":"1","author":"J Rouder","year":"2018","journal-title":"Adv Methods Pract Psychol Sci."},{"key":"ref55","doi-asserted-by":"publisher","first-page":"128-154","DOI":"10.1111\/insr.12107","article-title":"A review of modern computational algorithms for Bayesian optimal design.","volume":"84","author":"E Ryan","year":"2016","journal-title":"Int Stat Rev."},{"key":"ref56","doi-asserted-by":"publisher","first-page":"128-142","DOI":"10.3758\/s13423-017-1230-y","article-title":"Bayes Factor Design Analysis: planning for compelling evidence.","volume":"25","author":"F Sch\u00f6nbrodt","year":"2018","journal-title":"Psychon Bull Rev."},{"key":"ref57","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.18879706","article-title":"sarah407\/BFDA_Multi-Armed-Bandit: v1.1 (v1.1).","author":"S Schreiber","year":"2026","journal-title":"Zenodo."},{"key":"ref58","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-92bf1922-011","article-title":"Statsmodels: Econometric and Statistical Modeling with Python.","author":"S Seabold","year":"2010","journal-title":"SciPy 2010."},{"key":"ref59","doi-asserted-by":"publisher","first-page":"5833-5842","DOI":"10.1523\/JNEUROSCI.0053-12.2012","article-title":"Serotonin selectively modulates reward value in human decision-making.","volume":"32","author":"B Seymour","year":"2012","journal-title":"J Neurosci."},{"key":"ref60","doi-asserted-by":"publisher","first-page":"155-164","DOI":"10.1037\/xge0000250","article-title":"Charting the expansion of strategic exploratory behavior during adolescence.","volume":"146","author":"L Somerville","year":"2017","journal-title":"J Exp Psychol Gen."},{"key":"ref61","doi-asserted-by":"publisher","first-page":"1042-1058","DOI":"10.3758\/s13428-018-01189-8","article-title":"A tutorial on Bayes Factor Design Analysis using an informed prior.","volume":"51","author":"A Stefan","year":"2019","journal-title":"Behav Res Methods."},{"key":"ref62","doi-asserted-by":"publisher","first-page":"38330340","DOI":"10.1037\/met0000641","article-title":"Interim design analysis using Bayes factor forecasts.","author":"A Stefan","year":"2024","journal-title":"Psychol Methods."},{"key":"ref63","doi-asserted-by":"publisher","first-page":"61-70","DOI":"10.1016\/j.jclinepi.2016.11.003","article-title":"Potentially unnecessary and wasteful clinical trial research detected in cumulative meta-epidemiological and trial sequential analysis.","volume":"82","author":"P Storz-Pfennig","year":"2017","journal-title":"J Clin Epidemiol."},{"key":"ref64","doi-asserted-by":"publisher","first-page":"93-101","DOI":"10.1016\/0920-9964(96)88520-3","article-title":"Exploratory factor analysis of MRI brain structure measures in schizophrenia.","volume":"19","author":"A Tien","year":"1996","journal-title":"Schizophr Res."},{"key":"ref65","doi-asserted-by":"crossref","DOI":"10.1109\/DIAGNOSTIKA.2018.8526104","article-title":"Design of Experiments approach and its application in the evaluation of experiments.","author":"O T\u016fmov\u00e1","year":"2018","journal-title":"2018 International Conference on Diagnostics in Electrical Engineering (Diagnostika)"},{"key":"ref66","doi-asserted-by":"publisher","first-page":"e86224","DOI":"10.7554\/eLife.86224","article-title":"Designing optimal behavioral experiments using machine learning.","volume":"13","author":"S Valentin","year":"2024","journal-title":"eLife."},{"key":"ref67","doi-asserted-by":"publisher","first-page":"217-239","DOI":"10.1037\/met0000100","article-title":"A systematic review of Bayesian articles in psychology: the last 25 years.","volume":"22","author":"R Schoot","year":"2017","journal-title":"Psychol Methods."},{"key":"ref68","doi-asserted-by":"publisher","first-page":"90-100","DOI":"10.1016\/j.jtbi.2006.02.002","article-title":"Prior specification in Bayesian statistics: Three cautionary tales.","volume":"242","author":"S Van Dongen","year":"2006","journal-title":"Journal of Theoretical Biology."},{"key":"ref69","doi-asserted-by":"publisher","first-page":"235-243","DOI":"10.1177\/10731911241275256","article-title":"On the Use and Misuses of Preregistration: A Reply to Klonsky (2024).","volume":"32","author":"C Vize","year":"2025","journal-title":"Assessment."},{"key":"ref70","doi-asserted-by":"publisher","first-page":"1588-1589","DOI":"10.1097\/j.pain.0000000000000574","article-title":"The fear-avoidance model of pain.","volume":"157","author":"J Vlaeyen","year":"2016","journal-title":"Pain."},{"key":"ref71","doi-asserted-by":"publisher","first-page":"779-804","DOI":"10.3758\/bf03194105","article-title":"A practical solution to the pervasive problems of p values.","volume":"14","author":"E Wagenmakers","year":"2007","journal-title":"Psychon Bull Rev."},{"key":"ref72","doi-asserted-by":"publisher","first-page":"35-57","DOI":"10.3758\/s13423-017-1343-3","article-title":"Bayesian inference for psychology. Part I: theoretical advantages and practical ramifications.","volume":"25","author":"E Wagenmakers","year":"2018","journal-title":"Psychon Bull Rev."},{"key":"ref73","doi-asserted-by":"publisher","first-page":"18-39","DOI":"10.1162\/cpsy_a_00027","article-title":"Differential effects of psychotic illness on directed and random exploration.","volume":"4","author":"J Waltz","year":"2020","journal-title":"Comput Psychiatr."},{"key":"ref74","doi-asserted-by":"publisher","first-page":"2398212818772964","DOI":"10.1177\/2398212818772964","article-title":"Model-based and model-free pain avoidance learning.","volume":"2","author":"O Wang","year":"2018","journal-title":"Brain and Neuroscience Advances."},{"key":"ref75","doi-asserted-by":"publisher","first-page":"49-56","DOI":"10.1016\/j.cobeha.2020.10.001","article-title":"Balancing exploration and exploitation with information and randomization.","volume":"38","author":"R Wilson","year":"2021","journal-title":"Curr Opin Behav Sci."},{"key":"ref76","doi-asserted-by":"publisher","first-page":"2074-2081","DOI":"10.1037\/a0038199","article-title":"Humans use directed and random exploration to solve the explore-exploit dilemma.","volume":"143","author":"R Wilson","year":"2014","journal-title":"J Exp Psychol Gen."},{"key":"ref77","doi-asserted-by":"publisher","first-page":"499-508","DOI":"10.1016\/j.jmp.2010.08.002","article-title":"Optimal experimental design for a class of bandit problems.","volume":"54","author":"S Zhang","year":"2010","journal-title":"J Math Psychol."}],"container-title":["Wellcome Open Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2\/iparadigms","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T09:39:13Z","timestamp":1780652353000},"score":1,"resource":{"primary":{"URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,7]]},"references-count":76,"URL":"https:\/\/doi.org\/10.12688\/wellcomeopenres.22288.2","relation":{"has-version":[{"id-type":"doi","id":"10.12688\/wellcomeopenres.22288.1","asserted-by":"subject"}],"has-review":[{"id-type":"doi","id":"10.21956\/wellcomeopenres.28844.r155003","asserted-by":"subject"},{"id-type":"doi","id":"10.21956\/wellcomeopenres.28844.r155582","asserted-by":"subject"},{"id-type":"doi","id":"10.21956\/wellcomeopenres.28844.r155582","asserted-by":"object"},{"id-type":"doi","id":"10.21956\/wellcomeopenres.28844.r155003","asserted-by":"object"}]},"ISSN":["2398-502X"],"issn-type":[{"value":"2398-502X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,7]]},"assertion":[{"value":"Indexed","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2#article-reports","order":0,"name":"referee-status","label":"Referee status","group":{"name":"current-referee-status","label":"Current Referee Status"}},{"value":"10.21956\/wellcomeopenres.24565.r104233, Brandon S Coventry, University of Wisconsin-Madison, Madison, Wisconsin, 21 Oct 2024, version 1, 3 approved with reservations","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v1#referee-response-104233","order":0,"name":"referee-response-104233","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Sarah Schreiber<\/b>; \n<i>Posted: 25 Apr 2026<\/i>; \n<i>It was a pleasure to read your article detailing Bayes Factor design of experiments and its incorporation into latent variable modeling. Much well deserved interest has recently been received to incorporate Bayesian methods to the design of experiments, and I\u2019m pleased to see your method adding to the conversation. I believe that modeling experiments during the experimental design phase with well-defined models of decision processes can certainly help bound and inform the BFDA process in meaningful ways. However, there are several areas in the manuscript that require further refinement to enhance both its readability and the broader adoption of your model and methodology.<\/i> \n<b>Response<\/b>:\n<i> <\/i>We are thankful to Reviewer 1 for a generally positive evaluation of our work.  \n<i>Point 1. I believe that the description of the MAB task needs much better description. As it stands, it\u2019s placed in the context of a pain model that is never described, and the definition of a painful stimulus is vague at best.<\/i> \n<i>I also believe that this article often verges into vague descriptions of what is being done. I\u2019ve tried to highlight sections below, but I believe portions can be rewritten to explicitly state what is being done. This will greatly increase readability.<\/i> \n<b>Response:<\/b> We thank the reviewer for this feedback. We have revised the description of the multi-armed bandit (MAB) task to improve clarity and specificity. The revised text now provides a clearer definition of the task structure, the exploration-exploitation trade-off, and the nature of the task outcomes. We also explicitly describe the pain-related application of the binary MAB, specifying the delivery versus absence of a painful stimulus (e.g., electrocutaneous stimulation), and we clarify the fixed task horizon and trial structure. \n<b>Revised Sections:<\/b> \u201cTo validate the analysis pipelines and to provide a concrete example for the proposed approach, a binary two-armed bandit task was simulated. The MAB is a widely-used paradigm to investigate the exploration-exploitation dilemma in behavioural science (Danwitz \n<i>et al.<\/i>, 2022; Daw \n<i>et al.<\/i>, 2006; Gershman, 2019), in which the agent (for example the participant or a reinforcement learning agent) repeatedly chooses between multiple actions or \u201carms\u201d. Over time the agent learns the reward probabilities associated with each arm.\n<b> <\/b>To maximise the cumulative outcome across trials, the agent must balance exploring the available arms in order to reduce uncertainty about their outcomes (exploration) and using the information they have gathered so far to choose the optimal option (exploitation).\u201d \u201cThe outcomes can be continuous, for example in the form of a scalar monetary reward, where participants learn the reward distribution (e.g. mean and uncertainty), or binary, such as the presence versus absence of a food reward or a negative (aversive) stimulus. The binary version of the MAB task is often used in pain research, where a painful stimulus (e.g., an electrocutaneous shock) is administered. The two possible outcomes would be the delivery of an aversive (e.g. painful) stimulus and its absence (Krypotos \n<i>et al.<\/i>, 2022a; Krypotos \n<i>et al.<\/i>, 2022b). In the computational modelling of these outcomes, each outcome must be assigned a numerical value. Here, the outcome of each choice was coded as 0 or -1, where -1 represents an aversive outcome.\u201d \n<b><i>\u201c<\/i><\/b>To facilitate differentiation between exploration and exploitation, the agent has information on both options available to them prior to their first choice, so that the agent possesses information to base their exploitation on (Wilson et al., 2014). This is achieved by presenting the agent with four actions and their immediate results, after which they are free to make their own choices. The task was implemented with a fixed horizon of 10 choices, defining the number of trials within each game, in that each game consisted of four observed trials followed by 6 free choice trials. The outcome probabilities were combinations of the probabilities 0.1, 0.3, and 0.9, with the probability assigned to each arm changing after each game.\u201d  \n<i>Point 2. It should be noted that the use of Bayes factors in model comparisons can be tricky, with the potential to reduce meaningful Bayesian inference to a modified p-value. Many argue that the power of Bayesian thinking lies in using prior knowledge to build the best possible models and to avoid the temptation to just compare every model available. For a good discussion of these caveats, see (Gelman and Rubin, 1995; Gelman and Shalizi, 2013). Providing a discussion of and context to BFDA may help allay some of these criticisms of BF analyses.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this comment. We have added a discussion that contextualises our use of Bayes factors within the broader Bayesian literature, explicitly acknowledging common concerns about threshold-based interpretations. We clarify that Bayes factors are used here as a pragmatic tool for design analysis, rather than as a replacement for full Bayesian inference. \n<b>Revised Section:<\/b> \u201cIt is important to highlight that while this analysis summarizes Bayes factors using common thresholds, their interpretation is not inherently discrete. The primary strength of Bayes factors lies in their continuous nature and in their role within a broader Bayesian framework that emphasizes principled model construction and the use of prior knowledge (Aczel et al., 2020; Coventry &amp; Bartlett, 2024; Gelman &amp; Rubin, 1995; Gelman &amp; Shalizi, 2013). However, in the present work, Bayes factors are used specifically as a pragmatic tool for design analysis, where threshold-based summaries aid decision-making. In retrospective analyses, Bayes factors should still be interpreted within a full Bayesian framework rather than being reduced to a modified p-value.\u201d &nbsp;  \n<i>Point 3. I also believe this article would benefit greatly from following the Bayesian analysis reporting guidelines (BARG) for article clarity and reproducibility (Kruschke, 2021).<\/i> &nbsp; \n<b>Response<\/b>\n<i>: <\/i> We thank the reviewer for this helpful suggestion. We have now incorporated several relevant elements of the Bayesian Analysis Reporting Guidelines for reporting retrospective Bayesian inference to improve the clarity and transparency of our approach for prospective Bayes factor design analysis. Specifically, we now more clearly state the goals of the analysis, explicitly define the observed data and inferred parameters, and clarify the specification and justification of the priors used. We believe these additions improve readability and reproducibility while remaining appropriate to the prospective focus of the present work. \n<b>Revised Sections:<\/b> \n<b>\u201c<\/b>The goal of this work is a prospective design analysis, specifically to evaluate how sample size, number of games per participant, and effect size influence Bayes factors and parameter recovery for a latent exploration parameter in a multi-armed bandit task. We first analysed the accuracy of Bayesian parameter estimations of latent variables, within a population, as well as between two groups, using simulated behavioural data in a multi-armed bandit task (Krypotos \n<i>et al.<\/i>, 2024). In a second step, we combined BFDA with simulations of behavioural data to explore the relationship between sample size and the strength of evidence for both null and alternative hypothesis.\u201d \n<b>\u201c<\/b>To summarise, the primary outcome per game is the number of aversive outcomes (0\u20136), while the main inferred parameter is the exploration parameter \u03c4. The between group comparison focusses on the difference in the exploration parameter \u0394\u03bc_\u03c4. For both tau and the difference in population means (\u0394\u03bc_\u03c4), we used a discrete uniform prior defined over a bounded parameter space (\u03c4 \u2208 [0.01, 3] and \u0394\u03bc_\u03c4 \u2208 [-3, 3]).\u201d  \n<b>Specific Comments<\/b> \n<u>Abstract<\/u> \n<i>Point 4. Interest in the use of Bayesian statistics I believe extends well beyond just the fact that one can incorporate prior knowledge. While that is certainly a powerful component of Bayesian design, other advantageous features include the fact that inference is performed using the data on hand without implicit assumptions on population distributions, the ability to easily handle data distributions which are not strictly normal, direct quantification of uncertainty under inference, and the ability to iteratively update hypotheses as data is observed.<\/i> \n<b>Response<\/b>: We thank the reviewer for this comment. Due to abstract word limits, these points could not be fully included. However, we have reworded the Abstract to better reflect the broader advantages of the Bayesian framework and expanded the Introduction to highlight several of the reviewer\u2019s suggestions. \n<b>Revised Section: <\/b> \n<u>Abstract:<\/u> \u201cBayesian statistics offers a flexible framework that supports iterative updating of hypotheses and the incorporation of prior information, amongst other advantages.\u201d \n<u>Introduction:<\/u> \u201cConcurrently, Bayesian methods are gaining recognition for their advantages, which include the incorporation of prior knowledge into statistical processes, the ability to quantify evidence for both null and alternative hypotheses, accommodate non-normal data, and directly represent uncertainty through probability distributions (Jeffreys, 1935; Jeffreys, 1961; Wagenmakers, 2007; Blackwell &amp; Ramamoorthi, 1982;Gelman &amp; Shalizi, 2013; Kruschke, 2010; Kruschke, 2013).\u201d  \n<u>Introduction<\/u>: \n<i>Point 5.&nbsp; <\/i> \u201c\n<i>Designing an effective experiment can be challenging due to the number of parameters that need to be considered.\u201d While this is true, I believe this is a bit reductive, potentially to the point of trivializing design of experiments (DOE). Rewording of text and adding appropriate discussion of the goals of DOE would add important context to the reader, especially those relatively new to the science and admittedly art of DOE<\/i>. \n<b>Response<\/b>: We thank the reviewer for this comment and have revised the Introduction accordingly. \n<b>Revised Section:<\/b> \u201cDesigning an effective experiment is a multifaceted process that begins with formulating clear research questions and hypotheses, selecting appropriate methodologies, and aligning design choices with the underlying theoretical framework. Beyond these conceptual considerations, the appropriate analyses and practical parameters must be carefully determined to ensure valid, reliable, and adequately powered statistical inference (Czitrom 2012; Garud et al., 2017; T\u016fmov&aacute; et al., 2018). &nbsp;  \n<i>Point 6. \u201cTo improve a study\u2019s effectiveness, it is recommended to conduct a thorough prospective design analysis rather than relying purely on a retrospective approach\u201d. While true, I believe it\u2019s important to mention preregistration as part of the current initiatives to improve reproducibility. For a thorough discussion of the pros and cons of preregistration, see (Vize et al., 2024).<\/i> &nbsp; \n<b>Response<\/b>: &nbsp;We thank the reviewer for this suggestion. We have revised the manuscript to explicitly reference preregistration as part of broader reproducibility initiatives. \n<b>Revised Section: <\/b> \u201cProspective design analyses can help optimize the use of available resources, which has become increasingly important considering recent concerns about 'research waste' (Ioannidis \n<i>et al.<\/i>, 2014; Macleod \n<i>et al.<\/i>, 2014; Storz-Pfennig, 2017). These approaches align with efforts to improve reproducibility and replicability, such as preregistration and sharing of code and data (Munaf&ograve; et al., 2017; Vize et al., 2025). As BFDA requires the primary analysis to be specified a priori, it integrates well with these practices.\u201d &nbsp;  \n<i>Point 7. \u201cBayesian methods are gaining recognition for their advantages, as they allow the incorporation of prior knowledge into statistical processes.\u201d Again, I think much more can be said about the advantages of Bayesian approaches. See comments above in abstract section, and the following references (Blackwell and Ramamoorthi, 1982; Gelman and Shalizi, 2013; Kruschke, 2013, 2010).<\/i> &nbsp; \n<b>Response<\/b>: We agree with the reviewer and have revised this section to more clearly emphasize the general advantages of Bayesian statistics and their role in Bayes Factor Design Analysis. \n<b>Revised Sections:<\/b> \u201cConcurrently, Bayesian methods are gaining recognition for their advantages, which include the incorporation of prior knowledge into statistical processes, the ability to quantify evidence for both null and alternative hypotheses, accommodate non-normal data, and directly represent uncertainty through probability distributions (Jeffreys, 1935; Jeffreys, 1961; Wagenmakers, 2007; Blackwell &amp; Ramamoorthi, 1982;Gelman &amp; Shalizi, 2013; Kruschke, 2010; Kruschke, 2013). &nbsp;  \n<i>Point 8. \u201cTraditionally, sample size determination methods rely on frequentist statistics, but there has been an ongoing critique among statisticians and methodologists regarding these approaches.\u201d While true, I\u2019d argue that the primary goal of DOE is not just to determine sample size, but to determine sample size to ensure an experiment has the statistical power to detect if a desired effect is truly present. Sample size is a means to this end of the question experimenters are truly interested in.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this important clarification. We have revised the Introduction to more clearly emphasize that sample size determination is a means to ensuring adequate statistical power to detect meaningful effects, rather than an end in itself. &nbsp; \n<b>Revised Section: <\/b> \u201cDesigning an effective experiment is a multifaceted process that begins with formulating clear research questions and hypotheses, selecting appropriate methodologies, and aligning design choices with the underlying theoretical framework. Beyond these conceptual considerations, the appropriate analyses and practical parameters must be carefully determined to ensure valid, reliable, and adequately powered statistical inference (Czitrom 2012; Garud et al., 2017; T\u016fmov&aacute; et al., 2018).\n<b>\u201d<\/b> &nbsp; \u201cTraditionally, the focus of predictive analyses has been on determining the sample size required to ensure adequate statistical power to detect meaningful effects, which is a useful step to ensure the quality and validity of the experiment and all conclusions drawn from it. Methodologies for determining sample sizes have long primarily relied on frequentist statistics, but there has been an ongoing critique among statisticians and methodologists regarding these approaches. Among the main concerns are the common misinterpretation of \n<i>p<\/i>-values and significance testing (Benjamin \n<i>et al.<\/i>, 2018; Gigerenzer, 2004; Morrison &amp; Henkel, 2017; Wagenmakers, 2007; Wagenmakers \n<i>et al.<\/i>, 2018).\u201d  \n<i>Point 9. \u201ca-priori\u201d. Please change to&nbsp;a priori<\/i> &nbsp; \n<b>Response<\/b>: This has been revised. \n<b>Revised Section: <\/b> \u201cBFDA evaluates the distribution of the Bayes factors for a given experimental design, providing a powerful alternative to frequentist a priori power analyses.\u201d  \n<i>Point 10. \u201cBFDA assumes a population with predetermined attributes and conducts simulations on repeated sampling from this population\u201d. In the context of a tutorial article, I believe concrete examples are going to be needed to best orient the reader to the goals of BFDA. One way to do this is to provide a quick hypothetical experiment, outlining what example predetermined attributes may be, and how one might conduct simulations. Are there experiments where simulations are generally widely available and accepted? What about cases of a novel experiment, how might one deal with model selection? These are questions that should be addressed to best help the reader adopt this framework.<\/i> &nbsp; \n<b>Response<\/b>: We appreciate the reviewer\u2019s suggestion to include a hypothetical example. However, as the manuscript develops the example of the MAB in detail throughout the remainder of the tutorial, we believe that introducing an additional hypothetical scenario would be beyond the scope of the current article and may create confusion for the reader. Instead, we have revised this section to highlight the goal of BFDA, to clarify the notion of population-level assumptions and elaborated on how distributional properties may be specified in both established and novel paradigms. &nbsp; \n<b>Revised Section:<\/b> \u201cBFDA assumes that the variable of interest is defined at the population level, and that observations are sampled from this population. In well-established paradigms, distributional properties and effect size estimates of the variable at population level can be informed by previous research. In novel paradigms, standard distributions (e.g. the normal distribution) may be assumed and sensitivity analyses across possible effect sizes can be used to assess robustness (Sch&ouml;nbrodt &amp; Wagenmakers, 2018). For each sample generated under this population model, the comparative evidence between the null hypothesis and the alternative hypothesis is measured by calculating the Bayes factor. Repeating this process yields a distribution of Bayes factors, which can be used to evaluate and compare design approaches and thus optimise the experimental setup.\u201d  \n<i>Point 11. \u201cHowever, in psychology and neuroscience, we often deal with latent variables that first need to be inferred from the data collected.\u201dAbsolutely, and I\u2019m glad this is mentioned. It should be noted that in neuroscience and social sciences, initial passes where latent variables are truly hidden is often addressed with hierarchical Bayesian models. See (Cronin et al., 2010; Gelman, 2006; Gelman and Hill, 2006).<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for calling these references to our attention and have revised the section accordingly. \n<b>Revised Section: <\/b> \u201cHowever, in psychology and neuroscience, we often deal with latent variables that first need to be inferred from the data collected. Such cases are commonly addressed using computational models (e.g. of behaviour or neural responses), including hierarchical Bayesian models, which allow latent parameters to be estimated while accounting for variability at multiple levels (Cronin et al., 2010; Gelman, 2006; Gelman &amp; Hill, 2006).\n<b>\u201d<\/b> &nbsp; \n<u>Methods<\/u>\n<i>. <\/i>  \n<i>Point 12. \u201cThe prior distribution P(\u03b8) represents our knowledge about the distribution of \u03b8 ahead of data acquisition. The chosen prior can heavily influence the outcome of an analysis and should therefore be chosen carefully, as it will bias the estimation.\u201d True, especially if the observed data is weak, in which case priors dominate. However, some guidance of proper prior choice should be detailed in this article. Potentially adding a prior predictive checks to at least one model to help inform the reader of exactly this condition.<\/i> &nbsp; \n<b>Response<\/b>: We are grateful for this comment. While an extensive treatment and comparison of prior specification is beyond the scope of this tutorial, we have added references to relevant literature to provide readers with further guidance. \n<b>Revised Section: <\/b> \u201cThe chosen prior can heavily influence the outcome of an analysis and should therefore be chosen carefully, as it will bias the estimation. When prior knowledge about the underlying distribution is limited, researchers often employ uninformative or weakly informative priors, which can nevertheless reflect plausible bonds for the parameters. Given the subjective nature of priors, their selection should be transparently reported and sufficiently motivated. In addition, sensitivity analyses can be conducted to evaluate the robustness of results with respect to the chosen prior. For further discussions on priors, refer to further literature such as Gelman (2006), Van Dongen (2006), or Stefan (2019).\u201d  \n<i>Point 13. \u201cOnce the likelihoods have been calculated for each group, we can compare \u03b8 between two groups, for example between patients and healthy controls.\u201d Need to clarify for the reader whether you mean a point estimate of \u03b8 or the distribution of \u03b8.<\/i> \n<b>Response<\/b>: We thank the reviewer for this comment and have clarified that the comparison refers to distributions rather than point estimates. \n<b>Revised Section: <\/b> \u201cOnce the likelihoods have been calculated for each group, we can compare the distribution of \n<i>\u03b8<\/i> between two groups, for example between patients and healthy controls.\u201d  \n<i>Point 14. \u201cFigure 1 panel A\u201d I think this figure has the promise of being very good, but is a bit confusing to read as it stands. As an example, the box \u201cCompute the posterior\u201d is positioned right next to, what I believe is the incorporation of the prior into the likelihood to create the posterior, but it appears based on where the box is positioned that you mean to say P(theta) is the posterior.<\/i> &nbsp; \n<b>Response<\/b>: We appreciate these observations regarding Figure 1. This figure has now been revised to improve clarity and understanding for the reader. The figure legend has also been revised. &nbsp;  \n<i>Point 15. \u201cHDPI is a method of obtaining a credible interval for skewed probability distributions.\u201d Need to describe credible regions, and I believe you need to redo the definition of the HPDI, as it\u2019s not just for skewed distributions. I think I understand what you mean by skewed here, but as it stands its ambiguous given that one can easily define HPDIs for non-skewed distributions.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this helpful clarification. We have revised the definition of the HPDI to more accurately describe it and its average length. \n<b>Revised Section: <\/b> \u201cA further analysis evaluates the accuracy of the estimation of the difference between two groups by assessing the estimation error and the average length of the highest probability density interval (HPDI) to aid in determining an optimal sample size. The HPDI is a type of credible interval, representing a range of values within which the parameter has a certain probability of falling (e.g. 95%). Among all possible intervals, the HPDI is the credible interval that includes the highest probability densities. Therefore, a 95% HDPI would include the range of values with the highest densities and 95% probability. The interval is defined as  with An example of a HPDI is shown in Figure 1d. The length of the HPDI can serve as a measurement of uncertainty, with narrower intervals reflecting more precise estimates. The average length of the posterior\u2019s \n<i>HPDI<\/i> can be calculated in a two-step approach ( Joseph &amp; B&eacute;lisle, 1997). Let \n<i>l'<\/i> be the length of the \n<i>HPD<\/i> \n<sub>1\u2013\u03b1<\/sub> interval for a given dataset \n<i>Y <sub>i<\/sub> <\/i>. The average length \n<i>l<\/i>* can be calculated from \n<i>l'<\/i> by multiplying it by the probability of \n<i>Y <sub>i<\/sub> <\/i>being the outcome for this measurement and integrating over all possible outcomes \n<i>\u03b3<\/i> = [ \n<i>Y<\/i> \n<sub>1<\/sub>, \n<i>Y<\/i> \n<sub>2<\/sub>,...] . &nbsp;is the posterior predictive density and can be computed as .\u201d  \n<i>Point 16. \u201cFigure 1\u201d Consider reorganizing the panels of Figure 1 as they are referenced out of order in text, making the figure a little harder to read.<\/i> \n<b>Response<\/b>: This figure has now been revised to improve clarity and understanding for the reader. In-text references to Figure 1 have been revised to ensure that all panels are referred to in order. &nbsp;  \n<i>Point 17. \u201cThe probability of receiving a certain number of painful stimuli in one game is dependent on the choices of the participant on one hand and the reward probabilities of each arm on the other hand.\u201d So this is verging into confusion. I understand that MAB problems are used in models of pain as mentioned above, but if this terminology is going to be adapted in the method (painful stimuli), then this model needs to be fleshed out. Otherwise, I don\u2019t not have a well-defined view of what MAB model parameters, such as reward and painful stimuli mean. I\u2019d suggest either removing the pain model from the MAB context and explicitly defining model rewards and failures or totally rope the description of the MAB problem into the pain model context<\/i>. &nbsp; \n<b>Response<\/b>: We recognize that combining the MAB framework with a pain-related example may have been unclear. The aversive stimulus was chosen due to its relevance in the pain literature. To improve clarity, we revised the explanation and changed the terminology to consistently distinguish the general MAB model from its experimental instantiation, replacing \u201cpainful stimulus\u201d with \u201caversive outcome\u201d in the modelling context. This is also addressed in the response to point 1. \n<b>Revised Sections:<\/b> \u201cThe outcomes can be continuous, for example in the form of a scalar monetary reward, where participants learn the reward distribution (e.g. mean and uncertainty), or binary, such as the presence versus absence of a food reward or a negative (aversive) stimulus. The binary version of the MAB task is often used in pain research, where a painful stimulus (e.g., an electrocutaneous shock) is administered. The two possible outcomes would be the delivery of an aversive (e.g. painful) stimulus and its absence (Krypotos \n<i>et al.<\/i>, 2022a; Krypotos \n<i>et al.<\/i>, 2022b). In the computational modelling of these outcomes, each outcome must be assigned a numerical value. Here, the outcome of each choice was coded as 0 or -1, where -1 represents an aversive outcome.\u201d \u201cThe binary two-armed bandit task was simulated as described and the number of draws with an aversive outcome was recorded. The probability of receiving a certain number of aversive outcomes in one game is dependent on the choices of the participant on one hand and the reward probabilities of each arm on the other hand.\u201d  \n<i>Point 18. \u201cThe learning rate \u03b1 was fixed at 0.1\u201d This seems a bit high of a learning rate value. Why was this chosen and if there is appropriate literature, cite it here.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this comment. The value \u03b1 = 0.1 was selected as a fixed illustrative parameter to facilitate the simulations. We have adjusted the wording to make the considerations clearer. \n<b>Revised Section: <\/b> \u201cIn the present simulations, the learning rate \u03b1 was arbitrarily fixed at 0.1 for illustrative purposes. Although learning rates may vary, particularly in aversive contexts (Krypotos et al., 2022b; Wang et al., 2018), our conclusions are not contingent on this specific choice.\u201d  \n<i>Point 19. \u201cTherefore, there are seven possible outcomes, as the agent can receive between zero and six painful stimuli.\u201d Again, this needs much better defining.<\/i> \n<i>\u201cFor each choice, there are four possible combinations of arm chosen and aversive or neutral reward. For six free choice trials there are a total of 46 possible paths, or combinations of choices and outcomes.\u201d Moving this earlier in the text will help with understanding of the MAB model.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for these helpful comments. We have revised and restructured the entire MAB section, moving the description of possible choice-outcome combinations and clarifying the wording to improve overall comprehensibility. \n<u>Results<\/u>  \n<i>Point 20. \u201cThe use of the BFDA as a method for prospective design analysis allows us to consider a number of factors. However, it is essential to evaluate the algorithm estimating the latent variable as a preliminary step.\u201d This needs to be contextualized for understanding. What factors are you considering? What latent variables are you hoping to describe? What specific algorithm do you mean here? This sentence is quite vague. Being quite specific in what results you are showing will help readability greatly<\/i>. &nbsp; \n<b>Response<\/b>: We revised this section for clarity, more clearly specifying the goals of the BFDA, the outcome parameters, and the analytical approach. \n<b>Revised Section: <\/b> \u201cThe use of the BFDA as a method for prospective design analysis allows us to systematically evaluate design parameters, including sample size, number of games, task-specific features such as number of arms and aversive outcome probabilities, and the planned analysis approach. In our example experiment of the binary MAB with aversive outcomes, our hypothetical question is to compare the exploration parameter \u03c4 between two groups. This is a latent parameter which influences the agents\u2019 choices (eq.14). Our analysis consists of three main steps summarized in Figure 1c. First, we verify the estimations of population mean for one population. Next, we compare the population means between two group by examining the distribution of Bayes factors, which indicate whether there is evidence for or against a difference in population mean between groups. This analysis is carried out to determine possible effect sizes and the effect of experimental parameters, such as the number of participants, as well as the number of games per participant on the evidence. In the last step, we analyse potential errors when estimating a difference in means between groups and how they are affected by the experimental parameters (number of participants and number of games per participant).\u201d  \n<i>Point 21. \u201cOnce the algorithm has been validated, it can be used for the BFDA, in which repeated sampling from a population is simulated and the planned analysis carried out on the sample.\u201d A descriptive flow chart figure would be extremely useful here.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this helpful suggestion. Figure 1c presents a flowchart of the validation and BFDA procedures. We have now adjusted the text to improve clarity. \n<b>Revised Section: <\/b> \u201cIn our example experiment of the binary MAB with aversive outcomes, our hypothetical question is to compare the exploration parameter \u03c4 between two groups. This is a latent parameter which influences the agents\u2019 choices (eq.14). Our analysis consists of three main steps summarized in Figure 1c. First, we verify the estimations of population mean for one population. Next, we compare the population means between two group by examining the distribution of Bayes factors, which indicate whether there is evidence for or against a difference in population mean between groups.\u201d  \n<i>Point 22. \u201cThe individual exploration parameter for each simulated participant was drawn from a normal distribution with population mean \u03bc\u03c4\u201d. Point to the exact parameter referenced in the methods, as right now this is vague.<\/i> &nbsp; \n<b>Response<\/b>: We have adjusted the text to refer to the softmax algorithm, where \u03c4 was first introduced and to include more information on the population. \n<b>Revised Section: <\/b> \n<b>\u201c<\/b>For each simulated participant, the individual exploration parameter \u03c4, as defined in the softmax algorithm (14), was drawn from a normal distribution with population mean \n<i>\u03bc <sub>\u03c4<\/sub> <\/i>ranging from 0.05 to 1 in increments of 0.05, and a standard deviation of \n<i>\u03c3<\/i> = 0.02. The goal was to then estimate the population mean within the Bayesian framework as described (see Figure 1c, Validation of estimations of population mean).\u201d &nbsp; \n<i>Point 23. \u201cThe overall fit of the regression model was statistically significant (F(3,496) = 136300, p &lt; 0.001,R2 = 0.999)\u201d This regression model isn\u2019t described in the methods. Please detail what exactly is done here. It\u2019s also unclear why a frequentist approach here was used to validate the algorithm, given that the aim was a Bayes factor approach. This needs better detailing for why this was done.<\/i> \n<b>Response<\/b>: We thank the reviewer for this important point. We have revised the Methods section to explicitly describe the regression model used and clarify its role. We now state that the regression analysis is a descriptive check assessing the correspondence between the algorithm\u2019s estimates and the known generative values in simulation. The analysis is not part of the Bayesian inferential framework and is not intended to validate the Bayes factor approach. The frequentist formulation was used solely for transparency and ease of interpretation in this auxiliary verification step. \n<b>Revised Section:<\/b> \u201cTo evaluate the accuracy of this estimation, we compared the point estimates calculated as stated in (5) with the true exploration parameters. This was done using a linear regression model with point estimates as the dependent variable and true mean difference, sample size, and their interaction as predictors. The regression model, as well as the subsequent models, were estimated using ordinary least squares in the seaborn python package (Seabold &amp; Perktold, 2010). These regression analyses and correlations are descriptive summaries of the observed relationships and are not part of the Bayesian inferential framework or the Bayes factor-based design analysis. They are used solely as a descriptive check to assess whether the simulated data contain informative signal regarding the accuracy of the estimated population means. We then calculated the mean squared estimation error (MSEE) and evaluated its relationship with sample size using Spearman\u2019s rank-order correlation. We also computed a regression model with the MSEE as the dependent variable and the sample size, true population mean, and their interaction as predictors, allowing for nonlinear relationships. After inspection, the sample size was logarithmically transformed, and the population mean was exponentially transformed. The scaling constants used in these transformations were determined in prior curve-fitting procedures.\u201d \u201cNext, we analysed the estimated difference in mean between the two populations by comparing the estimated difference between the two populations to the true difference \u0394 \n<i>\u03bc<\/i> \n<sub>\u03c4<\/sub>. 250 simulations were run for each combination of \u0394 \n<i>\u03bc<\/i> \n<sub>\u03c4<\/sub> ranging from 0 to 1 in increments of 0.05 and sample size per group ranging from 10 to 58 in increments of 2. A linear regression model with point estimates of the difference as the dependent variable and true mean difference, sample size, and their interaction as predictors was used as a descriptive summary of the correspondence between estimated and generative values. Again, these analyses are not part of Bayesian inferential framework. The MSEE was calculated for the estimated difference and its relationship with sample size and mean difference was examined using a regression model with MSEE as the dependent variable and the sample size, true mean difference, and their interaction as predictors, incorporating nonlinear transformations of the predictors. Log transformation was applied to the sample size, and exponential transformation was applied to the difference in population mean, with scaling constants established in earlier curve-fitting operations.\u201d  \n<i>Point 24. \u201c250 simulations were run for each combination of \u0394\u03bc\u03c4 ranging from 0 to 1 in increments of 0.05 and sample size per group ranging from 10 to 60 in increments of 2.\u201d Please make this a little clearer. What simulation? Markov decision process? Is this MAB RL? Better descriptions of what is being done need to be added.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this comment. We have expanded the description of the simulation procedure to clarify that each replication involved simulating participant choices within MAB reinforcement learning framework, followed by computation of the Bayes factor as part of the BFDA. The revised text now more explicitly outlines the sequential steps of the simulation and analysis process. \n<b>Revised Section: <\/b> \n<b>\u201c<\/b>To evaluate the effects of sample size and difference in means on the evidence for a difference between groups, we performed the BFDA. First, we drew random samples of the exploration parameter \u03c4 from the two populations with different means, with each sampled value representing one simulated participant. Behavioural outcomes within the binary MAB were then generated based on \u03c4, and the distribution&nbsp;&nbsp;was computed as described in Equation 6, from which the Bayes factor was calculated (Equation 8). This procedure was repeated 250 times for each combination of the sample size of each group ranging from 10 to 58 in increments of 2 and difference in means ranging from 0 to 0.95 in increments of 0.05. To determine the population means for a given mean difference, the mean of the first population was randomly sampled from a uniform distribution between 0 and&nbsp;, and the mean of the second population was defined by adding the specified difference. The Bayes factor was calculated for each of the 250 simulations, yielding a distribution of Bayes factors reflecting the evidence for or against a difference in means at each combination of sample size and mean difference.\u201d  \n<i>Point 25. \u201cFigure 2\u201d This is interestingly. What is happening at 0.05? It seems that BF dominate predictions except in this small window. When there\u2019s 0 difference in the mean, Bayes is doing create, but minimal and not moderate uncertainty proves problematic?<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this observation. Because the null hypothesis is specified as a point hypothesis (i.e., \u0394\u03bc\u03c4 = 0), even extremely small deviations from zero imply that the null is technically false. In this region of very small effect sizes, the Bayes factor may not strongly favour the alternative unless the sample size is sufficiently large. As a result, moderate evidence can appear unstable in this narrow range, reflecting the difficulty of distinguishing very small effects from zero. We have explicitly added this fact to the text. \n<b>Revised Section:<\/b> \u201cWith an increase of \u0394 \n<i>\u03bc<\/i> \n<sub>\u03c4<\/sub> and \n<i>n<\/i> the Bayes factors are more likely to support the correct hypothesis and the stronger the evidence in favour of the correct hypothesis. When the difference in means is zero, the evidence correctly supports the null hypothesis. However, even very small deviations from zero technically contradict the null hypothesis, which can challenge the algorithm. Detecting and strongly supporting such minimal differences requires very large sample sizes.\n<b>\u201d<\/b>  \n<i>Point 26. \u201cFor \u0394\u03bc\u03c4 \u2260 0 this probability is the probability of wrongfully accepting the null hypothesis, which corresponds to the type II error.\u201d In Bayesian models, type I\/II errors are generally not considered for a variety of reasons related to how Bayesian inference is performed. Bayesian approaches instead talk of Type M or Type S errors generally. You can derive a type I\/II error for Bayesian approaches, but care must be taken in doing so. See (Gelman and Carlin, 2014).<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this comment and have revised the relevant sections to explicitly define and label Type S and Type M errors. \n<b>Revised Sections:<\/b> \u201cSimilarly, we can calculate the probability of receiving a Bayes factor \n<i>BF<\/i> \n<sub>10<\/sub> smaller than 1\/3 which would indicate at least moderate evidence in favour of the null hypothesis (Figure 3b). When this occurs despite a true nonzero mean difference, it constitutes a Type S (sign) error in the Bayesian framework.\u201d &nbsp; \u201cTo gain an understanding of how the Bayes factor relates to the true estimate of the difference in mean, we can determine the magnitude error for those simulations that result in a Bayes factor greater than 10. The magnitude error (Type M error) is calculated by dividing the difference between the estimated value and the true difference in mean by the true difference in mean. This reflects an overestimation of the true effect size in statistically significant results (Gelman &amp; Carlin, 2014).\u201d  \n<i>Point 27. \u201dFigure 3\u201d Please label color map axis in this figure.<\/i> \n<b>Response<\/b>: The colour map axis has been labelled in Figure 3 as requested. This change has also been applied to Figure 4. &nbsp; \n<i>Point 28. \u201cThis is supported by positive correlations between P(BF10 &gt; 10) with n and ngames (rs(23) = 0.74, p &lt; 0.001;rs(7) = 0.62, p &lt; 0.001) and negative correlations between P(BF10 &lt; 1\/10) with n and ngames (rs(23) = \u20130.49, p &lt; 0.001, rs(7) = \u20130.31, p &lt; 0.001\u201d I think a stronger rational needs to be given here for mixing Bayesian and frequentist approaches to understanding this problem. These are fundamentally different statistical paradigms with inference performed differently. While they may come to similar conclusions, care needs to be taken here.<\/i> \n<b>Response<\/b>: We thank the reviewer for raising this important point. As in Point 23, the reported correlations are not intended as inferential statistics but as descriptive summaries of simulation outcomes. We have clarified this distinction in the manuscript to explicitly separate these summaries from the Bayesian inferential framework. &nbsp; \n<b>Revised Section:<\/b> \u201cThese regression analyses and correlations are descriptive summaries of the observed relationships and are not part of the Bayesian inferential framework or the Bayes factor-based design analysis. They are used solely as a descriptive check to assess whether the simulated data contain informative signal regarding the accuracy of the estimated population means.\u201d &nbsp;  \n<u>Discussion<\/u> \n<i>Point 29. \u201cIn this work, we used BFDA to analyse the effect of sample size, number of games per participant, and effect size on the probability of obtaining significant evidence to support a null or alternative hypothesis, and the probabilities of incorrectly supporting either hypothesis<b>.\u201d<\/b> This needs to be contextualized for experiments that are modeled by MAB problems, and not strictly true for all possible experimental designs. I think this is where the real power of this model lies. If one is running experiments with similar designs and can, using prior knowledge, place reasonable predictions around model parameters, then one has a really nice and cheap way of assessing bounds and providing reasonable sample size prediction. However, the moment the experimental design is not MAB-like, then these results may not be strictly true.<\/i> \n<b>Response<\/b>: We thank the reviewer for this insightful comment. We agree that the present findings are specific to experiments modelled within a multi-armed bandit framework. We have revised the manuscript to more clearly contextualise the results within this task structure and to avoid overgeneralization. \n<b>Revised Section:<\/b> \u201cIn this work, we used BFDA in the example of a binary multi-armed bandit (MAB) task with aversive outcomes to analyse the effect of sample size, number of games per participant, and effect size on the probability of obtaining significant evidence to support a null or alternative hypothesis, and the probabilities of incorrectly supporting either hypothesis.\u201d &nbsp; \u201cIt is important to note that these results are specific to the task used. Although some findings may generalize to similar MAB learning tasks, they do not necessarily extend to other experimental paradigms.\u201d  \n<i>Point 30. \u201cexamine latent variables in relation to BFDA.\u201d Again, specific examples of this from results would be extremely beneficial.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this comment. We have revised the text accordingly to improve clarity. \n<b>Revised Sections:<\/b> \u201cThe example of a multi-armed bandit (MAB) task with prior information was used to examine latent variables in relation to BFDA. Specifically, we considered a hypothetical experiment designed to assess differences in the exploration parameter (\u03c4), estimated via the softmax decision-rule, between two groups. We showed how sample size, number of games per participant and effect size influence the probability of obtaining evidence for group differences in \u03c4.\u201d &nbsp; \u201cWe found that it can be challenging to determine the difference in means between two groups. Large sample sizes are needed for a strong probability of detecting evidence in favour of difference, where present. If there is strong evidence supporting the alternative hypothesis, the probability of overestimating the difference in means is relatively high. In the example we used an uninformative prior, but depending on the experiment and existing literature, this should be adjusted. A well-informed prior can lead to higher probabilities of detecting a difference between groups.\u201d  \n<i>Point 31. \u201cIn the example we used an uninformative prior\u201d This needs to be placed in methods section to orient the reader to what priors are being used. Add in uniform prior parameters as well. See (Kruschke, 2021).<\/i> \n<b>Response<\/b>: We thank the reviewer for pointing this out. We have revised the methods section to explicitly state the prior specification, including the parameters of the uniform prior. \n<b>Revised Section:<\/b> \n<b>\u201c<\/b>For both tau and the difference in population means (\u0394\u03bc_\u03c4), we used a discrete uniform prior defined over a bounded parameter space (\u03c4 \u2208 [0.01, 3] and \u0394\u03bc_\u03c4 \u2208 [-3, 3]).\u201d  \n<b>References<\/b> 1. Cronin B, Stevenson IH, Sur M, et al.:&nbsp;Hierarchical Bayesian modeling and Markov chain Monte Carlo sampling for tuning-curve analysis.&nbsp;\n<i>J Neurophysiol.&nbsp;<\/i>2010;\n<b>&nbsp;103<\/b>&nbsp;(1):&nbsp;591-602&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text 2. Gelman A:&nbsp;Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper).&nbsp;\n<i>Bayesian Analysis.&nbsp;<\/i>2006;\n<b>&nbsp;1<\/b>&nbsp;(3):&nbsp;|&nbsp;Publisher Full Text 3. Gelman A, Carlin J:&nbsp;Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.&nbsp;\n<i>Perspect Psychol Sci.&nbsp;<\/i>2014;\n<b>&nbsp;9<\/b>&nbsp;(6):&nbsp;641-51&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text 4. Gelman A, Rubin D:&nbsp;Avoiding Model Selection in Bayesian Social Research.&nbsp;\n<i>Sociological Methodology.&nbsp;<\/i>1995;\n<b>&nbsp;25<\/b>&nbsp;|&nbsp;Publisher Full Text 5. Gelman A, Shalizi CR:&nbsp;Philosophy and the practice of Bayesian statistics.&nbsp;\n<i>Br J Math Stat Psychol.&nbsp;<\/i>2013;\n<b>&nbsp;66<\/b>&nbsp;(1):&nbsp;8-38&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text 6. Kruschke JK:&nbsp;Bayesian Analysis Reporting Guidelines.&nbsp;\n<i>Nat Hum Behav.&nbsp;<\/i>2021;\n<b>&nbsp;5<\/b>&nbsp;(10):&nbsp;1282-1291&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text 7. Kruschke JK:&nbsp;Bayesian estimation supersedes the t test.&nbsp;\n<i>J Exp Psychol Gen.&nbsp;<\/i>2013;\n<b>&nbsp;142<\/b>&nbsp;(2):&nbsp;573-603&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text 8. Kruschke JK:&nbsp;What to believe: Bayesian methods for data analysis.&nbsp;\n<i>Trends Cogn Sci.&nbsp;<\/i>2010;\n<b>&nbsp;14<\/b>&nbsp;(7):&nbsp;293-300&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text 9. Vize CE, Phillips NL, Miller JD, et al.:&nbsp;On the Use and Misuses of Preregistration: A Reply to Klonsky (2024).&nbsp;\n<i>Assessment.&nbsp;<\/i>2024;&nbsp;10731911241275256&nbsp;PubMed Abstract&nbsp;|&nbsp;Publisher Full Text Gelman, A., Hill, J., 2006. Data Analysis Using Regression and Multilevel\/Hierarchical Models, 1st ed. Cambridge University Press, Cambridge, United Kingdom. Blackwell, D.L., Ramamoorthi, R.V., 1982. A Bayes but Not Classically Sufficient Statistic. The Annals of Statistics 10, 1025\u20131026.","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2#referee-comment-6532","order":1,"name":"referee-comment-6532","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.21956\/wellcomeopenres.24565.r95853, Dominik Bach, University of Bonn, Bonn, 26 Nov 2024, version 1, 3 approved with reservations","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v1#referee-response-95853","order":2,"name":"referee-response-95853","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"<b>Sarah Schreiber<\/b>; \n<i>Posted: 25 Apr 2026<\/i>; \n<i>Schreiber et al. give a tutorial-style introduction into, and worked example of, the established framework of Bayes Factor Design Analysis, applied in the context of a multi-armed bandit task. No new method is developed. Thus, the original scientific contribution appears relatively small, but the worked example provides a welcome illustration of this (suite of) method(s) for students of the field. My major concern is that two assumptions of the method are not sufficiently discussed; there are also a couple of minor technical concerns.<\/i> &nbsp; \n<b>Response<\/b>:\n<i> <\/i>We are thankful to Reviewer 2 for a generally positive evaluation of our work. &nbsp; \n<i>Point 1 &amp; 2:<\/i> \n<i>Any results hinge on the assumption of a specific data-generating process. If the true data-generating process (in a future experiment) is not the same as the one assumed in the simulation, then results such as required group size etc. will be misleading. The authors might want to refer to (Melinscak and Bach, 2020) as an example of how to use Bayesian Design Optimisation to plan experiments to decide between data-generating models, rather than parameters of the same model in different participant groups, as done here.<\/i> &nbsp; \n<i>The results also hinge on the (somewhat related but not equivalent) assumption that the data-generating process is exactly the same as the model used to analyse the data (i.e. the measurement model). The appropriateness of the measurement model can be assessed wihtout knowing the data-generating process, by running an experiment with a manipulation the effect of which is (somewhat) known, and testing different data analysis methods in terms of how well they reproduce the known effect. This method is known as experiment-based calibration (Bach et al., 2023, 2020), and the relevant metric as retrodictive validity (Bach and Melinscak, 2020).<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for highlighting this important point. In response, we have expanded the limitations section to more clearly discuss the dependence of BFDA on the assumed generative model. We now also explicitly refer to Bayesian Design Optimisation as an approach for comparing competing data-generating models, and to experiment-based calibration for evaluating the validity of the measurement model. \n<b>Revised Section:<\/b> \u201cThe dependence on an accurate generative model is a general limitation of using BFDA, as design recommendations are conditional on that model. If the true data-generating process deviates from the specified model, conclusions about the optimal design may be inaccurate. In settings with multiple competing computational models aiming to describe a process, Bayesian Design Optimisation can be used to compare candidate data-generating models (Melinscak and Bach, 2020). Independently, the adequacy of the analysis model can be evaluated empirically using experiment-based calibration (Bach &amp; Melinscak, 2020; Bach et al., 2020; Bach et al., 2023).\u201d &nbsp;  \n<i>Point 3. The notation of many integrals is incorrect; sometimes the integration variable (\u201edtheta\u201c) is not stated (e.g. eq. 1-2), often the integration interval is missing or incorrectly replaced with the integration variable (e.g. eq. 1 should read int_{integration_interval} rather than int_{integration_variable}.).<\/i> \n<b>Response<\/b>: We thank the reviewer for pointing this out and have revised the equations accordingly. &nbsp;  \n<i>Point 4. It is rather confusing that the same symbol P is used to refer to multiple functions depending on what argument they take. That is, P(theta) is supposed to be something different from, let\u2019s say, P(omega). This is an abuse of mathematical notation. A common and useful convention in mathematical statistics is to differentiate probability distributions by subscript (e.g. P_X|Y, P_theta, P_omega, etc.). Psychological statistics literature is obviously rife with this abuse of notation, but in a sophisticated and heavily mathematical manuscript like this, the authors can clearly do better.<\/i> &nbsp; \n<b>Response<\/b>: We thank the reviewer for this helpful comment. We have revised the notation throughout the manuscript to avoid overloading the symbol&nbsp;&nbsp;and making clear when probability distributions are being used. &nbsp; &nbsp; \n<b>References<\/b> 1. Dominik R. Bach, Filip Melinscak:&nbsp;Psychophysiological modelling and the measurement of fear conditioning.&nbsp;|&nbsp;Publisher Full Text 2. Dominik R. Bach ,Juliana Sporrer,Rany Abend,Tom Beckers,Joseph E. Dunsmoor,Miquel A. Fullana,Matthias Gamer,Dylan G. Gee,Alfons Hamm,Catherine A. Hartley,Ryan J. Herringa,Tanja Jovanovic,Raffael Kalisch,David C. Knight,Shmuel Lissek,Tina B. Lonsdorf,Christian J. Merz, Mohammed Milad, Jayne Morriss, Elizabeth A. Phelps, Daniela Schiller:&nbsp;Consensus design of a calibration experiment for human fear conditioning.&nbsp;|&nbsp;Publisher Full Text Melinscak, F., Bach, D.R., 2020. Computational optimization of associative learning experiments. PLoS Comput Biol. https:\/\/doi.org\/10.1371\/journal.pcbi.1007593. &nbsp;","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2#referee-comment-6533","order":3,"name":"referee-comment-6533","label":"Referee Comment","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.21956\/wellcomeopenres.28844.r155003, Jessica Schaaf, Medical Neuroscience, adboud University Donders Institute for Brain Cognition and Behaviour (Ringgold ID: 198328), Nijmegen, Gelderland, 05 Jun 2026, version 2, indexed","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2#referee-response-155003","order":4,"name":"referee-response-155003","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"10.21956\/wellcomeopenres.28844.r155582, Tommaso Costa, University of Turin, Turin, 05 Jun 2026, version 2, indexed","URL":"https:\/\/wellcomeopenresearch.org\/articles\/9-423\/v2#referee-response-155582","order":5,"name":"referee-response-155582","label":"Referee Report","group":{"name":"article-reports","label":"Article Reports"}},{"value":"The work was funded by Wellcome Trust (214251\/Z\/18\/Z, 203139\/Z\/16\/Z, and 203139\/A\/16\/Z), EPSRC (EP\/W03509X\/1), IITP (MSIT 2019-0-01371) and JSPS KAKENHI (22H04998). This research was also part supported by the NIHR Oxford Health Biomedical Research Centre. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. This work was supported by a fellowship of the German Academic Exchange Service (DAAD). W.Y. was funded by Versus Arthritis (ARUK-21537) and MRC 713 (MR\/W027593\/1), UK. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.","order":6,"name":"grant-information","label":"Grant Information"},{"value":"This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.","order":0,"name":"copyright-info","label":"Copyright"}]}}