{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T00:39:51Z","timestamp":1773707991561,"version":"3.50.1"},"reference-count":59,"publisher":"Public Library of Science (PLoS)","issue":"12","license":[{"start":{"date-parts":[[2020,12,23]],"date-time":"2020-12-23T00:00:00Z","timestamp":1608681600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1344256"],"award-info":[{"award-number":["IIS-1344256"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01MH118925"],"award-info":[{"award-number":["R01MH118925"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>The fate of scientific hypotheses often relies on the ability of a computational model to explain the data, quantified in modern statistical approaches by the likelihood function. The log-likelihood is the key element for parameter estimation and model evaluation. However, the log-likelihood of complex models in fields such as computational biology and neuroscience is often intractable to compute analytically or numerically. In those cases, researchers can often only estimate the log-likelihood by comparing observed data with synthetic observations generated by model simulations. Standard techniques to approximate the likelihood via simulation either use summary statistics of the data or are at risk of producing substantial biases in the estimate. Here, we explore another method, inverse binomial sampling (IBS), which can estimate the log-likelihood of an entire data set efficiently and without bias. For each observation, IBS draws samples from the simulator model until one matches the observation. The log-likelihood estimate is then a function of the number of samples drawn. The variance of this estimator is uniformly bounded, achieves the minimum variance for an unbiased estimator, and we can compute calibrated estimates of the variance. We provide theoretical arguments in favor of IBS and an empirical assessment of the method for maximum-likelihood estimation with simulation-based models. As case studies, we take three model-fitting problems of increasing complexity from computational and cognitive neuroscience. In all problems, IBS generally produces lower error in the estimated parameters and maximum log-likelihood values than alternative sampling methods with the same average number of samples. Our results demonstrate the potential of IBS as a practical, robust, and easy to implement method for log-likelihood evaluation when exact techniques are not available.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1008483","type":"journal-article","created":{"date-parts":[[2020,12,23]],"date-time":"2020-12-23T21:08:18Z","timestamp":1608757698000},"page":"e1008483","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":32,"title":["Unbiased and efficient log-likelihood estimation with inverse binomial sampling"],"prefix":"10.1371","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7169-4885","authenticated-orcid":true,"given":"Bas","family":"van Opheusden","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7471-7336","authenticated-orcid":true,"given":"Luigi","family":"Acerbi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9835-9083","authenticated-orcid":true,"given":"Wei Ji","family":"Ma","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2020,12,23]]},"reference":[{"issue":"1","key":"pcbi.1008483.ref001","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/S0022-2496(02)00028-7","article-title":"Tutorial on maximum likelihood estimation","volume":"47","author":"IJ Myung","year":"2003","journal-title":"Journal of Mathematical Psychology"},{"key":"pcbi.1008483.ref002","doi-asserted-by":"crossref","DOI":"10.1201\/b16018","volume-title":"Bayesian data analysis","author":"A Gelman","year":"2013"},{"key":"pcbi.1008483.ref003","doi-asserted-by":"crossref","unstructured":"Akaike H. A new look at the statistical model identification. In: Selected Papers of Hirotugu Akaike. Springer; 1974. p. 215\u2013222.","DOI":"10.1007\/978-1-4612-1694-0_16"},{"issue":"4","key":"pcbi.1008483.ref004","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1111\/1467-9868.00353","article-title":"Bayesian measures of model complexity and fit","volume":"64","author":"DJ Spiegelhalter","year":"2002","journal-title":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)"},{"issue":"Dec","key":"pcbi.1008483.ref005","first-page":"3571","article-title":"Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory","volume":"11","author":"S Watanabe","year":"2010","journal-title":"Journal of Machine Learning Research"},{"issue":"5","key":"pcbi.1008483.ref006","doi-asserted-by":"crossref","first-page":"1413","DOI":"10.1007\/s11222-016-9696-4","article-title":"Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC","volume":"27","author":"A Vehtari","year":"2017","journal-title":"Statistics and Computing"},{"key":"pcbi.1008483.ref007","volume-title":"Information theory, inference and learning algorithms","author":"DJ MacKay","year":"2003"},{"issue":"2","key":"pcbi.1008483.ref008","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1214\/aos\/1176344136","article-title":"Estimating the dimension of a model","volume":"6","author":"G Schwarz","year":"1978","journal-title":"The Annals of Statistics"},{"issue":"12","key":"pcbi.1008483.ref009","doi-asserted-by":"crossref","first-page":"1791","DOI":"10.1093\/oxfordjournals.molbev.a026091","article-title":"Population growth of human Y chromosomes: a study of Y chromosome microsatellites","volume":"16","author":"JK Pritchard","year":"1999","journal-title":"Molecular Biology and Evolution"},{"issue":"11","key":"pcbi.1008483.ref010","doi-asserted-by":"crossref","first-page":"e230","DOI":"10.1371\/journal.pcbi.0030230","article-title":"Using likelihood-free inference to compare evolutionary dynamics of the protein networks of H. pylori and P. falciparum","volume":"3","author":"O Ratmann","year":"2007","journal-title":"PLoS Computational Biology"},{"key":"pcbi.1008483.ref011","doi-asserted-by":"crossref","DOI":"10.1201\/b11812","volume-title":"Stochastic modelling for systems biology","author":"DJ Wilkinson","year":"2011"},{"issue":"4-5","key":"pcbi.1008483.ref012","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1007\/s00422-008-0263-8","article-title":"Minimal Hodgkin\u2013Huxley type models for different classes of cortical and thalamic neurons","volume":"99","author":"M Pospischil","year":"2008","journal-title":"Biological Cybernetics"},{"key":"pcbi.1008483.ref013","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511975899","volume-title":"Principles of computational modelling in neuroscience","author":"D Sterratt","year":"2011"},{"key":"pcbi.1008483.ref014","doi-asserted-by":"crossref","unstructured":"van Opheusden B, Bnaya Z, Galbiati G, Ma WJ. Do people think like computers? In: International Conference on Computers and Games. Springer; 2016. p. 212\u2013224.","DOI":"10.1007\/978-3-319-50935-8_20"},{"issue":"4","key":"pcbi.1008483.ref015","doi-asserted-by":"crossref","first-page":"2025","DOI":"10.1093\/genetics\/162.4.2025","article-title":"Approximate Bayesian computation in population genetics","volume":"162","author":"MA Beaumont","year":"2002","journal-title":"Genetics"},{"issue":"7310","key":"pcbi.1008483.ref016","doi-asserted-by":"crossref","first-page":"1102","DOI":"10.1038\/nature09319","article-title":"Statistical inference for noisy nonlinear ecological dynamic systems","volume":"466","author":"SN Wood","year":"2010","journal-title":"Nature"},{"issue":"3","key":"pcbi.1008483.ref017","doi-asserted-by":"crossref","first-page":"222","DOI":"10.1093\/biomet\/33.3.222","article-title":"On a method of estimating frequencies","volume":"33","author":"J Haldane","year":"1945","journal-title":"Biometrika"},{"issue":"1","key":"pcbi.1008483.ref018","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1214\/aoms\/1177706361","article-title":"Unbiased sequential estimation for binomial populations","volume":"30","author":"MH de Groot","year":"1959","journal-title":"The Annals of Mathematical Statistics"},{"issue":"3924","key":"pcbi.1008483.ref019","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1038\/155049b0","article-title":"A labour-saving method of sampling","volume":"155","author":"J Haldane","year":"1945","journal-title":"Nature"},{"issue":"1","key":"pcbi.1008483.ref020","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1214\/aoms\/1177731018","article-title":"Unbiased Estimates for Certain Binomial Sampling Problems with Applications","volume":"17","author":"M Girshick","year":"1946","journal-title":"The Annals of Mathematical Statistics"},{"key":"pcbi.1008483.ref021","volume-title":"Unbiased tests, unbiased estimators, and randomized similar regions","author":"R Dawson","year":"1953"},{"key":"pcbi.1008483.ref022","doi-asserted-by":"crossref","unstructured":"Duncan GM. Unbiased simulators for anaytic functions and maximum unbiased simulated likelihood estimation; 2004.","DOI":"10.2139\/ssrn.692921"},{"key":"pcbi.1008483.ref023","volume-title":"Handbook of mathematical functions with formulas, graphs, and mathematical tables","author":"M Abramowitz","year":"1948"},{"issue":"430","key":"pcbi.1008483.ref024","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1080\/01621459.1995.10476572","article-title":"Bayes factors","volume":"90","author":"RE Kass","year":"1995","journal-title":"Journal of the American Statistical Association"},{"key":"pcbi.1008483.ref025","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198503682.001.0001","volume-title":"The theory of probability","author":"H Jeffreys","year":"1998"},{"key":"pcbi.1008483.ref026","volume-title":"Model selection and multi-model inference: A practical information-theoretic approach","author":"D Anderson","year":"2002"},{"key":"pcbi.1008483.ref027","unstructured":"Acerbi L, Ma WJ. Practical Bayesian Optimization for Model Fitting with Bayesian Adaptive Direct Search. In: Advances in Neural Information Processing Systems; 2017. p. 1836\u20131846."},{"issue":"1","key":"pcbi.1008483.ref028","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/106365603321828970","article-title":"Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)","volume":"11","author":"N Hansen","year":"2003","journal-title":"Evolutionary Computation"},{"issue":"1","key":"pcbi.1008483.ref029","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1109\/TEVC.2008.924423","article-title":"A method for handling uncertainty in evolutionary optimization with an application to feedback control of combustion","volume":"13","author":"N Hansen","year":"2008","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"pcbi.1008483.ref030","doi-asserted-by":"crossref","unstructured":"Maximon LC. The dilogarithm function for complex argument. In: Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences. vol. 459. The Royal Society; 2003. p. 2807\u20132819.","DOI":"10.1098\/rspa.2003.1156"},{"issue":"1","key":"pcbi.1008483.ref031","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/biomet\/87.1.1","article-title":"Predicting the output from a complex computer code when fast approximations are available","volume":"87","author":"MC Kennedy","year":"2000","journal-title":"Biometrika"},{"issue":"2088","key":"pcbi.1008483.ref032","doi-asserted-by":"crossref","first-page":"3251","DOI":"10.1098\/rspa.2007.1900","article-title":"Multi-fidelity optimization via surrogate modelling","volume":"463","author":"AI Forrester","year":"2007","journal-title":"Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences"},{"issue":"4","key":"pcbi.1008483.ref033","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1023\/A:1008306431147","article-title":"Efficient global optimization of expensive black-box functions","volume":"13","author":"DR Jones","year":"1998","journal-title":"Journal of Global Optimization"},{"key":"pcbi.1008483.ref034","unstructured":"Snoek J, Larochelle H, Adams RP. Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems; 2012. p. 2951\u20132959."},{"issue":"1","key":"pcbi.1008483.ref035","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1109\/JPROC.2015.2494218","article-title":"Taking the human out of the loop: A review of Bayesian optimization","volume":"104","author":"B Shahriari","year":"2015","journal-title":"Proceedings of the IEEE"},{"key":"pcbi.1008483.ref036","volume-title":"Gaussian processes for machine learning","author":"CE Rasmussen","year":"2006"},{"key":"pcbi.1008483.ref037","unstructured":"Kandasamy K, Schneider J, P\u00f3czos B. Bayesian active learning for posterior estimation. In: Twenty-Fourth International Joint Conference on Artificial Intelligence; 2015."},{"key":"pcbi.1008483.ref038","unstructured":"Acerbi L. Variational Bayesian Monte Carlo. In: Advances in Neural Information Processing Systems; 2018. p. 8213\u20138223."},{"key":"pcbi.1008483.ref039","unstructured":"Acerbi L. Variational Bayesian Monte Carlo with Noisy Likelihoods. arXiv preprint arXiv:200608655. 2020."},{"key":"pcbi.1008483.ref040","unstructured":"J\u00e4rvenp\u00e4\u00e4 M, Gutmann M, Vehtari A, Marttinen P. Parallel Gaussian process surrogate method to accelerate likelihood-free inference. arXiv preprint arXiv:190501252. 2019."},{"issue":"6","key":"pcbi.1008483.ref041","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1037\/a0039400","article-title":"Is psychology suffering from a replication crisis? What does \u201cfailure to replicate\u201d really mean?","volume":"70","author":"SE Maxwell","year":"2015","journal-title":"American Psychologist"},{"issue":"1","key":"pcbi.1008483.ref042","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/BF01494395","article-title":"Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung","volume":"15","author":"JW Lindeberg","year":"1922","journal-title":"Mathematische Zeitschrift"},{"key":"pcbi.1008483.ref043","volume-title":"Probability and measure theory","author":"RB Ash","year":"2000"},{"issue":"1","key":"pcbi.1008483.ref044","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1137\/040603371","article-title":"Mesh adaptive direct search algorithms for constrained optimization","volume":"17","author":"C Audet","year":"2006","journal-title":"SIAM Journal on Optimization"},{"key":"pcbi.1008483.ref045","volume-title":"Signal detection theory and psychophysics","author":"DM Green","year":"1966"},{"issue":"8","key":"pcbi.1008483.ref046","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.3758\/BF03194544","article-title":"The psychometric function: I. Fitting, sampling, and goodness of fit","volume":"63","author":"FA Wichmann","year":"2001","journal-title":"Perception & Psychophysics"},{"issue":"6","key":"pcbi.1008483.ref047","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1167\/12.6.25","article-title":"The psychometric function: The lapse rate revisited","volume":"12","author":"N Prins","year":"2012","journal-title":"Journal of Vision"},{"issue":"22","key":"pcbi.1008483.ref048","doi-asserted-by":"crossref","first-page":"8780","DOI":"10.1073\/pnas.1117465109","article-title":"Variability in encoding precision accounts for visual short-term memory limitations","volume":"109","author":"R Van den Berg","year":"2012","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"pcbi.1008483.ref049","unstructured":"Acerbi L, Ma WJ, Vijayakumar S. A framework for testing identifiability of Bayesian models of perception. In: Advances in Neural Information Processing Systems; 2014. p. 1026\u20131034."},{"key":"pcbi.1008483.ref050","volume-title":"Numerical recipes in C++","author":"WH Press","year":"1992"},{"key":"pcbi.1008483.ref051","volume-title":"Elements of information theory","author":"TM Cover","year":"2012"},{"issue":"3","key":"pcbi.1008483.ref052","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1037\/h0056940","article-title":"Stimulus information as a determinant of reaction time","volume":"45","author":"R Hyman","year":"1953","journal-title":"Journal of Experimental Psychology"},{"issue":"6","key":"pcbi.1008483.ref053","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"L Paninski","year":"2003","journal-title":"Neural Computation"},{"issue":"1","key":"pcbi.1008483.ref054","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1093\/biomet\/57.1.97","article-title":"Monte Carlo sampling methods using Markov chains and their applications","volume":"57","author":"WK Hastings","year":"1970","journal-title":"Biometrika"},{"key":"pcbi.1008483.ref055","doi-asserted-by":"crossref","DOI":"10.1201\/b10905","volume-title":"Handbook of Markov Chain Monte Carlo","author":"S Brooks","year":"2011"},{"issue":"2","key":"pcbi.1008483.ref056","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1023\/A:1007665907178","article-title":"An introduction to variational methods for graphical models","volume":"37","author":"MI Jordan","year":"1999","journal-title":"Machine Learning"},{"key":"pcbi.1008483.ref057","unstructured":"Ranganath R, Gerrish S, Blei D. Black box variational inference. In: Artificial Intelligence and Statistics; 2014. p. 814\u2013822."},{"key":"pcbi.1008483.ref058","unstructured":"Acerbi L. An Exploration of Acquisition and Mean Functions in Variational Bayesian Monte Carlo. In: Symposium on Advances in Approximate Bayesian Inference; 2019. p. 1\u201310."},{"issue":"1","key":"pcbi.1008483.ref059","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1214\/16-BA1002","article-title":"Adapting the ABC distance function","volume":"12","author":"D Prangle","year":"2017","journal-title":"Bayesian Analysis"}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008483","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,20]],"date-time":"2024-08-20T01:33:04Z","timestamp":1724117584000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1008483"}},"subtitle":[],"editor":[{"given":"Daniele","family":"Marinazzo","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2020,12,23]]},"references-count":59,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2020,12,23]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1008483","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,23]]}}}