{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T19:38:17Z","timestamp":1780083497746,"version":"3.54.0"},"reference-count":60,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,4,30]],"date-time":"2021-04-30T00:00:00Z","timestamp":1619740800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Entropy makes it possible to measure the uncertainty about an information source from the distribution of its output symbols. It is known that the maximum Shannon\u2019s entropy of a discrete source of information is reached when its symbols follow a Uniform distribution. In cryptography, these sources have great applications since they allow for the highest security standards to be reached. In this work, the most effective estimator is selected to estimate entropy in short samples of bytes and bits with maximum entropy. For this, 18 estimators were compared. Results concerning the comparisons published in the literature between these estimators are discussed. The most suitable estimator is determined experimentally, based on its bias, the mean square error short samples of bytes and bits.<\/jats:p>","DOI":"10.3390\/e23050561","type":"journal-article","created":{"date-parts":[[2021,4,30]],"date-time":"2021-04-30T10:53:29Z","timestamp":1619780009000},"page":"561","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Selecting an Effective Entropy Estimator for Short Sequences of Bits and Bytes with Maximum Entropy"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0449-8611","authenticated-orcid":false,"given":"Lianet","family":"Contreras Rodr\u00edguez","sequence":"first","affiliation":[{"name":"Facultad de Matem\u00e1tica y Computaci\u00f3n, Instituto de Criptograf\u00eda, Universidad de la Habana, Habana 10400, Cuba"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5004-2960","authenticated-orcid":false,"given":"Evaristo Jos\u00e9","family":"Madarro-Cap\u00f3\u00a0","sequence":"additional","affiliation":[{"name":"Facultad de Matem\u00e1tica y Computaci\u00f3n, Instituto de Criptograf\u00eda, Universidad de la Habana, Habana 10400, Cuba"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6104-9671","authenticated-orcid":false,"given":"Carlos Miguel","family":"Leg\u00f3n-P\u00e9rez\u00a0","sequence":"additional","affiliation":[{"name":"Facultad de Matem\u00e1tica y Computaci\u00f3n, Instituto de Criptograf\u00eda, Universidad de la Habana, Habana 10400, Cuba"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0681-3833","authenticated-orcid":false,"given":"Omar","family":"Rojas","sequence":"additional","affiliation":[{"name":"Facultad de Ciencias Econ\u00f3micas y Empresariales, Universidad Panamericana, \u00c1lvaro del Portillo 49, Zapopan, Jalisco 45010, Mexico"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7793-896X","authenticated-orcid":false,"given":"Guillermo","family":"Sosa-G\u00f3mez","sequence":"additional","affiliation":[{"name":"Facultad de Ciencias Econ\u00f3micas y Empresariales, Universidad Panamericana, \u00c1lvaro del Portillo 49, Zapopan, Jalisco 45010, Mexico"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Cover, T.M., and Thomas, J.A. (2006). Elements of Information Theory, John Wiley & Sons.","DOI":"10.1002\/047174882X"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Verd\u00fa, S. (2019). Empil Estimation of Information Measures: A Literature guide. Entropy, 21.","DOI":"10.3390\/e21080720"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4039","DOI":"10.1002\/sim.2942","article-title":"Coverage-adjusted entropy estimation","volume":"26","author":"Vu","year":"2007","journal-title":"Stat. Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1162\/089976603321780272","article-title":"Estimation of entropy and mutual information","volume":"15","author":"Paninski","year":"2003","journal-title":"Neural Comput."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1002\/rsa.10019","article-title":"Convergence properties of functional estimates for discrete distributions","volume":"19","author":"Antos","year":"2001","journal-title":"Random Struct. Algorithms"},{"key":"ref_6","first-page":"2833","article-title":"Bayesian entropy estimation for countable discrete distributions","volume":"15","author":"Archer","year":"2014","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Timme, N.M., and Lapish, C. (2018). A tutorial for information theory in neuroscience. eNeuro, 5.","DOI":"10.1523\/ENEURO.0052-18.2018"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1007\/s10994-019-05795-1","article-title":"Efficient feature selection using shrinkage estimators","volume":"108","author":"Sechidis","year":"2019","journal-title":"Mach. Learn."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1007\/s11042-019-08088-w","article-title":"An empirical approach towards characterization of encrypted and unencrypted VoIP traffic","volume":"79","author":"Choudhury","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"13016","DOI":"10.1103\/PhysRevResearch.2.013016","article-title":"Efficient randomness certification by quantum probability estimation","volume":"2","author":"Zhang","year":"2020","journal-title":"Phys. Rev. Res."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-9-461","article-title":"Minet: A r\/bioconductor package for inferring large transcriptional networks using mutual information","volume":"9","author":"Meyer","year":"2008","journal-title":"BMC Bioinform."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"695","DOI":"10.3906\/elk-1312-90","article-title":"Comprehensive review of association estimators for the inference of gene networks","volume":"24","author":"Kurt","year":"2016","journal-title":"Turk. J. Electr. Eng. Comput. Sci."},{"key":"ref_13","first-page":"29","article-title":"Entropy: An Essential Component of Cryptographic Security","volume":"1","author":"Schulman","year":"2019","journal-title":"J. Cybersecur. Aware. Educ."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.3390\/e17031135","article-title":"Comparing security notions of secret sharing schemes","volume":"17","author":"Dai","year":"2015","journal-title":"Entropy"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1007\/s00453-016-0219-7","article-title":"On the Impossibility of Cryptography with Tamperable Randomness","volume":"79","author":"Austrin","year":"2017","journal-title":"Algorithmica"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Yasser, I., Mohamed, M.A., Samra, A.S., and Khalifa, F. (2020). A chaotic-based encryption\/decryption framework for secure multimedia communications. Entropy, 22.","DOI":"10.3390\/e22111253"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"25664","DOI":"10.1109\/ACCESS.2020.2970806","article-title":"An Efficient Image Encryption Scheme Based on the LSS Chaotic Map and Single S-Box","volume":"8","author":"Lu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_18","unstructured":"Knuth, D. (1981). The Art of Computer Programming: Volume 2, Seminumerical Algoritms, Addison-Wesley Professional."},{"key":"ref_19","unstructured":"(2021, April 18). Pseudorandom Number Sequence Test Program. Available online: http:\/\/www.fourmilab.ch\/random\/."},{"key":"ref_20","unstructured":"(2021, April 18). Marsaglia, George; The Marsaglia Random Number CDROM Including the Diehard Battery of Tests of Randomness; Natl. Sci. Found. (Grants DMS-8807976 DMS-9206972). Available online: http:\/\/stat.fsu.edu\/pub\/diehard\/."},{"key":"ref_21","unstructured":"Rukhin, A., Soto, J., Nechvatal, J., Miles, S., Barker, E., Leigh, S., Levenson, M., Vangel, M., Banks, D., and Heckert, A. (2021, April 22). SP800-22: A Statistical Test Suite for Random and Pseudorandom Number Generators for Cryptographic Applications, Available online: http:\/\/csrc.nist.gov\/groups\/ST\/toolkit\/rng\/documents\/SP800-22rev1a.pdf."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"L\u2019ecuyer, P., and Simard, R. (2007). TestU01: A C library for empirical testing of random number generators. ACM Trans. Math. Softw., 33.","DOI":"10.1145\/1268776.1268777"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Madarro-Cap\u00f3, E.J., Leg\u00f3n-P\u00e9rez, C.M., Rojas, O., Sosa-G\u00f3mez, G., and Socorro-Llanes, R. (2020). Bit independence criterion extended to stream ciphers. Appl. Sci., 10.","DOI":"10.3390\/app10217668"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Madarro Cap\u00f3, E.J., Cuellar, O.J., Leg\u00f3n P\u00e9rez, C.M., and G\u00f3mez, G.S. (2017, January 12\u201314). Evaluation of input\u2014Output statistical dependence PRNGs by SAC. Proceedings of the 2016 International Conference on Software Process Improvement (CIMPS), Aguascalientes, Mexico.","DOI":"10.1109\/CIMPS.2016.7802810"},{"key":"ref_25","first-page":"108","article-title":"Note on the bias of information estimates","volume":"71","author":"Miller","year":"1955","journal-title":"Inf. Theory Psychol. Probl. Methods"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Nemenman, I., Shafee, F., and Bialek, W. (2001). Entropy and Inference, Revisited. arXiv.","DOI":"10.7551\/mitpress\/1120.003.0065"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1063\/1.166191","article-title":"Entropy estimation of symbol sequences","volume":"6","author":"Grassberger","year":"1996","journal-title":"Chaos"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1023\/A:1026096204727","article-title":"Nonparametric estimation of Shannon\u2019s index of diversity when there are unseen species in sample","volume":"10","author":"Chao","year":"2003","journal-title":"Environ. Ecol. Stat."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2551","DOI":"10.1088\/0305-4470\/31\/11\/007","article-title":"Bayes\u2019 estimators of generalized entropies","volume":"31","author":"Holste","year":"1998","journal-title":"J. Phys. A. Math. Gen."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1109\/TIT.1981.1056331","article-title":"The Performance of Universal Encoding","volume":"27","author":"Krichevsky","year":"1981","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1214\/aoms\/1177706722","article-title":"Some problems of simultaneous minimax estimation","volume":"29","author":"Trybula","year":"1958","journal-title":"Ann. Math. Stat."},{"key":"ref_32","first-page":"1469","article-title":"Entropy inference and the james-stein estimator, with application to nonlinear gene association networks","volume":"10","author":"Hausser","year":"2009","journal-title":"J. Mach. Learn. Res."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3125643","article-title":"Estimating the unseen: Improved estimators for entropy and other properties","volume":"64","author":"Valiant","year":"2017","journal-title":"J. ACM"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1368","DOI":"10.1162\/NECO_a_00266","article-title":"Entropy estimation in Turing\u2019s perspective","volume":"24","author":"Zhang","year":"2012","journal-title":"Neural Comput."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Daub, C.O., Steuer, R., Selbig, J., and Kloska, S. (2004). Estimating mutual information using B-spline functions\u2014An improved similarity measure for analysing gene expression data. BMC Bioinform., 5.","DOI":"10.1186\/1471-2105-5-118"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-7-S1-S7","article-title":"ARACNE: An algorithm for the reconstruction of gene regulatory networks in a mammalian cellular context","volume":"7","author":"Margolin","year":"2006","journal-title":"BMC Bioinform."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1903","DOI":"10.1162\/0899766054323026","article-title":"Edgeworth approximation of multivariate differential entropy","volume":"17","year":"2005","journal-title":"Neural Comput."},{"key":"ref_38","first-page":"1054","article-title":"Estimation of the entropy on the basis of its polynomial representation","volume":"85","author":"Vinck","year":"2012","journal-title":"IEEE Int. Symp. Inf. Theory Proc."},{"key":"ref_39","first-page":"95","article-title":"Sample Estimate of the Entropy of a Random Vector","volume":"23","author":"Kozachenko","year":"1987","journal-title":"Probl. Inf. Transm."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"202001","DOI":"10.1088\/1751-8113\/41\/20\/202001","article-title":"Entropy estimates of small data sets","volume":"41","author":"Bonachela","year":"2008","journal-title":"J. Phys. A Math. Theor."},{"key":"ref_41","unstructured":"Grassberger, P. (2003). Entropy estimates from insufficient samplings. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"L295","DOI":"10.1088\/0305-4470\/37\/27\/L02","article-title":"Bias analysis in entropy estimation","volume":"37","year":"2004","journal-title":"J. Phys. A. Math. Gen."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1091","DOI":"10.1111\/2041-210X.12108","article-title":"Entropy and the species accumulation curve: A novel entropy estimator via discovery rates of new species","volume":"4","author":"Chao","year":"2013","journal-title":"Methods Ecol. Evol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1093\/biomet\/65.3.625","article-title":"Estimation of the Size of a Closed Population when Capture Probabilities vary Among Animals","volume":"65","author":"Burnham","year":"1978","journal-title":"Biometrika"},{"key":"ref_45","first-page":"1700","article-title":"Bayesian entropy estimation for binary spike train data using parametric prior knowledge","volume":"15","author":"Archer","year":"2013","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Valiant, G., and Valiant, P. (2011). Estimating the unseen: An n\/log(n)-sample estimator for entropy and support size, shown optimal via new CLTs. Proc. Annu. ACM Symp. Theory Comput., 685\u2013694.","DOI":"10.1145\/1993636.1993727"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"2013","DOI":"10.3390\/e13122013","article-title":"Coincidences and estimation of entropies of random variables with large cardinalities","volume":"13","author":"Nemenman","year":"2011","journal-title":"Entropy"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"88","DOI":"10.22237\/jmasm\/1446350940","article-title":"New entropy estimators with smaller root mean squared error","volume":"14","year":"2015","journal-title":"J. Mod. Appl. Stat. Methods"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"6841","DOI":"10.1103\/PhysRevE.52.6841","article-title":"Estimating functions of probability distributions from a finite set of samples","volume":"52","author":"Wolpert","year":"1995","journal-title":"Phys. Rev. E"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2097","DOI":"10.1162\/NECO_a_00775","article-title":"A note on entropy estimation","volume":"27","year":"2015","journal-title":"Neural Comput."},{"key":"ref_51","first-page":"131","article-title":"Influence of Statistical Estimators on the Large-Scale Causal Inference of Regulatory Networks","volume":"6","year":"2012","journal-title":"Stat. Mach. Learn. Approaches Netw. Anal."},{"key":"ref_52","unstructured":"M\u00fcller, S. (2021, April 22). Linux Random Number Generator-A New Approach. Available online: http:\/\/www.chronox.de\/lrng\/doc\/lrng.pdf."},{"key":"ref_53","first-page":"219","article-title":"Randomness in digital cryptography: A survey","volume":"13","author":"Marton","year":"2010","journal-title":"Rom. J. Inf. Sci. Technol."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"2570","DOI":"10.1162\/NECO_a_00646","article-title":"Nonparametric estimation of K\u00fcllback-Leibler divergence","volume":"26","author":"Zhang","year":"2014","journal-title":"Neural Comput."},{"key":"ref_55","unstructured":"(2021, March 15). GitHub\u2014Simomarsili\/ndd: Bayesian Entropy Estimation in Python\u2014Via the Nemenman-Schafee-Bialek Algorithm. Available online: https:\/\/github.com\/simomarsili\/ndd."},{"key":"ref_56","first-page":"1","article-title":"entropart: An R package to measure and partition diversity","volume":"11","author":"Marcon","year":"2015","journal-title":"J. Stat. Softw."},{"key":"ref_57","unstructured":"(2021, March 15). GitHub\u2014Pillowlab\/CDMentropy: Centered Dirichlet Mixture Entropy Estimator for Binary Data. Available online: https:\/\/github.com\/pillowlab\/CDMentropy."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Rosenblad, A. (2011). The Concise Encyclopedia of Statistics, Springer Science & Business Media.","DOI":"10.1080\/02664760903075614"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"8","DOI":"10.20982\/tqmp.11.1.p008","article-title":"Hierarchical Cluster Analysis: Comparison of Three Linkage Measures and Application to Psychological Data","volume":"11","author":"Yim","year":"2015","journal-title":"Quant. Methods Psychol."},{"key":"ref_60","unstructured":"Ma, X., and Dhavala, S. (2018). Hierarchical clustering with prior knowledge. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/5\/561\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:56:08Z","timestamp":1760162168000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/5\/561"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,30]]},"references-count":60,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["e23050561"],"URL":"https:\/\/doi.org\/10.3390\/e23050561","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,30]]}}}