{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T08:19:15Z","timestamp":1773735555799,"version":"3.50.1"},"reference-count":45,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2025,2,2]],"date-time":"2025-02-02T00:00:00Z","timestamp":1738454400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004837","name":"MCIN\/AEI\/ 10.13039\/501100011033","doi-asserted-by":"publisher","award":["TED2021-130369B-C32"],"award-info":[{"award-number":["TED2021-130369B-C32"]}],"id":[{"id":"10.13039\/501100004837","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>This study investigates the application of machine learning predictors for the estimation of min-entropy in random number generators (RNGs), a key component in cryptographic applications where accurate entropy assessment is essential for cybersecurity. Our research indicates that these predictors, and indeed any predictor that leverages sequence correlations, primarily estimate average min-entropy, a metric not extensively studied in this context. We explore the relationship between average min-entropy and the traditional min-entropy, focusing on their dependence on the number of target bits being predicted. Using data from generalized binary autoregressive models, a subset of Markov processes, we demonstrate that machine learning models (including a hybrid of convolutional and recurrent long short-term memory layers and the transformer-based GPT-2 model) outperform traditional NIST SP 800-90B predictors in certain scenarios. Our findings underscore the importance of considering the number of target bits in min-entropy assessment for RNGs and highlight the potential of machine learning approaches in enhancing entropy estimation techniques for improved cryptographic security.<\/jats:p>","DOI":"10.3390\/e27020156","type":"journal-article","created":{"date-parts":[[2025,2,3]],"date-time":"2025-02-03T05:36:32Z","timestamp":1738560992000},"page":"156","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Machine Learning Predictors for Min-Entropy Estimation"],"prefix":"10.3390","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-0635-953X","authenticated-orcid":false,"given":"Javier","family":"Blanco-Romero","sequence":"first","affiliation":[{"name":"Department of Telematic Engineering, Universidad Carlos III de Madrid, Legan\u00e9s, 28911 Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2077-6095","authenticated-orcid":false,"given":"Vicente","family":"Lorenzo","sequence":"additional","affiliation":[{"name":"Department of Telematic Engineering, Universidad Carlos III de Madrid, Legan\u00e9s, 28911 Madrid, Spain"},{"name":"Department of Applied Mathematics for ICT, Universidad Polit\u00e9cnica de Madrid, 28031 Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5232-2031","authenticated-orcid":false,"given":"Florina","family":"Almenares Mendoza","sequence":"additional","affiliation":[{"name":"Department of Telematic Engineering, Universidad Carlos III de Madrid, Legan\u00e9s, 28911 Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3323-6453","authenticated-orcid":false,"given":"Daniel","family":"D\u00edaz-S\u00e1nchez","sequence":"additional","affiliation":[{"name":"Department of Telematic Engineering, Universidad Carlos III de Madrid, Legan\u00e9s, 28911 Madrid, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2025,2,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bassham, L.E., Rukhin, A.L., Soto, J., Nechvatal, J.R., Smid, M.E., Barker, E.B., Leigh, S.D., Levenson, M., Vangel, M., and Banks, D.L. (2010). A statistical test suite for random and pseudorandom number generators for cryptographic applications, NIST Special Publication.","DOI":"10.6028\/NIST.SP.800-22r1a"},{"key":"ref_2","unstructured":"Marsaglia, G. (2024, June 28). The Marsaglia Random Number CDROM Including the Diehard Battery of Tests of Randomness. Available online: http:\/\/www.stat.fsu.edu\/pub\/diehard\/."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"045103","DOI":"10.1088\/1402-4896\/aaf36a","article-title":"Experimentally probing the algorithmic randomness and incomputability of quantum randomness","volume":"94","author":"Abbott","year":"2019","journal-title":"Phys. Scr."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"022102","DOI":"10.1103\/PhysRevA.82.022102","article-title":"Experimental evidence of quantum randomness incomputability","volume":"82","author":"Calude","year":"2010","journal-title":"Phys. Rev. A"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"127032","DOI":"10.1016\/j.physleta.2020.127032","article-title":"Searching for evidence of algorithmic randomness and incomputability in the output of quantum random number generators","volume":"388","author":"Kavulich","year":"2021","journal-title":"Phys. Lett. A"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"9243","DOI":"10.1007\/s00500-019-04450-0","article-title":"On the effects of pseudorandom and quantum-random number generators in soft computing","volume":"24","author":"Bird","year":"2020","journal-title":"Soft Comput."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Turan, M.S., Barker, E., Kelsey, J., McKay, K.A., Baish, M.L., and Boyle, M. (2018). Recommendation for the entropy sources used for random bit generation, NIST Special Publication.","DOI":"10.6028\/NIST.SP.800-90B"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5898","DOI":"10.1021\/acsnano.1c11084","article-title":"A High-Quality Entropy Source Using van der Waals Heterojunction for True Random Number Generation","volume":"16","author":"Abraham","year":"2022","journal-title":"ACS Nano"},{"key":"ref_9","first-page":"5144","article-title":"Using ECG signal as an entropy source for efficient generation of long random bit sequences","volume":"34","author":"Islam","year":"2022","journal-title":"J. King Saud Univ. Comput. Inf. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Li, X., Stanwicks, P., Provelengios, G., Tessier, R., and Holcomb, D. (2020, January 9\u201311). Jitter-based adaptive true random number generation for FPGAs in the cloud. Proceedings of the 2020 International Conference on Field-Programmable Technology (ICFPT), Maui, HI, USA.","DOI":"10.1109\/ICFPT51103.2020.00024"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1109\/TIFS.2018.2850770","article-title":"Machine learning cryptanalysis of a quantum random number generator","volume":"14","author":"Truong","year":"2018","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Yang, J., Zhu, S., Chen, T., Ma, Y., Lv, N., and Lin, J. (2018, January 8\u201310). Neural network based min-entropy estimation for random number generators. Proceedings of the International Conference on Security and Privacy in Communication Systems, Singapore.","DOI":"10.1007\/978-3-030-01704-0_13"},{"key":"ref_13","first-page":"4241713","article-title":"High-efficiency min-entropy estimation based on neural network for random number generators","volume":"2020","author":"Lv","year":"2020","journal-title":"Secur. Commun. Netw."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, C., Zhang, J., Sang, L., Gong, L., Wang, L., Wang, A., and Wang, Y. (2020). Deep learning-based security verification for a random number generator using white chaos. Entropy, 22.","DOI":"10.3390\/e22101134"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"163685","DOI":"10.1109\/ACCESS.2020.3022098","article-title":"Testing randomness using artificial neural network","volume":"8","author":"Feng","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1410","DOI":"10.1109\/TIFS.2023.3240859","article-title":"Improvement of Min-entropy evaluation based on pruning and quantized deep neural network","volume":"18","author":"Li","year":"2023","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kelsey, J., McKay, K.A., and S\u00f6nmez Turan, M. (2015, January 13\u201316). Predictive models for min-entropy estimation. Proceedings of the Cryptographic Hardware and Embedded Systems\u2014CHES 2015: 17th International Workshop, Saint-Malo, France. Proceedings 17.","DOI":"10.1007\/978-3-662-48324-4_19"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhu, S., Ma, Y., Chen, T., Lin, J., and Jing, J. (2017). Analysis and improvement of entropy estimators in NIST SP 800-90B for non-IID entropy sources. IACR Transactions on Symmetric Cryptology, Ruhr University Bochum.","DOI":"10.46586\/tosc.v2017.i3.151-168"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Dodis, Y., Reyzin, L., and Smith, A. (2004, January 2\u20136). Fuzzy extractors: How to generate strong keys from biometrics and other noisy data. Proceedings of the Advances in Cryptology-EUROCRYPT 2004: International Conference on the Theory and Applications of Cryptographic Techniques, Interlaken, Switzerland. Proceedings 23.","DOI":"10.1007\/978-3-540-24676-3_31"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jentsch, C., and Reichmann, L. (2019). Generalized binary time series models. Econometrics, 7.","DOI":"10.3390\/econometrics7040047"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/j.1538-7305.1951.tb01366.x","article-title":"Prediction and entropy of printed English","volume":"30","author":"Shannon","year":"1951","journal-title":"Bell Syst. Tech. J."},{"key":"ref_22","unstructured":"Cachin, C. (1997). Entropy Measures and Unconditional Security in Cryptography. [Ph.D. Thesis, ETH Zurich]."},{"key":"ref_23","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_24","unstructured":"Vijayakumar, A.K., Cogswell, M., Selvaraju, R.R., Sun, Q., Lee, S., Crandall, D., and Batra, D. (2016). Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Shao, L., Gouws, S., Britz, D., Goldie, A., Strope, B., and Kurzweil, R. (2017). Generating high-quality and informative conversation responses with sequence-to-sequence models. arXiv.","DOI":"10.18653\/v1\/D17-1235"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Fan, A., Lewis, M., and Dauphin, Y. (2018). Hierarchical neural story generation. arXiv.","DOI":"10.18653\/v1\/P18-1082"},{"key":"ref_27","unstructured":"Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019). The curious case of neural text degeneration. arXiv."},{"key":"ref_28","first-page":"147","article-title":"A learning algorithm for Boltzmann machines","volume":"9","author":"Ackley","year":"1985","journal-title":"Cogn. Sci."},{"key":"ref_29","unstructured":"Li, J., Monroe, W., and Jurafsky, D. (2017). Learning to decode for future success. arXiv."},{"key":"ref_30","unstructured":"Oord, A.v.d., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv."},{"key":"ref_31","unstructured":"Serdyuk, D., Ke, N.R., Sordoni, A., Trischler, A., Pal, C., and Bengio, Y. (2017). Twin networks: Matching the future for sequence generation. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Lawrence, C., Kotnis, B., and Niepert, M. (2019). Attending to future tokens for bidirectional sequence generation. arXiv.","DOI":"10.18653\/v1\/D19-1001"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Qi, W., Yan, Y., Gong, Y., Liu, D., Duan, N., Chen, J., Zhang, R., and Zhou, M. (2020). Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.217"},{"key":"ref_34","unstructured":"Gloeckle, F., Idrissi, B.Y., Rozi\u00e8re, B., Lopez-Paz, D., and Synnaeve, G. (2024). Better & Faster Large Language Models via Multi-token Prediction. arXiv."},{"key":"ref_35","first-page":"10107","article-title":"Blockwise parallel decoding for deep autoregressive models","volume":"31","author":"Stern","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","unstructured":"Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J.D., Chen, D., and Dao, T. (2024). Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv."},{"key":"ref_37","first-page":"1","article-title":"Markov chains","volume":"Volume 83","author":"Ching","year":"2006","journal-title":"Models, Algorithms and Applications"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"528","DOI":"10.1111\/j.2517-6161.1985.tb01383.x","article-title":"A model for high-order Markov chains","volume":"47","author":"Raftery","year":"1985","journal-title":"J. R. Stat. Soc. Ser. B Stat. Methodol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"972","DOI":"10.1002\/nla.2063","article-title":"Convergence of a transition probability tensor of a higher-order Markov chain to the stationary probability vector","volume":"23","author":"Bozorgmanesh","year":"2016","journal-title":"Numer. Linear Algebra Appl."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Bagini, V., and Bucci, M. (1999, January 12\u201313). A design of reliable true random number generator for cryptographic applications. Proceedings of the Cryptographic Hardware and Embedded Systems: First International Workshop, CHES\u201999, Worcester, MA, USA. Proceedings 1.","DOI":"10.1007\/3-540-48059-5_18"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Stip\u010devi\u0107, M., and Ursin, R. (2015). An on-demand optical quantum random number generator with in-future action and ultra-fast response. Sci. Rep., 5.","DOI":"10.1038\/srep10214"},{"key":"ref_42","unstructured":"QURSA-UC3M (2024, June 28). qursa-uc3m\/ml-predictors-min-entropy-estimation. Available online: https:\/\/github.com\/qursa-uc3m\/ml-predictors-min-entropy-estimation\/tree\/0.1.0."},{"key":"ref_43","unstructured":"OpenSSL Project (2024, June 28). RAND_bytes(3)\u2014OpenSSL 3.4. Available online: https:\/\/www.openssl.org\/docs\/man3.4\/man3\/RAND_bytes.html."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Meeker, W.Q., Hahn, G.J., and Escobar, L.A. (2017). Statistical Intervals: A Guide for Practitioners and Researchers, John Wiley & Sons.","DOI":"10.1002\/9781118594841"},{"key":"ref_45","unstructured":"Face, H. (2024, June 28). Transformers: State-of-the-Art Machine Learning for PyTorch, TensorFlow, and JAX. Available online: https:\/\/huggingface.co\/docs\/transformers\/index."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/27\/2\/156\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:25:54Z","timestamp":1760027154000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/27\/2\/156"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,2]]},"references-count":45,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,2]]}},"alternative-id":["e27020156"],"URL":"https:\/\/doi.org\/10.3390\/e27020156","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,2]]}}}