{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:31:33Z","timestamp":1760059893428,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T00:00:00Z","timestamp":1752796800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Swiss Accelerator Research and Technology programme (CHART)","award":["C20-10"],"award-info":[{"award-number":["C20-10"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>The dynamic aperture is an essential concept in circular particle accelerators, providing the extent of the phase space region where particle motion remains stable over multiple turns. The accurate prediction of the dynamic aperture is key to optimising performance in accelerators such as the CERN Large Hadron Collider and is crucial for designing future accelerators like the CERN Future Circular Hadron Collider. Traditional methods for computing the dynamic aperture are computationally demanding and involve extensive numerical simulations with numerous initial phase space conditions. In our recent work, we have devised surrogate models to predict the dynamic aperture boundary both efficiently and accurately. These models have been further refined by incorporating them into a novel active learning framework. This framework enhances performance through continual retraining and intelligent data generation based on informed sampling driven by error estimation. A critical attribute of this framework is the precise estimation of uncertainty in dynamic aperture predictions. In this study, we investigate various machine learning techniques for uncertainty estimation, including Monte Carlo dropout, bootstrap methods, and aleatory uncertainty quantification. We evaluated these approaches to determine the most effective method for reliable uncertainty estimation in dynamic aperture predictions using machine learning techniques.<\/jats:p>","DOI":"10.3390\/computers14070287","type":"journal-article","created":{"date-parts":[[2025,7,18]],"date-time":"2025-07-18T10:10:38Z","timestamp":1752833438000},"page":"287","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Machine Learning Techniques for Uncertainty Estimation in Dynamic Aperture Prediction"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3456-1477","authenticated-orcid":false,"given":"Carlo Emilio","family":"Montanari","sequence":"first","affiliation":[{"name":"Department of Physics and Astronomy, The University of Manchester, Manchester M13 9PL, UK"},{"name":"CERN, 1211 Geneva, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8033-1923","authenticated-orcid":false,"given":"Robert B.","family":"Appleby","sequence":"additional","affiliation":[{"name":"Department of Physics and Astronomy, The University of Manchester, Manchester M13 9PL, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1122-7919","authenticated-orcid":false,"given":"Davide","family":"Di Croce","sequence":"additional","affiliation":[{"name":"CERN, 1211 Geneva, Switzerland"},{"name":"Institute of Physics, \u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne, 1015 Lausanne, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2093-9395","authenticated-orcid":false,"given":"Massimo","family":"Giovannozzi","sequence":"additional","affiliation":[{"name":"CERN, 1211 Geneva, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3218-0048","authenticated-orcid":false,"given":"Tatiana","family":"Pieloni","sequence":"additional","affiliation":[{"name":"Institute of Physics, \u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne, 1015 Lausanne, Switzerland"}]},{"given":"Stefano","family":"Redaelli","sequence":"additional","affiliation":[{"name":"CERN, 1211 Geneva, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0301-821X","authenticated-orcid":false,"given":"Frederik F.","family":"Van der Veken","sequence":"additional","affiliation":[{"name":"CERN, 1211 Geneva, Switzerland"}]}],"member":"1968","published-online":{"date-parts":[[2025,7,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"104003","DOI":"10.1103\/PhysRevAccelBeams.22.104003","article-title":"Advances on the modeling of the time evolution of dynamic aperture of hadron circular accelerators","volume":"22","author":"Bazzani","year":"2019","journal-title":"Phys. Rev. Accel. Beams"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"024001","DOI":"10.1103\/PhysRevSTAB.15.024001","article-title":"A proposed scaling law for intensity evolution in hadron storage rings based on dynamic aperture variation with time","volume":"15","author":"Giovannozzi","year":"2012","journal-title":"Phys. Rev. Spec. Top. Accel. Beams"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"034002","DOI":"10.1103\/PhysRevAccelBeams.22.034002","article-title":"Innovative method to measure the extent of the stable phase-space region of proton synchrotrons","volume":"22","author":"Maclean","year":"2019","journal-title":"Phys. Rev. Accel. Beams"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.nima.2018.07.063","article-title":"Description of the luminosity evolution for the CERN LHC including dynamic aperture effects. Part I: The model","volume":"A905","author":"Giovannozzi","year":"2018","journal-title":"Nucl. Instrum. Methods Phys. Res."},{"key":"ref_5","unstructured":"Br\u00fcning, O.S., Collier, P., Lebrun, P., Myers, S., Ostojic, R., Poole, J., and Proudlock, P. (2004). LHC Design Report, CERN. CERN Yellow Reports: Monographs."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Apollinari, G., B\u00e9jar Alonso, I., Br\u00fcning, O., Fessia, P., Lamont, M., Rossi, L., and Tavian, L. (2017). High-Luminosity Large Hadron Collider (HL\u2013LHC), CERN. CERN Yellow Reports: Monographs.","DOI":"10.2172\/1767028"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1140\/epjst\/e2019-900045-4","article-title":"FCC-ee: The Lepton Collider","volume":"228","author":"Abada","year":"2019","journal-title":"Eur. Phys. J. Spec. Top."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"755","DOI":"10.1140\/epjst\/e2019-900087-0","article-title":"FCC-hh: The Hadron Collider: Future Circular Collider Conceptual Design Report Volume 3","volume":"228","author":"Abada","year":"2019","journal-title":"Eur. Phys. J. Spec. Top."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"074001","DOI":"10.1103\/PhysRevAccelBeams.24.074001","article-title":"Numerical optimization of dc wire parameters for mitigation of the long range beam-beam interactions in High Luminosity Large Hadron Collider","volume":"24","author":"Skoufaris","year":"2021","journal-title":"Phys. Rev. Accel. Beams"},{"key":"ref_10","unstructured":"Droin, C., Sterbini, G., Efthymiopoulos, I., Mounet, N., De Maria, R., Tomas, R., Kostoglou, S., and European Organization for Nuclear Research (2024, January 19\u201324). Status of beam-beam studies for the high-luminosity LHC. Proceedings of the IPAC\u201924\u201415th International Particle Accelerator Conference, Nashville, TN, USA."},{"key":"ref_11","unstructured":"Assmann, R.W., Jeanneret, J.B., and Kaltchev, D.I. (2002, January 3\u20137). Efficiency for the imperfect LHC collimation system. Proceedings of the 8th European Particle Accelerator Conference (EPAC 2002), Paris, France."},{"key":"ref_12","unstructured":"Bracco, C. (2008). Commissioning Scenarios and Tests for the LHC Collimation System. [Ph.D. Thesis, EPFL]."},{"key":"ref_13","unstructured":"Schenk, M., Coyle, L., Pieloni, T., Giovannozzi, M., Mereghetti, A., Krymova, E., and Obozinski, G. (2021, January 24\u201328). Modeling Particle Stability Plots for Accelerator Optimization Using Adaptive Sampling. Proceedings of the IPAC\u201921, Campinas, Brazil."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1140\/epjp\/s13360-023-04167-y","article-title":"Ensemble reservoir computing for dynamical systems: Prediction of phase-space stable region for hadron storage rings","volume":"138","author":"Casanova","year":"2023","journal-title":"Eur. Phys. J. Plus"},{"key":"ref_15","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Aftan, S., and Shah, H. (2023, January 26). A Survey on BERT and Its Applications. Proceedings of the 2023 20th Learning and Technology Conference, Jeddah, Saudi Arabia.","DOI":"10.1109\/LT58159.2023.10092289"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Di Croce, D., Giovannozzi, M., Montanari, C.E., Pieloni, T., Redaelli, S., and Van der Veken, F.F. (2024). Assessing the Performance of Deep Learning Predictions for Dynamic Aperture of a Hadron Circular Particle Accelerator. Instruments, 8.","DOI":"10.3390\/instruments8040050"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1007\/s00163-020-00336-7","article-title":"Managing computational complexity using surrogate models: A critical review","volume":"31","author":"Alizadeh","year":"2020","journal-title":"Res. Eng. Des."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sudret, B., Marelli, S., and Wiart, J. (2017, January 19\u201324). Surrogate models for uncertainty quantification: An overview. Proceedings of the 2017 11th European Conference on Antennas and Propagation (EUCAP), Paris, France.","DOI":"10.23919\/EuCAP.2017.7928679"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"084801","DOI":"10.1103\/PhysRevAccelBeams.27.084801","article-title":"Bayesian optimization algorithms for accelerator physics","volume":"27","author":"Roussel","year":"2024","journal-title":"Phys. Rev. Accel. Beams"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1513","DOI":"10.1007\/s10462-023-10562-9","article-title":"A survey of uncertainty in deep neural networks","volume":"56","author":"Gawlikowski","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1007\/s10462-023-10698-8","article-title":"A review of predictive uncertainty estimation with machine learning","volume":"57","author":"Tyralis","year":"2024","journal-title":"Artif. Intell. Rev."},{"key":"ref_23","unstructured":"Gal, Y., and Ghahramani, Z. (2016). Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference. arXiv."},{"key":"ref_24","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017, January 4\u20139). Concrete Dropout. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_25","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 19\u201324). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on International Conference on Machine Learning\u2014ICML\u201916, New York, NY, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1023\/A:1018054314350","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_27","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer. [1st ed.]. Information Science and Statistics."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"P04004","DOI":"10.1088\/1748-0221\/19\/04\/P04004","article-title":"Optimizing dynamic aperture studies with active learning","volume":"19","author":"Giovannozzi","year":"2024","journal-title":"J. Instrum."},{"key":"ref_29","unstructured":"Kaplan, L., Cerutti, F., Sensoy, M., Preece, A., and Sullivan, P. (2018). Uncertainty Aware AI ML: Why and How. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"111027","DOI":"10.1016\/j.asoc.2023.111027","article-title":"Uncertainty aware neural network from similarity and sensitivity","volume":"149","author":"Kabir","year":"2023","journal-title":"Appl. Soft Comput."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"6928","DOI":"10.1109\/TNNLS.2022.3213315","article-title":"An Optimized Uncertainty-Aware Training Framework for Neural Networks","volume":"35","author":"Tabarisaadi","year":"2024","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_32","first-page":"3432","article-title":"Dynamic aperture extrapolation in presence of tune modulation","volume":"E57","author":"Giovannozzi","year":"1998","journal-title":"Phys. Rev."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"012007","DOI":"10.1088\/1742-6596\/874\/1\/012007","article-title":"Multiparametric response of the HL-LHC Dynamic Aperture in presence of beam-beam effects","volume":"874","author":"Pellegrini","year":"2017","journal-title":"J. Phys. Conf. Ser."},{"key":"ref_34","unstructured":"Jing, Y.C., Litvinenko, V., and Trbojevic, D. (2015, January 4). Optimization of Dynamic Aperture for Hadron Lattices in eRHIC. Proceedings of the 6th International Particle Accelerator Conference (IPAC\u201915), Richmond, VA, USA."},{"key":"ref_35","unstructured":"Cruz Alaniz, E., Abelleira, J.L., van Riesen-Hauptpresenter, L., Seryi, A., Martin, R., and Tom\u00e1s, R. (2018, January 3). Methods to Increase the Dynamic Aperture of the FCC-hh Lattice. Proceedings of the 9th International Particle Accelerator Conference (IPAC\u201918), Vancouver, BC, Canada."},{"key":"ref_36","unstructured":"Iadarola, G., Latina, A., Abramov, A., Droin, C., Demetriadou, D., Soubelet, F., Van der Veken, F., Sterbini, G., Dilly, J., and Paraschou, K. (2024, January 19\u201324). Xsuite: An integrated beam physics simulation framework. Proceedings of the IPAC\u201924\u201415th International Particle Accelerator Conference, Nashville, TN, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Grote, H., and Schmidt, F. (2003, January 12\u201316). MAD-X\u2014An Upgrade from MAD8. Proceedings of the PAC\u201903, Portland, OR, USA.","DOI":"10.1109\/PAC.2003.1289960"},{"key":"ref_38","unstructured":"Deniau, L., Burkhardt, H., Maria, R.D., Giovannozzi, M., Jowett, J.M., Latina, A., Persson, T., Schmidt, F., Shreyber, I.S., and Skowro\u0144ski, P.K. (2018, January 20\u201324). Upgrade of MAD-X for HL-LHC Project and FCC Studies. Proceedings of the ICAP\u201918, Key West, FL, USA."},{"key":"ref_39","unstructured":"Maria, R.D., Latina, A., Schmidt, F., Dilly, J., Deniau, L., Skowronski, P., Berg, J., and Gl\u00e4\u00dfle, T. (2023, January 7\u201312). Status of MAD-X V5.09. Proceedings of the IPAC\u201923\u2014International Particle Accelerator Conference, Venice, Italy."},{"key":"ref_40","unstructured":"(2025, July 17). MAD\u2014Methodical Accelerator Design. Available online: https:\/\/mad.web.cern.ch\/mad\/."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Hostiadi, D.P., Atmojo, Y.P., Huizen, R.R., Susila, I.M.D., Pradipta, G.A., and Liandana, I.M. (2022, January 8\u20139). A New Approach Feature Selection for Intrusion Detection System Using Correlation Analysis. Proceedings of the 2022 4th International Conference on Cybernetics and Intelligent System (ICORIS), Prapat, Indonesia.","DOI":"10.1109\/ICORIS56080.2022.10031468"},{"key":"ref_42","first-page":"457","article-title":"Perusing The Effect of Discretization of Data on Accuracy of Predicting Na\u00efve Bayes Algorithm","volume":"1","author":"Parsaei","year":"2016","journal-title":"J. Curr. Res. Sci."},{"key":"ref_43","unstructured":"Schmidt, F., Forest, E., and McIntosh, E. (2002). Introduction to the Polymorphic Tracking Code: Fibre Bundles, Polymorphic Taylor Types and \u201cExact Tracking\u201d, CERN. Available online: https:\/\/cds.cern.ch\/record\/573082."},{"key":"ref_44","unstructured":"Schmidt, F., Chiu, C.Y., Goddard, B., Jacquet, D., Kain, V., Lamont, M., Mertens, V., Uythoven, J., and Wenninger, J. (2005, January 16\u201320). MAD-X PTC Integration. Proceedings of the 21st IEEE Particle Accelerator Conference (PAC 2005), Knoxville, TN, USA."},{"key":"ref_45","unstructured":"Bazzani, A., Servizi, G., Todesco, E., and Turchetti, G. (1994). A Normal Form Approach to the Theory of Nonlinear Betatronic Motion, CERN. CERN Yellow Reports: Monographs."},{"key":"ref_46","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2025, July 17). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: https:\/\/www.tensorflow.org\/."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"227","DOI":"10.3102\/1076998619872761","article-title":"Deep Learning with TensorFlow: A Review","volume":"45","author":"Pang","year":"2020","journal-title":"J. Educ. Behav. Stat."},{"key":"ref_48","unstructured":"Arora, R., Basu, A., Mianjy, P., and Mukherjee, A. (2018). Understanding Deep Neural Networks with Rectified Linear Units. arXiv."},{"key":"ref_49","unstructured":"Agarap, A.F. (2019). Deep Learning using Rectified Linear Units (ReLU). arXiv."},{"key":"ref_50","unstructured":"O\u2019Malley, T., Bursztein, E., Long, J., Chollet, F., Jin, H., and Invernizzi, L. (2025, July 17). KerasTuner. Available online: https:\/\/github.com\/keras-team\/keras-tuner."},{"key":"ref_51","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2023). Attention Is All You Need. arXiv."},{"key":"ref_52","unstructured":"Nguyen, S., Nguyen, D., Nguyen, K., Than, K., Bui, H., and Ho, N. (2021). Structured Dropout Variational Inference for Bayesian Neural Networks. arXiv."},{"key":"ref_53","unstructured":"Foong, A.Y.K., Burt, D.R., Li, Y., and Turner, R.E. (2020). On the Expressiveness of Approximate Inference in Bayesian Neural Networks. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Lovric, M. (2011). Kullback-Leibler Divergence. International Encyclopedia of Statistical Science, Springer.","DOI":"10.1007\/978-3-642-04898-2"},{"key":"ref_55","unstructured":"Murphy, K.P. (2012). Machine Learning: A Probabilistic Perspective, MIT Press."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/7\/287\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:11:57Z","timestamp":1760033517000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/7\/287"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,18]]},"references-count":55,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["computers14070287"],"URL":"https:\/\/doi.org\/10.3390\/computers14070287","relation":{},"ISSN":["2073-431X"],"issn-type":[{"type":"electronic","value":"2073-431X"}],"subject":[],"published":{"date-parts":[[2025,7,18]]}}}