{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,3]],"date-time":"2025-12-03T18:02:56Z","timestamp":1764784976723,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2022,10,10]],"date-time":"2022-10-10T00:00:00Z","timestamp":1665360000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"SESAR Joint Undertaking under grant agreement","award":["892618"],"award-info":[{"award-number":["892618"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Designing and developing artificial intelligence (AI)-based systems that can be trusted justifiably is one of the main issues aviation must face in the coming years. European Union Aviation Safety Agency (EASA) has developed a user guide that could be potentially transformed as means of compliance for future AI-based regulation. Designers and developers must understand how the learning assurance process of any machine learning (ML) model impacts trust. ML is a narrow branch of AI that uses statistical models to perform predictions. This work deals with the learning assurance process for ML-based systems in the field of air traffic control. A conflict detection tool has been developed to identify separation infringements among aircraft pairs, and the ML algorithm used for classification and regression was extreme gradient boosting. This paper analyses the validity and adaptability of EASA W-shaped methodology for ML-based systems. The results have identified the lack of the EASA W-shaped methodology in time-dependent analysis, by showing how time can impact ML algorithms designed in the case where no time requirements are considered. Another meaningful conclusion is, for systems that depend highly on when the prediction is made, classification and regression metrics cannot be one-size-fits-all because they vary over time.<\/jats:p>","DOI":"10.3390\/s22197680","type":"journal-article","created":{"date-parts":[[2022,10,11]],"date-time":"2022-10-11T00:50:01Z","timestamp":1665449401000},"page":"7680","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0112-9792","authenticated-orcid":false,"given":"Javier A.","family":"P\u00e9rez-Cast\u00e1n","sequence":"first","affiliation":[{"name":"ETSI Aeron\u00e1utica y del Espacio, Plaza Cardenal Cisneros, Universidad Polit\u00e9cnica de Madrid, 28008 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0046-4094","authenticated-orcid":false,"given":"Luis","family":"P\u00e9rez Sanz","sequence":"additional","affiliation":[{"name":"ETSI Aeron\u00e1utica y del Espacio, Plaza Cardenal Cisneros, Universidad Polit\u00e9cnica de Madrid, 28008 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marta","family":"Fern\u00e1ndez-Castellano","sequence":"additional","affiliation":[{"name":"ETSI Aeron\u00e1utica y del Espacio, Plaza Cardenal Cisneros, Universidad Polit\u00e9cnica de Madrid, 28008 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tomislav","family":"Radi\u0161i\u0107","sequence":"additional","affiliation":[{"name":"Faculty of Transport and Traffic Sciences, University of Zagreb, Borongajska Cesta, 10000 Zagreb, Croatia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1762-4899","authenticated-orcid":false,"given":"Kristina","family":"Samard\u017ei\u0107","sequence":"additional","affiliation":[{"name":"Faculty of Transport and Traffic Sciences, University of Zagreb, Borongajska Cesta, 10000 Zagreb, Croatia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ivan","family":"Tukari\u0107","sequence":"additional","affiliation":[{"name":"Faculty of Transport and Traffic Sciences, University of Zagreb, Borongajska Cesta, 10000 Zagreb, Croatia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,10]]},"reference":[{"key":"ref_1","unstructured":"EUROCONTROL (2022, July 07). EUROCONTROL Forecast Update 2021\u20132027, Available online: https:\/\/www.eurocontrol.int\/publication\/eurocontrol-forecast-update-2021-2027."},{"key":"ref_2","unstructured":"SESAR Joint Undertaking (2019). European ATM Master Plan\u2014Digitalising Europe\u2019s Aviation Infraestructure, SESAR Joint Undertaking."},{"key":"ref_3","unstructured":"European Union Aviation Safety Agaency (EASA) (2020). Artificial Intelligence Roadmap A Human-Centric Approach to AI in Aviation."},{"key":"ref_4","unstructured":"European Aviation Artificial Intelligence High Level Group (EAAI-HLG) (2020). The FLY AI Report Demystifying and Accelerating AI in Aviation\/ATM, European Organisation for the Safety of Air Navigation."},{"key":"ref_5","unstructured":"Geron, A. (2017). Hands-on Machine Learning with Scikit-Learn and Tensor Flow, O\u2019Reilly Media, Inc."},{"key":"ref_6","first-page":"243","article-title":"Prognostic Kalman Filter Based Bayesian Learning Model for Data Accuracy Prediction","volume":"72","author":"Karthik","year":"2022","journal-title":"Comput. Mater. Contin."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e2019EA000740","DOI":"10.1029\/2019EA000740","article-title":"Comparative Assessment of Various Machine Learning-Based Bias Correction Methods for Numerical Weather Prediction Model Forecasts of Extreme Air Temperatures in Urban Areas","volume":"7","author":"Cho","year":"2020","journal-title":"Earth Space Sci."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Uddin, M.F. (2019, January 20\u201321). Addressing accuracy paradox using enhanched weighted performance metric in machine learning. Proceedings of the Information Technology Trends: Emerging Technologies Blockchain and IoT (ITT 2019), Ras Al Khaimah, United Arab Emirates.","DOI":"10.1109\/ITT48889.2019.9075071"},{"key":"ref_9","unstructured":"European Union Aviation Safety Agaency (EASA) (2021). Concepts of Design Assurance for Neural Networks (CoDANN) II\u2014Public Extract."},{"key":"ref_10","unstructured":"European Union Aviation Safety Agaency (EASA) (2021). EASA Artificial Intelligence Roadmap 1.0 EASA, A Deliverable of the EASA AI Roadmap."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3138","DOI":"10.1109\/TITS.2015.2437452","article-title":"Machine Learning and Mass Estimation Methods for Ground-Based Aircraft Climb Prediction","volume":"16","author":"Alligier","year":"2015","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.trc.2018.08.012","article-title":"Learning Aircraft Operational Factors to Improve Aircraft Climb Prediction: A Large Scale Multi-Airport Study","volume":"96","author":"Alligier","year":"2018","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_13","unstructured":"Alligier, R., Gianazza, D., Durand, N., Alligier, R., Gianazza, D., Durand, N., Learning, M., Prediction, A., Alligier, R., and Gianazza, D. (2015, January 23\u201326). Machine learning applied to airspeed prediction during climb. Proceedings of the ATM Seminar 2015, 11th USA\/EUROPE Air Traffic Management R&D Seminar, FAA & Eurocontrol, Lisbon, Portugal."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sridhar, B. (2020, January 3\u20134). Applications of machine learning techniques to aviation operations: Promises and challenges. Proceedings of the International Conference on Artificial Intelligence and Data Analytics for Air Transportation (AIDA-AT 2020), Singapore.","DOI":"10.1109\/AIDA-AT48540.2020.9049205"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1109\/TSMCC.2011.2161285","article-title":"A Review on ensembles for the class imbalance problem: Bagging-, boosting-, and hybrid-based approaches","volume":"42","author":"Galar","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Haupt, S.E., Cowie, J., Linden, S., McCandless, T., Kosovic, B., and Alessandrini, S. (November, January 29). Machine Learning for Applied Weather Prediction. Proceedings of the IEEE 14th International Conference on eScience (e-Science 2018), Amsterdam, The Netherlands.","DOI":"10.1109\/eScience.2018.00047"},{"key":"ref_17","unstructured":"Krozel, J., Peters, M.E., and Hunter, G. (1997). Conflict Detection and Resolution for Future Air Transportation Management-TR97138-01, Seagull Technology."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Paielli, R.A., and Erzberger, H. (2017, January 5\u20139). Trajectory Specification for Terminal Air Traffic: Pairwise Conflict Detection and Resolution. Proceedings of the 17th AIAA Aviation Technology Integration, and Operations Conference, Denver, CO, USA.","DOI":"10.2514\/6.2017-4256"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1109\/6979.898217","article-title":"A Review of Conflict Detection and Resolution Modeling Methods","volume":"1","author":"Kuchar","year":"2000","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"930","DOI":"10.1177\/0954410011417347","article-title":"Automated Conflict Resolution, Arrival Management, and Weather Avoidance for Air Traffic Management","volume":"226","author":"Erzberger","year":"2012","journal-title":"Proc. Inst. Mech. Eng. G J. Aerosp. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"4839","DOI":"10.1177\/0954410019831198","article-title":"Design of a Conflict-Detection Air Traffic Control Tool for the Implementation of Continuous Climb Operations: A Case Study at Palma TMA","volume":"233","author":"Comendador","year":"2019","journal-title":"Proc. Inst. Mech. Eng. G J. Aerosp. Eng."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-Cast\u00e1n, J.A., P\u00e9rez-Sanz, L., Serrano-Mira, L., Sa\u00e9z-Hernando, F.J., Rodr\u00edguez Gauxachs, I., and G\u00f3mez-Comendador, V.F. (2022). Design of an ATC Tool for Conflict Detection Based on Machine Learning Techniques. Aerospace, 9.","DOI":"10.3390\/aerospace9020067"},{"key":"ref_23","unstructured":"International Civil Aviation Organization (ICAO) (1998). Doc 9689-AN\/953\u2014Manual on Airspace Planning Methodology for the Determination of Separation Minima, International Civil Aviation Organization."},{"key":"ref_24","unstructured":"AISA Consortium (2021). AISA 3.2: Conflict Detection Module, SESAR Joint Undertaking."},{"key":"ref_25","unstructured":"The OpenSky Network (2020, October 07). The OpenSky Network API. Available online: https:\/\/opensky-network.org\/apidoc\/index.html."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/2786984.2786995","article-title":"Scikit-Learn","volume":"19","author":"Varoquaux","year":"2015","journal-title":"GetMobile Mob. Comput. Commun."},{"key":"ref_27","unstructured":"Weiss, G., McCarthy, K., and Zabar, B. (2007, January 25\u201328). Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs?. Proceedings of the International Conference on Data Mining (DMIN), Las Vegas, NV, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/19\/7680\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:49:11Z","timestamp":1760143751000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/19\/7680"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,10]]},"references-count":27,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["s22197680"],"URL":"https:\/\/doi.org\/10.3390\/s22197680","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,10,10]]}}}