{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T16:17:37Z","timestamp":1774282657471,"version":"3.50.1"},"reference-count":34,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2022,5,9]],"date-time":"2022-05-09T00:00:00Z","timestamp":1652054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union\u2019s Horizon 2020 research and innovation program under the REMAP project","award":["769288"],"award-info":[{"award-number":["769288"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Mathematics"],"abstract":"<jats:p>Aircraft maintenance is a complex domain where designing new systems that include Machine Learning (ML) algorithms can become a challenge. In the context of designing a tool for Condition-Based Maintenance (CBM) in aircraft maintenance planning, this case study addresses (1) the use of Playful Probing approach to obtain insights that allow understanding of how to design for interaction with ML algorithms, (2) the integration of a Reinforcement Learning (RL) agent for Human\u2013AI collaboration in maintenance planning and (3) the visualisation of CBM indicators. Using a design science research approach, we designed a Playful Probe protocol and materials, and evaluated results by running a participatory design workshop. Our main contribution is to show how to elicit ideas for integration of maintenance planning practices with ML estimation tools and the RL agent. Through a participatory design workshop with participants\u2019 observation, in which they played with CBM artefacts, Playful Probes favour the elicitation of user interaction requirements with the RL planning agent to aid the planner to obtain a reliable maintenance plan and turn possible to understand how to represent CBM indicators and visualise them through a trajectory prediction.<\/jats:p>","DOI":"10.3390\/math10091604","type":"journal-article","created":{"date-parts":[[2022,5,10]],"date-time":"2022-05-10T00:30:28Z","timestamp":1652142628000},"page":"1604","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Playful Probes for Design Interaction with Machine Learning: A Tool for Aircraft Condition-Based Maintenance Planning and Visualisation"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8188-0721","authenticated-orcid":false,"given":"Jorge","family":"Ribeiro","sequence":"first","affiliation":[{"name":"CISUC\u2014Centre Informatics and Systems, Informatics Engineering Department, University of Coimbra, 3004-531 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7019-7721","authenticated-orcid":false,"given":"Pedro","family":"Andrade","sequence":"additional","affiliation":[{"name":"CISUC\u2014Centre Informatics and Systems, Informatics Engineering Department, University of Coimbra, 3004-531 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8323-3445","authenticated-orcid":false,"given":"Manuel","family":"Carvalho","sequence":"additional","affiliation":[{"name":"CISUC\u2014Centre Informatics and Systems, Informatics Engineering Department, University of Coimbra, 3004-531 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5656-0061","authenticated-orcid":false,"given":"Catarina","family":"Silva","sequence":"additional","affiliation":[{"name":"CISUC\u2014Centre Informatics and Systems, Informatics Engineering Department, University of Coimbra, 3004-531 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9770-7672","authenticated-orcid":false,"given":"Bernardete","family":"Ribeiro","sequence":"additional","affiliation":[{"name":"CISUC\u2014Centre Informatics and Systems, Informatics Engineering Department, University of Coimbra, 3004-531 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1911-2788","authenticated-orcid":false,"given":"Lic\u00ednio","family":"Roque","sequence":"additional","affiliation":[{"name":"CISUC\u2014Centre Informatics and Systems, Informatics Engineering Department, University of Coimbra, 3004-531 Coimbra, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2022,5,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Andrade, P., Silva, C., Ribeiro, B., and Santos, B.F. (2021). Aircraft maintenance check scheduling using reinforcement learning. Aerospace, 8.","DOI":"10.3390\/aerospace8040113"},{"key":"ref_2","first-page":"1","article-title":"Participatory design that matters\u2014Facing the big issues","volume":"25","author":"Kyng","year":"2018","journal-title":"ACM Trans. Comput. Hum. Interact."},{"key":"ref_3","unstructured":"B\u00f8dker, S., Roque, L., Larsen-Ledet, I., and Thomas, V. (2022, March 15). Taming a Run-Away Object: How to Maintain and Extend Human Control in Human-Computer Interaction?. Available online: https:\/\/pure.au.dk\/ws\/files\/135965176\/bodker_taming_a_runaway_object.pdf."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Aigner, W., Miksch, S., Schumann, H., and Al, E. (2011). Visualization of Time-Oriented Data, Springer.","DOI":"10.1007\/978-0-85729-079-3"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Munzner, T. (2015). Visualization Analysis & Design, CRC Press, Taylor & Francis Group.","DOI":"10.1201\/b17511"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Amershi, S., Inkpen, K., Teevan, J., Kikin-Gil, R., Horvitz, E., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., and Collisson, P. (2019, January 4\u20139). Guidelines for Human-AI Interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems\u2014CHI \u201919, Glasgow, UK.","DOI":"10.1145\/3290605.3300233"},{"key":"ref_7","unstructured":"Holbrook, J. (2020, April 16). Human-Centered Machine Learning. Available online: https:\/\/medium.com\/google-design\/human-centered-machine-learning-a770d10562cd."},{"key":"ref_8","unstructured":"Wright, A.P., Wang, Z.J., Park, H., Guo, G., Sperrle, F., El-Assady, M., Endert, A., Keim, D., and Chau, D.H. (2020, October 22). A Comparative Analysis of Industry Human-AI Interaction Guidelines, Available online: http:\/\/xxx.lanl.gov\/abs\/2010.11761."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Guzdial, M., Liao, N., Chen, J., Chen, S.Y., Shah, S., Shah, V., Reno, J., Smith, G., and Riedl, M.O. (2019, January 4\u20139). Friend, collaborator, student, manager: How design of an AI-driven game level editor affects creators. Proceedings of the Conference on Human Factors in Computing Systems\u2014Proceedings, Glasgow, UK. Available online: http:\/\/xxx.lanl.gov\/abs\/1901.06417.","DOI":"10.1145\/3290605.3300854"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., and Kankanhalli, M. (2018, January 21\u201326). Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda. Proceedings of the Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174156"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Wang, D., Yang, Q., Abdul, A., Lim, B.Y., and States, U. (2019, January 4\u20139). Designing Theory-Driven User-Centric Explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.","DOI":"10.1145\/3290605.3300831"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Zhou, J., Gandomi, A.H., Chen, F., and Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10.","DOI":"10.3390\/electronics10050593"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable ai: A review of machine learning interpretability methods. Entropy, 23.","DOI":"10.3390\/e23010018"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J.M., and Eckersley, P. (2020, January 27\u201330). Explainable machine learning in deployment. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Barcelona, Spain. Available online: http:\/\/xxx.lanl.gov\/abs\/1909.06342.","DOI":"10.1145\/3351095.3375624"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bramer, M., Petridis, M., and Hopgood, A. (2011). Reinforcement Learning for Scheduling of Maintenance. Research and Development in Intelligent Systems XXVII, Springer.","DOI":"10.1007\/978-0-85729-130-1"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/s10845-013-0864-5","article-title":"Multi-agent reinforcement learning based maintenance policy for a resource constrained flow line system","volume":"27","author":"Wang","year":"2016","journal-title":"J. Intell. Manuf."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1007\/s10845-016-1237-7","article-title":"Optimal preventive maintenance policy based on reinforcement learning of a fleet of military trucks","volume":"30","author":"Barde","year":"2019","journal-title":"J. Intell. Manuf."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"107056","DOI":"10.1016\/j.cie.2020.107056","article-title":"Reinforcement learning-driven maintenance strategy: A novel solution for long-term aircraft maintenance decision optimization","volume":"153","author":"Hu","year":"2021","journal-title":"Comput. Ind. Eng."},{"key":"ref_19","unstructured":"Mattelmaki, T., and Korkeakoulu, T. (2008). Design Probes, University of Art and Design."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1145\/1015530.1015555","article-title":"Cultural probes and the value of uncertainty","volume":"11","author":"Gaver","year":"2004","journal-title":"Interactions"},{"key":"ref_21","unstructured":"Wallace, J., McCarthy, J., Wright, P.C., and Olivier, P. (May, January 27). Making design probes work. Proceedings of the Conference on Human Factors in Computing Systems, Paris, France."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1162\/DESI_a_00441","article-title":"How Do User Stories Inspire Design? A Study of Cultural Probes","volume":"33","author":"Celikoglu","year":"2017","journal-title":"Des. Issues"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Lange-Nielsen, F., Lafont, X.V., Cassar, B., and Khaled, R. (2012, January 4\u20136). Involving players earlier in the game design process using cultural probes. Proceedings of the 4th International Conference on Fun and Games, Toulouse, France.","DOI":"10.1145\/2367616.2367622"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B.B., Druin, A., Plaisant, C., Beaudouin-Lafon, M., Conversy, S., Evans, H., and Hansen, H. (2003, January 5\u201310). Technology probes: Inspiring design for and with families. Proceedings of the Conference on Human Factors in Computing Systems, Ft. Lauderdale, FL, USA.","DOI":"10.1145\/642611.642616"},{"key":"ref_25","unstructured":"Huizinga, J. (2016). Homo Ludens: A Study of the Play-Element in Culture, Angelico Press."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Bernhaupt, R., Weiss, A., Obrist, M., and Tscheligi, M. (2007). Playful probing: Making probing more fun. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer. 4662 LNCS.","DOI":"10.1007\/978-3-540-74796-3_60"},{"key":"ref_27","unstructured":"Sjovoll, V., and Gulden, T. (2016, January 8\u20139). Play probes\u2014As a productive space and source for information. Proceedings of the 18th International Conference on Engineering and Product Design Education, Design Education: Collaboration and Cross-Disciplinarity, Aalborg, Denmark."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1145\/291224.291235","article-title":"Design: Cultural probes","volume":"6","author":"Gaver","year":"1999","journal-title":"Interactions"},{"key":"ref_29","unstructured":"Ribeiro, J., and Roque, L. (2020, January 26\u201328). Playfully probing practice-automation dialectics in designing new ML-tools. Proceedings of the VideoJogos 2020: 12th International Conference on Videogame Sciences and Arts, Mirandela, Portugal."},{"key":"ref_30","unstructured":"Vaishnavi, V.K., and Purao, S. (2009, January 7\u20138). DESRIST \u201909: Design Science Research in Information Systems and Technologies. Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology, Philadelphia, PA, USA."},{"key":"ref_31","unstructured":"Figma (2020, July 13). Where Teams Design Together. Available online: https:\/\/www.figma.com."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1109\/2.121503","article-title":"The Usability Engineering Life Cycle","volume":"25","author":"Nielsen","year":"1992","journal-title":"Computer"},{"key":"ref_33","unstructured":"Goebel, K., Saxena, A., Daigle, M., Celaya, J., Roychoudhury, I., and Clements, S. (2012, January 3\u20135). Introduction to prognostics. Proceedings of the European PHM Conference, Dresden, Germany."},{"key":"ref_34","unstructured":"EASA (2022, May 01). Certification Specifications and Guidance Material for Master Minimum Equipment List (CS-MMEL). Available online: https:\/\/www.easa.europa.eu\/document-library\/certification-specifications\/cs-mmel-issue-3."}],"container-title":["Mathematics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-7390\/10\/9\/1604\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:08:04Z","timestamp":1760137684000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-7390\/10\/9\/1604"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,5,9]]},"references-count":34,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["math10091604"],"URL":"https:\/\/doi.org\/10.3390\/math10091604","relation":{},"ISSN":["2227-7390"],"issn-type":[{"value":"2227-7390","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,5,9]]}}}