{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T14:53:22Z","timestamp":1772549602469,"version":"3.50.1"},"publisher-location":"Cham","reference-count":73,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031308253","type":"print"},{"value":"9783031308260","type":"electronic"}],"license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,4,20]],"date-time":"2023-04-20T00:00:00Z","timestamp":1681948800000},"content-version":"vor","delay-in-days":109,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>ML systems have become an essential tool for experts of many domains, data scientists and researchers, allowing them to find answers to many complex business questions starting from raw datasets. Nevertheless, the development of ML systems able to satisfy the stakeholders\u2019 needs requires an appropriate amount of knowledge about the ML domain. Over the years, several solutions have been proposed to automate the development of ML systems. However, an approach taking into account the new quality concerns needed by ML systems (like fairness, interpretability, privacy, and others) is still missing.<\/jats:p><jats:p>In this paper, we propose a new engineering approach for the quality-based development of ML systems by realizing a workflow formalized as a Software Product Line through Extended Feature Models to generate an ML System satisfying the required quality constraints. The proposed approach leverages an experimental environment that applies all the settings to enhance a given Quality Attribute, and selects the best one. The experimental environment is general and can be used for future quality methods\u2019 evaluations. Finally, we demonstrate the usefulness of our approach in the context of multi-class classification problem and fairness quality attribute.<\/jats:p>","DOI":"10.1007\/978-3-031-30826-0_5","type":"book-chapter","created":{"date-parts":[[2023,4,19]],"date-time":"2023-04-19T18:02:59Z","timestamp":1681927379000},"page":"88-110","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Democratizing Quality-Based Machine Learning Development through Extended Feature Models"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7388-890X","authenticated-orcid":false,"given":"Giordano","family":"d\u2019Aloisio","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7214-9945","authenticated-orcid":false,"given":"Antinisca","family":"Di Marco","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2092-0213","authenticated-orcid":false,"given":"Giovanni","family":"Stilo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,4,20]]},"reference":[{"key":"5_CR1","unstructured":"Conda website, https:\/\/docs.conda.io\/"},{"key":"5_CR2","unstructured":"Pickle documentation, https:\/\/docs.python.org\/3\/library\/pickle.html"},{"key":"5_CR3","unstructured":"Agarwal, A., Beygelzimer, A., Dudik, M., Langford, J., Wallach, H.: A Reductions Approach to Fair Classification. In: Proceedings of the 35th International Conference on Machine Learning. pp. 60\u201369. PMLR (Jul 2018), https:\/\/proceedings.mlr.press\/v80\/agarwal18a.html, iSSN: 2640-3498"},{"key":"5_CR4","unstructured":"Aly, M.: Survey on multiclass classification methods. Neural Netw 19(1-9), \u00a02 (2005)"},{"key":"5_CR5","doi-asserted-by":"publisher","unstructured":"Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software Engineering for Machine Learning: A Case Study. In: 2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). pp. 291\u2013300. IEEE, Montreal, QC, Canada (May 2019). https:\/\/doi.org\/10.1109\/ICSE-SEIP.2019.00042, https:\/\/ieeexplore.ieee.org\/document\/8804457\/","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"5_CR6","unstructured":"Apel, S., Batory, D., K\u00e4stner, C., Saake, G.: Feature-oriented software product lines. Springer (2016)"},{"key":"5_CR7","doi-asserted-by":"crossref","unstructured":"Azimi, S., Pahl, C.: A layered quality framework for machine learning-driven data and information models. In: ICEIS (1). pp. 579\u2013587 (2020)","DOI":"10.5220\/0009472305790587"},{"key":"5_CR8","doi-asserted-by":"crossref","unstructured":"Bellamy, R.K., Dey, K., Hind, M., Hoffman, S.C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovi\u0107, A., et\u00a0al.: Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development 63(4\/5), \u00a04\u20131 (2019)","DOI":"10.1147\/JRD.2019.2942287"},{"key":"5_CR9","doi-asserted-by":"publisher","unstructured":"Benavides, D., Segura, S., Ruiz-Cort\u00e9s, A.: Automated analysis of feature models 20 years later: A literature review. Information Systems 35(6), 615\u2013636 (Sep 2010). https:\/\/doi.org\/10.1016\/j.is.2010.01.001, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0306437910000025","DOI":"10.1016\/j.is.2010.01.001"},{"key":"5_CR10","doi-asserted-by":"publisher","unstructured":"Berthold, M.R., Cebron, N., Dill, F., Gabriel, T.R., K\u00f6tter, T., Meinl, T., Ohl, P., Thiel, K., Wiswedel, B.: Knime - the konstanz information miner: Version 2.0 and beyond. SIGKDD Explor. Newsl. 11(1), 26\u201331 (Nov 2009). https:\/\/doi.org\/10.1145\/1656274.1656280, https:\/\/doi-org.univaq.clas.cineca.it\/10.1145\/1656274.1656280","DOI":"10.1145\/1656274.1656280"},{"key":"5_CR11","unstructured":"Bird, S., Dud\u00edk, M., Edgar, R., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., Walker, K.: Fairlearn: A toolkit for assessing and improving fairness in AI. Tech. Rep. MSR-TR-2020-32, Microsoft (May 2020), https:\/\/www.microsoft.com\/en-us\/research\/publication\/fairlearn-a-toolkit-for-assessing-and-improving-fairness-in-ai\/"},{"key":"5_CR12","doi-asserted-by":"publisher","unstructured":"Bosch, J., Olsson, H.H., Crnkovic, I.: Engineering AI Systems: A Research Agenda (2021). https:\/\/doi.org\/10.4018\/978-1-7998-5101-1.ch001, https:\/\/www.igi-global.com\/chapter\/engineering-ai-systems\/www.igi-global.com\/chapter\/engineering-ai-systems\/266130, iSBN: 9781799851011 Pages: 1-19 Publisher: IGI Global","DOI":"10.4018\/978-1-7998-5101-1.ch001"},{"key":"5_CR13","doi-asserted-by":"publisher","unstructured":"Braiek, H.B., Khomh, F.: On testing machine learning programs. Journal of Systems and Software 164, 110542 (2020). https:\/\/doi.org\/10.1016\/j.jss.2020.110542, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121220300248","DOI":"10.1016\/j.jss.2020.110542"},{"key":"5_CR14","doi-asserted-by":"crossref","unstructured":"Buckland, M., Gey, F.: The relationship between recall and precision. Journal of the American society for information science 45(1), 12\u201319 (1994), publisher: Wiley Online Library","DOI":"10.1002\/(SICI)1097-4571(199401)45:1<12::AID-ASI2>3.0.CO;2-L"},{"key":"5_CR15","doi-asserted-by":"crossref","unstructured":"Carvalho, D.V., Pereira, E.M., Cardoso, J.S.: Machine learning interpretability: A survey on methods and metrics. Electronics 8(8), \u00a0832 (2019)","DOI":"10.3390\/electronics8080832"},{"key":"5_CR16","unstructured":"Caton, S., Haas, C.: Fairness in machine learning: A survey (2020)"},{"key":"5_CR17","doi-asserted-by":"crossref","unstructured":"Celis, L.E., Huang, L., Keswani, V., Vishnoi, N.K.: Classification with fairness constraints: A meta-algorithm with provable guarantees. In: Proceedings of the conference on fairness, accountability, and transparency. pp. 319\u2013328 (2019)","DOI":"10.1145\/3287560.3287586"},{"key":"5_CR18","doi-asserted-by":"crossref","unstructured":"Chakraborty, J., Majumder, S., Yu, Z., Menzies, T.: Fairway: A way to build fair ml software. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 654\u2013665 (2020)","DOI":"10.1145\/3368089.3409697"},{"key":"5_CR19","doi-asserted-by":"publisher","unstructured":"Chen, L., Ali\u00a0Babar, M., Nuseibeh, B.: Characterizing architecturally significant requirements. IEEE Software 30(2), 38\u201345 (2013). https:\/\/doi.org\/10.1109\/MS.2012.174","DOI":"10.1109\/MS.2012.174"},{"key":"5_CR20","unstructured":"Chen, Z., Zhang, J.M., Hort, M., Sarro, F., Harman, M.: Fairness Testing: A Comprehensive Survey and Analysis of Trends (Aug 2022), http:\/\/arxiv.org\/abs\/2207.10223, arXiv:2207.10223 [cs]"},{"key":"5_CR21","doi-asserted-by":"publisher","unstructured":"Clifton, C.: Privacy Metrics. In: LIU, L., \u00d6ZSU, M.T. (eds.) Encyclopedia of Database Systems, pp. 2137\u20132139. Springer US, Boston, MA (2009). https:\/\/doi.org\/10.1007\/978-0-387-39940-9_272, https:\/\/doi.org\/10.1007\/978-0-387-39940-9_272","DOI":"10.1007\/978-0-387-39940-9_272"},{"key":"5_CR22","unstructured":"d\u2019Aloisio, G., Marco, A.D., Stilo, G.: Manila github repository (Jan 2023), https:\/\/github.com\/giordanoDaloisio\/manila"},{"key":"5_CR23","doi-asserted-by":"publisher","unstructured":"d\u2019Aloisio, G., Marco, A.D., Stilo, G.: Manila zenodo repository (Jan 2023). https:\/\/doi.org\/10.5281\/zenodo.7525759, https:\/\/doi.org\/10.5281\/zenodo.7525759","DOI":"10.5281\/zenodo.7525759"},{"key":"5_CR24","doi-asserted-by":"publisher","unstructured":"Di\u00a0Sipio, C., Di\u00a0Rocco, J., Di\u00a0Ruscio, D., Nguyen, D.P.T.: A Low-Code Tool Supporting the Development of Recommender Systems. In: Fifteenth ACM Conference on Recommender Systems. pp. 741\u2013744. ACM, Amsterdam Netherlands (Sep 2021). https:\/\/doi.org\/10.1145\/3460231.3478885, https:\/\/dl.acm.org\/doi\/10.1145\/3460231.3478885","DOI":"10.1145\/3460231.3478885"},{"key":"5_CR25","doi-asserted-by":"publisher","unstructured":"Domingos, P., Pazzani, M.: On the Optimality of the Simple Bayesian Classifier under Zero-One Loss. Machine Learning 29(2), 103\u2013130 (Nov 1997). https:\/\/doi.org\/10.1023\/A:1007413511361, https:\/\/doi.org\/10.1023\/A:1007413511361","DOI":"10.1023\/A:1007413511361"},{"key":"5_CR26","doi-asserted-by":"publisher","unstructured":"d\u2019Aloisio, G., D\u2019Angelo, A., Di\u00a0Marco, A., Stilo, G.: Debiaser for Multiple Variables to enhance fairness in classification tasks. Information Processing & Management 60(2), 103226 (Mar 2023). https:\/\/doi.org\/10.1016\/j.ipm.2022.103226, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0306457322003272","DOI":"10.1016\/j.ipm.2022.103226"},{"key":"5_CR27","doi-asserted-by":"publisher","unstructured":"Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., Venkatasubramanian, S.: Certifying and Removing Disparate Impact. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 259\u2013268. ACM, Sydney NSW Australia (Aug 2015). https:\/\/doi.org\/10.1145\/2783258.2783311, https:\/\/dl.acm.org\/doi\/10.1145\/2783258.2783311","DOI":"10.1145\/2783258.2783311"},{"key":"5_CR28","doi-asserted-by":"crossref","unstructured":"Friedman, J.H.: Stochastic gradient boosting. Computational statistics & data analysis 38(4), 367\u2013378 (2002), publisher: Elsevier","DOI":"10.1016\/S0167-9473(01)00065-2"},{"key":"5_CR29","doi-asserted-by":"publisher","unstructured":"Galindo, J.A., Benavides, D., Trinidad, P., Guti\u00e9rrez-Fern\u00e1ndez, A.M., Ruiz-Cort\u00e9s, A.: Automated analysis of feature models: Quo vadis? Computing 101(5), 387\u2013433 (May 2019). https:\/\/doi.org\/10.1007\/s00607-018-0646-1, http:\/\/link.springer.com\/10.1007\/s00607-018-0646-1","DOI":"10.1007\/s00607-018-0646-1"},{"key":"5_CR30","doi-asserted-by":"crossref","unstructured":"Giray, G.: A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software p. 111031 (2021)","DOI":"10.1016\/j.jss.2021.111031"},{"key":"5_CR31","doi-asserted-by":"publisher","unstructured":"Goncalves Jr., P.M., Barros, R.S.M.: Automating data preprocessing with dmpml and kddml. In: 2011 10th IEEE\/ACIS International Conference on Computer and Information Science. pp. 97\u2013103 (2011). https:\/\/doi.org\/10.1109\/ICIS.2011.23","DOI":"10.1109\/ICIS.2011.23"},{"key":"5_CR32","unstructured":"Hamada, K., Ishikawa, F., Masuda, S., Myojin, T., Nishi, Y., Ogawa, H., Toku, T., Tokumoto, S., Tsuchiya, K., Ujita, Y., et\u00a0al.: Guidelines for quality assurance of machine learning-based artificial intelligence. In: SEKE. pp. 335\u2013341 (2020)"},{"key":"5_CR33","unstructured":"Hardt, M., Price, E., Price, E., Srebro, N.: Equality of Opportunity in Supervised Learning. In: Advances in Neural Information Processing Systems. vol.\u00a029. Curran Associates, Inc. (2016), https:\/\/proceedings.neurips.cc\/paper\/2016\/hash\/9d2682367c3935defcb1f9e247a97c0d-Abstract.html"},{"key":"5_CR34","doi-asserted-by":"publisher","unstructured":"He, X., Zhao, K., Chu, X.: Automl: A survey of the state-of-the-art. Knowledge-Based Systems 212, 106622 (2021). https:\/\/doi.org\/10.1016\/j.knosys.2020.106622, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0950705120307516","DOI":"10.1016\/j.knosys.2020.106622"},{"key":"5_CR35","doi-asserted-by":"crossref","unstructured":"Ishikawa, F.: Concepts in quality assessment for machine learning-from test data to arguments. In: International Conference on Conceptual Modeling. pp. 536\u2013544. Springer (2018)","DOI":"10.1007\/978-3-030-00847-5_39"},{"key":"5_CR36","unstructured":"ISO: ISO\/IEC 25010:2011. Tech. rep. (2011), https:\/\/www.iso.org\/cms\/render\/live\/en\/sites\/isoorg\/contents\/data\/standard\/03\/57\/35733.html"},{"key":"5_CR37","doi-asserted-by":"publisher","unstructured":"Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33(1), 1\u201333 (Oct 2012). https:\/\/doi.org\/10.1007\/s10115-011-0463-8, http:\/\/link.springer.com\/10.1007\/s10115-011-0463-8","DOI":"10.1007\/s10115-011-0463-8"},{"key":"5_CR38","doi-asserted-by":"crossref","unstructured":"Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature-oriented domain analysis (foda) feasibility study. Tech. rep., Carnegie-Mellon Univ Pittsburgh Pa Software Engineering Inst (1990)","DOI":"10.21236\/ADA235785"},{"key":"5_CR39","doi-asserted-by":"crossref","unstructured":"Kearns, M., Neel, S., Roth, A., Wu, Z.S.: An empirical study of rich subgroup fairness for machine learning. In: Proceedings of the conference on fairness, accountability, and transparency. pp. 100\u2013109 (2019)","DOI":"10.1145\/3287560.3287592"},{"key":"5_CR40","doi-asserted-by":"crossref","unstructured":"Kumeno, F.: Sofware engneering challenges for machine learning applications: A literature review. Intelligent Decision Technologies 13(4), 463\u2013476 (2019)","DOI":"10.3233\/IDT-190160"},{"key":"5_CR41","unstructured":"Kusner, M.J., Loftus, J., Russell, C., Silva, R.: Counterfactual fairness. In: Advances in Neural Information Processing Systems. vol.\u00a030. Curran Associates, Inc. (2017), https:\/\/proceedings.neurips.cc\/paper\/2017\/hash\/a486cd07e4ac3d270571622f4f316ec5-Abstract.html"},{"key":"5_CR42","doi-asserted-by":"crossref","unstructured":"Lim, T.S., Loh, W.Y., Shih, Y.S.: A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine learning 40(3), 203\u2013228 (2000), publisher: Springer","DOI":"10.1023\/A:1007608224229"},{"key":"5_CR43","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., Kotsiantis, S.: Explainable ai: A review of machine learning interpretability methods. Entropy 23(1), \u00a018 (2021)","DOI":"10.3390\/e23010018"},{"key":"5_CR44","doi-asserted-by":"crossref","unstructured":"Liu, J., Pacitti, E., Valduriez, P., Mattoso, M.: A survey of data-intensive scientific workflow management. Journal of Grid Computing 13(4), 457\u2013493 (2015)","DOI":"10.1007\/s10723-015-9329-8"},{"key":"5_CR45","unstructured":"Mart\u00ednez-Plumed, F., Contreras-Ochando, L., Ferri, C., Orallo, J.H., Kull, M., Lachiche, N., Quintana, M.J.R., Flach, P.A.: Crisp-dm twenty years later: From data mining processes to data science trajectories. IEEE Transactions on Knowledge and Data Engineering (2019)"},{"key":"5_CR46","doi-asserted-by":"publisher","unstructured":"Mart\u00ednez-Fern\u00e1ndez, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A.M., Wagner, S.: Software Engineering for AI-Based Systems: A Survey. ACM Transactions on Software Engineering and Methodology 31(2), 37e:1\u201337e:59 (Apr 2022). https:\/\/doi.org\/10.1145\/3487043, https:\/\/doi.org\/10.1145\/3487043","DOI":"10.1145\/3487043"},{"key":"5_CR47","doi-asserted-by":"publisher","unstructured":"Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys 54(6), 1\u201335 (Jul 2021). https:\/\/doi.org\/10.1145\/3457607, https:\/\/dl.acm.org\/doi\/10.1145\/3457607","DOI":"10.1145\/3457607"},{"key":"5_CR48","doi-asserted-by":"crossref","unstructured":"Menard, S.: Applied logistic regression analysis, vol.\u00a0106. Sage (2002)","DOI":"10.4135\/9781412983433"},{"key":"5_CR49","unstructured":"Molnar, C.: Interpretable machine learning. Lulu. com (2020)"},{"key":"5_CR50","doi-asserted-by":"publisher","unstructured":"Muccini, H., Vaidhyanathan, K.: Software Architecture for ML-based Systems: What Exists and What Lies Ahead. In: 2021 IEEE\/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN). pp. 121\u2013128 (May 2021). https:\/\/doi.org\/10.1109\/WAIN52551.2021.00026","DOI":"10.1109\/WAIN52551.2021.00026"},{"key":"5_CR51","unstructured":"Nations, U.: THE 17 GOALS $$|$$ Sustainable Development, https:\/\/sdgs.un.org\/goals"},{"key":"5_CR52","doi-asserted-by":"crossref","unstructured":"Noble, W.S.: What is a support vector machine? Nature biotechnology 24(12), 1565\u20131567 (2006), publisher: Nature Publishing Group","DOI":"10.1038\/nbt1206-1565"},{"key":"5_CR53","unstructured":"PalletsProject: Jinja website, https:\/\/jinja.palletsprojects.com\/"},{"key":"5_CR54","doi-asserted-by":"crossref","unstructured":"Patro, S., Sahu, K.K.: Normalization: A preprocessing stage. arXiv preprint arXiv:1503.06462 (2015)","DOI":"10.17148\/IARJSET.2015.2305"},{"key":"5_CR55","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825\u20132830 (2011)"},{"key":"5_CR56","unstructured":"Putzel, P., Lee, S.: Blackbox Post-Processing for Multiclass Fairness. arXiv:2201.04461 [cs] (Jan 2022), http:\/\/arxiv.org\/abs\/2201.04461, arXiv: 2201.04461"},{"key":"5_CR57","doi-asserted-by":"crossref","unstructured":"Refaeilzadeh, P., Tang, L., Liu, H.: Cross-validation. Encyclopedia of database systems 5, 532\u2013538 (2009)","DOI":"10.1007\/978-0-387-39940-9_565"},{"key":"5_CR58","doi-asserted-by":"publisher","unstructured":"Refaeilzadeh, P., Tang, L., Liu, H.: Cross-Validation, pp.\u00a01\u20137. Springer New York, New York, NY (2016). https:\/\/doi.org\/10.1007\/978-1-4899-7993-3_565-2","DOI":"10.1007\/978-1-4899-7993-3_565-2"},{"key":"5_CR59","doi-asserted-by":"publisher","unstructured":"Refaeilzadeh, P., Tang, L., Liu, H.: Cross-Validation. In: Encyclopedia of Database Systems, pp.\u00a01\u20137. Springer New York, New York, NY (2016). https:\/\/doi.org\/10.1007\/978-1-4899-7993-3_565-2","DOI":"10.1007\/978-1-4899-7993-3_565-2"},{"key":"5_CR60","unstructured":"Rosenfield, G., Fitzpatrick-Lins, K.: A coefficient of agreement as a measure of thematic classification accuracy. Photogrammetric Engineering and Remote Sensing 52(2), 223\u2013227 (1986), http:\/\/pubs.er.usgs.gov\/publication\/70014667"},{"key":"5_CR61","doi-asserted-by":"crossref","unstructured":"R\u00f6nkk\u00f6, M., Heikkinen, J., Kotovirta, V., Chandrasekar, V.: Automated preprocessing of environmental data. Future Generation Computer Systems 45, 13\u201324 (2015). https:\/\/doi.org\/10.1016\/j.future.2014.10.011, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0167739X14002040","DOI":"10.1016\/j.future.2014.10.011"},{"key":"5_CR62","doi-asserted-by":"crossref","unstructured":"Sahay, A., Indamutsa, A., Di\u00a0Ruscio, D., Pierantonio, A.: Supporting the understanding and comparison of low-code development platforms. In: 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). pp. 171\u2013178. IEEE (2020)","DOI":"10.1109\/SEAA51224.2020.00036"},{"key":"5_CR63","unstructured":"Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577 (2018)"},{"key":"5_CR64","doi-asserted-by":"crossref","unstructured":"Siebert, J., Joeckel, L., Heidrich, J., Trendowicz, A., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., Aoyama, M.: Construction of a quality model for machine learning systems. Software Quality Journal pp. 1\u201329 (2021)","DOI":"10.1007\/s11219-021-09557-y"},{"key":"5_CR65","doi-asserted-by":"crossref","unstructured":"de\u00a0Souza\u00a0Nascimento, E., Ahmed, I., Oliveira, E., Palheta, M.P., Steinmacher, I., Conte, T.: Understanding development process of machine learning systems: Challenges and solutions. In: 2019 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). pp.\u00a01\u20136. IEEE (2019)","DOI":"10.1109\/ESEM.2019.8870157"},{"key":"5_CR66","doi-asserted-by":"crossref","unstructured":"Studer, S., Bui, T.B., Drescher, C., Hanuschkin, A., Winkler, L., Peters, S., M\u00fcller, K.R.: Towards crisp-ml (q): a machine learning process model with quality assurance methodology. Machine Learning and Knowledge Extraction 3(2), 392\u2013413 (2021)","DOI":"10.3390\/make3020020"},{"key":"5_CR67","doi-asserted-by":"publisher","unstructured":"Taha, A.A., Hanbury, A.: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging 15(1), \u00a029 (Aug 2015). https:\/\/doi.org\/10.1186\/s12880-015-0068-x, https:\/\/doi.org\/10.1186\/s12880-015-0068-x","DOI":"10.1186\/s12880-015-0068-x"},{"key":"5_CR68","doi-asserted-by":"crossref","unstructured":"Th\u00fcm, T., K\u00e4stner, C., Benduhn, F., Meinicke, J., Saake, G., Leich, T.: Featureide: An extensible framework for feature-oriented software development. Science of Computer Programming 79, 70\u201385 (2014)","DOI":"10.1016\/j.scico.2012.06.002"},{"key":"5_CR69","doi-asserted-by":"crossref","unstructured":"Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.P., Humbert, M., Juels, A., Lin, H.: Fairtest: Discovering unwarranted associations in data-driven applications. In: 2017 IEEE European Symposium on Security and Privacy (EuroS &P). pp. 401\u2013416. IEEE (2017)","DOI":"10.1109\/EuroSP.2017.29"},{"key":"5_CR70","doi-asserted-by":"crossref","unstructured":"Villamizar, H., Escovedo, T., Kalinowski, M.: Requirements engineering for machine learning: A systematic mapping study. In: SEAA. pp. 29\u201336 (2021)","DOI":"10.1109\/SEAA53835.2021.00013"},{"key":"5_CR71","unstructured":"Xu, R., Baracaldo, N., Joshi, J.: Privacy-Preserving Machine Learning: Methods, Challenges and Directions. arXiv:2108.04417 [cs] (Sep 2021), http:\/\/arxiv.org\/abs\/2108.04417, arXiv: 2108.04417"},{"key":"5_CR72","unstructured":"Zhang, J.M., Harman, M., Ma, L., Liu, Y.: Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering (2020)"},{"key":"5_CR73","doi-asserted-by":"publisher","unstructured":"Zhou, J., Gandomi, A.H., Chen, F., Holzinger, A.: Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics. Electronics 10(5), \u00a0593 (Jan 2021). https:\/\/doi.org\/10.3390\/electronics10050593, https:\/\/www.mdpi.com\/2079-9292\/10\/5\/593, number: 5 Publisher: Multidisciplinary Digital Publishing Institute","DOI":"10.3390\/electronics10050593"}],"container-title":["Lecture Notes in Computer Science","Fundamental Approaches to Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-30826-0_5","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,22]],"date-time":"2023-05-22T22:02:30Z","timestamp":1684792950000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-30826-0_5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"ISBN":["9783031308253","9783031308260"],"references-count":73,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-30826-0_5","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023]]},"assertion":[{"value":"20 April 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"FASE","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Fundamental Approaches to Software Engineering","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Paris","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"France","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"22 April 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"27 April 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"fase2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/etaps.org\/2023\/fase","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Double-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"EasyChair","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"50","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"12","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"24% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"3","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"6-7","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"Yes","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"The proceedings also include 2 tool papers, 2 NIER papers, and 2 competition papers","order":10,"name":"additional_info_on_review_process","label":"Additional Info on Review Process","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}