{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T13:45:29Z","timestamp":1778247929303,"version":"3.51.4"},"reference-count":83,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2019,3,22]],"date-time":"2019-03-22T00:00:00Z","timestamp":1553212800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>When performing a regression or classification analysis, one needs to specify a statistical model. This model should avoid the overfitting and underfitting of data, and achieve a low generalization error that characterizes its prediction performance. In order to identify such a model, one needs to decide which model to select from candidate model families based on performance evaluations. In this paper, we review the theoretical framework of model selection and model assessment, including error-complexity curves, the bias-variance tradeoff, and learning curves for evaluating statistical models. We discuss criterion-based, step-wise selection procedures and resampling methods for model selection, whereas cross-validation provides the most simple and generic means for computationally estimating all required entities. To make the theoretical concepts transparent, we present worked examples for linear regression models. However, our conceptual presentation is extensible to more general models, as well as classification problems.<\/jats:p>","DOI":"10.3390\/make1010032","type":"journal-article","created":{"date-parts":[[2019,3,25]],"date-time":"2019-03-25T06:56:52Z","timestamp":1553497012000},"page":"521-551","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":116,"title":["Evaluation of Regression Models: Model Assessment, Model Selection and Generalization Error"],"prefix":"10.3390","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0745-5641","authenticated-orcid":false,"given":"Frank","family":"Emmert-Streib","sequence":"first","affiliation":[{"name":"Predictive Society and Data Analytics Lab, Faculty of Information Technolgy and Communication Sciences, Tampere University, 33100 Tampere, Finland"},{"name":"Institute of Biosciences and Medical Technology, 33520 Tampere, Finland"}]},{"given":"Matthias","family":"Dehmer","sequence":"additional","affiliation":[{"name":"Institute for Intelligent Production, Faculty for Management, University of Applied Sciences Upper Austria, Steyr Campus, 4400 Steyr, Austria"},{"name":"Department of Mechatronics and Biomedical Computer Science, University for Health Sciences, Medical Informatics and Technology, 6060 Hall in Tirol, Austria"},{"name":"College of Computer and Control Engineering, Nankai University, Tianjin 300071, China"}]}],"member":"1968","published-online":{"date-parts":[[2019,3,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/j.dss.2013.08.008","article-title":"Understanding the paradigm shift to computational social science in the presence of big data","volume":"63","author":"Chang","year":"2014","journal-title":"Decis. Support Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1089\/big.2013.1508","article-title":"Data science and its relationship to big data and data-driven decision making","volume":"1","author":"Provost","year":"2013","journal-title":"Big Data"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1080\/00031305.2015.1077729","article-title":"Data science in statistics curricula: Preparing students to \u2018think with data\u2019","volume":"69","author":"Hardin","year":"2015","journal-title":"Am. Stat."},{"key":"ref_4","first-page":"12","article-title":"The process of analyzing data is the emergent feature of data science","volume":"7","author":"Moutari","year":"2016","journal-title":"Front. Genet."},{"key":"ref_5","first-page":"235","article-title":"Defining data science by a data-driven quantification of the community","volume":"1","author":"Dehmer","year":"2019","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Dehmer, M., and Emmert-Streib, F. (2017). Frontiers Data Science, CRC Press.","DOI":"10.1201\/9781315156408"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.nbt.2008.12.009","article-title":"Next-generation DNA sequencing techniques","volume":"25","author":"Ansorge","year":"2009","journal-title":"New Biotechnol."},{"key":"ref_8","first-page":"15","article-title":"The gene regulatory network for breast cancer: Integrated regulatory landscape of cancer hallmarks","volume":"5","author":"Mullan","year":"2014","journal-title":"Front. Genet."},{"key":"ref_9","first-page":"506","article-title":"A review of connectivity mapping and computational approaches in pharmacogenomics","volume":"19","author":"Musa","year":"2017","journal-title":"Brief. Bioinf."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1146\/annurev.genom.9.081307.164359","article-title":"Next-generation DNA sequencing methods","volume":"9","author":"Mardis","year":"2008","journal-title":"Ann. Rev. Genom. Hum. Genet."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12859-016-0979-8","article-title":"Comparison of module detection algorithms in protein networks and investigation of the biological meaning of predicted modules","volume":"17","author":"Tripathi","year":"2016","journal-title":"BMC Bioinf."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1140\/epjst\/e2012-01697-8","article-title":"Manifesto of computational social science","volume":"214","author":"Conte","year":"2012","journal-title":"Eur. Phys. J.-Spec. Top."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1126\/science.1167742","article-title":"Life in the network: The coming age of computational social science","volume":"323","author":"Lazer","year":"2009","journal-title":"Science"},{"key":"ref_14","first-page":"1","article-title":"Data analytics applications for streaming data from social media: What to predict?","volume":"1","author":"Dehmer","year":"2018","journal-title":"Front. Big Data"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging Predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Clarke, B., Fokoue, E., and Zhang, H.H. (2009). Principles and Theory for Data Mining and Machine Learning, Springer.","DOI":"10.1007\/978-0-387-98135-2"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Harrell, F.E. (2001). Regression Modeling Strategies, Springer.","DOI":"10.1007\/978-1-4757-3462-1"},{"key":"ref_18","unstructured":"Haste, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference and Prediction, Springer."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"359","DOI":"10.3390\/make1010021","article-title":"High-dimensional LASSO-based computational regression models: Regularization, shrinkage, and selection","volume":"1","author":"Dehmer","year":"2019","journal-title":"Mach. Learn. Knowl. Extr."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Sch\u00f6lkopf, B., and Smola, A. (2002). Learning with Kernels: Support Vector Machines, Regulariztion, Optimization and Beyond, The MIT Press.","DOI":"10.7551\/mitpress\/4175.001.0001"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1109\/MSP.2018.2867638","article-title":"Model selection techniques: An overview","volume":"35","author":"Ding","year":"2018","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1006\/jmps.1999.1284","article-title":"Key concepts in model selection: Performance and generalizability","volume":"44","author":"Forster","year":"2000","journal-title":"J. Math. Psychol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1214\/09-SS054","article-title":"A survey of cross-validation procedures for model selection","volume":"4","author":"Arlot","year":"2010","journal-title":"Stat. Surv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1177\/0049124104268644","article-title":"Multimodel inference: Understanding AIC and BIC in model selection","volume":"33","author":"Burnham","year":"2004","journal-title":"Sociol. Methods Res."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1198\/016214504000000269","article-title":"Methods and criteria for model selection","volume":"99","author":"Kadane","year":"2004","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"111","DOI":"10.2307\/271063","article-title":"Bayesian model selection in social research","volume":"25","author":"Raftery","year":"1995","journal-title":"Sociol. Methodol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1111\/j.1467-9574.2012.00530.x","article-title":"\u2018All models are wrong\u2026\u2019: An introduction to model uncertainty","volume":"66","author":"Wit","year":"2012","journal-title":"Stat. Neerl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1890\/13-1452.1","article-title":"Model selection for ecologists: The worldviews of AIC and BIC","volume":"95","author":"Aho","year":"2014","journal-title":"Ecology"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1006\/jmps.1999.1276","article-title":"An introduction to model selection","volume":"44","author":"Zucchini","year":"2000","journal-title":"J. Math. Psych."},{"key":"ref_30","unstructured":"R Development Core Team (2008). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Sheather, S. (2009). A Modern Approach to Regression With R, Springer Science & Business Media.","DOI":"10.1007\/978-0-387-09608-7"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J. R. Stat. Soc. Ser. B"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Hastie, T., Tibshirani, R., and Wainwright, M. (2015). Statistical Learning with Sparsity: The Lasso And Generalizations, CRC Press.","DOI":"10.1201\/b18401"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/00401706.1970.10488634","article-title":"Ridge regression: Biased estimation for nonorthogonal problems","volume":"12","author":"Hoerl","year":"1970","journal-title":"Technometrics"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Friedman, J., Hastie, T., and Tibshirani, R. (2009). Glmnet: Lasso and elastic-net regularized generalized linear models. R Package Version, 1.","DOI":"10.32614\/CRAN.package.glmnet"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1418","DOI":"10.1198\/016214506000000735","article-title":"The adaptive lasso and its oracle properties","volume":"101","author":"Zou","year":"2006","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1111\/j.1467-9868.2005.00503.x","article-title":"Regularization and variable selection via the elastic net","volume":"67","author":"Zou","year":"2005","journal-title":"J. R. Stat. Soc. Ser. B (Stat. Methodol.)"},{"key":"ref_38","unstructured":"Abu-Mostafa, Y.S., Magdon-Ismail, M., and Lin, H.T. (2012). Learning from Data, AMLBook."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/neco.1992.4.1.1","article-title":"Neural networks and the bias\/variance dilemma","volume":"4","author":"Geman","year":"1992","journal-title":"Neural Comput."},{"key":"ref_40","unstructured":"Kohavi, R., and Wolpert, D.H. (1996, January 3\u20136). Bias plus variance decomposition for zero-one loss functions. Proceedings of the 13th International Conference on Machine Learning, Bari, Italy."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Geurts, P. (2009). Bias vs. variance decomposition for regression and classification. Data Mining and Knowledge Discovery Handbook, Springer.","DOI":"10.1007\/978-0-387-09823-4_37"},{"key":"ref_42","unstructured":"Weinberger, K. (2019, January 01). Lecture Notes in Machine Learning (CS4780\/CS5780). Available online: http:\/\/www.cs.cornell.edu\/courses\/cs4780\/2017sp\/lectures\/lecturenote11.html."},{"key":"ref_43","unstructured":"Nicholson, A.M. (2002). Generalization Error Estimates and Training Data Valuation. [Ph.D. Thesis, California Institute of Technology]."},{"key":"ref_44","first-page":"569","article-title":"Estimation of generalization error: Random and fixed inputs","volume":"16","author":"Wang","year":"2006","journal-title":"Stat. Sin."},{"key":"ref_45","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"S124","DOI":"10.1086\/341840","article-title":"Predictive accuracy as an achievable goal of science","volume":"69","author":"Forster","year":"2002","journal-title":"Philos. Sci."},{"key":"ref_47","unstructured":"Draper, N.R., and Smith, H. (2014). Applied Regression Analysis, John Wiley & Sons."},{"key":"ref_48","first-page":"557","article-title":"Correlation of causation","volume":"20","author":"Wright","year":"1921","journal-title":"J. Agric. Res."},{"key":"ref_49","first-page":"49","article-title":"The interpretation of Mallows\u2019s C_p-statistic","volume":"45","author":"Gilmour","year":"1996","journal-title":"J. R. Stat. Soc. Ser. D (Stat.)"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/147078539203400204","article-title":"Mallows? Cp statistic and model selection in multiple linear regression","volume":"34","author":"Zuccaro","year":"1992","journal-title":"Mark. Res. Soc. J."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Akaike, H. (1974). Akaike, H. A new look at the statistical model identification. Selected Papers of Hirotugu Akaike, Springer.","DOI":"10.1007\/978-1-4612-1694-0_16"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1007\/s00265-010-1037-6","article-title":"A brief guide to model selection, multimodel inference and model averaging in behavioural ecology using Akaike\u2019s information criterion","volume":"65","author":"Symonds","year":"2011","journal-title":"Behav. Ecol. Sociobiol."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1214\/aos\/1176344136","article-title":"Estimating the dimension of a model","volume":"6","author":"Schwarz","year":"1978","journal-title":"Ann. Stat."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1002\/wics.199","article-title":"The Bayesian information criterion: Background, derivation, and applications","volume":"4","author":"Neath","year":"2012","journal-title":"Wiley Interdiscip. Rev. Comput. Stat."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"773","DOI":"10.1080\/01621459.1995.10476572","article-title":"Bayes factors","volume":"90","author":"Kass","year":"1995","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1016\/j.jmp.2015.11.001","article-title":"The philosophy of Bayes factors and the quantification of statistical evidence","volume":"72","author":"Morey","year":"2016","journal-title":"J. Math. Psychol."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1080\/00031305.1999.10474443","article-title":"Bayes factors: What they are and what they are not","volume":"53","author":"Lavine","year":"1999","journal-title":"Am. Stat."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Jaynes, E.T. (2003). Probability Theory: The Logic of Science, Cambridge University Press.","DOI":"10.1017\/CBO9780511790423"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1037\/a0027127","article-title":"Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC)","volume":"17","author":"Vrieze","year":"2012","journal-title":"Psychol. Methods"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"937","DOI":"10.1093\/biomet\/92.4.937","article-title":"Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation","volume":"92","author":"Yang","year":"2005","journal-title":"Biometrika"},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1177\/0049124103262065","article-title":"AIC and BIC: Comparisons of assumptions and performance","volume":"33","author":"Kuha","year":"2004","journal-title":"Sociol. Methods Res."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1093\/biomet\/54.3-4.357","article-title":"The discarding of variables in multivariate analysis","volume":"54","author":"Beale","year":"1967","journal-title":"Biometrika"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1111\/j.2044-8317.1992.tb00992.x","article-title":"Backward, forward and stepwise automated subset selection algorithms: Frequency of obtaining authentic and noise variables","volume":"45","author":"Derksen","year":"1992","journal-title":"Br. J. Math. Stat. Psychol."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1080\/01621459.1975.10479865","article-title":"The predictive sample reuse method with applications","volume":"70","author":"Geisser","year":"1975","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","article-title":"Cross-validatory choice and assessment of statistical predictions","volume":"36","author":"Stone","year":"1974","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"},{"key":"ref_66","unstructured":"Good, P.I. (2006). Resampling Methods, Springer."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"2813","DOI":"10.1002\/(SICI)1097-0258(19971230)16:24<2813::AID-SIM701>3.0.CO;2-Z","article-title":"Resampling and cross-validation techniques: A tool to reduce bias caused by model building?","volume":"16","author":"Schumacher","year":"1997","journal-title":"Stat. Med."},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Efron, B. (1982). The Jackknife, the Bootstrap, and Other Resampling Plans, Siam.","DOI":"10.1137\/1.9781611970319"},{"key":"ref_69","doi-asserted-by":"crossref","unstructured":"Efron, B., and Tibshirani, R. (1994). An Introduction to the Bootstrap, Chapman and Hall\/CRC.","DOI":"10.1201\/9780429246593"},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/S0169-7439(00)00102-7","article-title":"The bootstrap: A tutorial","volume":"54","author":"Wehrens","year":"2000","journal-title":"Chemometr. Intel. Lab. Syst."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1186\/1758-2946-6-10","article-title":"Cross-validation pitfalls when selecting and assessing regression and classification models","volume":"6","author":"Krstajic","year":"2014","journal-title":"J. Cheminform."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"3301","DOI":"10.1093\/bioinformatics\/bti499","article-title":"Prediction error estimation: A comparison of resampling methods","volume":"21","author":"Molinaro","year":"2005","journal-title":"Bioinformatics"},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"605","DOI":"10.1162\/neco.1992.4.4.605","article-title":"Four types of learning curves","volume":"4","author":"Amari","year":"1992","journal-title":"Neural Comput."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/0893-6080(93)90013-M","article-title":"A universal theorem on learning curves","volume":"6","author":"Amari","year":"1993","journal-title":"Neural Netw."},{"key":"ref_75","first-page":"2079","article-title":"On over-fitting in model selection and subsequent selection bias in performance evaluation","volume":"11","author":"Cawley","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_76","first-page":"61","article-title":"Model selection: Beyond the bayesian\/frequentist divide","volume":"11","author":"Guyon","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"711","DOI":"10.1007\/s11222-016-9649-y","article-title":"Comparison of Bayesian predictive methods for model selection","volume":"27","author":"Piironen","year":"2017","journal-title":"Stat. Comput."},{"key":"ref_78","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1098\/rspa.1977.0069","article-title":"Explicativity: A mathematical theory of explanation with statistical applications","volume":"354","author":"Good","year":"1977","journal-title":"Proc. R. Soc. Lond. A"},{"key":"ref_79","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.2307\/41703503","article-title":"Business intelligence and analytics: From big data to big impact","volume":"36","author":"Chen","year":"2012","journal-title":"MIS Q."},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1016\/j.jbusres.2015.07.001","article-title":"Big Data consumer analytics and the transformation of marketing","volume":"69","author":"Erevelles","year":"2016","journal-title":"J. Bus. Res."},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.bdr.2015.01.006","article-title":"Significance and challenges of big data research","volume":"2","author":"Jin","year":"2015","journal-title":"Big Data Res."},{"key":"ref_82","doi-asserted-by":"crossref","unstructured":"Holzinger, A., Kieseberg, P., Weippl, E., and Tjoa, A.M. (2018, January 27\u201330). Current advances, trends and challenges of machine learning and knowledge extraction: From machine learning to explainable ai. Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Hamburg, Germany.","DOI":"10.1007\/978-3-319-99740-7_1"},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1038\/455028a","article-title":"Big data: How do your data grow?","volume":"455","author":"Lynch","year":"2008","journal-title":"Nature"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/1\/1\/32\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:40:05Z","timestamp":1760186405000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/1\/1\/32"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,22]]},"references-count":83,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,3]]}},"alternative-id":["make1010032"],"URL":"https:\/\/doi.org\/10.3390\/make1010032","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3,22]]}}}