{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T03:09:36Z","timestamp":1775012976220,"version":"3.50.1"},"reference-count":44,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2020,11,17]],"date-time":"2020-11-17T00:00:00Z","timestamp":1605571200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Acquiring knowledge about users\u2019 opinion and what they say regarding specific features within an app, constitutes a solid steppingstone for understanding their needs and concerns. App review utilization helps project management teams to identify threads and opportunities for app software maintenance, optimization and strategic marketing purposes. Nevertheless, app user review classification for identifying valuable gems of information for app software improvement, is a complex and multidimensional issue. It requires foresight and multiple combinations of sophisticated text pre-processing, feature extraction and machine learning methods to efficiently classify app reviews into specific topics. Against this backdrop, we propose a novel feature engineering classification schema that is capable to identify more efficiently and earlier terms-words within reviews that could be classified into specific topics. For this reason, we present a novel feature extraction method, the DEVMAX.DF combined with different machine learning algorithms to propose a solution in app review classification problems. One step further, a simulation of a real case scenario takes place to validate the effectiveness of the proposed classification schema into different apps. After multiple experiments, results indicate that the proposed schema outperforms other term extraction methods such as TF.IDF and \u03c72 to classify app reviews into topics. To this end, the paper contributes to the knowledge expansion of research and practitioners with the purpose to reinforce their decision-making process within the realm of app reviews utilization.<\/jats:p>","DOI":"10.3390\/e22111310","type":"journal-article","created":{"date-parts":[[2020,11,17]],"date-time":"2020-11-17T11:02:12Z","timestamp":1605610932000},"page":"1310","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["How to Utilize My App Reviews? A Novel Topics Extraction Machine Learning Schema for Strategic Business Purposes"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5273-0855","authenticated-orcid":false,"given":"Ioannis","family":"Triantafyllou","sequence":"first","affiliation":[{"name":"Research Lab of Information Management, Department of Archival, Library Science and Information Studies, University of West Attica, Ag. Spyridonos, Egaleo, 12243 Athens, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2407-9502","authenticated-orcid":false,"given":"Ioannis C.","family":"Drivas","sequence":"additional","affiliation":[{"name":"Research Lab of Information Management, Department of Archival, Library Science and Information Studies, University of West Attica, Ag. Spyridonos, Egaleo, 12243 Athens, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1659-3504","authenticated-orcid":false,"given":"Georgios","family":"Giannakopoulos","sequence":"additional","affiliation":[{"name":"Research Lab of Information Management, Department of Archival, Library Science and Information Studies, University of West Attica, Ag. Spyridonos, Egaleo, 12243 Athens, Greece"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,17]]},"reference":[{"key":"ref_1","unstructured":"Statista (2020, October 05). Annual Number of Global Mobile App Downloads 2016\u20132019. Available online: https:\/\/www.statista.com\/statistics\/271644\/worldwide-free-and-paid-mobile-app-store-downloads\/."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Falaki, H., Mahajan, R., Kandula, S., Lymberopoulos, D., Govindan, R., and Estrin, D. (2010, January 15\u201318). Diversity in smartphone usage. Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services, San Francisco, CA, USA.","DOI":"10.1145\/1814433.1814453"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"D\u0105browski, J., Letier, E., Perini, A., and Susi, A. Finding and analyzing app reviews related to specific features: A research preview. Proceedings of the International Working Conference on Requirements Engineering: Foundation for Software Quality, Essen, Germany, 18\u201321 March 2019.","DOI":"10.1007\/978-3-030-15538-4_14"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1007\/s00766-016-0251-9","article-title":"On the automatic classification of app reviews","volume":"21","author":"Maalej","year":"2016","journal-title":"Requir. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Morales-Ramirez, I., Munante, D., Kifetew, F., Perini, A., Susi, A., and Siena, A. (2017, January 4\u20138). Exploiting user feedback in tool-supported multi-criteria requirements prioritization. Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE), Lisbon, Portugal.","DOI":"10.1109\/RE.2017.41"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/MS.2015.153","article-title":"Toward data-driven requirements engineering","volume":"33","author":"Maalej","year":"2015","journal-title":"IEEE Softw."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Martens, D., and Johann, T. (2017, January 21\u201321). On the emotion of users in app reviews. Proceedings of the 2017 IEEE\/ACM 2nd International Workshop on Emotion Awareness in Software Engineering (SEmotion), Buenos Aires, Argentina.","DOI":"10.1109\/SEmotion.2017.6"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Panichella, S., di Sorbo, A., Guzman, E., Visaggio, C.A., Canfora, G., and Gall, H.C. (October, January 29). How can i improve my app? Classifying user reviews for software maintenance and evolution. Proceedings of the 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), Bremen, Germany.","DOI":"10.1109\/ICSM.2015.7332474"},{"key":"ref_9","unstructured":"Intercom.com (2020, October 02). How to Make Product Improvements. Available online: https:\/\/www.intercom.com\/blog\/ways-to-improve-a-product\/."},{"key":"ref_10","unstructured":"Statista (2020, October 06). Mobile Application User Retention Rate Worldwide from 2012 to 2019. Available online: https:\/\/www.statista.com\/statistics\/751532\/worldwide-application-user-retention-rate\/."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Iacob, C., and Harrison, R. (2013, January 18\u201319). Retrieving and analyzing mobile apps feature requests from online reviews. Proceedings of the 2013 10th Working Conference on Mining Software Repositories (MSR), San Francisco, CA, USA.","DOI":"10.1109\/MSR.2013.6624001"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Johann, T., Stanik, C., and Maalej, W. (2017, January 4\u20138). Safe: A simple approach for feature extraction from app descriptions and app reviews. Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE), Lisbon, Portugal.","DOI":"10.1109\/RE.2017.71"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Maalej, W., and Hadeer, N. (2015, January 24\u201328). Bug report, feature request, or simply praise on automatically classifying app reviews. Proceedings of the 2015 IEEE 23rd international requirements engineering conference (RE), Ottawa, ON, Canada.","DOI":"10.1109\/RE.2015.7320414"},{"key":"ref_14","unstructured":"Vu, P.M., Nguyen, T.T., Pham, H.V., and Nguyen, T.T. (2015, January 9\u201313). Mining user opinions in mobile app reviews: A keyword-based approach (t). Proceedings of the 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE), Lincoln, NE, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Vu, P.M., Pham, H.V., Nguyen, T.T., and Nguyen, T.T. (2016, January 3\u20137). Phrase-based extraction of user opinions in mobile app reviews. Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering, Singapore.","DOI":"10.1145\/2970276.2970365"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Di Sorbo, A., Panichella, S., Alexandru, C.V., Shimagaki, J., Visaggio, C.A., Canfora, G., and Gall, H.C. (2016, January 13\u201318). What would users change in my app? Summarizing app reviews for recommending software changes. Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, Seattle, WA, USA.","DOI":"10.1145\/2950290.2950299"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Guzman, E., El-Haliby, M., and Bruegge, B. (2015, January 9\u201313). Ensemble methods for app review classification: An approach for software evolution (n). Proceedings of the 2015 30th IEEE\/ACM International Conference on Automated Software Engineering (ASE), Lincoln, NE, USA.","DOI":"10.1109\/ASE.2015.88"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Rustam, F., Ashraf, I., Mehmood, A., Ullah, S., and Choi, G.S. (2019). Tweets classification on the base of sentiments for US airline companies. Entropy, 21.","DOI":"10.3390\/e21111078"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"30234","DOI":"10.1109\/ACCESS.2020.2972632","article-title":"Classification of Shopify App User Reviews Using Novel Multi Text Features","volume":"8","author":"Rustam","year":"2020","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Suprayogi, E., Budi, I., and Mahendra, R. (2018, January 27\u201328). Information Extraction for Mobile Application User Review. Proceedings of the 2018 International Conference on Advanced Computer Science and Information Systems (ICACSIS), Yogyakarta, Indonesia.","DOI":"10.1109\/ICACSIS.2018.8618164"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Shah, F.A., Sirts, K., and Pfahl, D. Simplifying the classification of app reviews using only lexical features. Proceedings of the International Conference on Software Technologies, Lisbon, Portugal, 18\u201322 June 2018.","DOI":"10.1007\/978-3-030-29157-0_8"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Bakiu, E., and Guzman, E. (2017, January 4\u20138). Which feature is unusable? Detecting usability and user experience issues from user reviews. Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW), Lisbon, Portugal.","DOI":"10.1109\/REW.2017.76"},{"key":"ref_23","unstructured":"Panichella, S., and Adelina, C. (2020, October 15). Replication Package for: Analyzing Reviews and Code of Mobile Apps for a Better Release Planning Organization. Available online: https:\/\/github.com\/panichella\/UserReviewReference-Replication-Package\/tree\/URR-v1.0."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Ciurumelea, A., Schaufelb\u00fchl, A., Panichella, S., and Gall, H.C. (2017, January 20\u201324). Analyzing reviews and code of mobile apps for better release planning. Proceedings of the 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), Klagenfurt, Austria.","DOI":"10.1109\/SANER.2017.7884612"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1016\/j.compind.2015.10.008","article-title":"A distributional approach to open questions in market research","volume":"78","author":"Evert","year":"2016","journal-title":"Comput. Ind."},{"key":"ref_26","unstructured":"Triantafyllou, I., Demiros, I., and Piperidis, S. (2001, January 5\u20137). Two level self-organizing approach to text classification. Proceedings of the RANLP-2001: Recent Advances in NLP, Tzigov Chark, Bulgaria."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Aggarwal, C.C., and Zhai, C. (2012). Mining Text Data, Springer Science & Business Media.","DOI":"10.1007\/978-1-4614-3223-4"},{"key":"ref_28","first-page":"16","article-title":"Hypatia Digital Library: A novel text classification approach for small text fragments","volume":"4","author":"Triantafyllou","year":"2019","journal-title":"J. Integr. Inf. Manag."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Vorgia, F., Triantafyllou, I., and Koulouris, A. (2017). Hypatia Digital Library: A text classification approach based on abstracts. Strategic Innovative Marketing, Springer.","DOI":"10.1007\/978-3-319-33865-1_89"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Nargesian, F., Samulowitz, H., Khurana, U., Khalil, E.B., and Turaga, D.S. (2017, January 19\u201325). Learning Feature Engineering for Classification. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, Melbourne, Australia.","DOI":"10.24963\/ijcai.2017\/352"},{"key":"ref_31","unstructured":"Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. (2015, January 7\u201312). Efficient and robust automated machine learning. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1108\/00220410410560582","article-title":"Understanding inverse document frequency: On theoretical arguments for IDF","volume":"60","author":"Robertson","year":"2004","journal-title":"J. Doc."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1108\/eb026526","article-title":"A statistical interpretation of term specificity and its application in retrieval","volume":"28","author":"Jones","year":"1972","journal-title":"J. Doc."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2758","DOI":"10.1016\/j.eswa.2010.08.066","article-title":"A comparative study of TF* IDF, LSI and multi-words for text classification","volume":"38","author":"Zhang","year":"2011","journal-title":"Expert Syst. Appl."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Saengsiri, P., Meesad, P., Wichian, S.N., and Herwig, U. (2010, January 24\u201325). Comparison of hybrid feature selection models on gene expression data. Proceedings of the 2010 Eighth International Conference on ICT and Knowledge Engineering, Bangkok, Thailand.","DOI":"10.1109\/ICTKE.2010.5692905"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Haruechaiyasak, C., Jitkrittum, W., Sangkeettrakarn, C., and Damrongrat, C. (2008, January 9\u201312). Implementing news article category browsing based on text categorization technique. Proceedings of the 2008 IEEE\/WIC\/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Sydney, Australia.","DOI":"10.1109\/WIIAT.2008.61"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.compind.2015.09.005","article-title":"Natural language processing for aviation safety reports: From classification to interactive analysis","volume":"78","author":"Tanguy","year":"2016","journal-title":"Comput. Ind."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.compind.2015.12.001","article-title":"Text analytics in industry: Challenges, desiderata and trends","volume":"78","author":"Ittoo","year":"2016","journal-title":"Comput. Ind."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Hirave, T., Malgaonkar, S., Alwash, M., Cheriyan, J., and Surve, S. (2019, January 20\u201321). Analysis and Prioritization of App Reviews. Proceedings of the 2019 International Conference on Advances in Computing, Communication and Control (ICAC3), Mumbai, India.","DOI":"10.1109\/ICAC347590.2019.9036801"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Guzman, E., and Maalej, W. (2014, January 25\u201329). How do users like this feature? A fine grained sentiment analysis of app reviews. Proceedings of the 2014 IEEE 22nd International Requirements Engineering Conference (RE), Karlskrona, Sweden.","DOI":"10.1109\/RE.2014.6912257"},{"key":"ref_41","unstructured":"Google Developers (2020, October 26). Google Analytics for Mobile Apps. In Google Analytics Implementation Guides and Solutions. Available online: https:\/\/developers.google.com\/analytics\/solutions\/mobile."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1109\/MS.2013.124","article-title":"Does involving users in software development really influence system success?","volume":"30","author":"Abelein","year":"2013","journal-title":"IEEE Softw."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Drivas, I.C., Giannakopoulos, G.A., and Sakas, D.P. Learning Analytics in Big Data Era. Exploration, Validation and Predictive Models Development. Proceedings of the International Conference on Intelligent Tutoring Systems, Athens, Greece, 8\u201312 June 2020.","DOI":"10.1007\/978-3-030-49663-0_50"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1016\/j.ins.2019.02.022","article-title":"Probabilistic linguistic TODIM method for selecting products through online product reviews","volume":"485","author":"Liu","year":"2019","journal-title":"Inf. Sci."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/11\/1310\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:34:39Z","timestamp":1760178879000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/22\/11\/1310"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,17]]},"references-count":44,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2020,11]]}},"alternative-id":["e22111310"],"URL":"https:\/\/doi.org\/10.3390\/e22111310","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,17]]}}}