{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T14:03:12Z","timestamp":1765807392858,"version":"build-2065373602"},"reference-count":59,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2017,1,12]],"date-time":"2017-01-12T00:00:00Z","timestamp":1484179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>This study develops a model for essay scoring and article relevancy. Essay scoring is a costly process when we consider the time spent by an evaluator. It may lead to inequalities of the effort by various evaluators to apply the same evaluation criteria. Bibliometric research uses the evaluation criteria to find relevancy of articles instead. Researchers mostly face relevancy issues while searching articles. Therefore, they classify the articles manually. However, manual classification is burdensome due to time needed for evaluation. The proposed model performs automatic essay evaluation using multi-text features and ensemble machine learning. The proposed method is implemented in two data sets: a Kaggle short answer data set for essay scoring that includes four ranges of disciplines (Science, Biology, English, and English language Arts), and a bibliometric data set having IoT (Internet of Things) and non-IoT classes. The efficacy of the model is measured against the Tandalla and AutoP approach using Cohen\u2019s kappa. The model achieves kappa values of 0.80 and 0.83 for the first and second data sets, respectively. Kappa values show that the proposed model has better performance than those of earlier approaches.<\/jats:p>","DOI":"10.3390\/sym9010011","type":"journal-article","created":{"date-parts":[[2017,1,12]],"date-time":"2017-01-12T10:02:08Z","timestamp":1484215328000},"page":"11","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Prognosis Essay Scoring and Article Relevancy Using Multi-Text Features and Machine Learning"],"prefix":"10.3390","volume":"9","author":[{"given":"Arif","family":"Mehmood","sequence":"first","affiliation":[{"name":"Department of Information and Communication Engineering, Yeungnam University, Gyeongsan 38542, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Byung-Won","family":"On","sequence":"additional","affiliation":[{"name":"Department of Statistics and Computer Science, Kunsan National University, Gunsan 54150, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ingyu","family":"Lee","sequence":"additional","affiliation":[{"name":"Sorrel College of Business, Troy University, Troy, AL 36082, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0854-768X","authenticated-orcid":false,"given":"Gyu","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Yeungnam University, Gyeongsan 38542, Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,1,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1007\/s10972-013-9371-y","article-title":"Translating current science into materials for high school via a scientist\u2013teacher partnership","volume":"25","author":"Brown","year":"2014","journal-title":"J. Sci. Teach. Educ."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1080\/03098770601167922","article-title":"E-assessment by design: Using multiple-choice tests to good effect","volume":"31","author":"Nicol","year":"2007","journal-title":"J. Furth. High. Educ."},{"key":"ref_3","unstructured":"Mitchell, T., Russell, T., Broomhead, P., and Aldridge, N. (, January July). Towards Robust Computerised Marking of Free-Text Responses. Proceedings of the 6th CAA Conference, Loughborough University, London, UK."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1016\/j.compedu.2010.02.012","article-title":"A comparison of human and computer marking of short free-text student responses","volume":"55","author":"Butcher","year":"2010","journal-title":"Comput. Educ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1145\/1348246.1348248","article-title":"Image retrieval: Ideas, influences, and trends of the new age","volume":"40","author":"Datta","year":"2008","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_6","first-page":"5-3","article-title":"Recent trends in user studies: Action research and qualitative methods","volume":"5","author":"Wilson","year":"2000","journal-title":"Inf. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1007\/s11192-016-1931-4","article-title":"Proving ground for social network analysis in the emerging research area \u201cInternet of Things\u201d(IoT)","volume":"109","author":"Mehmood","year":"2016","journal-title":"Scientometrics"},{"key":"ref_8","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v036.i11","article-title":"Feature selection with the Boruta package","volume":"36","author":"Kursa","year":"2010","journal-title":"J. Stat. Softw."},{"key":"ref_10","first-page":"5","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mob. Learn."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Stat."},{"key":"ref_12","unstructured":"Tandalla, L. Scoring Short Answer Essays. Available online: https:\/\/kaggle2.blob.core.windows.net\/competitions\/kaggle\/2959\/media\/TechnicalMethodsPaper.pdf."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ramachandran, L., Cheng, J., and Foltz, P. (2015, January 04). Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, Denver, CO, USA.","DOI":"10.3115\/v1\/W15-0612"},{"key":"ref_14","unstructured":"Joachims, T.A. (,  1997). Probabilistic of the Rocchio Algorithm with TFIDF for Text Categorization. Proceedings of the 14th Internationa1 Conference on Machine Learning (ICML\u201997), Nasvile, TN, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Manning, C.D., Raghavan, P., and Schutze, H. (2008). Introduction to Information Retrieval, Cambridge University Press.","DOI":"10.1017\/CBO9780511809071"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"330","DOI":"10.1016\/j.patcog.2010.08.011","article-title":"Mining data with random forests: A survey and results of new tests","volume":"44","author":"Verikas","year":"2011","journal-title":"Pattern Recognit."},{"key":"ref_17","first-page":"477","article-title":"Boosting algorithms: Regularization, prediction and model fitting","volume":"22","author":"BUhlmann","year":"2007","journal-title":"Stat. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1080\/00220973.1994.9943835","article-title":"Computer grading of student prose, using modern concepts and software","volume":"62","author":"Page","year":"1994","journal-title":"J. Exp. Educ."},{"key":"ref_19","unstructured":"Mason, O., and Grove-Stephensen, I. (, January July). Automated Free Text Marking with Paperless School. Proceeding of the 6th Conference, Loughborough University, London, UK."},{"key":"ref_20","first-page":"939","article-title":"The intelligent essay assessor: Applications to educational technology","volume":"1","author":"Foltz","year":"1999","journal-title":"Interact. Multimedia Electron. J. Comput.-Enhanc. Learn."},{"key":"ref_21","unstructured":"Mayfield, E., and Rose, C.P. (2013). Invited Chapter in the Handbook of Automated Essay Grading, Routledge Academic Press."},{"key":"ref_22","first-page":"693","article-title":"Automatically grading essays with Markit\\copyright","volume":"1","author":"Williams","year":"2004","journal-title":"J. Issues Inf. Sci. Inf. Technol."},{"key":"ref_23","unstructured":"Attali, Y., and Burstein, J. (2006). Automated essay scoring with e-rater\u00ae V.2. J. Technol. Learn. Assess., 4, Available online: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.800.4773&rep=rep1&type=pdf."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Shermis, M.D., and Burstein, J.C. (2013). Handbook of Automated Essay Evaluation: Current Applications and New Directions, Routledge.","DOI":"10.4324\/9780203122761"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Shermis, M.D., and Burstein, J. (2013). Handbook of Automated Essay Evaluation: Current Applications and New Directions, Routledge.","DOI":"10.4324\/9780203122761"},{"key":"ref_26","unstructured":"Shermis, M.D., and Hamner, B. (2012). Annual National Council on Measurement in Education Meeting, National Council on Measurement in Education."},{"key":"ref_27","first-page":"238","article-title":"The imminence of grading essays by computer","volume":"47","author":"Page","year":"1966","journal-title":"Phi Delta Kappan"},{"key":"ref_28","unstructured":"Rudner, L.M., and Liang, T. (2002). Automated essay scoring using Bayes\u2019 theorem. J. Technol. Learn. Assess., 1, Available online: http:\/\/ejournals.bc.edu\/ojs\/index.php\/jtla\/article\/viewFile\/1668\/1512."},{"key":"ref_29","unstructured":"Christie, J.R. (, January June). Automated essay marking-for both style and content. Proceedings of the Third Annual Computer Assisted Assessment Conference, Loughborough University, Loughborough, UK."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Thomas, P., Haley, D., DeRoeck, A., and Petre, M. (2004, January 28). E-assessment using latent semantic analysis in the computer science domain: A pilot study. Proceedings of the Workshop on eLearning for Computational Linguistics and Computational Linguistics for eLearning, Geneva, Switzerland.","DOI":"10.3115\/1610028.1610034"},{"key":"ref_31","unstructured":"Islam, M.M., and Hoque, A.L. (2010, January 23\u201325). Automated essay scoring using generalized latent semantic analysis. Proceedings of the 2010 13th International Conference on Computer and Information Technology (ICCIT), Dhaka, Bangladesh."},{"key":"ref_32","first-page":"3","article-title":"The state of educational data mining in 2009: A review and future visions","volume":"1","author":"Baker","year":"2009","journal-title":"JEDM"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Romero, C., Ventura, S., Pechenizkiy, M., and Baker, R.S. (2010). Handbook of Educational Data Mining, CRC Press.","DOI":"10.1201\/b10274"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1432","DOI":"10.1016\/j.eswa.2013.08.042","article-title":"Educational data mining: A survey and a data mining-based analysis of recent works","volume":"41","year":"2014","journal-title":"Exp. Syst. Appl."},{"key":"ref_35","unstructured":"Kim, J., Shaw, E., Xu, H., and Adarsh, G.V. (2012, January June). Assisting Instructional Assessment of Undergraduate Collaborative Wiki and SVN Activities. Proceedings of the 5th International Conference on Educational Data Mining (EDM), Chania, Greece."},{"key":"ref_36","unstructured":"Lopez, M.I., Luna, J.M., Romero, C., and Ventura, S. (2012, January June). Classification via Clustering for Predicting Final Marks Based on Student Participation in Forums. Proceedings of the 5th International Conference on Educational Data Mining (EDM), Chania, Greece."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"933","DOI":"10.1007\/s11251-012-9262-1","article-title":"Patterns in elementary school students\u2032 strategic actions in varying learning situations","volume":"41","author":"Malmberg","year":"2013","journal-title":"Instr. Sci."},{"key":"ref_38","first-page":"111","article-title":"Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds","volume":"4","author":"Gobert","year":"2012","journal-title":"JEDM"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.compedu.2014.03.001","article-title":"A system for formative assessment and monitoring of students\u2019 progress","volume":"76","author":"Rodrigues","year":"2014","journal-title":"Comput. Educ."},{"key":"ref_40","first-page":"383","article-title":"Advances in the Field of Automated Essay Evaluation","volume":"39","author":"Zupanc","year":"2015","journal-title":"Informatica"},{"key":"ref_41","unstructured":"ASAP-SAS Scoring Short Answer Essays. ASAP Short Answer Scoring Competition System Description, 2012. Available online: http:\/\/www.kaggle.com\/c\/asap-sas\/."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.knosys.2012.06.005","article-title":"A novel probabilistic feature selection method for text classification","volume":"36","author":"Uysal","year":"2012","journal-title":"Knowl.-Based Syst."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1016\/j.ipm.2013.08.006","article-title":"The impact of preprocessing on text classification","volume":"50","author":"Uysal","year":"2014","journal-title":"Inf. Process. Manag."},{"key":"ref_44","unstructured":"Norvig, P. How to Write a Spelling Corrector. Available online: http:\/\/norvig.com\/spell-correct.html."},{"key":"ref_45","unstructured":"Porter, M. Porter Stemming Algorithm. Available online: http:\/\/tartarus.org\/martin\/PorterStemmer\/."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1016\/j.jbi.2007.11.004","article-title":"Evaluation of preprocessing techniques for chief complaint classification","volume":"41","author":"Dara","year":"2008","journal-title":"J. Biomed. Inf."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1016\/j.engappai.2012.08.002","article-title":"An automatic approach for ontology-based feature extraction from heterogeneous textual resources","volume":"26","author":"Vicient","year":"2013","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1016\/j.knosys.2014.10.013","article-title":"Novel feature selection method based on harmony search for email classification","volume":"73","author":"Wang","year":"2015","journal-title":"Knowl.-Based Syst."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1016\/j.datak.2011.05.002","article-title":"Classifying text streams by keywords using classifier ensemble","volume":"70","author":"Yang","year":"2011","journal-title":"Data Knowl. Eng."},{"key":"ref_50","first-page":"412","article-title":"A comparative study on feature selection in text categorization","volume":"97","author":"Yang","year":"1997","journal-title":"ICML"},{"key":"ref_51","first-page":"589","article-title":"Consistent feature selection for pattern recognition in polynomial time","volume":"8","author":"Nilsson","year":"2007","journal-title":"J. Mach. Learn. Res."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Pandey, U., and Chakravarty, S. (2010, January 9\u201311). A survey on text classification techniques for e-mail filtering. Proceedings of the 2010 Second International Conference on Machine Learning and Computing (ICMLC), Banglore, India.","DOI":"10.1109\/ICMLC.2010.61"},{"key":"ref_53","unstructured":"Deng, X.B., Ye, Y.M., Li, H.B., and Huang, J.Z. (2008, January 12\u201315). An improved random forest approach for detection of hidden web search interfaces. Proceedings of the 2008 International Conference on Machine Learning and Cybernetics, Kunming, China."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Bissacco, A., Yang, M.H., and Soatto, S. (2007, January 17\u201322). Fast human pose estimation using appearance and motion via multi-dimensional boosting regression. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.","DOI":"10.1109\/CVPR.2007.383129"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1609\/aaai.v25i1.7801","article-title":"Incorporating Boosted Regression Trees into Ecological Latent Variable Models","volume":"11","author":"Hutchinson","year":"2011","journal-title":"AAAI"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Pittman, S.J., and Brown, K.A. (2011). Multi-scale approach for predicting fish species distributions across coral reef seascapes. PLoS ONE, 6.","DOI":"10.1371\/journal.pone.0020583"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"942","DOI":"10.1109\/TPAMI.2013.159","article-title":"Learning nonlinear functions using regularized greedy forest","volume":"36","author":"Johnson","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Shermis, M.D., and Burstein, J. (2013). Handbook of Automated Essay Evaluation: Current Applications and New Directions, Routledge.","DOI":"10.4324\/9780203122761"},{"key":"ref_59","unstructured":"Guthrie, D., Allison, B., Liu, W., Guthrie, L., and Wilks, Y. (2006, January 24\u201326). A closer look at skip-gram modelling. Proceedings of the 5th international Conference on Language Resources and Evaluation (LREC-2006), Genoa, Italy."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/9\/1\/11\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:26:03Z","timestamp":1760207163000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/9\/1\/11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,1,12]]},"references-count":59,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,1]]}},"alternative-id":["sym9010011"],"URL":"https:\/\/doi.org\/10.3390\/sym9010011","relation":{},"ISSN":["2073-8994"],"issn-type":[{"type":"electronic","value":"2073-8994"}],"subject":[],"published":{"date-parts":[[2017,1,12]]}}}