{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T22:13:24Z","timestamp":1792534404264,"version":"4.5.0"},"reference-count":59,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2024,6,30]],"date-time":"2024-06-30T00:00:00Z","timestamp":1719705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Transfer learning has gained significant traction in natural language processing due to the emergence of state-of-the-art pre-trained language models (PLMs). Unlike traditional word embedding methods such as TF-IDF and Word2Vec, PLMs are context-dependent and outperform conventional techniques when fine-tuned for specific tasks. This paper proposes an innovative hard voting classifier to enhance crash severity classification by combining machine learning and deep learning models with various word embedding techniques, including BERT, RoBERTa, Word2Vec, and TF-IDF. Our study involves two comprehensive experiments using motorists\u2019 crash data from the Missouri State Highway Patrol. The first experiment evaluates the performance of three machine learning models\u2014XGBoost (XGB), random forest (RF), and naive Bayes (NB)\u2014paired with TF-IDF, Word2Vec, and BERT feature extraction techniques. Additionally, BERT and RoBERTa are fine-tuned with a Bidirectional Long Short-Term Memory (Bi-LSTM) classification model. All models are initially evaluated on the original dataset. The second experiment repeats the evaluation using an augmented dataset to address the severe data imbalance. The results from the original dataset show strong performance for all models in the \u201cFatal\u201d and \u201cPersonal Injury\u201d classes but a poor classification of the minority \u201cProperty Damage\u201d class. In the augmented dataset, while the models continued to excel with the majority classes, only XGB\/TFIDF and BERT-LSTM showed improved performance for the minority class. The ensemble model outperformed individual models in both datasets, achieving an F1 score of 99% for \u201cFatal\u201d and \u201cPersonal Injury\u201d and 62% for \u201cProperty Damage\u201d on the augmented dataset. These findings suggest that ensemble models, combined with data augmentation, are highly effective for crash severity classification and potentially other textual classification tasks.<\/jats:p>","DOI":"10.3390\/a17070284","type":"journal-article","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T08:17:29Z","timestamp":1719821849000},"page":"284","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Ensemble Learning with Pre-Trained Transformers for Crash Severity Classification: A Deep NLP Approach"],"prefix":"10.3390","volume":"17","author":[{"given":"Shadi","family":"Jaradat","sequence":"first","affiliation":[{"name":"Centre for Accident Research & Road Safety, Queensland University of Technology, Brisbane, QLD 4000, Australia"},{"name":"Centre of Data Science, Queensland University of Technology, Brisbane, QLD 4000, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9954-0159","authenticated-orcid":false,"given":"Richi","family":"Nayak","sequence":"additional","affiliation":[{"name":"Centre of Data Science, Queensland University of Technology, Brisbane, QLD 4000, Australia"},{"name":"School of Computer Science, Queensland University of Technology, Brisbane, QLD 4000, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1217-9808","authenticated-orcid":false,"given":"Alexander","family":"Paz","sequence":"additional","affiliation":[{"name":"School of Civil Engineering, Queensland University of Technology, Brisbane, QLD 4000, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2634-4576","authenticated-orcid":false,"given":"Mohammed","family":"Elhenawy","sequence":"additional","affiliation":[{"name":"Centre for Accident Research & Road Safety, Queensland University of Technology, Brisbane, QLD 4000, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,6,30]]},"reference":[{"key":"ref_1","unstructured":"Oestergaard, F., Beck Kinman, S., and Ravn Pedersen, S. (2013). Control your data or drown trying. I.B.M. Nordic Blog, Available online: https:\/\/www.ibm.com\/blogs\/nordic-msp\/control-your-data-or-drown-trying\/."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"19","DOI":"10.21248\/jlcl.20.2005.68","article-title":"A brief survey of text mining","volume":"20","author":"Hotho","year":"2005","journal-title":"J. Lang. Technol. Comput. Linguist."},{"key":"ref_3","unstructured":"Ramos, J. (2003, January 23\u201324). Using TF-IDF to determine word relevance in document queries. Proceedings of the First Instructional Conference On Machine Learning, Los Angeles, CA, USA."},{"key":"ref_4","unstructured":"Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013, January 5\u201310). Distributed representations of words and phrases and their compositionality. Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_5","unstructured":"Devlin, J. (2019). BERT: Pre-training of deep bidirectional transformers for understanding. arXiv."},{"key":"ref_6","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv."},{"key":"ref_7","unstructured":"He, P., Liu, X., Gao, J., and Chen, W. (2021). DeBERTa: Decoding-Enhanced BERT with disentangled attention. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1186\/s40537-016-0043-6","article-title":"A survey of transfer learning","volume":"3","author":"Weiss","year":"2016","journal-title":"J. Big Data"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"14760","DOI":"10.1016\/j.eswa.2011.05.004","article-title":"A multiobjective simulated annealing approach for classifier ensemble: Named entity recognition in Indian languages as case studies","volume":"38","author":"Ekbal","year":"2011","journal-title":"Expert Syst. Appl."},{"key":"ref_10","unstructured":"Gundapu, S., and Mamidi, R. (2021). Transformer based automatic COVID-19 fake news detection system. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3439726","article-title":"Deep learning\u2013based text classification","volume":"54","author":"Minaee","year":"2022","journal-title":"ACM Comput. Surv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Rustam, F., Ashraf, I., Mehmood, A., Ullah, S., and Choi, G.S. (2019). Tweets classification on the base of sentiments for U.S. airline companies. Entropy, 21.","DOI":"10.3390\/e21111078"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3498123","DOI":"10.1155\/2022\/3498123","article-title":"A fine-tuned BERT-based transfer learning approach for text classification","volume":"2022","author":"Qasim","year":"2022","journal-title":"J. Healthc. Eng."},{"key":"ref_14","unstructured":"Sanh, L.D.V., Chaumond, J., and Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster; cheaper; lighter. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1093\/tse\/tdaa010","article-title":"Application of machine learning tools in classifying pedestrian crash types: A case study","volume":"2","author":"Das","year":"2020","journal-title":"Transp. Saf. Environ."},{"key":"ref_16","first-page":"54","article-title":"The seven tools of causal inference, with reflections on machine learning","volume":"62","author":"Pearl","year":"2019","journal-title":"Commun. A.C.M."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1080\/03081069008717439","article-title":"Estimating truck accident rate and involvements using linear and Poisson regression models","volume":"15","author":"Joshua","year":"1990","journal-title":"Transp. Plan. Technol."},{"key":"ref_18","unstructured":"Fan, W.D., Gong, L., Washing, E.M., Yu, M., and Haile, E. (2016, January 10\u201314). Identifying and quantifying factors affecting vehicle crash severity at highway-rail grade crossings: Models and their comparison. Proceedings of the Transportation Research Board 95th Annual Meeting, Washington, DC, USA."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1016\/j.aap.2014.10.022","article-title":"Motor vehicle drivers\u2019 injuries in train\u2013motor vehicle crashes","volume":"74","author":"Zhao","year":"2015","journal-title":"Accid. Anal. Prev."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1016\/0001-4575(94)90013-2","article-title":"Risk factors of fatality in motor vehicle traffic accidents","volume":"26","author":"Shibata","year":"1994","journal-title":"Accid. Anal. Prev."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1016\/j.aap.2009.07.013","article-title":"Empirical assessment of the impact of highway design exceptions on the frequency and severity of vehicle accidents","volume":"42","author":"Malyshkina","year":"2010","journal-title":"Accid. Anal. Prev."},{"key":"ref_22","first-page":"72","article-title":"Comparing three commonly used crash severity models on sample size requirements: Multinomial logit, ordered probit and mixed logit models","volume":"1","author":"Ye","year":"2014","journal-title":"Anal. Methods Accid. Res."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1016\/S0001-4575(01)00073-2","article-title":"Using logistic regression to estimate the influence of accident factors on accident severity","volume":"34","year":"2002","journal-title":"Accid. Anal. Prev."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1016\/j.aap.2007.11.010","article-title":"A mixed generalized ordered response model for examining pedestrian and bicyclist injury severity level in traffic crashes","volume":"40","author":"Eluru","year":"2008","journal-title":"Accid. Anal. Prev."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"640","DOI":"10.1177\/0361198119844456","article-title":"Prediction and factor identification for crash severity: Comparison of discrete choice and tree-based models","volume":"2673","author":"Wang","year":"2019","journal-title":"Transp. Res. Rec."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.ijtst.2020.01.001","article-title":"Factors affecting injury severity in vehicle-pedestrian crashes: A day-of-week analysis using random parameter ordered response models and Artificial Neural Networks","volume":"9","author":"Mokhtarimousavi","year":"2020","journal-title":"Int. J. Transp. Sci. Technol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"104492","DOI":"10.1016\/j.ssci.2019.09.015","article-title":"Applications of machine learning methods for engineering risk assessment\u2014A review","volume":"122","author":"Hegde","year":"2020","journal-title":"Saf. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"e100262","DOI":"10.1136\/bmjhci-2020-100262","article-title":"Applying natural language processing and machine learning techniques to patient experience feedback: A systematic review","volume":"28","author":"Khanbhai","year":"2021","journal-title":"B.M.J. Health Care Inform."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"478","DOI":"10.1016\/j.aap.2011.08.016","article-title":"Using support vector machine models for crash injury severity analysis","volume":"45","author":"Li","year":"2012","journal-title":"Accid. Anal. Prev."},{"key":"ref_30","first-page":"89","article-title":"Traffic accident analysis using machine learning paradigms","volume":"29","author":"Chong","year":"2005","journal-title":"Informatica"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.ijtst.2019.10.002","article-title":"Analyzing injury severity of motorcycle at-fault crashes using machine learning techniques, decision tree and logistic regression models","volume":"9","author":"Rezapour","year":"2020","journal-title":"Int. J. Transp. Sci. Technol."},{"key":"ref_32","first-page":"273","article-title":"Modelling pedestrian crash severity at mid-blocks","volume":"13","author":"Pour","year":"2017","journal-title":"Transp. A Transp. Sci."},{"key":"ref_33","first-page":"432","article-title":"Modern data sources and techniques for analysis and forecast of road accidents: A review","volume":"7","author":"Pedraza","year":"2020","journal-title":"J. Traffic Transp. Eng."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1080\/17457300.2016.1224902","article-title":"Clustering-based classification of road traffic accidents using hierarchical clustering and artificial neural networks","volume":"24","author":"Taamneh","year":"2017","journal-title":"Int. J. Inj. Control. Saf. Promot."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"104988","DOI":"10.1016\/j.ssci.2020.104988","article-title":"Injury severity on traffic crashes: A text mining with an interpretable machine-learning approach","volume":"132","author":"Arteaga","year":"2020","journal-title":"Saf. Sci."},{"key":"ref_36","first-page":"100176","article-title":"Vehicle involvements in hydroplaning crashes: Applying interpretable machine learning","volume":"6","author":"Das","year":"2020","journal-title":"Transp. Res. Interdiscip. Perspect."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Nayak, R., Piyatrapoomi, N., and Weligamage, J. (2009, January 28\u201330). Application of text mining in analysing road crashes for road asset management. Proceedings of the Engineering Asset Lifecycle Management, Athens, Greece.","DOI":"10.1007\/978-0-85729-320-6_7"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/j.ins.2018.09.001","article-title":"An analysis of hierarchical text classification using word embeddings","volume":"471","author":"Stein","year":"2019","journal-title":"Inf. Sci."},{"key":"ref_39","unstructured":"Gao, L. (2013, January 13\u201317). verb-based text mining of road crash report. Proceedings of the 92nd Annual Meeting of the Transportation Research Board, Washington, DC, USA."},{"key":"ref_40","first-page":"1338","article-title":"Identifying secondary crashes using text mining techniques","volume":"12","author":"Zhang","year":"2019","journal-title":"J. Transp. Saf. Secur."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"105835","DOI":"10.1016\/j.aap.2020.105835","article-title":"Injury severity analysis of pedestrian and bicyclist trespassing crashes at non-crossings: A hybrid predictive text analytics and heterogeneity-based statistical modeling approach","volume":"150","author":"Wali","year":"2021","journal-title":"Accid. Anal. Prev."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"106211","DOI":"10.1016\/j.aap.2021.106211","article-title":"Identification and analysis of misclassified work-zone crashes using text mining techniques","volume":"159","author":"Sayed","year":"2021","journal-title":"Accid. Anal. Prev."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Ganguli, R., Miller, P., and Pothina, R. (2021). Effectiveness of natural language processing based machine learning in analyzing incident narratives at a mine. Minerals, 11.","DOI":"10.3390\/min11070776"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1016\/j.aap.2019.07.021","article-title":"Topic analysis of Road Safety Inspections using latent Dirichlet allocation: A case study of roadside safety in Irish main roads","volume":"131","author":"Roque","year":"2019","journal-title":"Accid. Anal. Prev."},{"key":"ref_45","unstructured":"Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., and Amodei, D. (2020, January 6\u201312). Language models are few-shot learners. Proceedings of the Advances in Neural Information Processing Systems 33 (NeurIPS 2020), NeurIPS, Vancouver, BC, Canada."},{"key":"ref_46","unstructured":"Parmar, V.S., Jones, U., Gomez, A.N., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_47","unstructured":"Ho, T.K. (1995, January 14\u201316). Random dDecision fForests. Proceedings of the 3rd International Conference on Document Analysis and Recognition, Montreal, QC, Canada."},{"key":"ref_48","unstructured":"Rish, I. (2001, January 4). An empirical study of the naive Bayes classifier. Proceedings of the Workshop on Empirical Methods in Artificial Intelligence, Seattle, WA, USA."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Khusna, W., and Murfi, H. (2020, January 7). An analysis of the proportion of feature subsampling on XGBoost\u2014A case study of claim prediction in car insurance. Proceedings of the International Conference on Science and Applied Science (ICSAS2020), Surakarta, Indonesia.","DOI":"10.1063\/5.0031366"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.dss.2017.10.001","article-title":"Decision support from financial disclosures with deep neural networks and transfer learning","volume":"104","author":"Kraus","year":"2017","journal-title":"Decis. Support Syst."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1016\/j.iatssr.2021.01.001","article-title":"Applying interpretable machine learning to classify tree and utility pole related crash injury types","volume":"45","author":"Das","year":"2021","journal-title":"IATSS Res."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Davis, J., and Goadrich, M. (2006, January 25\u201329). The relationship between Precision-Recall and R.O.C. curves. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.","DOI":"10.1145\/1143844.1143874"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Saito, T., and Rehmsmeier, M. (2015). The Precision-Recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0118432"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/MCAS.2006.1688199","article-title":"Ensemble based systems in decision making","volume":"6","author":"Polikar","year":"2006","journal-title":"IEEE Circuits Syst. Mag."},{"key":"ref_55","unstructured":"Bishop, C.M., and Nasrabadi, N.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Kuncheva, L.I. (2014). Combining Pattern Classifiers: Methods and Algorithms, John Wiley & Sons.","DOI":"10.1002\/9781118914564"},{"key":"ref_57","first-page":"4","article-title":"A review of machine learning algorithms for text-documents classification","volume":"1","author":"Khan","year":"2010","journal-title":"J. Adv. Inf. Technol."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1080\/01431160600746456","article-title":"A survey of image classification methods and techniques for improving classification performance","volume":"28","author":"Lu","year":"2007","journal-title":"Int. J. Remote Sens."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Lin, H., Yang, Z., Wang, J., Sun, Y., Xu, B., and Zhao, Z. (2019). Neural network-based approaches for biomedical relation classification: A review. J. Biomed. Inform., 99.","DOI":"10.1016\/j.jbi.2019.103294"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/7\/284\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:08:16Z","timestamp":1760108896000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/7\/284"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,30]]},"references-count":59,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["a17070284"],"URL":"https:\/\/doi.org\/10.3390\/a17070284","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,30]]}}}