{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T15:54:50Z","timestamp":1764863690337,"version":"3.46.0"},"reference-count":52,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T00:00:00Z","timestamp":1764806400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Research Ireland (Science Foundation Ireland","award":["16\/IA\/4605"],"award-info":[{"award-number":["16\/IA\/4605"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["BDCC"],"abstract":"<jats:p>In the context of remanufacturing, particularly mobile device refurbishing, effective operator training is crucial for accurate defect identification and process inspection efficiency. This study examines the application of Natural Language Processing (NLP) techniques to evaluate operator expertise based on subjective textual responses gathered during a defect analysis task. Operators were asked to describe screen defects using open-ended questions, and their responses were compared with expert responses to evaluate their accuracy and consistency. We employed four NLP models, including finetuned Sentence-BERT (SBERT), pre-trained SBERT, Word2Vec, and Dice similarity, to determine their effectiveness in interpreting short, domain-specific text. A novel disagreement tagging framework was introduced to supplement traditional similarity metrics with explainable insights. This framework identifies the root causes of model\u2013human misalignment across four categories: defect type, severity, terminology, and location. Results show that a finetuned SBERT model significantly outperforms other models by achieving Pearsons\u2019s correlation of 0.93 with MAE and RMSE scores of 0.07 and 0.12, respectively, providing more accurate and context-aware evaluations. In contrast, other models exhibit limitations in semantic understanding and consistency. The results highlight the importance of finetuning NLP models for domain-specific applications and demonstrate how qualitative tagging methods can enhance interpretability and model debugging. This combined approach indicates a scalable and transparent methodology for the evaluation of operator responses, supporting the development of more effective training programmes in industrial settings where remanufacturing and sustainability generally are a key performance metric.<\/jats:p>","DOI":"10.3390\/bdcc9120312","type":"journal-article","created":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T15:14:06Z","timestamp":1764861246000},"page":"312","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Subjective Evaluation of Operator Responses for Mobile Defect Identification in Remanufacturing: Application of NLP and Disagreement Tagging"],"prefix":"10.3390","volume":"9","author":[{"given":"Abbirah","family":"Ahmed","sequence":"first","affiliation":[{"name":"Department of Electronic and Computer Engineering, University of Limerick, V94 TP9X Limerick, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0509-0051","authenticated-orcid":false,"given":"Reenu","family":"Mohandas","sequence":"additional","affiliation":[{"name":"Department of Electronic and Computer Engineering, University of Limerick, V94 TP9X Limerick, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0767-4302","authenticated-orcid":false,"given":"Arash","family":"Joorabchi","sequence":"additional","affiliation":[{"name":"Department of Electronic and Computer Engineering, University of Limerick, V94 TP9X Limerick, Ireland"}]},{"given":"Martin J.","family":"Hayes","sequence":"additional","affiliation":[{"name":"Department of Electronic and Computer Engineering, University of Limerick, V94 TP9X Limerick, Ireland"}]}],"member":"1968","published-online":{"date-parts":[[2025,12,4]]},"reference":[{"key":"ref_1","unstructured":"Ahmed, A., Mohandas, R., Joorabchi, A., and Hayes, M. (2025, January 9\u201310). Subjective Assessment of Operator Responses for Mobile Defect Identification in Remanufacturing. Proceedings of the Irish Signals and Systems Conference, Letterkenny, Ireland."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"105398","DOI":"10.1016\/j.resconrec.2021.105398","article-title":"Promoting consumer\u2019s attitude toward refurbished mobile phones: A social media analytics approach","volume":"167","author":"Sharifi","year":"2021","journal-title":"Resour. Conserv. Recycl."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Dhiman, D., Jaiswal, A., Sail, K.S., and Wavhale, A. (2024, January 21\u201323). Unlocking Access: The Role of Refurbished Mobile Phones in Digital Inclusion and Sustainable Development. Proceedings of the 2024 ITU Kaleidoscope: Innovation and Digital Transformation for a Sustainable World (ITU K), New Delhi, India.","DOI":"10.23919\/ITUK62727.2024.10772979"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"12","DOI":"10.37934\/sijap.1.1.1224","article-title":"Sustainable Tech: Exploring Consumer Valuation and Investment in Refurbished Smartphones","volume":"1","author":"Kamil","year":"2024","journal-title":"Semarak Int. J. Appl. Psychol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1605","DOI":"10.1007\/s00170-014-5871-y","article-title":"Defect inspection and extraction of the mobile phone cover glass based on the principal components analysis","volume":"73","author":"Li","year":"2014","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1007\/s10579-018-9431-1","article-title":"MedSTS: A resource for clinical semantic textual similarity","volume":"54","author":"Wang","year":"2020","journal-title":"Lang. Resour. Eval."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"102755","DOI":"10.1016\/j.inffus.2024.102755","article-title":"Natural language processing in finance: A survey","volume":"115","author":"Du","year":"2025","journal-title":"Inf. Fusion"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"61642","DOI":"10.1109\/ACCESS.2020.2983584","article-title":"Identifying banking transaction descriptions via support vector machine short-text classification based on a specialized labelled corpus","volume":"8","author":"Seara","year":"2020","journal-title":"IEEE Access"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"4295","DOI":"10.1007\/s10115-023-01892-9","article-title":"GradeAid: A framework for automatic short answers grading in educational contexts\u2014Design, implementation and evaluation","volume":"65","author":"Guarino","year":"2023","journal-title":"Knowl. Inf. Syst."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ahmed, A., Joorabchi, A., and Hayes, M.J. (2022, January 22\u201324). On Deep Learning Approaches to Automated Assessment: Strategies for Short Answer Grading. Proceedings of the CSEDU (2), Virtual.","DOI":"10.5220\/0011082100003182"},{"key":"ref_11","first-page":"115","article-title":"Short answer grading using string similarity and corpus-based similarity","volume":"3","author":"Gomaa","year":"2012","journal-title":"Int. J. Adv. Comput. Sci. Appl. (IJACSA)"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"4205","DOI":"10.1080\/00207543.2020.1759838","article-title":"Manufacturing process similarity measurement model and application based on process constituent elements","volume":"59","author":"Wu","year":"2021","journal-title":"Int. J. Prod. Res."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Ahn, H., and Chang, T.-W. (2018, January 8\u201311). Measuring Similarity for Manufacturing Process Models. Proceedings of the IFIP International Conference on Advances in Production Management Systems, Chemnitz, Germany.","DOI":"10.1007\/978-3-319-99707-0_28"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Akhbardeh, F., Desell, T., and Zampieri, M. (2020, January 4\u20137). NLP tools for predictive maintenance records in MaintNet. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, Suzhou, China.","DOI":"10.18653\/v1\/2020.aacl-demo.5"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.54569\/aair.1142568","article-title":"Semantic similarity comparison between production line failures for predictive maintenance","volume":"3","author":"Omurca","year":"2023","journal-title":"Adv. Artif. Intell. Res."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s13243-024-00147-2","article-title":"Deep learning enabled computer vision in remanufacturing and refurbishment applications: Defect detection and grading for smart phones","volume":"15","author":"Mohandas","year":"2025","journal-title":"J. Remanufacturing"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1007\/s00170-021-06592-8","article-title":"Recent advances in surface defect inspection of industrial products using deep learning techniques","volume":"113","author":"Zheng","year":"2021","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Saberironaghi, A., Ren, J., and El-Gindy, M. (2023). Defect detection methods for industrial products using deep learning techniques: A review. Algorithms, 16.","DOI":"10.3390\/a16020095"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"040801","DOI":"10.1115\/1.4049535","article-title":"Image-based surface defect detection using deep learning: A review","volume":"21","author":"Bhatt","year":"2021","journal-title":"J. Comput. Inf. Sci. Eng."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Tabernik, D., \u0160ela, S., Skvar\u010d, J., and Sko\u010daj, D. (2019, January September). Deep-learning-based computer vision system for surface-defect detection. Proceedings of the Computer Vision Systems: 12th International Conference, ICVS 2019, Thessaloniki, Greece.","DOI":"10.1007\/978-3-030-34995-0_44"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1109\/TII.2019.2945403","article-title":"Deep-learning-based small surface defect detection via an exaggerated local variation-based generative adversarial network","volume":"16","author":"Lian","year":"2019","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Chen, Y., Ding, Y., Zhao, F., Zhang, E., Wu, Z., and Shao, L. (2021). Surface defect detection methods for industrial products: A review. Appl. Sci., 11.","DOI":"10.3390\/app11167657"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"43370","DOI":"10.1109\/ACCESS.2023.3271748","article-title":"Deep learning for automatic vision-based recognition of industrial surface defects: A survey","volume":"11","author":"Prunella","year":"2023","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Bhattacharya, M., Penica, M., O\u2019Connell, E., Southern, M., and Hayes, M. (2023). Human-in-loop: A review of smart manufacturing deployments. Systems, 11.","DOI":"10.3390\/systems11010035"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"112614","DOI":"10.1016\/j.measurement.2023.112614","article-title":"Defect transformer: An efficient hybrid transformer architecture for surface defect detection","volume":"211","author":"Wang","year":"2023","journal-title":"Measurement"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3126","DOI":"10.1109\/TUFFC.2021.3081750","article-title":"Automated defect detection from ultrasonic images using deep learning","volume":"68","author":"Medak","year":"2021","journal-title":"IEEE Trans. Ultrason. Ferroelectr. Freq. Control."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"103834","DOI":"10.1016\/j.compind.2022.103834","article-title":"MaMiNet: Memory-attended multi-inference network for surface-defect detection","volume":"145","author":"Luo","year":"2023","journal-title":"Comput. Ind."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1995","DOI":"10.1007\/s10845-021-01878-w","article-title":"Automated surface defect detection framework using machine vision and convolutional neural networks","volume":"34","author":"Singh","year":"2023","journal-title":"J. Intell. Manuf."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2135","DOI":"10.1109\/TII.2014.2359416","article-title":"Saliency-based defect detection in industrial images by using phase spectrum","volume":"10","author":"Bai","year":"2014","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"14531","DOI":"10.1109\/ACCESS.2020.2966881","article-title":"Review of pavement defect detection methods","volume":"8","author":"Cao","year":"2020","journal-title":"IEEE Access"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s13243-020-00093-9","article-title":"Achieving remanufacturing inspection using deep learning","volume":"11","author":"Nwankpa","year":"2021","journal-title":"J. Remanufacturing"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1016\/j.procir.2022.02.062","article-title":"A concept for autonomous quality control for core inspection in remanufacturing","volume":"105","author":"Kaiser","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Saiz, F.A., Alfaro, G., and Barandiaran, I. (2021). An inspection and classification system for automotive component remanufacturing industry based on ensemble learning. Information, 12.","DOI":"10.3390\/info12120489"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zizic, M.C., Mladineo, M., Gjeldum, N., and Celent, L. (2022). From industry 4.0 towards industry 5.0: A review and analysis of paradigm shift for the people, organization and technology. Energies, 15.","DOI":"10.3390\/en15145221"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2082","DOI":"10.1016\/j.procs.2024.02.029","article-title":"Natural language processing approaches in industrial maintenance: A systematic literature review","volume":"232","author":"Zhong","year":"2024","journal-title":"Procedia Comput. Sci."},{"key":"ref_36","unstructured":"Bernabei, M., Colabianchi, S., and Costantino, F. (2022, January 7\u20139). Natural language processing applications in manufacturing: A systematic literature review. Proceedings of the Summer School Francesco Turco, Proceedings, Riviera dei Fiori, Italy."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Besharati Moghaddam, F., Lopez, A.J., De Vuyst, S., and Gautama, S. (2024). Natural Language Processing in Knowledge-Based Support for Operator Assistance. Appl. Sci., 14.","DOI":"10.3390\/app14072766"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1881","DOI":"10.1109\/TCPMT.2024.3453597","article-title":"Enhancing Defect Detection in Circuit Board Assembly Using AI and Text Analytics for Component Failure Classification","volume":"14","author":"Sheikh","year":"2024","journal-title":"IEEE Trans. Compon. Packag. Manuf. Technol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.procir.2022.10.071","article-title":"Applying natural language processing in manufacturing","volume":"115","author":"May","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Kobayashi, D., Yasuda, S., Iuti, T., and Ito, S. (2022, January 12\u201313). Application of natural language processing in semiconductor manufacturing. Proceedings of the 2022 International Symposium on Semiconductor Manufacturing (ISSM), Tokyo, Japan.","DOI":"10.1109\/ISSM55802.2022.10026893"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1016\/j.cirp.2021.04.045","article-title":"Text mining for AI enhanced failure detection and availability optimization in production systems","volume":"70","author":"Ansari","year":"2021","journal-title":"CIRP Ann."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"4548","DOI":"10.1080\/00207543.2021.1951868","article-title":"Using deep learning to value free-form text data for predictive maintenance","volume":"60","author":"Lamouri","year":"2022","journal-title":"Int. J. Prod. Res."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1551","DOI":"10.1016\/j.procir.2022.05.190","article-title":"Analysis and relevance of service reports to extend predictive maintenance of large-scale plants","volume":"107","author":"Solak","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_44","unstructured":"Goldberg, Y., and Levy, O. (2014). word2vec Explained: Deriving Mikolov et al.\u2019s negative-sampling word-embedding method. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2018). Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. arXiv.","DOI":"10.18653\/v1\/D17-1070"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Neculoiu, P., Versteegh, M., and Rotaru, M. (2016, January 11). Learning text similarity with siamese recurrent networks. Proceedings of the 1st Workshop on Representation Learning for NLP, Berlin, Germany.","DOI":"10.18653\/v1\/W16-1617"},{"key":"ref_47","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks, Association for Computational Linguistics.","DOI":"10.18653\/v1\/D19-1410"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Ahmed, A., Joorabchi, A., and Hayes, M.J. (2022, January 9\u201310). On the Application of Sentence Transformers to Automatic Short Answer Grading in Blended Assessment. Proceedings of the 2022 33rd Irish Signals and Systems Conference (ISSC), Cork, Ireland.","DOI":"10.1109\/ISSC55427.2022.9826194"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"297","DOI":"10.2307\/1932409","article-title":"Measures of the amount of ecologic association between species","volume":"26","author":"Dice","year":"1945","journal-title":"Ecology"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Mohler, M., and Mihalcea, R. (April, January 30). Text-to-text semantic similarity for automatic short answer grading. Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), Athens, Greece.","DOI":"10.3115\/1609067.1609130"},{"key":"ref_52","unstructured":"Mohler, M., Bunescu, R., and Mihalcea, R. (2011, January 19\u201324). Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA."}],"container-title":["Big Data and Cognitive Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/12\/312\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,4]],"date-time":"2025-12-04T15:49:54Z","timestamp":1764863394000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-2289\/9\/12\/312"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,4]]},"references-count":52,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["bdcc9120312"],"URL":"https:\/\/doi.org\/10.3390\/bdcc9120312","relation":{},"ISSN":["2504-2289"],"issn-type":[{"value":"2504-2289","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,4]]}}}