{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,14]],"date-time":"2026-07-14T15:03:11Z","timestamp":1784041391531,"version":"3.55.0"},"reference-count":30,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2025,9,14]],"date-time":"2025-09-14T00:00:00Z","timestamp":1757808000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Aeronautics and Space Administration (NASA)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Semantic similarity, the task of determining whether two sentences convey the same meaning, is central to applications such as paraphrase detection, semantic search, and question answering. Despite the widespread adoption of transformer-based models for this task, their performance is influenced by both the choice of similarity measure and BERT (bert-base-nli-mean-tokens), RoBERTa (all-roberta-large-v1), and MPNet (all-mpnet-base-v2) on the Microsoft Research Paraphrase Corpus (MRPC). Sentence embeddings were compared using cosine similarity, dot product, Manhattan distance, and Euclidean distance, with thresholds optimized for accuracy, balanced accuracy, and F1-score. Results indicate a consistent advantage for MPNet, which achieved the highest accuracy (75.6%), balanced accuracy (71.0%), and F1-score (0.836) when paired with cosine similarity at an optimized threshold of 0.671. BERT and RoBERTa performed competitively but exhibited greater sensitivity to the choice of Similarity metric, with BERT notably underperforming when using cosine similarity compared to Manhattan or Euclidean distance. Optimal thresholds varied widely (0.334\u20130.867), underscoring the difficulty of establishing a single, generalizable cut-off for paraphrase classification. These findings highlight the value of fine-tuning of both Similarity metrics and thresholds alongside model selection, offering practical guidance for designing high-accuracy semantic similarity systems in real-world NLP applications.<\/jats:p>","DOI":"10.3390\/computers14090385","type":"journal-article","created":{"date-parts":[[2025,9,15]],"date-time":"2025-09-15T10:51:33Z","timestamp":1757933493000},"page":"385","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Transformer Models for Paraphrase Detection: A Comprehensive Semantic Similarity Study"],"prefix":"10.3390","volume":"14","author":[{"given":"Dianeliz","family":"Ortiz Martes","sequence":"first","affiliation":[{"name":"Department of Mathematics and Systems Engineering, Florida Institute of Technology, Melbourne, FL 32901, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1408-2565","authenticated-orcid":false,"given":"Evan","family":"Gunderson","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, Florida Institute of Technology, Melbourne, FL 32901, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8134-9692","authenticated-orcid":false,"given":"Caitlin","family":"Neuman","sequence":"additional","affiliation":[{"name":"Department of Ocean Engineering and Marine Sciences, Florida Institute of Technology, Melbourne, FL 32901, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9397-1807","authenticated-orcid":false,"given":"Nezamoddin N.","family":"Kachouie","sequence":"additional","affiliation":[{"name":"Department of Mathematics and Systems Engineering, Florida Institute of Technology, Melbourne, FL 32901, USA"},{"name":"Department of Electrical Engineering and Computer Science, Florida Institute of Technology, Melbourne, FL 32901, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,14]]},"reference":[{"key":"ref_1","unstructured":"Dolan, W.B., and Brockett, C. (2005, January 14). Automatically Constructing a Corpus of Sentential Paraphrases. Proceedings of the Third International Workshop on Paraphrasing (IWP2005), Jeju Island, Republic of Korea."},{"key":"ref_2","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv."},{"key":"ref_3","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv.","DOI":"10.18653\/v1\/D19-1410"},{"key":"ref_5","unstructured":"Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. (2020). MPNet: Masked and Permuted Pre-training for Language Understanding. arXiv."},{"key":"ref_6","unstructured":"Marchenko, O., and Vrublevskyi, V. (2023). Comparison of Transformer-based Deep Learning Methods for the Paraphrase Identification Task. CEUR Workshop Proceedings, Proceedings of the Information Technology and Implementation (IT&I-2023), Kyiv, Ukraine, 20\u201321 November 2023, CEUR. Available online: https:\/\/ceur-ws.org\/Vol-3624\/Short_5.pdf#:~:text=1.%20Traditional%20Rule,learning%20tec-niques%2C%20such%20as%20Support."},{"key":"ref_7","unstructured":"Samuel, F., and Stevenson, M. (2025, May 08). A Semantic Similarity Approach to Paraphrase Detection. Available online: https:\/\/www.researchgate.net\/publication\/228616213_A_Semantic_Similarity_Approach_to_Paraphrase_Detection."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"03016","DOI":"10.1051\/e3sconf\/202451503016","article-title":"Evaluation of Bert and CHATGPT Models in Inference, Paraphrase and Similarity Tasks","volume":"515","author":"Kim","year":"2024","journal-title":"E3S Web Conf."},{"key":"ref_9","unstructured":"Andrianos, M., Clematide, S., and Opitz, J. (2024). PARAPHRASUS: A Comprehensive Benchmark for Evaluating Paraphrase Detection Models. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"65797","DOI":"10.1109\/ACCESS.2025.3556899","article-title":"Paraphrase identification with deep learning: A review of datasets and methods","volume":"13","author":"Zhou","year":"2025","journal-title":"IEEE Access"},{"key":"ref_11","unstructured":"Nie, Z., Feng, Z., Li, M., Zhang, C., Zhang, Y., Long, D., and Zhang, R. (2025). When Text Embedding Meets Large Language Model: A Comprehensive Survey. arXiv."},{"key":"ref_12","unstructured":"Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv."},{"key":"ref_13","unstructured":"Taku, K., and Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv."},{"key":"ref_14","unstructured":"Rico, S., Haddow, B., and Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv."},{"key":"ref_15","first-page":"23","article-title":"A new algorithm for data compression","volume":"12","author":"Philip","year":"1994","journal-title":"C Users J."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Tedo, V., and Me\u0161trovi\u0107, A. (2020). Corpus-based paraphrase detection experiments and review. Information, 11.","DOI":"10.3390\/info11050241"},{"key":"ref_17","unstructured":"You, K. (2025). Semantics at an Angle: When Cosine Similarity Works Until It Doesn\u2019t. arXiv."},{"key":"ref_18","unstructured":"(2024). Pooling and Attention: What are Effective Designs for LLM-based Embedding Models?. arXiv, Available online: https:\/\/arxiv.org\/html\/2409.02727v2."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Aslam, M.A., Khan, K., Khan, W., Khan, S.U., Albanyan, A., and Algamdi, S.A. (2015). Paraphrase detection for Urdu language text using fine-tune BiLSTM framework. Sci. Rep., 15.","DOI":"10.1038\/s41598-025-93260-6"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Tsinganos, N., Fouliras, P., and Mavridis, I. (2022). Applying BERT for early-stage recognition of persistence in chat-based social engineering attacks. Appl. Sci., 12.","DOI":"10.3390\/app122312353"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"e13386","DOI":"10.1111\/exsy.13386","article-title":"Comparison study of unsupervised paraphrase detection: Deep learning\u2014The key for semantic similarity detection","volume":"40","author":"Vrbanec","year":"2023","journal-title":"Expert Syst."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1017\/S1351324923000189","article-title":"Urdu paraphrase detection: A novel DNN-based implementation using a semi-automatically generated corpus","volume":"30","author":"Iqbal","year":"2024","journal-title":"Nat. Lang. Eng."},{"key":"ref_23","unstructured":"Antonio, S., Ponti, A., Candelieri, A., Giordani, I., and Archetti, F. (2023). A Bayesian approach for prompt optimization in pre-trained language models. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Perez, G., Mostaccio, C., Maltempo, G., and Antonelli, L. (2024). Evaluation of natural language processing models to measure similarity between scenarios written in Spanish. Cad. Do IME-S\u00e9rie Inform\u00e1tica, 50.","DOI":"10.12957\/cadinf.2024.87935"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"111441","DOI":"10.1016\/j.engappai.2025.111441","article-title":"A semi-supervised Multi-View Siamese Network with dual-contextual attention and knowledge distillation for cross-lingual low-resource paraphrase detection","volume":"158","author":"Ahmed","year":"2025","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Regilan, S., Gajalakshmi, P., Weslin, D., Vijay, J., Kadhiravan, D., and Jenitha, J. (2025, January 5\u20137). Benchmarking AI-Driven Resume Screening: An Evaluation of Precision and Efficiency. Proceedings of the 2025 11th International Conference on Communication and Signal Processing (ICCSP), Melmaruvathur, India.","DOI":"10.1109\/ICCSP64183.2025.11089249"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1214\/aos\/1176344552","article-title":"Bootstrap methods: Another look at the jackknife","volume":"7","author":"Efron","year":"1979","journal-title":"Ann. Stat."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Efron, B., and Tibshirani, R. (1993). An Introduction to the Bootstrap, Chapman & Hall\/CRC.","DOI":"10.1007\/978-1-4899-4541-9"},{"key":"ref_29","unstructured":"Good, P. (2005). Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses, Springer."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1214\/088342304000000396","article-title":"Permutation methods: A basis for exact inference","volume":"19","author":"Ernst","year":"2004","journal-title":"Stat. Sci."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/9\/385\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:45:27Z","timestamp":1760035527000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/14\/9\/385"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,14]]},"references-count":30,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["computers14090385"],"URL":"https:\/\/doi.org\/10.3390\/computers14090385","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,14]]}}}