{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,13]],"date-time":"2026-09-13T04:55:10Z","timestamp":1789275310124,"version":"build-2803163510"},"reference-count":63,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,4,13]],"date-time":"2025-04-13T00:00:00Z","timestamp":1744502400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Key Scientific Research Project of Higher Education Institutions in Henan Province","award":["24A520060"],"award-info":[{"award-number":["24A520060"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The division of retrieval into multiple stages has evolved to balance efficiency and effectiveness among various ranking models. Faster but less accurate models are used to retrieve results from the entire corpus. Slower yet more precise models refine the ranking within the top candidate list. This study proposes a multi-stage learning-to-rank (MultiLTR) method. MultiLTR applies learning-to-rank techniques across multiple stages. It incorporates text from different fields such as titles, body content, and abstracts to produce a more comprehensive and accurate ranking. MultiLTR iteratively refines ranking accuracy through sequential processing phases. It dynamically selects top-performing rankers from a diverse candidate pool at each stage. Experiments were carried out on benchmark datasets, MQ2007 and MQ2008, using three categories of learning-to-rank algorithms. The results demonstrate that MultiLTR outperforms state-of-the-art ranking approaches, particularly in field-based ranking tasks. This study improves ranking accuracy and offers new insights into enhancing multi-stage ranking strategies.<\/jats:p>","DOI":"10.3390\/info16040308","type":"journal-article","created":{"date-parts":[[2025,4,14]],"date-time":"2025-04-14T04:42:07Z","timestamp":1744605727000},"page":"308","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["MultiLTR: Text Ranking with a Multi-Stage Learning-to-Rank Approach"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6720-4831","authenticated-orcid":false,"given":"Hua","family":"Yang","sequence":"first","affiliation":[{"name":"School of Artificial Intelligence, Zhongyuan University of Technology, Zhengzhou 450007, China"},{"name":"VISTA Lab, Algoritmi Center, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1323-0249","authenticated-orcid":false,"given":"Teresa","family":"Gon\u00e7alves","sequence":"additional","affiliation":[{"name":"VISTA Lab, Algoritmi Center, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"},{"name":"Department of Computer Science, University of \u00c9vora, 7000-671 \u00c9vora, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,4,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"110183","DOI":"10.1016\/j.asoc.2023.110183","article-title":"Field features: The impact in learning to rank approaches","volume":"138","author":"Yang","year":"2023","journal-title":"Appl. Soft Comput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1007\/s10791-016-9279-1","article-title":"Assessing efficiency\u2013effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments","volume":"19","author":"Clarke","year":"2016","journal-title":"Inf. Retr. J."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhang, L., Zhang, Y., Long, D., Xie, P., Zhang, M., and Zhang, M. (2024, January 11\u201316). A two-stage adaptation of large language models for text ranking. Proceedings of the Findings of the Association for Computational Linguistics ACL 2024, Bangkok, Thailand.","DOI":"10.18653\/v1\/2024.findings-acl.706"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Zheng, K., Zhao, H., Huang, R., Zhang, B., Mou, N., Niu, Y., Song, Y., Wang, H., and Gai, K. (2024, January 13\u201317). Full stage learning to rank: A unified framework for multi-stage systems. Proceedings of the ACM Web Conference 2024, Singapore.","DOI":"10.1145\/3589334.3645523"},{"key":"ref_5","unstructured":"Liu, Z., Li, C., Xiao, S., Li, C., Lian, D., and Shao, Y. (2025). Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture with Configurable Depth and Width. arXiv."},{"key":"ref_6","unstructured":"Nogueira, R., Yang, W., Cho, K., and Lin, J. (2019). Multi-stage document ranking with BERT. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1561\/1500000100","article-title":"Pre-training methods in information retrieval","volume":"16","author":"Fan","year":"2022","journal-title":"Found. Trends\u00ae Inf. Retr."},{"key":"ref_8","unstructured":"Lu, J., Hall, K., Ma, J., and Ni, J. (2022). HYRR: Hybrid Infused Reranking for Passage Retrieval. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, B., Li, M., Zeng, Z., Zhuo, J., Wang, S., Xu, S., Long, B., and Yan, W. (2023). Learning Multi-Stage Multi-Grained Semantic Embeddings for E-Commerce Search. arXiv.","DOI":"10.1145\/3543873.3584638"},{"key":"ref_10","unstructured":"Huang, P.S., He, X., Gao, J., Deng, L., Acero, A., and Heck, L. (November, January 27). Learning deep structured semantic models for web search using clickthrough data. Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, San Francisco, CA, USA."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhang, H., Wang, S., Zhang, K., Tang, Z., Jiang, Y., Xiao, Y., Yan, W., and Yang, W.Y. (2020, January 25\u201330). Towards personalized and semantic retrieval: An end-to-end solution for e-commerce search via embedding learning. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.","DOI":"10.1145\/3397271.3401446"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Qiu, Y., Zhao, C., Zhang, H., Zhuo, J., Li, T., Zhang, X., Wang, S., Xu, S., Long, B., and Yang, W.Y. (2022, January 17\u201321). Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA.","DOI":"10.1145\/3511808.3557670"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Hai Le, N., Gerald, T., Formal, T., Nie, J.Y., Piwowarski, B., and Soulier, L. (2023, January 2\u20136). CoSPLADE: Contextualizing SPLADE for Conversational Information Retrieval. Proceedings of the European Conference on Information Retrieval, Dublin, Ireland.","DOI":"10.1007\/978-3-031-28244-7_34"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yang, D., Zhang, Y., and Fang, H. (2023). An exploration study of mixed-initiative query reformulation in conversational passage retrieval. arXiv.","DOI":"10.6028\/NIST.SP.500-338.cast-udel_fang"},{"key":"ref_15","unstructured":"Gao, L., Dai, Z., and Callan, J. (2021). Rethink training of BERT rerankers in multi-stage retrieval pipeline. Advances in Information Retrieval: Proceedings of the 43rd European Conference on IR Research, ECIR 2021, Virtual Event, 28 March\u20131 April 2021, Proceedings, Part II 43, Springer."},{"key":"ref_16","unstructured":"Nogueira, R., and Cho, K. (2019). Passage Re-ranking with BERT. arXiv."},{"key":"ref_17","unstructured":"Yilmaz, Z.A., Wang, S., Yang, W., Zhang, H., and Lin, J. (2019, January 3\u20137). Applying BERT to document retrieval with birch. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, Hong Kong, China."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1145\/3446426","article-title":"Multi-stage conversational passage retrieval: An approach to fusing term importance estimation and neural query rewriting","volume":"39","author":"Lin","year":"2021","journal-title":"ACM Trans. Inf. Syst. (TOIS)"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"102067","DOI":"10.1016\/j.ipm.2019.102067","article-title":"A deep look into neural ranking models for information retrieval","volume":"57","author":"Guo","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Craswell, N., Mitra, B., Yilmaz, E., Campos, D., and Lin, J. (2021, January 11\u201315). Ms marco: Benchmarking ranking models in the large-data regime. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.","DOI":"10.1145\/3404835.3462804"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Yates, A., Nogueira, R., and Lin, J. (2021, January 8\u201312). Pretrained transformers for text ranking: BERT and beyond. Proceedings of the 14th ACM International Conference on Web Search and Data Mining, Virtual.","DOI":"10.1145\/3437963.3441667"},{"key":"ref_22","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_23","unstructured":"Leonhardt, J., Beringer, F., and Anand, A. (2021). Exploiting Sentence-Level Representations for Passage Ranking. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mitra, B., Diaz, F., and Craswell, N. (2017, January 3\u20137). Learning to match using local and distributed representations of text for web search. Proceedings of the 26th International Conference on World Wide Web, Perth, Australia.","DOI":"10.1145\/3038912.3052579"},{"key":"ref_25","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ahmadi, K., Gathwala, A., Osajima, J., Hsiao, D., and Das, P. (2024, January 15\u201318). SLLIM-Rank: A Multi-Stage Item-to-Item Recommendation Model using Learning-to-Rank. Proceedings of the 2024 IEEE International Conference on Big Data (BigData), Washington, DC, USA.","DOI":"10.1109\/BigData62323.2024.10826085"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Lee, J., Bernier-Colborne, G., Maharaj, T., and Vajjala, S. (2024, January 16\u201321). Methods, Applications, and Directions of Learning-to-Rank in NLP Research. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico.","DOI":"10.18653\/v1\/2024.findings-naacl.123"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Dato, D., MacAvaney, S., Nardini, F.M., Perego, R., and Tonellotto, N. (2022, January 11\u201315). The Istella22 Dataset: Bridging Traditional and Neural Learning to Rank Evaluation. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain.","DOI":"10.1145\/3477495.3531740"},{"key":"ref_30","unstructured":"Qin, Z., Yan, L., Zhuang, H., Tay, Y., Pasumarthi, R.K., Wang, X., Bendersky, M., and Najork, M. (2021, January 3\u20137). Are neural rankers still outperformed by gradient boosted decision trees?. Proceedings of the ICLR\u20192021, Virtual."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1214\/aos\/1013203451","article-title":"Greedy function approximation: A gradient boosting machine","volume":"29","author":"Friedman","year":"2001","journal-title":"Ann. Statist."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., and Hullender, G. (2005, January 7\u201311). Learning to rank using gradient descent. Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany.","DOI":"10.1145\/1102351.1102363"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Xu, J., and Li, H. (2007, January 23\u201327). Adarank: A boosting algorithm for information retrieval. Proceedings of the 30th annual international ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands.","DOI":"10.1145\/1277741.1277809"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1007\/s10791-006-9019-z","article-title":"Linear feature-based models for information retrieval","volume":"10","author":"Metzler","year":"2007","journal-title":"Inf. Retr."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1007\/s10791-009-9112-1","article-title":"Adapting boosting for information retrieval measures","volume":"13","author":"Wu","year":"2010","journal-title":"Inf. Retr."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Cao, Z., Qin, T., Liu, T.Y., Tsai, M.F., and Li, H. (2007, January 20\u201324). Learning to rank: From pairwise approach to listwise approach. Proceedings of the 24th International Conference on Machine Learning, Corvalis, OR, USA.","DOI":"10.1145\/1273496.1273513"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"K\u00f6ppel, M., Segner, A., Wagener, M., Pensel, L., Karwath, A., and Kramer, S. (2019, January 16\u201320). Pairwise learning to rank by neural networks revisited: Reconstruction, theoretical analysis and practical performance. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, W\u00fcrzburg, Germany.","DOI":"10.1007\/978-3-030-46133-1_15"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Jia, Y., Wang, H., Guo, S., and Wang, H. (2021, January 19\u201323). Pairrank: Online pairwise learning to rank by divide-and-conquer. Proceedings of the Web Conference 2021, Ljubljana, Slovenia.","DOI":"10.1145\/3442381.3449972"},{"key":"ref_40","unstructured":"Yuan, K., and Kuang, D. (2021). Deep Pairwise Learning To Rank For Search Autocomplete. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Ai, Q., Bi, K., Guo, J., and Croft, W.B. (2018, January 8\u201312). Learning a deep listwise context model for ranking refinement. Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, MI, USA.","DOI":"10.1145\/3209978.3209985"},{"key":"ref_42","unstructured":"Sharma, A. (2020). Listwise Learning to Rank with Deep Q-Networks. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Pang, L., Xu, J., Ai, Q., Lan, Y., Cheng, X., and Wen, J. (2020, January 25\u201330). Setrank: Learning a permutation-invariant ranking model for information retrieval. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual.","DOI":"10.1145\/3397271.3401104"},{"key":"ref_44","unstructured":"Swezey, R., Grover, A., Charron, B., and Ermon, S. (2021). PiRank: Scalable Learning To Rank via Differentiable Sorting. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_45","unstructured":"Chen, Z., and Eickhoff, C. (2021). PoolRank: Max\/Min Pooling-based Ranking Loss for Listwise Learning & Ranking Balance. arXiv."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"102962","DOI":"10.1016\/j.ipm.2022.102962","article-title":"ListMAP: Listwise learning to rank as maximum a posteriori estimation","volume":"59","author":"Keshvari","year":"2022","journal-title":"Inf. Process. Manag."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Chen, F., and Fang, H. (2022, January 15\u201319). An Exploration of Learning-to-re-rank Using a Two-step Framework for Fair Ranking. Proceedings of the TREC, Online.","DOI":"10.6028\/NIST.SP.500-338.fair-udel_fang"},{"key":"ref_48","unstructured":"Han, S., Wang, X., Bendersky, M., and Najork, M. (2020). Learning-to-Rank with BERT in TF-Ranking. arXiv."},{"key":"ref_49","unstructured":"Awan, Z., Kahlke, T., Ralph, P., and Kennedy, P. (2023). Bi-Encoders based Species Normalization\u2013Pairwise Sentence Learning to Rank. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Richards, J.A. (2022). Feature reduction. Remote Sensing Digital Image Analysis, Springer.","DOI":"10.1007\/978-3-030-82327-6"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1561\/1500000016","article-title":"Learning to rank for information retrieval","volume":"3","author":"Liu","year":"2009","journal-title":"Found. Trends\u00ae Inf. Retr."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Cabello-Solorzano, K., Ortigosa de Araujo, I., Pe na, M., Correia, L., and Tall\u00f3n-Ballesteros, A.J. (2023, January 5\u20137). The impact of data normalization on the accuracy of machine learning algorithms: A comparative analysis. Proceedings of the International Conference on Soft Computing Models in Industrial and Environmental Applications, Salamanca, Spain.","DOI":"10.1007\/978-3-031-42536-3_33"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/582415.582418","article-title":"Cumulated gain-based evaluation of IR techniques","volume":"20","year":"2002","journal-title":"ACM Trans. Inf. Syst. (TOIS)"},{"key":"ref_54","unstructured":"Qin, T., and Liu, T.Y. (2013). Introducing LETOR 4.0 datasets. arXiv."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1007\/s10791-009-9123-y","article-title":"LETOR: A benchmark collection for research on learning to rank for information retrieval","volume":"13","author":"Qin","year":"2010","journal-title":"Inf. Retr."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Zhang, E., and Zhang, Y. (2009). Average Precision. Encyclopedia of Database Systems, Springer.","DOI":"10.1007\/978-0-387-39940-9_482"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1017\/S1351324909005129","article-title":"Christopher D. Manning, Prabhakar Raghavan, Hinrich Sch\u00fctze, Introduction to Information Retrieval, Cambridge University Press. 2008. ISBN-13 978-0-521-86571-5, xxi+ 482 pages","volume":"16","author":"Sanderson","year":"2010","journal-title":"Nat. Lang. Eng."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Beitzel, S.M., Jensen, E.C., and Frieder, O. (2009). MAP. Encyclopedia of Database Systems, Springer.","DOI":"10.1007\/978-0-387-39940-9_492"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Xiong, C., Dai, Z., Callan, J., Liu, Z., and Power, R. (2017, January 7\u201311). End-to-end neural ad-hoc ranking with kernel pooling. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Tokyo, Japan.","DOI":"10.1145\/3077136.3080809"},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Fan, Y., Guo, J., Lan, Y., Xu, J., Zhai, C., and Cheng, X. (2018, January 8\u201312). Modeling diverse relevance patterns in ad-hoc retrieval. Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, MI, USA.","DOI":"10.1145\/3209978.3209980"},{"key":"ref_61","unstructured":"Tang, Z., and Yang, G.H. (February, January 27). Deeptilebars: Visualizing term distribution for neural information retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_62","unstructured":"Fox, E.A., and Shaw, J.A. (2025, February 12). Combination of multiple searches, NIST Special Publications SP, Available online: https:\/\/trec.nist.gov\/pubs\/trec2\/t2_proceedings.html."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1007\/s10791-024-09435-8","article-title":"Injecting the score of the first-stage retriever as text improves BERT-based re-rankers","volume":"27","author":"Askari","year":"2024","journal-title":"Discov. Comput."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/4\/308\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:13:44Z","timestamp":1760030024000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/4\/308"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,13]]},"references-count":63,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["info16040308"],"URL":"https:\/\/doi.org\/10.3390\/info16040308","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,13]]}}}