{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T19:09:56Z","timestamp":1768590596149,"version":"3.49.0"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,8,17]],"date-time":"2016-08-17T00:00:00Z","timestamp":1471392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100008242","name":"NICTA","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100008242","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2016,9,14]]},"abstract":"<jats:p>Current random-forest (RF)-based learning-to-rank (LtR) algorithms use a classification or regression framework to solve the ranking problem in a pointwise manner. The success of this simple yet effective approach coupled with the inherent parallelizability of the learning algorithm makes it a strong candidate for widespread adoption. In this article, we aim to better understand the effectiveness of RF-based rank-learning algorithms with a focus on the comparison between pointwise and listwise approaches.<\/jats:p>\n          <jats:p>We introduce what we believe to be the first listwise version of an RF-based LtR algorithm. The algorithm directly optimizes an information retrieval metric of choice (in our case, NDCG) in a greedy manner. Direct optimization of the listwise objective functions is computationally prohibitive for most learning algorithms, but possible in RF since each tree maximizes the objective in a coordinate-wise fashion. Computational complexity of the listwise approach is higher than the pointwise counterpart; hence for larger datasets, we design a hybrid algorithm that combines a listwise objective in the early stages of tree construction and a pointwise objective in the latter stages. We also study the effect of the discount function of NDCG on the listwise algorithm.<\/jats:p>\n          <jats:p>Experimental results on several publicly available LtR datasets reveal that the listwise\/hybrid algorithm outperforms the pointwise approach on the majority (but not all) of the datasets. We then investigate several aspects of the two algorithms to better understand the inevitable performance tradeoffs. The aspects include examining an RF-based unsupervised LtR algorithm and comparing individual tree strength. Finally, we compare the the investigated RF-based algorithms with several other LtR algorithms.<\/jats:p>","DOI":"10.1145\/2866571","type":"journal-article","created":{"date-parts":[[2016,8,26]],"date-time":"2016-08-26T12:25:39Z","timestamp":1472214339000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["Comparing Pointwise and Listwise Objective Functions for Random-Forest-Based Learning-to-Rank"],"prefix":"10.1145","volume":"34","author":[{"given":"Muhammad","family":"Ibrahim","sequence":"first","affiliation":[{"name":"Monash University, Australia"}]},{"given":"Mark","family":"Carman","sequence":"additional","affiliation":[{"name":"Monash University, Australia"}]}],"member":"320","published-online":{"date-parts":[[2016,8,17]]},"reference":[{"key":"e_1_2_2_1_1","first-page":"1063","article-title":"Analysis of a random forests model","volume":"13","author":"Biau G\u00e9rard","year":"2012","unstructured":"G\u00e9rard Biau . 2012 . Analysis of a random forests model . Journal of Machine Learning Research 13 , 1 (2012), 1063 -- 1095 . G\u00e9rard Biau. 2012. Analysis of a random forests model. Journal of Machine Learning Research 13, 1 (2012), 1063--1095.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1390681.1442799"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273513"},{"key":"e_1_2_2_5_1","first-page":"1","article-title":"Yahoo! learning to rank challenge overview","volume":"14","author":"Chapelle Olivier","year":"2011","unstructured":"Olivier Chapelle and Yi Chang . 2011 . Yahoo! learning to rank challenge overview . Journal of Machine Learning Research-Proceedings Track 14 (2011), 1 -- 24 . Olivier Chapelle and Yi Chang. 2011. Yahoo! learning to rank challenge overview. Journal of Machine Learning Research-Proceedings Track 14 (2011), 1--24.","journal-title":"Journal of Machine Learning Research-Proceedings Track"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/11776420_44"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1561\/0600000035"},{"key":"e_1_2_2_9_1","volume-title":"Decision Forests for Computer Vision and Medical Image Analysis","author":"Criminisi Antonio","unstructured":"Antonio Criminisi and Jamie Shotton . 2013. Decision Forests for Computer Vision and Medical Image Analysis . Springer . Antonio Criminisi and Jamie Shotton. 2013. Decision Forests for Computer Vision and Medical Image Analysis. Springer."},{"key":"e_1_2_2_10_1","volume-title":"Search Engines: Information Retrieval in Practice","author":"Croft W. Bruce","year":"2010","unstructured":"W. Bruce Croft , Donald Metzler , and Trevor Strohman . 2010 . Search Engines: Information Retrieval in Practice . Addison-Wesley Reading . W. Bruce Croft, Donald Metzler, and Trevor Strohman. 2010. Search Engines: Information Retrieval in Practice. Addison-Wesley Reading."},{"key":"e_1_2_2_11_1","volume-title":"Narrowing the gap: Random forests in theory and in practice. arXiv Preprint arXiv:1310.1415","author":"Denil Misha","year":"2013","unstructured":"Misha Denil , David Matheson , and Nando De Freitas . 2013. Narrowing the gap: Random forests in theory and in practice. arXiv Preprint arXiv:1310.1415 ( 2013 ). Misha Denil, David Matheson, and Nando De Freitas. 2013. Narrowing the gap: Random forests in theory and in practice. arXiv Preprint arXiv:1310.1415 (2013)."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/951949.952144"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/945365.964285"},{"key":"e_1_2_2_14_1","volume-title":"Schapire","author":"Freund Yoav","year":"1995","unstructured":"Yoav Freund and Robert E . Schapire . 1995 . A decision-theoretic generalization of on-line learning and an application to boosting. In Computational Learning Theory. Springer , 23--37. Yoav Freund and Robert E. Schapire. 1995. A decision-theoretic generalization of on-line learning and an application to boosting. In Computational Learning Theory. Springer, 23--37."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-9473(01)00065-2"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jspi.2006.06.002"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009932"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2002945.2002947"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/10485252.2012.677843"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-006-6226-1"},{"key":"e_1_2_2_22_1","volume-title":"JMLR: Workshop and Conference Proceedings","volume":"14","author":"Geurts Pierre","year":"2011","unstructured":"Pierre Geurts and Gilles Louppe . 2011 . Learning to rank with extremely randomized trees . In JMLR: Workshop and Conference Proceedings , Vol. 14 . Pierre Geurts and Gilles Louppe. 2011. Learning to rank with extremely randomized trees. In JMLR: Workshop and Conference Proceedings, Vol. 14."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TGRS.2004.842481"},{"key":"e_1_2_2_24_1","volume-title":"The Elements of Statistical Learning: Data Mining, Inference and Prediction","author":"Hastie Trevor","unstructured":"Trevor Hastie , Robert Tibshirani , and Jerome Friedman . 2009. The Elements of Statistical Learning: Data Mining, Inference and Prediction . Springer , 2 nd Edition. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer, 2nd Edition.","edition":"2"},{"key":"e_1_2_2_25_1","volume-title":"Proceedings of the 12th Australasian Data Mining Conference. Australian Computer Society, 91--99","author":"Ibrahim Muhammad","year":"2014","unstructured":"Muhammad Ibrahim and Mark Carman . 2014 . Improving scalability and performance of random forest based learning-to-rank algorithms by aggressive subsampling . In Proceedings of the 12th Australasian Data Mining Conference. Australian Computer Society, 91--99 . Muhammad Ibrahim and Mark Carman. 2014. Improving scalability and performance of random forest based learning-to-rank algorithms by aggressive subsampling. In Proceedings of the 12th Australasian Data Mining Conference. Australian Computer Society, 91--99."},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-014-5451-2"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/345508.345545"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553449"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/2018740"},{"key":"e_1_2_2_31_1","first-page":"897","article-title":"Learning to rank using classification and gradient boosting","volume":"19","author":"Li Ping","year":"2007","unstructured":"Ping Li , C. Burges , and Qiang Wu . 2007 . Learning to rank using classification and gradient boosting . Advances in Neural Information Processing Systems 19 (2007), 897 -- 904 . Ping Li, C. Burges, and Qiang Wu. 2007. Learning to rank using classification and gradient boosting. Advances in Neural Information Processing Systems 19 (2007), 897--904.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1198\/016214505000001230"},{"key":"e_1_2_2_33_1","volume-title":"Learning to Rank for Information Retrieval","author":"Liu Tie-Yan","unstructured":"Tie-Yan Liu . 2011. Learning to Rank for Information Retrieval . Springerverlag , Berlin . Tie-Yan Liu. 2011. Learning to Rank for Information Retrieval. Springerverlag, Berlin."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-006-9019-z"},{"key":"e_1_2_2_35_1","volume-title":"An Empirical Analysis on Point-Wise Machine Learning Techniques Using Regression Trees for Web-Search Ranking. Master\u2019s thesis","author":"Mohan Ananth","unstructured":"Ananth Mohan . 2010. An Empirical Analysis on Point-Wise Machine Learning Techniques Using Regression Trees for Web-Search Ranking. Master\u2019s thesis . Washington University in St. Louis. Ananth Mohan. 2010. An Empirical Analysis on Point-Wise Machine Learning Techniques Using Regression Trees for Web-Search Ranking. Master\u2019s thesis. Washington University in St. Louis."},{"key":"e_1_2_2_36_1","first-page":"77","article-title":"Web-search ranking with initialized gradient boosted regression trees","volume":"14","author":"Mohan Ananth","year":"2011","unstructured":"Ananth Mohan , Zheng Chen , and Kilian Q Weinberger . 2011 . Web-search ranking with initialized gradient boosted regression trees . Journal of Machine Learning Research-Proceedings Track 14 (2011), 77 -- 89 . Ananth Mohan, Zheng Chen, and Kilian Q Weinberger. 2011. Web-search ranking with initialized gradient boosted regression trees. Journal of Machine Learning Research-Proceedings Track 14 (2011), 77--89.","journal-title":"Journal of Machine Learning Research-Proceedings Track"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009006"},{"key":"e_1_2_2_38_1","volume-title":"Ensemble Machine Learning","author":"Yanjun Qi.","unstructured":"Yanjun Qi. 2012. Random forest for bioinformatics . In Ensemble Machine Learning . Springer , 307--323. Yanjun Qi. 2012. Random forest for bioinformatics. In Ensemble Machine Learning. Springer, 307--323."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9124-x"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9123-y"},{"key":"e_1_2_2_41_1","first-page":"193","article-title":"Learning to rank with nonsmooth cost functions","volume":"19","author":"Quoc C.","year":"2007","unstructured":"C. Quoc and Viet Le . 2007 . Learning to rank with nonsmooth cost functions . Proceedings of the Advances in Neural Information Processing Systems 19 (2007), 193 -- 200 . C. Quoc and Viet Le. 2007. Learning to rank with nonsmooth cost functions. Proceedings of the Advances in Neural Information Processing Systems 19 (2007), 193--200.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_42_1","volume-title":"Machine Learning: ECML","author":"Robnik-\u0160ikonja Marko","year":"2004","unstructured":"Marko Robnik-\u0160ikonja . 2004. Improving random forests . In Machine Learning: ECML 2004 . Springer , 359--370. Marko Robnik-\u0160ikonja. 2004. Improving random forests. In Machine Learning: ECML 2004. Springer, 359--370."},{"key":"e_1_2_2_43_1","volume-title":"On the asymptotics of random forests. arXiv Preprint arXiv:1409.2090","author":"Scornet Erwan","year":"2014","unstructured":"Erwan Scornet . 2014. On the asymptotics of random forests. arXiv Preprint arXiv:1409.2090 ( 2014 ). Erwan Scornet. 2014. On the asymptotics of random forests. arXiv Preprint arXiv:1409.2090 (2014)."},{"key":"e_1_2_2_44_1","volume-title":"Consistency of random forests. arXiv Preprint arXiv:1405.2881","author":"Scornet Erwan","year":"2014","unstructured":"Erwan Scornet , G\u00e9rard Biau , and Jean-Philippe Vert . 2014. Consistency of random forests. arXiv Preprint arXiv:1405.2881 ( 2014 ). Erwan Scornet, G\u00e9rard Biau, and Jean-Philippe Vert. 2014. Consistency of random forests. arXiv Preprint arXiv:1405.2881 (2014)."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487630"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341544"},{"key":"e_1_2_2_48_1","volume-title":"Asymptotic theory for random forests. arXiv Preprint arXiv:1405.0352","author":"Wager Stefan","year":"2014","unstructured":"Stefan Wager . 2014. Asymptotic theory for random forests. arXiv Preprint arXiv:1405.0352 ( 2014 ). Stefan Wager. 2014. Asymptotic theory for random forests. arXiv Preprint arXiv:1405.0352 (2014)."},{"key":"e_1_2_2_49_1","volume-title":"A theoretical analysis of ndcg type ranking measures. arXiv Preprint arXiv:1304.6480","author":"Wang Yining","year":"2013","unstructured":"Yining Wang , Liwei Wang , Yuanzhi Li , Di He , Tie-Yan Liu , and Wei Chen . 2013. A theoretical analysis of ndcg type ranking measures. arXiv Preprint arXiv:1304.6480 ( 2013 ). Yining Wang, Liwei Wang, Yuanzhi Li, Di He, Tie-Yan Liu, and Wei Chen. 2013. A theoretical analysis of ndcg type ranking measures. arXiv Preprint arXiv:1304.6480 (2013)."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9112-1"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30217-6_13"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277809"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277790"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2866571","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2866571","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:15:05Z","timestamp":1750281305000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2866571"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,8,17]]},"references-count":52,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,9,14]]}},"alternative-id":["10.1145\/2866571"],"URL":"https:\/\/doi.org\/10.1145\/2866571","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,8,17]]},"assertion":[{"value":"2015-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-08-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}