{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T12:23:18Z","timestamp":1777897398154,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":34,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,4,20]],"date-time":"2024-04-20T00:00:00Z","timestamp":1713571200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"name":"National Research Foundation, Singapore"},{"name":"Cyber Security Agency under Singapore's National Cybersecurity R&D Programme","award":["NCRP25-P04-TAICeN"],"award-info":[{"award-number":["NCRP25-P04-TAICeN"]}]},{"name":"NRF Investigatorship","award":["NRF-NRFI06-2020-0001"],"award-info":[{"award-number":["NRF-NRFI06-2020-0001"]}]},{"name":"RIE2020 Industry Alignment Fund ? Industry Collaboration Projects (IAF-ICP) Funding Initiative"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,4,20]]},"DOI":"10.1145\/3643795.3648387","type":"proceedings-article","created":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T13:46:19Z","timestamp":1725975979000},"page":"119-126","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["LLMs for Relational Reasoning: How Far are We?"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3453-5698","authenticated-orcid":false,"given":"Zhiming","family":"Li","sequence":"first","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5694-608X","authenticated-orcid":false,"given":"Yushi","family":"Cao","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2564-6660","authenticated-orcid":false,"given":"Xiufeng","family":"Xu","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3476-3260","authenticated-orcid":false,"given":"Junzhe","family":"Jiang","sequence":"additional","affiliation":[{"name":"Hong Kong Polytechnic University, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2708-0584","authenticated-orcid":false,"given":"Xu","family":"Liu","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5081-6712","authenticated-orcid":false,"given":"Yon Shin","family":"Teo","sequence":"additional","affiliation":[{"name":"Continental Automotive Singapore Pte. Ltd., Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9726-3434","authenticated-orcid":false,"given":"Shang-Wei","family":"Lin","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7300-9215","authenticated-orcid":false,"given":"Yang","family":"Liu","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}]}],"member":"320","published-online":{"date-parts":[[2024,9,10]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"International Conference on Learning Representations.","author":"Balog Matej","year":"2016","unstructured":"Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016. DeepCoder: Learning to Write Programs. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_2_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901."},{"key":"e_1_3_2_1_3_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901."},{"key":"e_1_3_2_1_4_1","first-page":"19930","article-title":"GALOIS: boosting deep reinforcement learning via generalizable logic synthesis","volume":"35","author":"Cao Yushi","year":"2022","unstructured":"Yushi Cao, Zhiming Li, Tianpei Yang, Hao Zhang, Yan Zheng, Yi Li, Jianye Hao, and Yang Liu. 2022. GALOIS: boosting deep reinforcement learning via generalizable logic synthesis. Advances in Neural Information Processing Systems 35 (2022), 19930--19943.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_5_1","unstructured":"Karl Cobbe Vineet Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun Lukasz Kaiser Matthias Plappert Jerry Tworek Jacob Hilton Reiichiro Nakano et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)."},{"key":"e_1_3_2_1_6_1","volume-title":"Ryan D Honomichl, Kalina Christoff, and Silvia A Bunge.","author":"Crone Eveline A","year":"2009","unstructured":"Eveline A Crone, Carter Wendelken, Linda Van Leijenhorst, Ryan D Honomichl, Kalina Christoff, and Silvia A Bunge. 2009. Neurocognitive development of relational reasoning. Developmental science 12, 1 (2009), 55--66."},{"key":"e_1_3_2_1_7_1","volume-title":"Pentestgpt: An llm-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782","author":"Deng Gelei","year":"2023","unstructured":"Gelei Deng, Yi Liu, Victor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2023. Pentestgpt: An llm-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782 (2023)."},{"key":"e_1_3_2_1_8_1","volume-title":"International conference on machine learning. PMLR, 990--998","author":"Devlin Jacob","year":"2017","unstructured":"Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. 2017. Robustfill: Neural program learning under noisy i\/o. In International conference on machine learning. PMLR, 990--998."},{"key":"e_1_3_2_1_9_1","volume-title":"Neural Logic Machines. In International Conference on Learning Representations.","author":"Dong Honghua","unstructured":"Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. [n. d.]. Neural Logic Machines. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.5714"},{"key":"e_1_3_2_1_11_1","volume-title":"Learning probabilistic relational models. Relational data mining","author":"Getoor Lise","year":"2001","unstructured":"Lise Getoor, Nir Friedman, Daphne Koller, and Avi Pfeffer. 2001. Learning probabilistic relational models. Relational data mining (2001), 307--335."},{"key":"e_1_3_2_1_12_1","volume-title":"International Conference on Machine Learning. PMLR, 7583--7615","author":"Glanois Claire","year":"2022","unstructured":"Claire Glanois, Zhaohui Jiang, Xuening Feng, Paul Weng, Matthieu Zimmer, Dong Li, Wulong Liu, and Jianye Hao. 2022. Neuro-symbolic hierarchical rule induction. In International Conference on Machine Learning. PMLR, 7583--7615."},{"key":"e_1_3_2_1_13_1","volume-title":"Edward Grefenstette, Tiago Ramalho, John Agapiou, et al.","author":"Graves Alex","year":"2016","unstructured":"Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwi\u0144ska, Sergio G\u00f3mez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature 538, 7626 (2016), 471--476."},{"key":"e_1_3_2_1_14_1","volume-title":"Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences 21, 6","author":"Halford Graeme S","year":"1998","unstructured":"Graeme S Halford, William H Wilson, and Steven Phillips. 1998. Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and brain sciences 21, 6 (1998), 803--831."},{"key":"e_1_3_2_1_15_1","volume-title":"Logic synthesis and verification","author":"Hassoun Soha","unstructured":"Soha Hassoun and Tsutomu Sasao. 2001. Logic synthesis and verification. Vol. 654. Springer Science & Business Media."},{"key":"e_1_3_2_1_16_1","volume-title":"An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Huang Kai","unstructured":"Kai Huang, Xiangxin Meng, Jian Zhang, Yang Liu, Wenjie Wang, Shuhao Li, and Yuqing Zhang. 2023. An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE Computer Society, 1162--1174."},{"key":"e_1_3_2_1_17_1","volume-title":"Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199--22213."},{"key":"e_1_3_2_1_18_1","unstructured":"Daphne Koller Nir Friedman Sa\u0161o D\u017eeroski Charles Sutton Andrew McCallum Avi Pfeffer Pieter Abbeel Ming-Fai Wong Chris Meek Jennifer Neville et al. 2007. Introduction to statistical relational learning. MIT press."},{"key":"e_1_3_2_1_19_1","volume-title":"The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691","author":"Lester Brian","year":"2021","unstructured":"Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)."},{"key":"e_1_3_2_1_20_1","volume-title":"Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737","author":"Li Belinda Z","year":"2021","unstructured":"Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737 (2021)."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2021.103504"},{"key":"e_1_3_2_1_22_1","volume-title":"Inductive logic programming. New generation computing 8","author":"Muggleton Stephen","year":"1991","unstructured":"Stephen Muggleton. 1991. Inductive logic programming. New generation computing 8 (1991), 295--318."},{"key":"e_1_3_2_1_23_1","volume-title":"Are NLP models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191","author":"Patel Arkil","year":"2021","unstructured":"Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 (2021)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474381"},{"key":"e_1_3_2_1_25_1","volume-title":"Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri\u00e0 Garriga-Alonso, et al.","author":"Srivastava Aarohi","year":"2022","unstructured":"Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri\u00e0 Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022)."},{"key":"e_1_3_2_1_26_1","volume-title":"Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937","author":"Talmor Alon","year":"2018","unstructured":"Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 (2018)."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3410307"},{"key":"e_1_3_2_1_28_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_3_2_1_29_1","volume-title":"Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","author":"Valmeekam Karthik","year":"2023","unstructured":"Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2023. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track."},{"key":"e_1_3_2_1_30_1","volume-title":"International Conference on Learning Representations.","author":"Wei Jason","year":"2021","unstructured":"Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_31_1","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824--24837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_32_1","volume-title":"Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564","author":"Wu Shijie","year":"2023","unstructured":"Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023)."},{"key":"e_1_3_2_1_33_1","volume-title":"FinGPT: Open-Source Financial Large Language Models. arXiv preprint arXiv:2306.06031","author":"Yang Hongyang","year":"2023","unstructured":"Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. FinGPT: Open-Source Financial Large Language Models. arXiv preprint arXiv:2306.06031 (2023)."},{"key":"e_1_3_2_1_34_1","volume-title":"Differentiable Logic Machines. Transactions on Machine Learning Research","author":"Zimmer Matthieu","year":"2023","unstructured":"Matthieu Zimmer, Xuening Feng, Claire Glanois, Zhaohui JIANG, Jianyi Zhang, Paul Weng, Dong Li, Jianye HAO, and Wulong Liu. 2023. Differentiable Logic Machines. Transactions on Machine Learning Research (2023). https:\/\/openreview.net\/forum?id=mXfkKtu5JA"}],"event":{"name":"LLM4Code '24: 1st International Workshop on Large Language Models for Code","location":"Lisbon Portugal","acronym":"LLM4Code '24","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS","Faculty of Engineering of University of Porto"]},"container-title":["Proceedings of the 1st International Workshop on Large Language Models for Code"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643795.3648387","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643795.3648387","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:57:45Z","timestamp":1750294665000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643795.3648387"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,20]]},"references-count":34,"alternative-id":["10.1145\/3643795.3648387","10.1145\/3643795"],"URL":"https:\/\/doi.org\/10.1145\/3643795.3648387","relation":{},"subject":[],"published":{"date-parts":[[2024,4,20]]},"assertion":[{"value":"2024-09-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}