{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T09:26:02Z","timestamp":1770283562821,"version":"3.49.0"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"ETRA","license":[{"start":{"date-parts":[[2023,5,17]],"date-time":"2023-05-17T00:00:00Z","timestamp":1684281600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["CCF-2211428,CCF-2100035"],"award-info":[{"award-number":["CCF-2211428,CCF-2100035"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,5,17]]},"abstract":"<jats:p>Neural source code summarization is the task of generating natural language descriptions of source code behavior using neural networks. A fundamental component of most neural models is an attention mechanism. The attention mechanism learns to connect features in source code to specific words to use when generating natural language descriptions. Humans also pay attention to some features in code more than others. This human attention reflects experience and high-level cognition well beyond the capability of any current neural model. In this paper, we use data from published eye-tracking experiments to create a model of this human attention. The model predicts which words in source code are the most important for code summarization. Next, we augment a baseline neural code summarization approach using our model of human attention. We observe an improvement in prediction performance of the augmented approach in line with other bio-inspired neural models.<\/jats:p>","DOI":"10.1145\/3591136","type":"journal-article","created":{"date-parts":[[2023,5,18]],"date-time":"2023-05-18T20:21:03Z","timestamp":1684441263000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7475-7899","authenticated-orcid":false,"given":"Aakash","family":"Bansal","sequence":"first","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5178-7160","authenticated-orcid":false,"given":"Bonita","family":"Sharif","sequence":"additional","affiliation":[{"name":"University of Nebraska - Lincoln, Lincoln, NE, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0887-1083","authenticated-orcid":false,"given":"Collin","family":"McMillan","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN"}]}],"member":"320","published-online":{"date-parts":[[2023,5,18]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314111.3319834"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00052"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359591.3359735"},{"key":"e_1_2_2_4_1","volume-title":"International Conference on Learning Representations","author":"Allamanis Miltiadis","year":"2018","unstructured":"Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018. Learning to represent programs with graphs. International Conference on Learning Representations (2018)."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2729548"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00030"},{"key":"e_1_2_2_7_1","volume-title":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization. 65--72","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization. 65--72."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","unstructured":"Aakash Bansal. 2023. HumanAttn-Artifacts. https:\/\/doi.org\/10.17605\/OSF.IO\/B9SJZ","DOI":"10.17605\/OSF.IO"},{"key":"e_1_2_2_9_1","volume-title":"Project-Level Encoding for Neural Source Code Summarization of Subroutines. International Conference on Program Comprehension","author":"Bansal Aakash","year":"2021","unstructured":"Aakash Bansal, Sakib Haque, and Collin McMillan. 2021. Project-Level Encoding for Neural Source Code Summarization of Subroutines. International Conference on Program Comprehension (2021)."},{"key":"e_1_2_2_10_1","volume-title":"A neurobiological cross-domain evaluation metric for predictive coding networks. arXiv preprint arXiv:1805.10726","author":"Blanchard Nathaniel","year":"2018","unstructured":"Nathaniel Blanchard, Jeffery Kinnison, Brandon RichardWebster, Pouya Bashivan, and Walter J Scheirer. 2018. A neurobiological cross-domain evaluation metric for predictive coding networks. arXiv preprint arXiv:1805.10726 (2018)."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2009.104"},{"key":"e_1_2_2_12_1","volume-title":"Peter Battaglia, Nando de Freitas, Darshan Kumaran, Joel Z Leibo, Timothy Lillicrap, Joseph Modayil, Shakir Mohamed, and Neil C Rabinowitz.","author":"Botvinick Matthew","year":"2017","unstructured":"Matthew Botvinick, David GT Barrett, Peter Battaglia, Nando de Freitas, Darshan Kumaran, Joel Z Leibo, Timothy Lillicrap, Joseph Modayil, Shakir Mohamed, and Neil C Rabinowitz. 2017. Building machines that learn and think for themselves. Behavioral and Brain Sciences , Vol. 40 (2017)."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2015.36"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2011.19"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.48797"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X17000115"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion52605.2021.00038"},{"key":"e_1_2_2_18_1","volume-title":"Proceedings of the third international workshop on paraphrasing (IWP2005)","author":"Finch Andrew","year":"2005","unstructured":"Andrew Finch, Young-Sook Hwang, and Eiichiro Sumita. 2005. Using machine translation evaluation techniques to determine sentence-level semantic equivalence. In Proceedings of the third international workshop on paraphrasing (IWP2005)."},{"key":"e_1_2_2_19_1","volume-title":"Bio-inspired artificial intelligence: theories, methods, and technologies","author":"Floreano Dario","unstructured":"Dario Floreano and Claudio Mattiussi. 2008. Bio-inspired artificial intelligence: theories, methods, and technologies. MIT press."},{"key":"e_1_2_2_20_1","volume-title":"Using human brain activity to guide machine learning. Scientific reports","author":"Fong Ruth C","year":"2018","unstructured":"Ruth C Fong, Walter J Scheirer, and David D Cox. 2018. Using human brain activity to guide machine learning. Scientific reports, Vol. 8, 1 (2018), 1--10."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/585058.585065"},{"key":"e_1_2_2_22_1","volume-title":"Shake-shake regularization. arXiv preprint arXiv:1705.07485","author":"Gastaldi Xavier","year":"2017","unstructured":"Xavier Gastaldi. 2017. Shake-shake regularization. arXiv preprint arXiv:1705.07485 (2017)."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCRE.2010.13"},{"key":"e_1_2_2_24_1","volume-title":"Action Word Prediction for Neural Source Code Summarization. 28th IEEE International Conference on Software Analysis, Evolution and Reengineering","author":"Haque Sakib","year":"2021","unstructured":"Sakib Haque, Aakash Bansal, Lingfei Wu, and Collin McMillan. 2021. Action Word Prediction for Neural Source Code Summarization. 28th IEEE International Conference on Software Analysis, Evolution and Reengineering (2021)."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387449"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33018425"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2010.5596778"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICBK.2018.00016"},{"key":"e_1_2_2_29_1","volume-title":"Building machines that learn and think like people. Behavioral and brain sciences","author":"Lake Brenden M","year":"2017","unstructured":"Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and brain sciences , Vol. 40 (2017)."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3387904.3389268"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00087"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1394"},{"key":"e_1_2_2_33_1","volume-title":"International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=zv-typ1gPxA","author":"Liu Shangqing","year":"2021","unstructured":"Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021. Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=zv-typ1gPxA"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/DEVLRN.2018.8761032"},{"key":"e_1_2_2_35_1","unstructured":"Graham Neubig. [n. d.]. Survey of Methods to Generate Natural Language from Source Code. ( [n. d.])."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-50347-5_25"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3145904"},{"key":"e_1_2_2_38_1","volume-title":"Towards bridging human and artificial cognition: Hybrid variational predictive coding of the physical world, the body and the brain. Advances in Neural Information Processing Systems","author":"Ofner Andr\u00e9","year":"2018","unstructured":"Andr\u00e9 Ofner and Sebastian Stober. 2018. Towards bridging human and artificial cognition: Hybrid variational predictive coding of the physical world, the body and the brain. Advances in Neural Information Processing Systems (2018)."},{"key":"e_1_2_2_39_1","volume-title":"Zhiguang Qin, Son Tung Ngo, Thang Van Doan, Son Ba Nguyen, Son Hoang Nguyen, and Giang Hoang Nguyen.","author":"Oluwasammi Ariyo","year":"2021","unstructured":"Ariyo Oluwasammi, Muhammad Umar Aftab, Zhiguang Qin, Son Tung Ngo, Thang Van Doan, Son Ba Nguyen, Son Hoang Nguyen, and Giang Hoang Nguyen. 2021. Features to Text: A Comprehensive Survey of Deep Learning on Semantic Segmentation and Image Captioning. Complexity , Vol. 2021 (2021)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG47880.2020.00068"},{"key":"e_1_2_2_41_1","volume-title":"Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features. arXiv preprint arXiv:1810.10974","author":"Palazzo Simone","year":"2018","unstructured":"Simone Palazzo, Concetto Spampinato, Isaak Kavasidis, Daniela Giordano, and Mubarak Shah. 2018. Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features. arXiv preprint arXiv:1810.10974 (2018)."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678712"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.334"},{"key":"e_1_2_2_45_1","volume-title":"Eye movements and attention in reading, scene perception, and visual search. The quarterly journal of experimental psychology","author":"Rayner Keith","year":"2009","unstructured":"Keith Rayner. 2009. Eye movements and attention in reading, scene perception, and visual search. The quarterly journal of experimental psychology, Vol. 62, 8 (2009), 1457--1506."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2442238"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2015.7321188"},{"key":"e_1_2_2_48_1","unstructured":"Paige Rodeghero Collin McMillan Paul W. McBurney Nigel Bosch and Sidney D'Mello. 2014a. Artifacts for Improving Automated Source Code Summarization via an Eye-Tracking Study of Programmers. https:\/\/notredame.box.com\/s\/bhyoqle1i90vuz75pu4ufch3rpz1lpij"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568247"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468588"},{"key":"e_1_2_2_51_1","volume-title":"Semantic Similarity Metrics for Evaluating Source Code Summarization. In 30th International Conference on Program Comprehension (ICPC'22)","author":"Sakib Haque Aakash Bansal","year":"2022","unstructured":"Aakash Bansal Sakib Haque, Zachary Eberhart and Collin McMillan. 2022. Semantic Similarity Metrics for Evaluating Source Code Summarization. In 30th International Conference on Program Comprehension (ICPC'22)."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09829--4"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.06.008"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2168556.2168642"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1987856.1987872"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016--9486--9"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2019.08.034"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2009.5306335"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1167\/19.1.11"},{"key":"e_1_2_2_60_1","volume-title":"International conference on machine learning. PMLR, 1058--1066","author":"Wan Li","year":"2013","unstructured":"Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. 2013. Regularization of neural networks using dropconnect. In International conference on machine learning. PMLR, 1058--1066."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3380625.3380649"},{"key":"e_1_2_2_62_1","volume-title":"International Conference on Learning Representations. io","author":"Z\u00fcgner Daniel","year":"2021","unstructured":"Daniel Z\u00fcgner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan G\u00fcnnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In International Conference on Learning Representations. io"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591136","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3591136","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3591136","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:31Z","timestamp":1750178251000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591136"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,17]]},"references-count":62,"journal-issue":{"issue":"ETRA","published-print":{"date-parts":[[2023,5,17]]}},"alternative-id":["10.1145\/3591136"],"URL":"https:\/\/doi.org\/10.1145\/3591136","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,17]]},"assertion":[{"value":"2023-05-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}